Podcast Summary
Rethinking User Research: Moving from Validation to Falsification: User research should be used to challenge assumptions and drive better decision-making, rather than simply confirming existing beliefs. Businesses need to prioritize learning from mistakes and align research with business goals.
The field of user research is going through a reckoning. Judd Antin, a renowned expert in the field, believes that the discipline has gotten it wrong over the past decade. He argues that too often, product managers and designers rely on user research to validate their assumptions and check the box, rather than truly leveraging it to make better decisions. This phenomena, which Judd calls "user-centered performance," is a performance of customer obsession without the genuine desire to be wrong and learn from it. To adapt and thrive, user research needs to shift its focus to falsification rather than validation. It's time for the discipline to change, companies to reevaluate how they leverage user research, and researchers to think about business goals in addition to user needs.
Reevaluating User Research: Criticisms, Debates, and a Need for Change: User research needs a shift in focus towards strategic and business-oriented macro research, as well as technical usability micro research, in order to drive value and impact for businesses.
The word "reckoning" may have caused unnecessary drama in the conversation about user research. Despite this, the response to the conversation was largely positive and resonated with many people. However, there were also criticisms, with some feeling that researchers were being blamed for doing their job incorrectly and not taking enough responsibility themselves. Surprisingly, there were also critics from the anti-capitalist crew who disagreed with the notion that researchers should be profit-focused. The main message is that user research has been heavily impacted by layoffs, indicating a potential flaw in the system. Research is not driving the value and impact it should because it focuses too much on the wrong type of research. Researchers need to shift their focus towards macro research (strategic and business-oriented) and micro research (technical usability) rather than spending too much time on middle range research that may be interesting but lacks impact for businesses.
Revolutionizing Research in Technology Companies: Research is no longer undervalued in technology companies; it is now seen as crucial. To maximize its impact, companies need to integrate research into their product development process consistently.
Research in technology companies has often been mismanaged and undervalued, leading to ineffective results and a lack of impact. The idea of hiring researchers without understanding their role or how to leverage their skills has been a common practice. However, this approach is changing. Research is now seen as crucial, but it requires a new perspective. Middle range research, which focuses on specific questions or constituencies, may yield interesting insights but often lacks the necessary alignment with business goals and strategies. This can delay decision-making and create a reliance on researchers for every little detail. To address this, companies need to restructure their product development process to integrate research throughout, fostering consistent relationships and maximizing the impact of researchers' work.
Integrating Researchers: A Key to Effective Product Development: Researchers need to be engaged and integrated from the beginning of the product process to maximize their impact and the effectiveness of research.
In short, one big takeaway from the conversation is that the lack of integration and engagement of researchers in the product process is a major problem. When research is treated as a service function that is called in only at the end, it limits their impact and effectiveness. Many companies hired researchers with good intentions but didn't know how to integrate them properly. As a result, researchers lack input on the questions to ask and are unable to build the necessary relationships and drive insights. This leads to reactive and less impactful research, which then perpetuates the belief that researchers are not valuable. To break this cycle, constant engagement and integration of researchers from the beginning of the product process is crucial. The best researchers are multi-method, using a range of tools such as qualitative and quantitative research, survey design, applied statistics, and technical skills.
Balancing User Needs and Business Profitability: A Shift in Research Mindset: Researchers must understand the business side of things by familiarizing themselves with company strategies and metrics to identify research opportunities that contribute to business goals and align with organizational objectives.
Researchers need to be more business-oriented and consider the needs of the company alongside those of the user. While user-centeredness and empathy are valuable, there is an important overlap between user needs and business profitability that researchers must acknowledge. To achieve this, researchers should make an effort to understand the business side of things by reading quarterly reports, listening to shareholder calls, and familiarizing themselves with company strategies and metrics. By doing so, they can identify opportunities for research that directly contribute to business goals and address areas where competitors may be outperforming. This shift towards a more business-oriented mindset is crucial for researchers to have a greater impact and align their work with the overall objectives of the organization.
User-centered performance in product development: the illusion of customer obsession without informed decisions.: Embrace a mindset of being wrong and actively seek research to expose blind spots and improve intuition for better product outcomes.
User-centered performance is a common phenomenon in product development. It refers to the practice of symbolically showing customer obsession without actually focusing on learning and making informed decisions. This often occurs when PMs ask researchers to conduct quick user studies just to validate assumptions at the end of a product process, or when executives request focus groups to feel closer to customers. User-centered performance can also be driven by cognitive biases, ego, and a desire to validate rather than falsify assumptions. To combat this, it's important to embrace a mindset of wanting to be wrong and actively seeking research that exposes blind spots and improves intuition. Ultimately, integrating research effectively can lead to better outcomes and impactful product decisions.
Developing intuition while avoiding biases and leveraging diverse perspectives for effective decision-making.: Great leaders recognize the limitations of intuition, engage in analytical thinking, and utilize diverse perspectives to make informed decisions that improve overall decision quality.
While intuition and gut instincts are valuable tools, they can be limited and biased. Great leaders develop intuition through pattern matching and experience, but they must also be aware of the cognitive biases and thorny factors that can lead them astray. To check your gut, it's important to engage System 2 thinking, which involves slow and methodical analytic thinking. Additionally, leveraging the wisdom of the crowd can be a powerful strategy. Bringing diverse perspectives and evidence-based opinions into decision-making discussions can help mitigate biases and improve the overall quality of decisions. Researchers excel in leading these discussions and can have a huge impact when they develop strong, trusting relationships and become active participants in the decision-making process. Similarly, product managers should aim to be valued team members that others want on their team.
The importance of collaboration between product managers and user researchers in product development.: Effective collaboration between researchers and product managers can lead to faster and more successful product development by ensuring the right decisions are made from the start and saving time and effort in the long run.
Having a great product manager can make everyone's life easier. The role of a product manager is like the grease that makes everything run smoothly. However, the relationship between product managers and user researchers can be challenging. Many researchers feel excluded from the PMs' process, which leads to misunderstandings. One common trope is that research slows down the product development process, but this is not true. Good research actually speeds up the process by ensuring that the right decisions are made from the start, saving time and effort in the long run. Micro-level research, which can be done quickly, can provide valuable insights that drive significant business value. For example, in just 48 hours, research at Airbnb helped optimize a button's text, resulting in millions of dollars in increased revenue. So, the key takeaway is that effective collaboration between researchers and product managers can lead to faster and more successful product development.
Ahrefs Webmaster Tools: Improving Website Rankings and Optimizing for Google: Utilizing Ahrefs Webmaster Tools can enhance website visibility by providing keyword insights, backlink analysis, and automated site audits, while also emphasizing the importance of combining micro research and middle range research for effective product development.
Ahrefs Webmaster Tools is a valuable tool for improving website rankings and optimizing for Google. It provides insights on keywords, backlinks, and performs automatic site audits to identify issues that may hinder website performance. By setting up a free account and connecting your website, you can receive detailed explanations and advice on how to address these issues. Additionally, the conversation between Lenny and Judd Antin highlights the importance of both micro research (such as usability research) and middle range research in product development. Usability improvements can lead to improved business metrics, and researchers play a vital role in turning user feedback into valuable insights. AB testing is valuable for causal claims, but user research provides valuable data on how and why things happen. So, both methods should be used in combination for optimal results.
The Importance of Collaboration, Questioning Assumptions, and User Research in Product Development: Collaborating with data scientists and insights professionals, questioning widely quoted statements, avoiding post-hoc bias, and adopting a humble and open-minded approach to user research can greatly improve product development.
A strong partnership between data scientists and research and insights professionals is crucial for building better products. By involving these stakeholders from the beginning, companies can avoid making costly mistakes and gain valuable insights into the hows and whys behind their products' success. Additionally, it is important to question widely quoted statements and examine the truth behind them. In the case of the famous Henry Ford quote, it turns out to be apocryphal, highlighting the significance of relying on thorough research rather than anecdotal evidence. Furthermore, researchers caution against post-hoc bias, where individuals construct narratives that conveniently fit their desired outcomes. The book "Everything is Obvious If You Already Know the Answer" by Duncan Watts sheds light on this bias and emphasizes the need to challenge intuitive assumptions in decision making. Ultimately, a humble and open-minded approach to user research can lead to a stronger mental model of the user and, consequently, better product development.
Moving beyond insights: The process of effective research: Effective research involves understanding the "what," the "so what," and the "then what," while striking a balance between conclusions and collaboration. Developing diverse skills and fostering an integrated approach are crucial for success.
Effective research goes beyond simply uncovering insights. It involves a process of understanding the "what," the "so what," and the "then what." However, it's important to strike a balance and avoid going too far in one direction. Researchers should not jump to conclusions or dictate what should be built based solely on their findings. Instead, they should engage in communication and collaboration with the team to consider all aspects of the situation. The goal is to develop a framework based on actual evidence and then work together to experiment and create a successful product. Additionally, researchers should strive to develop diverse research skills, deepen their business knowledge, build strong relationships with team members, and focus on the most important tasks. On the other hand, companies should foster an integrated approach, breaking down silos between different insights disciplines and valuing contributions from various sources of data and expertise.
Integrating research disciplines and breaking down silos for an integrated insights function.: Companies should prioritize integrating different research disciplines, establishing a unified process, and collaborating between teams to leverage the full potential of user research and create successful products.
Companies should prioritize integrating different research disciplines and breaking down silos to create an integrated insights function. This means combining UX research, market analysis, competitor research, and utilizing data science and customer service feedback. By fueling one insights machine, companies can extract maximum value from research and avoid the problem of it coming too late or not knowing the right questions to ask. Additionally, companies need to establish a unified, lean process where researchers are involved from the beginning to the end of product development. This collaboration between researchers, product managers, design teams, and engineering teams is crucial for leveraging the full potential of user research and creating successful products.
Challenging Assumptions and Gaining Valuable Insights: Listening to individual experiences, questioning assumptions, and fostering strong relationships are essential in uncovering valuable information and driving the need for resources in research projects.
Assumptions can be shattered and new insights can be gained from just one example or anecdote. Statistical significance is not always necessary to uncover valuable information. For example, in a Facebook app study, a participant's behavior of hiding stories because she had already seen them challenged the common assumption about the meaning of "hides" as a signal. This revelation led to a better solution and a more nuanced understanding. Similarly, in another study, hosts' reluctance to connect Facebook with their accounts highlighted widespread concerns about data privacy. These stories emphasize the importance of listening to individual experiences and questioning assumptions. When it comes to determining the number of researchers needed in a team, the guiding principle should be fostering strong relationships and ensuring researchers have the capacity to fully engage in meaningful projects, even if it means creating some pain or scarcity to demonstrate the value of research and drive the need for more resources.
The Importance of Researchers in Business Growth: Researchers play a crucial role in answering complex questions, driving impactful decisions, and empowering a company's growth by providing valuable insights and challenging the status quo.
Companies, whether in the early stages or late stages, often overlook the value of having a researcher on their team. The lack of researchers on the list of first 10 employees in various B2B companies is concerning. A good researcher can bring incredible value to a company by answering questions that are impossible to answer through other means. They can help a company go faster and drive impactful decisions. As a company grows, it is important to ensure that researchers have strong partners in key areas of the business. While there may be trade-offs in hiring researchers over other roles, having a Swiss army knife of research tools can greatly benefit a company's decision-making process. Moreover, researchers can empower themselves by focusing on the right kind of research that contributes to business value, challenging the broken system, and driving the evolution of research.
The importance of effective communication in research and business.: Effective communication skills are essential for researchers and companies to convey their findings to diverse audiences and to question the validity of commonly used metrics like NPS.
Researchers and companies need to prioritize effective communication and develop the necessary skills to do so. It's not enough to just conduct rigorous research; researchers also need to be excellent communicators who can effectively convey their findings to different audiences. Companies should value researchers who can teach by example and communicate research findings in a way that is appropriate for the specific audience. Additionally, it's important to question the validity and usefulness of commonly used metrics like NPS. The consensus in the survey science community is that NPS is flawed and there are better alternatives, such as customer satisfaction metrics. Comparing NPS across industries is problematic due to variations in how it's asked and the inconsistent nature of the metric. Ultimately, researchers and companies can contribute to a brighter future by developing and honing their communication skills while building more inclusive models.
The Importance of Considering User Perspectives in Product Management: Product managers should be mindful of their biases and not rely solely on their own experiences when prioritizing and addressing issues in their products. User perspectives and needs should be considered for effective evaluation and improvement.
Product managers should dog food their own products and conduct walkthroughs to identify potential issues. However, it is important for them to recognize that they are not the typical user and may have biases that affect their perception of what is good or bad in the product. Some problems may require a pulse or context of use that PMs may not have solely based on their own usage. Therefore, while dog fooding is valuable, PMs should be cautious about relying solely on their own intuition or expertise when prioritizing and addressing issues. It is important to consider the perspectives and needs of various users when evaluating and improving the product.
Embracing Stoicism: Focusing on What We Can Control: Letting go of worries about things outside our control and prioritizing what we can influence promotes serenity and personal growth.
Focusing on the things we can control and ignoring the rest is a valuable lesson from stoicism. Judd Antin emphasizes the importance of this motto, which helps reduce stress and worry caused by things outside our control. He points out that much of the pain and stress in life comes from worrying about things we cannot influence. By letting go of those things and focusing on the areas where we have control, we can experience greater serenity and peace of mind. This principle aligns with the concept of circles of control, influence, and no control discussed in the book "7 Habits of Highly Effective People." Therefore, embracing this mindset can bring about positive transformation in our lives.
