Generative vs Evaluative Research in UX

As a UX researcher, one’s always looking for ways to improve products and create better user experiences. But how do you know which research method to use? Let's explore these two essential approaches, along with what is formative (generative) vs. summative (evaluative) research.

What is Generative UX Research?

Generative UX research is a foundational approach in user experience design that focuses on uncovering user needs, behaviors, and motivations. This type of research aims to generate new ideas and insights that can inform product development from its earliest stages.

What is a generative approach to research?

A generative approach to research is about exploring possibilities and generating new ideas. It's open-ended and focuses on understanding the user's world. Instead of testing existing concepts, you're creating new ones based on user insights.

Why is generative research important?

Generative research is crucial because it helps you avoid the trap of solving the wrong problem. It lets you step back and see the bigger picture. By understanding your users' needs, goals, and pain points, you can create products that truly resonate with them.

Generative Research Methodologies

There are several ways to conduct generative research:

• Interviews: Have in-depth conversations with users to understand their experiences and needs.
• Ethnographic studies: Observe users in their natural environment to see how they interact with products.
• Diary studies: Ask users to record their experiences over time, giving you a long-term view of their behavior.
• Participatory design workshops: Involve users in the design process to generate ideas together.
• Card sorting: Use this method to understand how users organize and categorize information.

What is evaluative research?

Evaluative research measures the effectiveness, efficiency, and satisfaction levels of a specific design or product. It provides quantitative and qualitative data about how well a solution meets user needs and expectations.

What is evaluative UX research?

Evaluative UX research, in contrast to generative research, assesses existing designs or prototypes. It's a critical component in refining and improving user experiences.

Why is evaluative research important?

Evaluative research is your quality control. It helps you fine-tune your designs and make them sing. It gives you the facts and figures to back up your design decisions and show stakeholders the value of your work.

Evaluative Research Methodologies

Here are some common evaluative research methods.

• Usability testing: Observe users as they interact with your product to identify issues and areas for improvement.
• A/B testing: Compare two versions of a design to see which performs better.
• Surveys: Collect quantitative data about user satisfaction and preferences.
• Heuristic evaluation: Have experts review your design based on established usability principles.
• Analytics review: Analyze user behavior data to identify patterns and issues.

What is Formative (Generative) vs. Summative (Evaluative) Research?

Formative research, which includes generative research, happens early in the design process. It shapes and informs the development of ideas and concepts. Summative research, which includes evaluative research, happens later and assesses the final or near-final product.

What is the difference between evaluative and generative research?

The main difference lies in their goals and timing. 

Evaluative vs Generative Research: Key Differences

• Goals: Generative research aims to create new ideas and uncover user needs, while evaluative research assesses existing designs.
• Timing: Generative research typically occurs early in the design process, while evaluative research happens later.
• Outcomes: Generative research produces insights and ideas, while evaluative research yields performance metrics and improvement recommendations.

When should each method be used?

Use generative research when:

• You're starting a new project
• You need to understand user needs and behaviors
• You want to explore new ideas and possibilities

Use evaluative research when:

• You have a specific design or prototype to test
• You need to measure the effectiveness of your solution
• You want to identify and fix usability issues

What's the best research method: generative or evaluative?

There's no "best" method – both are important for creating great user experiences. It's not about choosing one over the other, but knowing when to use each approach.

How do you conduct generative research?

Conducting generative research is a structured process that requires careful planning and execution. Here's a detailed step-by-step guide.

1. Define research objectives: Clearly articulate what you want to learn about your users and their needs. These objectives will guide your entire research process.
2. Choose appropriate methods: Select research methods that align with your objectives. This could include in-depth interviews, ethnographic studies, diary studies, or participatory design workshops.
3. Recruit participants: Identify and recruit participants who represent your target user base. Aim for a diverse sample to capture a range of perspectives.
4. Prepare research materials: Develop interview guides, observation protocols, or workshop plans. Ensure all materials align with your research objectives.
5. Conduct the research: Implement your chosen methods. Be open to unexpected insights and willing to explore new avenues of inquiry as they arise.
6. Document thoroughly: Record interviews, take detailed notes, and capture visual data where appropriate. The richer your documentation, the more valuable your analysis will be.
7. Analyze the data: Review all collected data, looking for patterns, themes, and unexpected insights. Use techniques like affinity mapping to organize your findings.
8. Generate insights: Transform your raw data into actionable insights. These should be clear, concise statements that capture key learnings about user needs, behaviors, or pain points.
9. Ideate solutions: Based on your insights, brainstorm potential solutions or product ideas that address the identified user needs.
10. Share findings: Communicate your insights and ideas to stakeholders in a compelling way, using storytelling techniques to bring the user's perspective to life.

What are the pros and cons of generative research?

Pros:

1. Generative research can reveal needs or problems that users themselves may not have articulated.
2. It helps teams avoid making unfounded assumptions about user needs or behaviors.
3. Generative research offers rich, contextual information about users' lives and experiences.

Cons:

1. Generative research often requires significant time investment for data collection and analysis.
2. It may require specialized skills and tools, making it potentially costly.
3. The open-ended nature of the data can make analysis complex and time-consuming.
4. Researchers must be careful not to introduce their own biases into the data collection or analysis.
5. The qualitative nature of much generative research can make it challenging to present findings in quantitative terms to stakeholders.

Tools to help with generative research

1. Looppanel: Looppanel is an AI-powered research assistant and research repository platform that makes research 10x easier, according to its customers. It streamlines your interview process with features for transcription, analysis, and insight generation.
2. Ethosapp: This research platform helps you conduct different kinds of generative research, gathering rich, contextual data from users. It's particularly useful for diary studies and mobile ethnography.
3. Mural: A digital workspace for visual collaboration, Mural is excellent for remote participatory design workshops and organizing research insights.

How do you conduct evaluative research?

Conducting evaluative research involves a systematic approach to assess the effectiveness of a design or product. 

1. Define evaluation criteria: Clearly articulate what aspects of the design you're evaluating (e.g., usability, user satisfaction, task completion rates).
2. Choose appropriate methods: Select methods that align with your evaluation criteria. This could include usability testing, surveys, A/B testing, or expert reviews.
3. Develop test materials: Create task scenarios, survey questions, or testing protocols that will effectively measure your evaluation criteria.
4. Recruit participants: For user-based methods, recruit participants who represent your target users. For expert-based methods, identify appropriate evaluators.
5. Set up the testing environment: Prepare the necessary equipment and ensure the testing environment is as realistic as possible.
6. Conduct the evaluation: Run your chosen evaluation methods, ensuring consistent procedures across all sessions or participants. Gather both quantitative (e.g., task completion times, success rates) and qualitative (e.g., user comments, observations) data.
7. Analyze results: Review all collected data, looking for patterns and insights. Use statistical analysis for quantitative data where appropriate.
8. Report results: Compile your findings, analysis, and recommendations into a clear, compelling report for stakeholders.

What are the pros and cons of evaluative research?

Pros:

• Provides concrete data
• Identifies specific usability issues
• Measures improvement over time

Cons:

• May miss broader user needs
• Can be affected by artificial test environments
• May focus too much on current solutions

Tools to help with evaluative research

1. UserTesting: This platform allows researchers to conduct both moderated and unmoderated usability tests, providing video recordings of user interactions and feedback.
2. SurveyMonkey: A versatile tool for creating and distributing surveys, SurveyMonkey is useful for gathering quantitative data about user satisfaction and preferences.
3. Hotjar: This tool provides heatmaps and session recordings, allowing researchers to visualize user behavior on websites and identify usability issues.

Frequently Asked Questions (FAQs)

What is the difference between generative and exploratory research?

In the context of generative vs evaluative research, generative and exploratory research are closely related. Both fall under the umbrella of formative (generative) vs. summative (evaluative) research. Generative research focuses on creating new ideas and solutions, while exploratory research aims to understand the current situation. Both are part of the broader category of generative research vs evaluative research, occurring earlier in the design process than evaluative methods.

What is the difference between evaluation research and evaluative research?

When discussing evaluative vs generative research, the terms evaluation research and evaluative research are often used interchangeably. Both are part of summative (evaluative) research in the formative (generative) vs. summative (evaluative) research spectrum. In UX, we typically use "evaluative research" to describe methods that test and improve designs, contrasting it with generative research in the generative vs evaluative research UX framework.

What is the difference between generative and foundational research?

In the context of generative vs. evaluative research, generative research creates new ideas based on user insights, while foundational research builds a knowledge base about users and their context. Both are part of formative (generative) research in the formative (generative) vs. summative (evaluative) research model. Generative research often builds upon foundational research, and both contrast with evaluative approaches in the generative research vs evaluative research paradigm.

What is the opposite of generative research?

In the evaluative vs generative research framework, evaluative research is often considered the opposite of generative research. While generative research creates new ideas and explores possibilities, evaluative research assesses existing designs. This distinction is key to understanding the generative research vs evaluative dichotomy in UX research. However, it's important to note that in practice, these approaches complement each other in the generative vs evaluative research UX process.