Respondent Removal

Project Overview

Design and research of feature that enabled the manual removal of poor data from survey results on a market research platform.

Role

UX Researcher

Test Script & Recruitment
Usability Testing
Analysis

UX Designer

Exploration
Wireframing
Prototyping
Presentation

Final Design

Key Impact and Results

This new feature allowed the center of excellence team (COE) and clients to remove bots and poor performers from survey data on the same platform that delivers the questionnaire results.

The launched feature saw strong adoption, with 64% of customers using it in the first 10 months and 7,500 reported uses among large enterprises and Fortune 500 companies.

Constraints

In order to design this new feature, I worked closely with the project manager to understand the business goals, as well as market researchers, to understand their needs.

These new features would scale out to all market research projects on the platform, including quantitative and qualitative research.

Business requirements were in conflict with what clients actually needed and engineering's capabilities which meant navigating ambiguity throughout the research and design process.

The Challenge

Clients used this site to collect survey results but sometimes struggled with the infiltration of bots and straightliners, compromising data quality. COE and clients were utilizing tools off platform to manually remove poor performers from survey data for clients, which was tedious and time consuming. This business did not want to lose clients to competitors and was looking for a way to improve the data quality.

What feature could I design that would empower COE and clients to identify and remove poor respondents from their survey results efficiently and effectively?

TLDR

Over two months I spoke to clients to better understand their needs and expectations around data cleaning. Then, I mocked up Wireframes exploring different methods of interaction for removal, Prototyped flows, and conducted three rounds of Usability Testing in order to understand clients' mental model around data cleaning and Engineering's capabilities to create a minimal viable product.

The launched feature saw strong adoption, with 64% of customers using it in the first 10 months and 7,500 reported uses.

What is Data Cleaning?

Data Cleaning is the process of finding and removing inaccurate, incomplete, or irrelevant data in a dataset in order to improve data quality and reliability, in order to execute more accurate decisions.

The focus of this MVP was to create an in-platform tool to remove poor respondents from survey data. The MVP required a new page where clients could manually remove respondents from a survey and view the history of updates in-platform.

Business Analysis stipulated two methods to remove respondents:

Copy and paste a user ID
Bulk removal via template

Empathize

I spoke to 5 market researchers about their survey data usage and needs in order to better understand their expectations around survey results.

Key Takeaways

Market researchers requested the following:

A way to remove poor performers while the survey was live and completed in-platform.
Ability to remove 1 - 500 respondents from survey data at a time.
Confirmation that respondents are removed.
History of submissions to track who was removed.

Design

I mocked up a prototype following the upload media patterns on the platform. Creating a new tab for Data Cleaning, I added a content switch for clients to select their method of removal. Clients reported the IDs of respondents to be scrubbed from survey data. Upon submission, a report was generated in the table below.

Usability Testing

The project was then usability tested by 5 internal participants.

All confirmed the feature was intuitive, "easy to use", and inspired high confidence, validating its readiness for development.

Delivery

I reviewed the results with the PM and Stakeholders and attained buy in. Then, I annotated the designs for engineering and handed off the mid-fidelity wires to the UI Design team.

The launched feature saw strong adoption, with 64% of customers using it in the first 10 months and 7,500 reported uses.

UI Design Phase I

Second Release

Shortly after the feature's release, Business wanted to understand why clients were removing respondents from surveys.

According to my conversations with the Center of Excellence team, client's typically removed respondents for the following reasons:

Incomplete responses
Red herrings
Streaking
Poor open-end responses
Bots

To investigate, I mocked up a design that prompts client's to select a reason for removal from a dropdown.

Next, I usability tested the form interaction and predefined options with 5 COE and CSMs.

Clients weren't just using this feature to removed poor respondents from their survey data, but to capture accurate quotas.
Client's wanted a designated time period during which they could remove respondents before the survey closed.
Since respondents could be removed and backfilled while a survey was live, clients did not know how to determine the total number of participants in a survey.

Working with Product and Engineering, I iterated on the following insights:

Renaming "Data Cleaning" to "Respondent Removal"
Added a table that shows:
- The target number of respondents for the survey
- Number of current completes
- Number of removed respondents
Created a new status in the progress stepper for a "Reviewable" period so clients are notified when the data is ready.
Added a control to manually backfill responses.

Of these insights, the first 2 were implemented in the scope for this update, while the others were assigned as a follow up assignment several months later.

Next Steps

Findings

Exploration & Testing

UXUI Design

Phase II UXUI Design

Third Release

This iteration was inspired by insights gained during the previous usability test.

Internal researchers requested:

A desire to restore removed respondents.
The capacity to cancel a removal in progress.
A method to backfill survey respondents before the survey closed.

Regarding restoration, I mocked up several versions to be used exclusively by admins per business requirements. Then worked to add a new status to the progress stepper for a "Reviewable" period so clients could be notified when the data was ready to be refined. Next, I mocked up button placements to backfill a survey. A smoother experience may have been to automate this process, but due to limitations placed on us from the business side, it would need to be a manual process. I then made a few updates to the report history to account for these changes.

After conferring with the Experience Team and Project Manager, I usability tested the updates on Zoom with 3 internal employees.

Users understood the purpose of the new status in the progress stepper.
Users understood the table copy for the report history.
Users enjoyed being able to cancel an upload, backfill their survey results, and restore removed respondents.
Users reported the confirmation modal copy was clear for cancelling and restoring responses.
Users found the CTA to cancel difficult because of its small size.
Although having a backfill limit makes sense from a business perspective, tracking reports the average backfill is currently around 5% for clients. Participants were uneasy about the 10% limit allotted.

Added a backfill touchpoint to the 'Update Status' menu on the details page.
Added an 'Update Status' menu to the Respondent Removal page for parity.
Added the survey close date and time on the respondent removal page.
Updated cancel icon to an 'x' on the row where the removal is happening.
Updated backfill modal copy to clarify how many responses the survey will backfill in order to reduce the fear about limitations.
Added a popover in the page header including a link to educate users about respondent removal.
Added popover copy to explain what each term means for respondent removal.

I designed the above updates while working closely with the Project Manager to update the alert clients received when results were ready for review.

Iteration

Findings

Usability Testing

Exploration

Solution