Quality Checker for Survey Responses
The Quality Checker view provides a comprehensive overview of each individual response in your experiment, allowing you to review survey answers, detailed status, and a summary table of risk signals. With this information, you can assess response quality and manually include or exclude specific responses in your analysis.
![Quality checker view of experiment Quality checker view of experiment](https://conjointly.com/img/quality-checker-view-example.png)
Accessing the Quality Checker view
There are multiple ways to access the Quality Checker view of your launched experiment:
- Via the experiment setting page.
![Access the Quality Checker view via experiment setting page Access the Quality Checker view via experiment setting page](https://conjointly.com/img/quality-checker-via-launched-experiment-setting.png)
- Via the Review participants page. You can also click on the icon next to each participant to directly access the Quality Checker view for that specific participant.
![Access the Quality Checker view via Review participants page Access the Quality Checker view via Review participants page](https://conjointly.com/img/quality-checker-via-review-participant.png)
Viewing only responses used in analysis
The Quality Checker view includes all recorded responses for your experiment, even those automatically filtered out based on automated checks and your response quality management settings, which are labeled with a red cross ❌. To view only the responses used in your analysis, tick the checkbox labeled Viewing only responses used in analysis.
![Viewing only responses used in analysis Viewing only responses used in analysis](https://conjointly.com/img/only-response-used-in-analysis.gif)
Reviewing response status
The top-right corner of the Quality checker view displays detailed information about each response, including:
- Starting time and survey duration.
- Status: Opened or completed.
- Risk signals (if detected): Low quality, duplicate IP, duplicate cookie, duplicate GET variable, touch screen.
- Device type.
- Browser details.
- Location signals.
![Detailed response status Detailed response status](https://conjointly.com/img/quality-checker-response-status.png)
Manually exclude or include response
To ensure accurate results, you can manually assess the quality of each response and decide whether to include or exclude it from your analysis through one of the following options:
- The
You can find these icons below Assessment of quality section. Click the icon to mark response as high-quality and include it in analysis or the icon to mark response as low-quality and exclude it from analysis. Using these icons helps train our system for more accurate quality assessments in the future. After assessing a few responses, the system will generate a prompt that allows you to mark the remaining participants as having either good or bad quality responses. or icon - The
Conjointly recommends using the ‘Exclude’ button in the top-right corner to remove responses due to non-quality issues, such as quota fulfilment or ineligibility. button
![Detailed response status Detailed response status](https://conjointly.com/img/mark-participant-good-or-bad.png)
These options are also available on the Review Participants page.
Alternatively, you can include or exclude respondents from the analysis by uploading a special external variable called CONJOINTLY_ACTION
.
Checking risk signals
Underneath the survey answers and response status sections, you will find a summary table of risk signals that displays the details of any detected risk signals, including:
- Datetime: The exact date and time when the risk signal was detected.
- Occasion: Indicates when the risk signal was identified:
at-start
: Signal detected at the beginning of the survey.on-response
: Signal detected when saving the response.on-review
: Signal detected during automated checks.
- Signal: The specific type of risk signal identified.
- Value: The specific value associated with the detected risk signal.
- Description: A detailed explanation of the risk signal.
![Example of a summary table of risk signal Example of a summary table of risk signal](https://conjointly.com/img/quality-checker-risk-signal-table-example.png)
Common survey rule violations and associated risk signals
The table below provides a list of survey rule violations and the specific risk signals triggered when each violation occurs:
Survey rule violation | Signal | |
---|---|---|
Participant activates developer’s tool | dev-tools-open | |
Low-quality responses | ||
Short response time | short-length-of-interview | |
Short answer for a question | quick-answer-for-question | |
Insufficient scrolling on conjoint sets | insufficient-scrolling-in-conjoint | |
Short average duration on conjoint sets | short-average-duration-in-conjoint | |
Quick answers on conjoint sets | short-duration-in-some-conjoint-questions | |
Insufficient mouse movements on conjoint sets | insufficient-mouse-movement-in-conjoint | |
Gibberish characters or keyjamming | jibberish-text-keyjamming | |
Deviation between key presses and text input length | key-press-deviation | |
Duplicates of open-ended responses across respondents | duplicate-open-end | |
Low-quality open-ends detected by the large language model | ||
Responses that appear to be test submissions | open-end-test | |
Responses that are overly brief, vague, or fail to address the question adequately | open-end-careless | |
Responses that appear to be copied and pasted from other sources | open-end-copy-pasted | |
Responses containing nonsensical or gibberish characters | open-end-keyjamming | |
Responses that are relevant and related to the survey question | open-end-good | |
Duplicate survey submissions | ||
Duplicates by IP | duplicate-ip | |
Duplicates by Cookie | duplicate-cookie | |
Duplicates by GET variable | duplicate-get-variable | |
Location inconsistencies | ||
Differences in country | country-mismatch | |
Differences in timezone | timezone-mismatch | |
Very large distance by location | location-mismatch | |
Other errors | ||
Wrong device type | wrong-device | |
Screened-out responses with incorrect itemId | inconsistent-event | |
Error while answering conjoint sets | incorrect-conjoint-data | |
Error while loading image heatmaps | incomplete-tag |
General information and associated signals
The table of risk signals also record the following information:
General variables or information | Signal | |
---|---|---|
Browser environment variable | browser-environment-variable | |
Tab switch | tab-switches | |
Answers submitted from JavaScript | manual_submit | |
Google Translate detected | google-translate | |
General device information | ||
Device info languages | device-info-languages | |
Screen width | screen-width | |
Screen height | screen-height | |
Screen color depth | screen-color-depth | |
Screen pixel depth | screen-pixel-depth | |
Screen pixel ratio | screen-pixel-ratio | |
Touch screen support | screen-is-touched | |
Hardware concurrency | hardware-concurrency | |
Hardware memory | hardware-memory | |
Hardware platform | hardware-platform | |
Connection type | connection-type | |
Browser user agent | browser-user-agent | |
Browser app version | browser-app-version |