How do I interpret preference scores in Claims Test/PVS?
Preference scores used in a Claims Test or a Product Variant Selector experiment are essentially the same as partworth utilities in conjoint or MaxDiff studies. They measure how much customers liked each claim or product variant. Items that are strongly preferred by customers are assigned higher scores, items that perform poorly (in comparison to other items) are assigned lower scores. In this article, we deep-dive into how preference scores work.
Identifying a significant difference between claims
In Claims Test experiments, the “Summary of preferences and diagnostics for each claim” module displays your claim’s score and ranking. Although the ranking depends on the score, a claim that is higher ranked than the others isn’t necessarily higher performing since there is still noise in the data. Therefore, we compute if there are statistically significant differences between the scores and diagnostics. We use 90% bootstrap confidence intervals across all significance checks on the system.
In a Claims Test report, to check the significance of the difference between each of your claims:
Select two values under the row labelled “Score”.
Refer to the text displayed in the blue box which appears in your left-hand menu.
You can also apply the same procedure to check the significant difference between two or more diagnostic scores.
Alternatively, you can view the confidence interval of each value by hovering your cursor over it.
The above outlined steps and interpretation also apply when analysing a Product Variant Selector report.
Comparing a claim across different segments
In a claims test, scores should be compared within each segment (because each score is relative to the other scores within each segment). With this in mind, you should only compare scores across segments at a high level, as the difference in sample size alters the confidence interval across segments.
However, it is possible to compare how claims are ranked differently across segments which can be achieved through the crosstab feature.