by Michael Mora, 26. August 2013
http://www.relevantinsights.com/why-conjoint-analysis-results-dont-always-match-reality
Over the past year I have been involved in several conjoint studies either for price or feature optimization or both in which clients are faced with many unknowns and struggle to reconcile the research results with the real world. The thing is that conjoint analysis can produce results in which preference shares align closely with market shares if the right information is available and embedded into the experimental design and market simulations. Here are some of the variables that have an impact on how well conjoint results could match reality: Attributes and Levels: This seems to be a basic consideration, but it is often the culprit of misalignment of the conjoint results. Sometimes attributes are simplified when too many levels are possible or simply omitted when they are difficult to represent in a conjoint survey setting, or there are too many attributes for a respondent to consider. Other times, preliminary research, which can pinpoint relevant attributes and levels, is skipped for budgetary or other reasons, and the products are evaluated on attributes and levels that don’t do a good job at representing how products are evaluated in the real world. External Factors: A conjoint analysis may well capture preferences accurately based on the right attributes and levels; however factors such as level of awareness, marketing strategy and tactics, advertising of product benefits, sales process, distribution, category inertia, and competitors’ reaction, among others, are often unknown at the time of the study design or analysis. Any discrepancy between the assumptions made in the research and what’s happening in the marketplace regarding these factors can make conjoint analysis results inaccurate. Sampling Plan: Including the right type of respondents in the study is an important part of the study design, and by that I mean including people who are at least active in the category. Disengaged respondents due to lack of knowledge or experience in the category are not able to provide relevant answers and you may ended up underestimating preference share. Survey Design: The challenge is always to design surveys that mimic how consumers think about the product that is being evaluated to obtain realistic answers. However, this can be difficult when we have to balance the amount and type of information provided, respondent burden, and survey length (which has cost implications) particularly when we are evaluating complex products or services with many attributes and levels, competing alternatives or new products. There are occasions when the list of attributes is reduced to minimize respondent burden and shorten the survey. This could leave relevant attributes out. There are other situations in which to make it easy for the respondent to go through a conjoint exercise that requires a lot of information processing, of either complex or new information or both, respondents are often “primed” by exposing them to the attributes and levels through introductory questions before they get to the conjoint section. In this way when they get there, they are more familiar with the product attributes and can make more thoughtful decisions. However, in the real world, consumers often don’t have the time or will to get acquainted with all the characteristics of a product and make decisions on limited information. “Priming” can also lead to higher awareness of certain attributes than the advertising and marketing strategy and tactics would never produce, which then could lead to inflated importance of those attributes and over-prediction of a product’s preference share. I believe that techniques like conjoint analysis have many useful applications, but should be used with caution and full knowledge of its limitations based on the information available at the time of design and analysis in order to manage expectations on the validity of the results in the real world. I hope these issues don’t discourage you from using conjoint analysis, but make you aware of the information needed and the work required to make its results valuable and actionable.