Best-Worst Scaling

Best-Worst Scaling (BWS)[1] techniques involve Choice modelling (or Discrete Choice Experiment - "DCE") and were invented by Jordan Louviere in 1987 while on the faculty at the University of Alberta. In general with BWS, survey respondents are shown a subset of items from a master list and are asked to indicate the best and worst items (or most and least important, or most and least appealing, etc.). The task is repeated a number of times, varying the particular subset of items in a systematic way, typically according to a statistical design. Analysis is typically conducted, as with DCEs more generally, assuming that respondents makes choices according to a random utility model (RUM). RUMs assume that an estimate of how much a respondent prefers item A over item B is provided by how often item A is chosen over item B in repeated choices. Thus, choice frequencies estimate the utilities on the relevant latent scale. BWS essentially aims to provide more choice information at the lower end of this scale without having to ask additional questions that are specific to lower ranked items.

History

Louviere attributes the idea to the early work of Anthony A. J. Marley in his PhD thesis, who together with Duncan Luce in the 1960s produced much of the ground-breaking research in mathematical psychology and psychophysics to axiomatise utility theory. Marley had encountered problems axiomatising certain types of ranking data and speculated in the discussion of his thesis that examination of the "inferior" and "superior" items in a list might be a fruitful topic for future research. The idea then languished for three decades until the first working papers and publications appeared in the early 1990s. The definitive textbook describing the theory, methods and applications was published in September 2015 (Cambridge University Press) by Jordan Louviere (University of South Australia), Terry N Flynn (TF Choices Ltd) and Anthony A. J Marley (University of Victoria and University of South Australia).[1] The book brings together the disparate research from various academic and practical disciplines, in the hope that replication and mistakes in implementation are avoided. The three authors have (individually and together) already published many of the key academic peer-reviewed articles describing BWS theory,[2][3][4] practice,[5][6] and a number of applications in health,[5] social care,[7] marketing,[6] transport, voting,[8] and environmental economics.[9] However, the method has now become popular in the wider research and practitioner communities, with other researchers exploring its use in areas as diverse as student evaluation of teaching,[10] marketing of wine,[11] quantification of concerns over ADHD medication,[12] the importance of environmental sustainability,[13] and priority-setting in genetic testing.[14]

Purposes of Best-Worst Scaling

There are two different purposes of BWS – as a method of data collection, and/or as a theory of how people make choices when confronted with three or more items. This distinction is crucial, given the continuing misuse of the term maxdiff to describe the method. As Marley and Louviere note, maxdiff is a long-established academic mathematical theory with very specific assumptions about how people make choices:[2] it assumes that respondents evaluate all possible pairs of items within the displayed set and choose the pair that reflects the maximum difference in preference or importance.

BWS as a theory of process (theory of decision-making)

Consider a set in which a respondent evaluates four items: A, B, C and D. If the respondent says that A is best and D is worst, these two responses inform us about five of six possible implied paired comparisons:

A > B, A > C, A > D, B > D, C > D

The only paired comparison that cannot be inferred is B vs. C. In a choice among five items, MaxDiff questioning informs on seven of ten implied paired comparisons. Thus BWS may be thought of as a variation of the method of Paired Comparisons.

Yet respondents can produce best-worst data in any of a number of ways. Instead of evaluating all possible pairs (the maxdiff model), they might choose the best from n items, the worst from the remaining n-1, or vice versa. Or indeed they may use another method entirely. Thus it should be clear that maxdiff is a subset of BWS. The maxdiff model has proved to be useful in proving the properties of a number of estimators in BWS.[2][3][4] However, its realism as a description of how humans might actually provide best and worst data can be questioned for the following reason. As the number of items increases, the number of possible pairs increases in a multiplicative fashion: n items produces n(n-1) pairs (where best-worst order matters). To assume that respondents do evaluate all possible pairs is a strong assumption and in 14 years of presentations, the three co-authors have virtually never found a course or conference participant who admitted to using this method to decide their best and worst choices.[1] Virtually all admitted to using sequential models (best then worst or worst then best).[15]

Early work (including that of Louviere himself) did use the term maxdiff to refer to BWS, but with the recruitment of Marley to the team developing the method, correct academic terminology has been disseminated throughout Europe and Asia-Pacific (if not North America, which continues to use the maxdiff term). Indeed, it is an open question whether major software manufacturers of discrete choice maxdifff routines actually implement maxdiff models in estimating parameters, despite this continuing advertising of maxdiff capabilities.

BWS as a method of data collection

The second use of BWS is as a method of data collection (rather than as a theory of how humans produce a best and a worst item). BWS can, particularly in the age of web-based surveys, be used to collect data in a systematic way that (1) forces all respondents to provide best and worst data in the same way (by, for instance, asking best first, greying out the chosen option, then asking worst); (2) Enables collection of a full ranking, if repeated BWS questioning is implemented to collect the "inner rankings". In many contexts, BWS for data collection has been regarded merely as a way to obtain such data in order to facilitate data expansion (to estimate conditional logit models with far more choice sets) or to estimate conventional rank ordered logit models.[16]

Types ("cases") of Best-Worst Scaling

The renaming of the method, to make clear that maxdiff scaling is BWS but BWS is not necessarily maxdiff, was decided by Louviere in consultation with his two key contributors (Flynn and Marley) in preparation for the book, and was presented in an article by Flynn.[17] That paper also took the opportunity to make clear that there are, in fact, three types ("cases") of BWS: Case 1 (the "object case"), Case 2 (the "profile case") and Case 3 (the "multi-profile case"). These three cases differ largely in the complexity of the choice items on offer.

Case 1 (the "object case") of BWS

Case 1 presents items that may be attitudinal statements, policy goals, marketing slogans or any type of item that has no attribute and level structure. It is primarily used to avoid scale biases known to affect rating (Likert) scale data.[18][19] It is particularly useful when eliciting the degree of importance or agreement that respondents ascribe from a set of statements and when the researcher wishes to ensure that the items compete with each other (so that respondents cannot easily rate multiple items as being of the same importance).

Case 2 (the "profile case") of BWS

Case 2 has predominated in health and the items are the attribute levels describing a single profile of the type familiar to choice modellers. Instead of making choices between profiles, the respondent must make best and worst (most and least) choices within a profile. Thus, for the example of a mobile (cell) phone, the choices would be the most acceptable and least acceptable features of a given phone. Case 2 has proved to be powerful in eliciting preferences among vulnerable groups, such as the elderly,[20][21] older carers,[22] and children,[23] who find conventional multi-profile discrete choice experiments difficult. Indeed, the first comparison of Case 2 with a DCE in a single model found that whilst the vast majority of (older) respondents provided usable data from the BWS task, only around one half do so for the DCE.[20]

Case 3 (the "multi-profile case") of BWS

Case 3 is perhaps the most familiar to choice modellers, being merely an extension of a discrete choice model: the number of profiles must be three or more, and instead of simply choosing the one the respondent would purchase, (s)he chooses the best and worst profile.

Designs for Best-Worst Scaling studies

Case 1 BWS studies typically use Balanced Incomplete Block Designs (BIBDs). These cause every item to appear the same number of times and also force every item to compete with every other the same number of times. These features are attractive since the respondent is prevented from inferring erroneous information about the items (what items the designer is "really" interested in).[1] They also ensure that there can be no "ties" in importance/salience at the very top or bottom of the scale.

Case 2 BWS studies can use Orthogonal Main Effects Plans (OMEPs) or efficient designs, although the former has predominated to date.

Case 3 BWS studies may use any of the types of design typically used for a DCE, with the proviso that the number of profiles (alternatives) in a choice set must be three or more for the BWS task to make sense.

Recent history of Best-Worst Scaling

Steve Cohen introduced BWS to the marketing research world in a paper presented at an ESOMAR Conference in Barcelona in 2002 entitled, "Renewing market segmentation: Some new tools to correct old problems." This paper was nominated for Best paper at that conference. In 2003 at the ESOMAR Latin America Conference in Punta del Este, Uruguay, Steve and his co-author, Dr. Leopldo Neira, compared BWS results to those obtained by rating scale methods. This paper won Best Methodological Paper at that conference. Later the same year, it was selected as winner of the John and Mary Goodyear Award for Best Paper at all ESOMAR Conferences in 2003 and then it was published as the lead article in "Excellence in International Research 2004," published by ESOMAR. At the 2003 Sawtooth Conference, Steve Cohen's paper "Maximum Difference Scaling: Improved Measures of Importance and Preference for Segmentation," was selected as Best Presentation. Steve and Bryan agreed that MaxDiff should be part of the Sawtooth package and it was introduced later that year. Later in 2004, Steve Cohen and Bryan Orme of Sawtooth Software won the David K. Hardin Award from the AMA for their paper which was published in Marketing Research Magazine entitled, "What's your preference? Asking survey respondents about their preferences creates new scaling decisions."

In parallel to this, Emma McIntosh and Jordan Louviere introduced BWS (case 2) to the health community at the 2002 Health Economists' Study Group conference. This prompted the collaboration with Flynn and ultimately the link-up with Marley, who had begun working with Louviere independently to prove the properties of BWS estimators. The popularity of the three cases has largely varied by academic discipline, with case 1 proving popular in marketing and food research, case 2 largely being adopted in health, and case 3 being used across a variety of disciplines that already use DCEs. It was partly this lack of understanding in many disciplines that there are actually three cases of BWS that prompted the three main developers to write the textbook.

The book contains an introductory chapter summarising the history of BWS and the three cases, together with why the respondent must think whether (s)he wishes to use it to understand theory (processes) of decision-making and/or merely to collect data in a systematic way. Three chapters, one for each case, follow, detailing the intuition and application of each. A chapter bringing together Marley's work proving the properties of the key estimators and laying out some open issues then follows. After laying out open issues for further analysis, nine chapters (three per case – describing applications from a variety of disciplines) then follow.

Conducting a BWS study

The basic steps in conducting all types of BWS study are:

Analysis

Estimation of the utility function is performed using any of a variety of methods.

  1. multinomial discrete choice analysis, in particular multinomial logit (strictly speaking the conditional logit, although the two terms are now used interchangeably). The multinomial logit (MNL) model is often the first stage in analysis and provides a measure of average utility for the attribute levels or objects (depending on the Case).
  2. In many cases, particularly cases 1 and 2, simple observation and plotting of choice frequencies should actually be the first step, as it is very useful in identifying preference heterogeneity and respondents using decision-rules based on a single attribute.
  3. Several algorithms could be used in this estimation process, including maximum likelihood, neural networks, and the Hierarchical Bayes model. The Hierarchical Bayes model is beneficial because it allows for borrowing across the data, although since BWS often allows the estimation of individual level models, the benefits of Bayesian models are heavily attenuated. Response time models have recently been shown to replicate the utility estimates of BWS, which represents a major step forward in the validation of stated preferences generally, and BWS preferences specifically.[25][26]

Advantages of Best-Worst Scaling

BWS questionnaires are relatively easy for most respondents to understand. Furthermore, humans are much better at judging items at extremes than in discriminating among items of middling importance or preference[citation needed]. And since the responses involve choices of items rather than expressing strength of preference, there is no opportunity for scale use bias.

Respondents find these ratings scales very easy but they do tend to deliver results which indicate that everything is "quite important", making the data not especially actionable.[citation needed] BWS on the other hand forces respondents to make choices between options, while still delivering rankings showing the relative importance of the items being rated. It also produces:

Disadvantages of Best-Worst Scaling

Best-worst scaling involves the collection of at least two sets of data: at a minimum, first-best and first-worst, and in some cases additional ranks (second best, second worst, etc...) The issue of how to combine these data is pertinent. Early work assumed best was simply the inverse of worst: that respondents had an internal ranking of all items and just chose the highest/lowest ranked item in a given question. More recent work has suggested that in some contexts this is not the case: a person might (for instance) choose according to traditional economic theory for best (trading across attributes) but choose worst using an elimination by attributes strategy (choosing as worst the item that is simply unacceptable on one attribute). In the presence of such different decision rules it becomes impossible to know how to combine the data: at what point does the person, when moving down the rankings, move from "economic trading" to "elimination by aspects".

This presents a clear problem for the data augmentation motivation for BWS but not necessarily for BWS when used as a way to understand process (decision-making). Psychologists in particular would be particularly interested in the different types of decision-making. Marketers, also, might wish to know if a given product had an unacceptable feature. Work is ongoing to investigate when different decision rules arise, and whether/how data from such different sources may be combined.

BWS also suffers from the same disadvantages of all stated preference techniques. It is unknown if the preferences are consistent with choices made in the real world (revealed preferences). In some instances revealed preferences (typically real market decisions) are available, providing a test of the BWS choices. In others, quite often health, there are no revealed preference data and validation appears impossible. More recently attempts have been made to validate SP data using physiological data, such as eye-tracking and response times.[25] Early work suggests that response time models are consistent with results from BWS models in health care but more research is required in other contexts.

References

  1. 1 2 3 4 "Best-Worst Scaling". Cambridge University Press. Retrieved 2015-10-01.
  2. 1 2 3 Marley, A. A. J.; Louviere, J. J. (2005-12-01). "Some probabilistic models of best, worst, and best–worst choices". Journal of Mathematical Psychology. Special Issue Honoring Jean-Claude Falmagne: Part 1Special Issue Honoring Jean-Claude Falmagne: Part 1 49 (6): 464–480. doi:10.1016/j.jmp.2005.05.003.
  3. 1 2 Marley, A. A. J.; Flynn, Terry N.; Louviere, J. J. (2008-10-01). "Probabilistic models of set-dependent and attribute-level best–worst choice". Journal of Mathematical Psychology 52 (5): 281–296. doi:10.1016/j.jmp.2008.02.002.
  4. 1 2 Marley, A. A. J.; Pihlens, D. (2012-02-01). "Models of best–worst choice and ranking among multiattribute options (profiles)". Journal of Mathematical Psychology 56 (1): 24–34. doi:10.1016/j.jmp.2011.09.001.
  5. 1 2 Flynn, Terry N.; Louviere, Jordan J.; Peters, Tim J.; Coast, Joanna (2007-01-01). "Best–worst scaling: What it can do for health care research and how to do it". Journal of Health Economics 26 (1): 171–189. doi:10.1016/j.jhealeco.2006.04.002.
  6. 1 2 Louviere, Jordan; Lings, Ian; Islam, Towhidul; Gudergan, Siegfried; Flynn, Terry (2013-09-01). "An introduction to the application of (case 1) best–worst scaling in marketing research". International Journal of Research in Marketing 30 (3): 292–303. doi:10.1016/j.ijresmar.2012.10.002.
  7. ↑ Potoglou, Dimitris; Burge, Peter; Flynn, Terry; Netten, Ann; Malley, Juliette; Forder, Julien; Brazier, John E. (2011-05-01). "Best–worst scaling vs. discrete choice experiments: An empirical comparison using social care data". Social Science & Medicine 72 (10): 1717–1727. doi:10.1016/j.socscimed.2011.03.027.
  8. ↑ García-Lapresta, José Luis; Marley, A. a. J.; Martínez-Panero, Miguel (2009-09-12). "Characterizing best–worst voting systems in the scoring context". Social Choice and Welfare 34 (3): 487–496. doi:10.1007/s00355-009-0417-1. ISSN 0176-1714.
  9. ↑ Scarpa, Riccardo; Notaro, Sandra; Louviere, Jordan; Raffaelli, Roberta (2011-06-19). "Exploring Scale Effects of Best/Worst Rank Ordered Choice Data to Estimate Benefits of Tourism in Alpine Grazing Commons". American Journal of Agricultural Economics: aaq174. doi:10.1093/ajae/aaq174. ISSN 0002-9092.
  10. ↑ Huybers, Twan (2014-05-19). "Student evaluation of teaching: the use of best–worst scaling". Assessment & Evaluation in Higher Education 39 (4): 496–513. doi:10.1080/02602938.2013.851782. ISSN 0260-2938.
  11. ↑ "Applying best‐worst scaling to wine marketingnull". International Journal of Wine Business Research 21 (1): 8–23. 2009-03-20. doi:10.1108/17511060910948008. ISSN 1751-1062.
  12. ↑ Ross, Melissa; Bridges, John F. P.; Ng, Xinyi; Wagner, Lauren D.; Frosch, Emily; Reeves, Gloria; dosReis, Susan (2014-11-17). "A Best-Worst Scaling Experiment to Prioritize Caregiver Concerns About ADHD Medication for Children". Psychiatric Services 66 (2): 208–211. doi:10.1176/appi.ps.201300525. ISSN 1075-2730.
  13. ↑ Mueller Loose, Simone; Lockshin, Larry (2013-03-01). "Testing the robustness of best worst scaling for cross-national segmentation with different numbers of choice sets". Food Quality and Preference. Ninth Pangborn Sensory Science Symposium 27 (2): 230–242. doi:10.1016/j.foodqual.2012.02.002.
  14. ↑ Severin, Franziska; Schmidtke, Jörg; Mühlbacher, Axel; Rogowski, Wolf H. (2013-11-01). "Eliciting preferences for priority setting in genetic testing: a pilot study comparing best-worst scaling and discrete-choice experiments". European Journal of Human Genetics 21 (11): 1202–1208. doi:10.1038/ejhg.2013.36. ISSN 1018-4813. PMC 3798841. PMID 23486538.
  15. ↑ Flynn, Terry N.; Louviere, Jordan J.; Peters, Tim J.; Coast, Joanna (2008-11-18). "Estimating preferences for a dermatology consultation using Best-Worst Scaling: Comparison of various methods of analysis". BMC Medical Research Methodology 8 (1): 76. doi:10.1186/1471-2288-8-76. ISSN 1471-2288. PMC 2600822. PMID 19017376.
  16. ↑ Louviere, Jordan J.; Street, Deborah; Burgess, Leonie; Wasi, Nada; Islam, Towhidul; Marley, Anthony A. J. (2008-01-01). "Modeling the choices of individual decision-makers by combining efficient choice experiment designs with extra preference information". Journal of Choice Modelling 1 (1): 128–164. doi:10.1016/S1755-5345(13)70025-3.
  17. ↑ Flynn, Terry N. (2010-06-01). "Valuing citizen and patient preferences in health: recent developments in three types of best–worst scaling". Expert Review of Pharmacoeconomics & Outcomes Research 10 (3): 259–267. doi:10.1586/erp.10.29. ISSN 1473-7167.
  18. ↑ Baumgartner, Hans; Steenkamp, Jan-Benedict E.M. (2001-05-01). "Response Styles in Marketing Research: A Cross-National Investigation". Journal of Marketing Research 38 (2): 143–156. doi:10.1509/jmkr.38.2.143.18840. ISSN 0022-2437.
  19. ↑ Steenkamp, Jan‐Benedict E. M.; Baumgartner, Hans (1998-06-01). "Assessing Measurement Invariance in Cross‐National Consumer Research". Journal of Consumer Research 25 (1): 78–107. doi:10.1086/209528.
  20. 1 2 3 N. Flynn, Terry; J. Peters, Tim; Coast, Joanna (2013-03-01). "Quantifying response shift or adaptation effects in quality of life by synthesising best-worst scaling and discrete choice data". Journal of Choice Modelling 6: 34–43. doi:10.1016/j.jocm.2013.04.004.
  21. ↑ Coast, Joanna; Flynn, Terry N.; Natarajan, Lucy; Sproston, Kerry; Lewis, Jane; Louviere, Jordan J.; Peters, Tim J. (2008-09-01). "Valuing the ICECAP capability index for older people". Social Science & Medicine. Part Special Issue: Ethics and the ethnography of medical research in Africa 67 (5): 874–882. doi:10.1016/j.socscimed.2008.05.015.
  22. ↑ Al-Janabi, Hareth; Flynn, Terry N.; Coast, Joanna (2011-05-01). "Estimation of a Preference-Based Carer Experience Scale". Medical Decision Making 31 (3): 458–468. doi:10.1177/0272989X10381280. ISSN 0272-989X. PMID 20924044.
  23. ↑ Ratcliffe, Professor Julie; Flynn, Terry; Terlich, Frances; Stevens, Katherine; Brazier, John; Sawyer, Michael (2012-12-23). "Developing Adolescent-Specific Health State Values for Economic Evaluation". PharmacoEconomics 30 (8): 713–727. doi:10.2165/11597900-000000000-00000. ISSN 1170-7690.
  24. ↑ Coast, Joanna; Al-Janabi, Hareth; Sutton, Eileen J.; Horrocks, Susan A.; Vosper, A. Jane; Swancutt, Dawn R.; Flynn, Terry N. (2012-06-01). "Using qualitative methods for attribute development for discrete choice experiments: issues and recommendations". Health Economics 21 (6): 730–741. doi:10.1002/hec.1739. ISSN 1099-1050.
  25. 1 2 Hawkins, Guy E.; Marley, A.a.j.; Heathcote, Andrew; Flynn, Terry N.; Louviere, Jordan J.; Brown, Scott D. (2014-05-01). "Integrating Cognitive Process and Descriptive Models of Attitudes and Preferences". Cognitive Science 38 (4): 701–735. doi:10.1111/cogs.12094. ISSN 1551-6709.
  26. ↑ "The best of times and the worst of times are interchangeable.". APA PsycNET. Retrieved 2015-10-01.
This article is issued from Wikipedia - version of the Sunday, March 06, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.