MaxDiff design settings

Behind the scenes, the Discover exercise designer selects items to show across multiple tasks according to the following goals:  

1. Each item should appear an equal number of times.
2. Each item should appear with each other an equal number of times.
3. Each item should appear in each position (top, middle, bottom) of MaxDiff questions an equal number of times.

Depending on your list of items and number of tasks to show, it may not be possible to achieve perfection on all three goals; but MaxDiff exercises that are almost-but-not-quite perfect are still very efficient and you will obtain excellent results in practice.  

For more information about Discover-MaxDiff designs, please see this white paper: MaxDiff in Discover vs. Lighthouse Studio

For a typical MaxDiff study where a high degree of precision is desired at the individual respondent level, we have found that each respondent should ideally see each list item 3 or more times throughout the MaxDiff exercise.

For studies with modest to small sample sizes (e.g., N of 50 to 400) and when you wish to drill down by segment to compare preferences across respondent groups, it is valuable to have a relatively high degree of precision in the individual-level MaxDiff scores by showing each item 3 or more times per respondent.

In other studies with relatively large sample sizes (e.g., N of 600 or more) and less need to drill down by segments to analyze differences across respondent groups, you may opt for a design that shows each item around one to two times per respondent.

There are three variables that affect the exercise design, and subsequently the number of times each item is shown per respondent:

1. Total number of list items
2. Number of items per task
3. Number of tasks

MaxDiff exercises are based on lists of items for the respondent to compare typically in terms of importance or preference; and the length of your list (total number of items) is a key factor in determining its design.

MaxDiff is a robust research method that can allow hundreds of list items to be evaluated. MaxDiff in Discover requires a minimum of 6 items to create a viable design.

Though you can use a longer list, we recommend keeping list lengths to about 30 items or less for most applications.  Exceeding 30 items can lead to longer surveys that increase respondent fatigue and reduce score precision. Exceptions could involve situations in which you have relatively large sample sizes, where you plan to show each item around one to two times per respondent, and when you do not need to drill down with a high degree of accuracy at the individual level or to compare differences among subgroups.

The Number of items per task input allows you to change how many list items are shown to respondents on each set (or task) of the MaxDiff exercise. 

For most situations, we recommend displaying three to five items at a time (per task) in MaxDiff exercises. Showing more than about five items at a time can lead to greater respondent fatigue and response error. 

Additionally, we recommend that you do not show more than half of the total number of items per set; this reduces the precision of your MaxDiff scores. For example, if you have just eight items in your MaxDiff study, you should not display more than four items per set. 

The Number of tasks input allows you to determine how many total sets (or tasks) are shown to your respondents.  

This setting is a bit more flexible than items per task, though we highly recommend that you keep it within a reasonable number. Excessive tasks often lead to respondent fatigue and response error. 

There is a tradeoff between how many times each item is seen per respondent and the precision we obtain at the individual level for utility score estimation. Showing each item less than 2x per respondent leads to less precision at the individual level; but if you have relatively large sample size to compensate and if you don’t necessarily need high precision at the individual level, then you may decide to proceed.

We strongly recommend against showing more than half as many items per task as total items in the respondent’s MaxDiff list. If you do so, you will obtain relatively less information to distinguish the items of middling preference for the respondent.

The potential added information for considering more than 7 items at a time is probably counteracted by the increase in respondent error and fatigue. In general, we recommend showing 5 or fewer items per MaxDiff set.

For it to make sense to ask respondents to mark the best and worst items in a set, we must show set sizes of at least 3 items.

You cannot display more items in a task than exist in the respondent’s MaxDiff list.

Rather than repeating a MaxDiff task (showing the same items) for the same respondent, the software prohibits this from occurring. If it isn’t possible to create any more unique sets of items, the software tells you.

The software requires at least one MaxDiff task when inserting a MaxDiff exercise.

Connectivity means that within a respondent’s given set of MaxDiff tasks, each item is either directly or indirectly compared to all other items in the list. This is a desirable property when individual-level analysis is the default (Discover defaults to individual-level HB analysis). If you plan to use aggregate (pooled) approaches to item score estimation such as aggregate logit or latent class MNL, you might decide that it’s OK to sacrifice connectivity at the individual respondent level, because connectivity will be established when pooling across respondents.