With Hierarchical Bayes estimation, we don’t just look at an individual’s choices in isolation to come up with a utility model that fits those choices. We also estimate average utilities of the entire sample, along with what the distribution of utilities might look like for all the respondents. The information coming from other respondents is used along with each individual respondent’s choices to estimate their utility values.
If a new respondent is added or removed and HB estimation is run again, this will result in very small changes to our sample and distribution estimates, which will ripple and cause (usually) very small changes to each respondent’s utilities.
Let’s use a simple example to illustrate (note this is not how HB works in practice). Assume our final utilities for each respondent is a simple approach where we combine the entire sample utilities with a set of utilities that fit each individual respondent, with a weight of 20% sample and 80% individual. The entire sample chose the color Red more often than the color Blue, while respondent 1 chose Red just slightly more often than Blue. We might end up with utilities like these:
| Sample | Individual | ||
|---|---|---|---|
| Red | Blue | Red | Blue |
| 1 | -1 | 0.2 | -0.2 |
Our simple math would then produce an final estimate of the Individual scores as:
| Individual | |
|---|---|
| Red | Blue |
| 0.36 | -0.36 |
However, if we remove a respondent from the sample, our Sample estimates will change in a very small way, practically no difference, but will also cause a very small change in the individual estimates.
| Sample | Individual | ||
|---|---|---|---|
| Red | Blue | Red | Blue |
| 0.9982 | -0.9982 | 0.2 | -0.2 |
| Individual | |
|---|---|
| Red | Blue |
| 0.35964 | -0.35964 |
HB estimation is a much more complicated procedure in practice, with an algorithm that usually involves tens of thousands of iterations where we update the estimates of the sample averages, distribution, and individual utilities. In practice with reasonably consistent respondents and sample sizes of hundreds or more, adding or removing individual respondents will have very little practical effect on the final respondent utility estimations. But, adding or removing a single respondent will always have some very small effect.
If you would like to explore HB analysis in more detail, we have some links to technical papers below:
Understanding HB: An Intuitive Approach