Anchoring vignettes and measures of health: adjusting for systematic differences in reporting behaviour

Preference-based measures of health are often used in economic evaluations to calculate quality adjusted life years (QALYs).  More recently they have been used to measure the performance of hospitals (for example PROMS in the UK) and included in general population surveys.  Increasingly they have been used as a measure of population health status; for example, comparisons of EQ-5D utility values have been made by socio-economic status and behavioural risk factors.  The EQ-5D as a subjective measure is likely to suffer from mis-reporting and reporting heterogeneity, like other self-rated measures of health.  This paper explores the use of anchoring vignettes to test and correct for systematic differences in responses and improve inter-group comparisons of EQ-5D profiles.  We chose this measure for comparison as it is widely used internationally.

Given the EQ-5D has five specific domains which require completion in order to estimate a utility value, we specifically designed vignettes to represent separately the five EQ-5D domains (version A) but also a broader health state description, encompassing all domains, as a whole (version B), and therefore test for differences in responses when utilising the two versions to adjust respondents’ own EQ-5D health state.  Using a large online survey panel, we collected data on own health state, ratings of hypothetical health states, objective assessments of health and standard demographics. We also gathered information on possible response consistency.  Response consistency is a necessary assumption of anchoring vignettes, and requires that a respondent uses the response categories in the same way for the vignettes as themselves.  In earlier pilot work we qualitatively examined response consistency, and found that many individuals failed to respond consistently.  As such we included a number of reflective questions in the online survey to provide an important initial indication of response consistency, before applying a parametric approach.

We apply a hierarchical ordered probit (HOPIT) model to the individual domain responses, and estimate the cut-point thresholds.  Notably while the HOPIT suggests that response consistency is confirmed, our reflective response consistency questions suggest otherwise.  Response consistency aside, we find that systematic differences are apparent across individuals, and the more holistic vignette (version B) appears to adjust for these differences better than the individual domain level vignettes (version A).

Our results suggest that anchoring vignettes may be an appropriate tool to adjust for differential item functioning in the EQ-5D.  We also contribute to the literature on the validity of the underlying assumptions of the anchoring vignettes approach.