Managing Intelligence: Skilled Experts and AI in Markets for Complex Products

In numerous high stakes markets skilled experts play a key role in facilitating consumer choice of complex products (e.g. doctors in health care or financial planners in choosing investment products). New artificial intelligence technologies can either be a complement to or substitute for expertise. We study the role of technology in expert productivity in the market for health insurance, where consumer choices are widely known to be suboptimal. We leverage unique, granular data on insurance agent effort, agent recommendations, enrollee health and enrollee preferences for the Medicare Advantage plans from the largest private Medicare marketplace. During our sample period, customers are randomly assigned to agents and the agent environment changes from one with little technology to one where agents are mandated to use a new AI-based recommendation tool assessing individual specific plan matches. We study the implications of technology adoption for agent recommendation quality and productivity, both on average and across the underlying skill distribution. We find that, prior to the mandated use of the predictive technology, skilled experts in this market make large mistakes costing the average customer $1,261 annually and there is widespread heterogeneity in skill. The choices made weight premiums over expected out-of-pocket spending by a factor of approximately 6:1 on average, with heterogeneity ranging from 3:1 to 18:1 across productivity quintiles. The mandated use of sophisticated decision support improves outcomes by $278 on average and substantially reduces heterogeneity in broker performance. The weight on premium versus expected out-of-pocket cost is 1:1 across the distribution of broker productivity. The lowest skilled agents make recommendations with AI using less effort that exceed the recommendation quality of the highest skilled agents without the tool. Technology is a substitute for skill and enhances productivity in our setting. We investigate the micro-foundations for broker behavior before and after mandated use of technology, study the implications of technology design, and discuss the implications of our study for the labor market for skilled experts.