AI & Knowledge Work: Boosting Productivity or Disrupting Performance?

I just re-read the 2023 study Navigating the Jagged Technological Frontier that explores GenAI's impact on knowledge-intensive tasks and thought it's worthwhile to share a summary and my key take aways. Conducted by Harvard Business School and Boston Consulting Group, this large-scale field experiment with 758 consultants examines how AI augments productivity and quality—and when it might backfire.

Study Summary

Design: Consultants tackled 18 realistic tasks, split into two categories—those AI could handle well (inside the frontier) and those beyond AI's capabilities (outside the frontier). Participants were randomly assigned access to GPT-4, with or without training.

Key Insight: AI creates a "jagged technological frontier," excelling at some tasks while underperforming at others.

Study Findings

Inside the Frontier:

• Consultants using GenAI completed 12% more tasks and were 25% faster.

• AI improved response quality by over 40%.

• Lower-skilled consultants benefitted the most, with a 43% performance boost.

Outside the Frontier:

• Consultants relying on GenAI were 19% less likely to achieve correct outcomes.

• Blind adoption of GenAI suggestions led to reduced effectiveness.

The research team also frames two different Human-AI Integration Models:

1. Centaurs: Strategically delegate between human and AI sub-tasks

2. Cyborgs: Integrate AI and human capabilities at a granular, sub-task level

Key take aways for Business Leaders

• Upskill your team and get smart about the potential of GenAI: Ensure your team becomes proficient in using available GenAI tools. Optimize workflows around GenAI's strengths to leverage it for tasks within its frontier, such as ideation, writing, and analysis.

• Provide guidance on governance: Clarify for what use cases your team is allowed to use free, publicly available GenAI tools and for which an enterprise version is required. There is a difference between experimentation vs. embedding tools into your company's workflows or products.

• Do not use GenAI blindly: GenAI excels at some use cases but can be terrible at others. Ensure controls are in place. Even several years after GenAI went mainstream, the best approaches to using GenAI are not fully understood (and are also constantly evolving) and need to be more deeply examined by scholars and practitioners. The field of LLM Evals - understanding how an LLM performs for a specific use case - is growing both within the tech companies as well as in the startup space.

As GenAI reshapes knowledge work, leaders must strategize its integration thoughtfully to unlock productivity gains while minimizing risks.

Study citation

Dell'Acqua, Fabrizio and McFowland III, Edward and Mollick, Ethan R. and Lifshitz-Assaf, Hila and Kellogg, Katherine and Rajendran, Saran and Krayer, Lisa and Candelon, François and Lakhani, Karim R., Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (September 15, 2023). Harvard Business School Technology & Operations Mgt. Unit Working Paper No. 24-013, The Wharton School Research Paper, Available at SSRN: https://ssrn.com/abstract=4573321 or http://dx.doi.org/10.2139/ssrn.4573321