Back
Research: quantifying GitHub Copilot’s impact on developer productivity and happiness
webRelevant to AI safety discussions around human-AI collaboration dynamics, automation effects on human agency, and how capable AI tools reshape professional workflows; useful as an empirical reference point for deployment and productivity impact studies.
Metadata
Importance: 38/100blog postprimary source
Summary
GitHub published a controlled study examining how Copilot, an AI pair programmer, affects developer productivity and wellbeing. The research found that developers using Copilot completed coding tasks significantly faster (55% faster in some tasks) and reported higher satisfaction and reduced frustration. The study provides empirical evidence on how AI code generation tools change human workflows and perceived productivity.
Key Points
- •Developers using GitHub Copilot completed tasks up to 55% faster than those without it in controlled experiments.
- •88% of Copilot users reported feeling more productive, and a majority reported reduced mental effort on repetitive tasks.
- •The study highlights improved developer happiness and reduced frustration as key qualitative benefits alongside speed gains.
- •Findings suggest AI assistance shifts developer focus toward higher-level problem-solving rather than boilerplate code writing.
- •The research provides a baseline empirical framework for measuring human-AI collaboration outcomes in professional software contexts.
Cited by 2 pages
| Page | Type | Quality |
|---|---|---|
| Autonomous Coding | Capability | 63.0 |
| AI-Human Hybrid Systems | Approach | 91.0 |
Cached Content Preview
HTTP 200Fetched Apr 9, 202613 KB
Research: quantifying GitHub Copilot’s impact on developer productivity and happiness - The GitHub Blog
Eirini Kalliamvakou · @ikaliam
September 7, 2022
|
Updated May 21, 2024
|
8 minutes
Share:
Everyday, we use tools and form habits to achieve more with less. Software development produces such a high number of tools and technologies to make work efficient, to the point of inducing decision fatigue. When we first launched a technical preview of GitHub Copilot in 2021, our hypothesis was that it would improve developer productivity and, in fact, early users shared reports that it did. In the months following its release, we wanted to better understand and measure its effects with quantitative and qualitative research. To do that, we first had to grapple with the question: what does it mean to be productive?
Why is developer productivity so difficult to measure?
When it comes to measuring developer productivity, there is little consensus and there are far more questions than answers. For example:
What are the “right” productivity metrics? [ 1 , 2 ]
How valuable are self-reports of productivity? [ 3 ]
Is the traditional view of productivity—outputs over inputs—a good fit for the complex problem solving and creativity involved in development work? [ 4 ].
In a 2021 study, we found that developers’ own view of productivity has a twist–it’s more akin to having a good day . The ability to stay focused on the task at hand, make meaningful progress, and feel good at the end of a day’s work make a real difference in developers’ satisfaction and productivity.
This isn’t a one-off finding, either. Other academic research shows that these outcomes are important for developers [ 5 ] and that satisfied developers perform better [ 6 , 7 ]. Clearly, there’s more to developer productivity than inputs and outputs.
How do we think about developer productivity at GitHub?
Because AI-assisted development is a relatively new field, as researchers we have little prior research to draw upon. We wanted to measure GitHub Copilot’s effects, but what are they? After early observations and interviews with users, we surveyed more than 2,000 developers to learn at scale about their experience using GitHub Copilot. We designed our research approach with three points in mind:
Look at productivity holistically. At GitHub we like to think broadly and sustainably about developer productivity and the many factors that influence it. We used the SPACE productivity framework to pick which aspects to investigate.
Include developers’ first-hand perspective. We conducted multiple rounds of research including qualitative (perceptual) and quantitative (observed) data to assemble the full picture. We wanted to verify: (a) Do users’
... (truncated, 13 KB total)Resource ID:
c197fcb1b49328ab | Stable ID: NjU2YmJiYz