👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. If you’ve been forwarded this email, you can subscribe here. AI Tooling for Software Engineers in 2024: Reality Check (Part 1)How do software engineers utilize GenAI tools in their software development workflow? We sidestep the hype, and look to the reality of tech professionals using LLMs for coding and other tasks.
In April last year we published The Productivity Impact of AI Coding Tools based on a survey of subscribers to this newsletter, about how new AI tools were helping developers with coding. Back then, ChatGPT and GitHub Copilot were the dominant tools and more were on the way during that time of experimentation. Based on readers’ feedback at the time, the article advised playing around with AI coding tools to find out what worked or not, and predicted: “AI coding tools will move the industry forward.” Fast forward to 2024, and AI coding tools are more widespread than ever. GitHub Copilot has passed 1 million paying customers (safe to assume mostly developers,) and there’s been a surge in startups building AI software engineering tools, along with no shortage of hype. A recent peak of the AI hype cycle saw some startups raise funding to “replace developers with AI engineers.” This publication’s take on that “mission” and similar ones is that they’re overheated marketing slogans in response to the popularity of GitHub Copilot, and aren’t the reality. But how are engineers really using these tools in 2024? In order to sidestep the hype and tackle that question, we recently launched a new survey asking software engineers and engineering managers about your hands-on experience with AI tooling; which tools are being used this year, what parts of the development workflow are AI-augmented, what works well, and what doesn’t? As far as we know, this is the biggest survey yet on how tech professionals are using AI tools, and this publication’s business model means we’re free from bias on the subject. The volume of responses was such that it’s taken a couple of months to compile the data, but today we present it! We analyze input from subscribers to this newsletter and seek to offer a balanced, pragmatic, and detailed view of where LLM-powered development tooling is today. This article covers:
The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online. 1. OverviewA total of 211 tech professionals took part in the survey, an increase on the 175 responses to last year’s AI tooling questionnaire. Positions: most respondents are individual contributors (circa 62%.) The remainder occupy various levels of engineering management:
We asked engineering leaders additional questions because these people are in a great position to see the impact of AI tooling on teams. We’ll cover the engineering leaderships’ views in a follow-up issue. Experience: The lengths of respondents’ careers is segmented into five-year periods, up to 20+ years of experience (YOE) in the tech industry:
Fun fact: one respondent had 60 years of professional development experience (!!) This developer is now semi-retired. His outlook on these tools – which he plans to use in the future – is positive, writing:
Time spent using AI tools: again, a pretty even split between professionals who are new to using AI tools (6 months or less,) people who have used them for 6-12 months, and those who have done so for over a year.
Responses from people who don’t use AI tooling may seem a bit out of place in a survey about these tools, but we wanted to reach such developers and ask specific questions, in order to find out why some folks do not, will not, or cannot employ them. We’ll dive into this in a future article. Size of company: a roughly even split in size from tiny businesses, all the way to mega-corporations. Companies with between 1-50 people are the majority.
2. Popular software engineering AI toolsChatGPT and GitHub Copliot need little introduction as the market leaders in the software engineering sector. But exactly how popular are they, and which other tools do developers use? The ToolsAs just mentioned, ChatGPT and GitHub Copilot are heavily represented in the survey. The surprise was the degree of their popularity:
The responses reveal that as many professionals are using both ChatGPT and GitHub Copilot as all other tools combined! Below is a breakdown of this “other” category; note, the next most popular tool, Google’s Gemini, has only 14% of the mentions garnered by GitHub Copilot:
The difference in mindshare across respondents is significant, as becomes clear by visualizing all mentions together:
This chart lists all tools with at least two mentions in the survey. Those mentioned only once include Microsoft Teams AI, Amazon Q, Meta AI, Vercel’s v0.dev (UI generation from prompts,) Databricks AI assistant, Replit Ghostwriter, Ellipsis.dev (AI code reviews & bugfixes,) Mutable.ai (creating living documentation,) CodeRabbit AI (AI-powered code reviews,) StartCoder (code completion,) and Aider (AI pair programming in a terminal.) It’s great that so many tools are being tried out! Favorite AI coding toolsWe asked developers which tools are their favorites. GitHub Copilot and ChatGPT. As the charts show, these got the most mentions. Here’s a selection:
GitHub Copilot Chat is mentioned quite a lot, mostly positively. A big plus is that it offers an alternative to opening a browser to use ChatGTP. Still, not everyone is blown away by what the market-leading AI can do, including a senior software engineer who’s saltiness is hard to miss:
Another respondent shares that their company evaluated 8 AI coding tools and settled on GitHub Copilot, which seems to have an advantage with companies buying department-wide licenses, and developers moving over to it from other tools:
Several respondents say ChatGPT is the only tool they use, and that they like it. A software engineer at an eSports company shares:
Other tools earned honorable mentions as some devs’ favorite tools:
3. AI-assisted software engineering workflowsWe asked respondents who have used AI tools for more than six months what an AI-assisted workflow looks like for them and/or their team. Some trends can be observed:... Subscribe to The Pragmatic Engineer to unlock the rest.Become a paying subscriber of The Pragmatic Engineer to get access to this post and other subscriber-only content. A subscription gets you:
|
AI Tooling for Software Engineers in 2024: Reality Check (Part 1)
Subscribe to:
Post Comments (Atom)
Thank you for supporting The Pragmatic Engineer
Thank you for reading The Pragmatic Engineer. As a token of our appreciation, we're offering you a limited-time offer of 20% off a paid...
No comments:
Post a Comment