👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. If you’ve been forwarded this email, you can subscribe here. AI Tooling for Software Engineers: Rolling Out Company-Wide (Part 3)How large tech companies are using internal AI tools. Also: guidelines and practical approaches for embracing LLM tools for software development on the individual dev, and organizational level
Before we start: you can now see use “table of contents” quick navigation on the right side of each article, when reading the newsletter on the web. Just click on the sidebar, and you can navigate this article — and all other The Pragmatic Engineer articles. See it in action on the web. Happy browsing! There’s no shortage of big claims about what LLM tools will be able to do, or should be able to do in the software engineering field. But what do they actually do, right now? We asked software engineers who regularly use these tools, and engineering leaders who oversee these tools in their organizations. This article is based on a survey of 216 professionals and is the third and final part of a mini-series on GenAI tooling. It covers how these tools are being used ‘day-to-day’ in tech workplaces, and what engineers think about them. Today, we cover:
In Part 1 of this series, we covered:
Part 2 was about:
Now, let’s dive into this final part of this mini-series. The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online. 1. AI usage guidelines across companiesWe asked survey participants “how is AI tooling used for development at your company?” The responses reveal different approaches:
The most referenced approaches:
Common features of guidelines across workplaces, based on survey responses:
It’s pretty clear some guidelines are responses to fears that LLMs may retain the data employees input and use it for training. This is also a reason why a handful of respondents shared that their companies go through the added complexity of running LLMs on their own infrastructure. It’s a reminder that LLM solutions which don’t store company data have a strong selling point for tech companies. 2. Internal LLMs at Meta, Netflix, Pinterest, StripeOnly a fraction of respondents say their companies strongly encourage the use of LLM tools, but some of these are cutting-edge market leaders in tech. Let’s take a look at how a well-built internal LLM can help a business. MetaThe social media giant has been investing heavily in ML and AI since before ChatGPT was released. Back in 2022, we covered how Meta was already preparing for AI/ML ‘wartime’ by investing heavily both in AI hardware, and hiring large numbers of AI and ML engineers. This investment has not slowed down since, and it’s little surprise that Meta seems to have built one of the leading in-house AI tools. Meta’s internal tool is called Metamate. Director of Engineering Esther Crawford describes it:
Esther explains what Metamate does:
Here’s a practical example on how useful Meta’s tool is, from Shana Britt E, director of strategic initiatives:
MicrosoftThe company offers Microsoft Copilot for Microsoft 365 for enterprises, and is dogfooding this system inside the company. I talked with software engineers who confirmed that the internal Microsoft Copilot is integrated with internal documents, and can thus provide more relevant context. It is also used in places like pull request reviews – although for this use case, I heard it’s more of a hit-and-miss in the quality of feedback. StripeThe fintech company has a similar system to Metamate. Miles Matthias, product manager, shares:
NetflixThe company has a place to access Netflix-provided versions of LLMs. A senior software engineer told us:
The company builds internal LLM tools. One clever utility is called Text-to-SQL: a feature where internal users can use plain text to ask for a type of query, and the tool generates the right SQL to be used with the company’s internal data store called Querybook. The engineering team improved the first version with RAG, to help identify the right table names to use (we previously did a deepdive on applied RAG). The results are promising. As the company shares:
Vendors offering similar capabilitiesThere are plenty of vendors offering a “Metamate-like” experience out of the box. Glean seems to be the leader in this area. Other options include Village Labs, Microsoft Copilot for M365, Coveo and Akooda. This category is relatively new and there are plenty of up-and-coming startups. Search for terms like “AI company knowledge management tools” to find them. The productivity perception of these systems rarely matches reality. Despite being a leader in the AI field, Meta is just figuring out how these tools can help it operate more efficiently. Metamate sounds impressive – and it’s ahead of what most companies have – but it doesn’t work optimally just yet, as we hear. I got this detail from talking with current engineers working at Meta. The reason companies like Meta are investing so much into this area was articulated by CEO Mark Zuckerberg two months ago, on the company’s earnings call. He talked about how AI investments will take years to pay off, and Meta wants to be early. He said:
3. Reservations and concernsWhen starting to use AI tooling, companies and developers often need to overcome reservations, or find workarounds. Let’s start by summarizing these reservations. Reasons for not using AI toolingReasons for disallowing – or heavily limiting – AI tools include security and privacy worries; especially about internal, confidential information, and proprietary code being leaked. A few respondents also mention customer data. Several larger companies have worked around these concerns by using in-house, self-hosted, LLMs, and their security and compliance teams add filtering to the inputs and outputs of these tools. This approach is clever:
The obvious downside is that it’s not trivial to build and maintain. However, given that leading tech companies already have internal models and are heavy users, it’s likely other businesses will follow by either building in house, or using a vendor offering hosted LLMs with capability for internal security teams to tweak filters. Developers’ reservationsBut it’s not just companies dragging their feet; developers are also hesitant about LLMs in the survey: Commonly cited ethical and environmental concerns:
These criticisms are valid. Large language models are known to be trained on copyrighted code, as well as on copyleft-licensed code, where the license is not complied with. And the surge in energy usage is also real, as covered in Is GenAI creating more carbon pollution by cloud providers?:
There are clear benefits to GenAI, but also technological downsides. The ethical concerns seem to have no easy answers, while the history of computing has been about making computers more energy efficient, so we should expect the same here. At the same time, it’s concerning that GenAI is used to justify creating data centers which consume massive amounts of energy, or considering nuclear-powered data centers to keep up with computing demand. Not enough utility, yet: We previously summarized negative sentiments in “Unimpressed” critiques in Part 2 of this survey. Common complaints about AI from engineers include:
Here are two more comments from engineers who stopped using AI tools:
These reservations are valid, but survey responses show that using LLM tools for 6+ months changes the views of many developers: mostly to a more positive, or more grounded, viewpoint. If you have an underwhelming first impression of these tools, it might be worth trying them daily for a bit before making up your mind. Why do devs start using LLMs?We asked tech professionals why they started using these tools. The most common responses listed by frequency:
An interesting detail for us is that company mandates and pushes are the single most-cited reasons for starting to use AI tools. It seems these do work – at least for that initial “push” to give the tools a go. 4. Advice for devs to get started with AI tools...Subscribe to The Pragmatic Engineer to unlock the rest.Become a paying subscriber of The Pragmatic Engineer to get access to this post and other subscriber-only content. A subscription gets you:
|
AI Tooling for Software Engineers: Rolling Out Company-Wide (Part 3)
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment