👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. If you’ve been forwarded this email, you can subscribe here. How GenAI is reshaping tech hiringLarge language models are forcing tech hiring managers to adapt software engineering interview processes, fast. We look into how this is happening, and what to expect in the near future
Veteran engineering manager Karthik Hariharan advises leaders at startups, and recently shared an interesting observation with me:
LLMs have taken the industry by storm in two short years, and our recent survey found that around 80% of software engineers use LLMs daily. The most popular are ChatGPT and GitHub Copilot, and there’s a long tail of other tools in use — with “GenAI-first” IDEs like Cursor, Windsurf and Zed also seeing a surge in popularity. It’s rare for a new technology to be so rapidly adopted as AI tools have been. Coincidentally or not, ChatGPT is very good at solving Leetcode-style algorithmic interview questions. As a rule, LLM tools make for strong coding companions. This means many interview processes for software engineers which currently focus on algorithmic coding are rapidly ceasing to be useful at identifying coding talent when candidates have access to these LLMs. But how are employers reacting to this development, and changing their processes to identify the best candidates? This article tackles the issue with detailed contributions from 49 tech professionals, via direct messages and form responses. Sixty five percent of respondents are hiring managers (engineering managers, directors-or-above, founders/cofounders), and 35% are software engineers. Thank you to everyone who contributed! Also thanks for the additional input from the cofounders of technical assessment vendors Cookd (a new type of technical testing), Equip (vet qualified candidates), interviewing.io (anonymous mock interviews with senior engineers from FAANG), Woven Teams (human-powered technical assessments). As usual, this publication has no affiliation with companies mentioned. See the ethics statement for more information. We cover:
The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online. 1. Impact on recruitmentThere are common themes about the impact of AI tooling mentioned by respondents who are recruiters, hiring managers, and interviewers: More focus on catching fakersThis is a major gripe of respondents:
Recruitment is more effortHiring managers and recruiters alike say that GenAI tools create extra work during the recruitment process. More resume screening:
Noiser. Jayanth Neelakanta, cofounder and CEO of pre-employment tech assessment startup, Equip, shares:
Harder to evaluate less experienced candidates. An interesting observation from a director at a UK digital agency:
Harder to get reliable signal. At heart, the recruitment process is about learning enough about potential hires in order to accurately assess them. GenAI makes this harder because it’s unclear how much signal comes from a candidate, and how much from the tool.
More time in interviews. Ali Alobaidi, cofounder at Cookd shares:
Recruitment tooling vendors’ viewpointCompanies want problems for candidates to tackle which can’t be solved by googling it. Equip cofounder and CEO, Jayanth Neelakanta, shares:
Around 5% of companies allow GenAI tools, and this number isn’t growing. Woven offers technical assessments for hiring software engineers, and customers can choose to allow or ban GenAI tools. Founder and CEO, Wes Winler, is surprised by how few opt in for AI:
2. Faking it with GenAI toolsHow common is it for candidates to use LLMs when the recruitment processes explicitly forbids use of AI? Well, a study has established a predictable but challenging motive for using chatbots against the rules: it works. Aline Learner, CEO of interview startup interviewing.io, told us about a study they ran to find out how easy it is to cheat with ChatGPT. The company set up 32 audio-only interviews to ensure anonymity. Interviewees were told to use ChatGTP, but not to tell the interviewer. The questions were Leetcode, slightly modified, and custom ones. The results:
Based on these results, ChatGPT is very good at passing coding interviews where the question – and some solutions – can be found online. The experiment proved cheating is profitable – because it results in better pass rate, while being undetected. Interviewees had to rate how worried they were about being caught: 81% had no worries, while only 6% were very worried. Surprisingly, no interviewer reported noticing anything underhand taking place. Remember, interviewers were instructed: "conduct the interview just as you typically would" and to "evaluate the candidate as you usually do." Nothing was mentioned about interviewers sneakily using ChatGPT – which is in line with common interview expectations. The interviewing.io team say:
Just a reminder that this test didn’t cover video interviews; it was held in an audio interview environment. Check out the full study. Faking it on cameraIncidents of candidates using AI tools during a video interview against the rules, is increasingly common, according to Wes Winler - Founder and CEO of Woven:
Some hiring managers admit to flying blind, like a head of engineering at a fintech company shares:
Interviewers are increasingly suspicious of “spat out” answers. A security engineer in Big Tech says:
GenAI is even reportedly used in systems design and behavioral interviews. Kwinten Van de Broeck, Director of Engineering, Cognite shares:
CluesSome things make interviewers suspicious that a candidate is using AI tools for an interview, and respondents shared some tell-tale signs :
Banning GenAI doesn’t always workWoven is a startup that runs technical screenings, and when a customer wants no GenAI tools to be used, the company can detect if candidates break the rule. Cofounder and CEO Wes Winler shares:
3. Impact on resume screeningGenAI is heavily used for writing resumes and sending mass applications. This means more inbound applications, and more which are tailored to specific positions. It’s not hard to see where it leads: a bigger-than-ever pile of applications for recruiters and hiring managers, with more noise and less signal, making it harder to find qualified candidates. So hiring managers are adapting: Weaker resumes and cover lettersA common observation – and complaint – of tech company hiring managers focuses on the standard of written applications. Cover letters are almost all AI-generated, and therefore useless. This is a common sentiment:
LLMs are increasingly ubiquitous in resume-writing, but it’s unclear they add value. Too many resumes look similar, with uniform wording and phrasing:
Pedigree is more importantWes Winler, cofounder at Woven:
Using AI to filter resumes?No respondents say they use AI to filter resumes, but plenty reckon that others do. Meanwhile, Leo Franchi, Head of Engineering at Pilot.com, is uncertain how effective the tools will perform during a trial run:
This is a tricky area in Europe due to regulation. In 18 months, it will be mandatory to register AI models used for “high risk” cases such as making decisions about job applications. Companies could suffer reputational damage if they use AI to reject candidates, especially if the tool is revealed to have biases. In other regions, there is no such regulation. At the same time, with more candidates using AI to mass apply to jobs, it’s hard to imagine companies not developing automated filtering systems to weed out AI-generated applications. 4. Changing take-homes and coding interviewsInterviewing is changing with the times and tools in a few ways:... Subscribe to The Pragmatic Engineer to unlock the rest.Become a paying subscriber of The Pragmatic Engineer to get access to this post and other subscriber-only content. A subscription gets you:
|
How GenAI is reshaping tech hiring
What is happening in Syria and Aleppo?
Don’t have a lot of Time? Skip to my Idiot’s Guide to Syrian Civil War Factions below:
To start, YouTube demonetized this video, so thank you to everybody who gives me $5 a month on Substack. If you have the $5, I would really appreciate you joining here: The Dog’s BreakfastSyria is a mess with up to 70 factions fighting. The typical maps of Syria don’t do the conflict justice and show the sheer scale of independent factions as well as factions within factions. But why did anti-government forces make such quick gains after years of static front lines? Here is a basic rundown of the conflict in a nutshell: Bashar al-Assad is the President and Dictator of Syria.
He became the president of Syria in 2000 after the death of his father, Hafez al-Assad
Who ruled as President / Dictator of Syria from 1971 until the year 2000. Initially, Bashar al-Assad had no political ambitions, deciding to become a doctor and later an ophthalmologist in London. Most of the political power was directed at his older brother, Bassel al-Assad.
The older son, Bassel al-Assad died in a car crash in 1994, and Bashar al-Assad returned home to Syria to become the heir and consolidate power. He was introduced to the nation as the next leader by his father. Bashar immediately begins fighting “Corruption” - which is defined as anyone who opposes his rise to power. Bashar also becomes the president of the “Syrian Computer Club” and introduces the internet into Syria.
Hafez al-Assad dies in June of 2000. Bashar al-Assad becomes president in a bit of constitutional trickery that lowers the minimum age to become President of Syria from 40 to 34 - Bashar al-Assad’s age. The “Damascus Spring” begins after Bashar attains power. Artists and intellectuals in Syria demand more reforms. Bashar al-Assad responds by throwing thousands of people in prison. By 2011, the self-immolation death of Tunisian fruit vendor Mohamed Bouazizi sets the “Arab Spring” loose in the middle east. Syria faces waves of protests calling for government reforms.
Some Army factions defect from the military. Islamist and Kurdish militia groups form, openly battling government forces.
Syrian Forces use air power to attack rebel strongholds and civilian areas who support the rebels. This sees the advent of “Barrel Bombs” - 55 gallon drums that are packed with explosives and pushed out of helicopters.
The strategic city of Aleppo, which is rebel held and the crossroads of the M4 and M5 highways, becomes the main focus of pro-Assad forces.
ISIS, or the Islamic State of Iraq and Syria takes advantage of this power vacuum to rise to power in norther Iraq and eastern Syria.
The Russian Air Force and Army Steps in to fight both ISIS and anti-Assad Forces.
Forces loyal to Bashar Al-Assad capture Aleppo in 2016.
Factions in a nutshellFactions can be broken two into two main camps:
Among the factions that support Bashar al-Assad are:
Among the factions that do not support Bashar al-Assad:
Moderate Anti-Assad rebel groups
So what happened in Aleppo?On November 27th, elements of HTS or Hayat Tahrir al-Sham entered Aleppo eventually pushing as far as the Airport.
Where they captured some Syrian Air Force aircraft, although the disposition of those aircraft an HTS’s ability to operate them is unknown.
HTS has also dropped instruction to pro-government forces on how to safely surrender. This is the first time I have ever seen a rebel army drop leaflets and the first time I have seen a QR code used in war.  HTS leaflets Note that the pro-Syrian Army has suffered comparatively few losses compared with the land gained by the HTS - roughly 200 or so casualties. This means that pro-Assad forces are likely fleeing rather than fighting. Why is this happening now?There are three possibilities, or it is a combination of the three:
So the HTS may be performing a shaping operation - not against Assad, but to better position itself if Kurdish supported groups ave their funding cut off. Note that I am just touching on several of the almost 70 factions fighting in Syria. It is impossible to cover all of them and the intricacies involved in their relations. But it’s best to just keep at at a high level. Here is a list of the factions in an handy infographic
You're currently a free subscriber to The official Ryan McBeth Substack. For the full experience, upgrade your subscription.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
How GenAI is reshaping tech hiring
Large language models are forcing tech hiring managers to adapt software engineering interview processes, fast. We look into how this is hap...