How Perplexity builds productJohnny Ho, co-founder and chief strategy officer, explains how he organizes his teams like slime mold, uses AI to build their AI company, and much more
👋 Hey, Lenny here! Welcome to this month’s ✨ free edition ✨ of Lenny’s Newsletter. Each week I tackle reader questions about building product, driving growth, and accelerating your career. If you’re not a subscriber, here’s what you missed this month: For $150 a year, get access to these posts and every prior post, along with an invite to a private Slack community with global meetups, a mentor matching program, interview prep support, live AMAs, and more. I guarantee you’ll get 100x the value of a subscription or your money back. Founded less than two years ago, Perplexity has become a many-times-a-day-use product for me, replacing many of my Google searches—and I’m not alone. With fewer than 50 employees, the company has a user base that’s grown to more than (an estimated) 50 million. They’re also generating over $20 million ARR and taking on both Google and OpenAI in the battle for the future of search. Their recent fundraise of $63m values the company at more than $1 billion, and their investors include Nvidia, Jeff Bezos, Andrej Karpathy, Garry Tan, Dylan Field, Elad Gil, Nat Friedman, Daniel Gross, and Naval Ravikant (but sadly not me 😭). Nvidia CEO Jensen Huang said he uses the product “almost every day.” I sat down with Johnny Ho, the company’s co-founder and head of product, to give you an inside look at how Perplexity builds product—which to me feels like what the future of product development will look like for many companies:
For more, check out Perplexity. And they’re hiring! For more stories of how the best product teams operate, don’t miss my deep dives into Linear, Shopify, Figma, Notion, Duolingo, Ramp, Miro, Coda, Gong, and Snowflake. P.S. I’m collaborating with Perplexity on a deep dive into how product managers use Perplexity, and we’d love to hear from you. Fill out this short survey if you use Perplexity regularly, and they’ll reach out to conduct a user interview. How Perplexity builds product
1. How do you use AI tools within Perplexity to build Perplexity?Honestly, at the very beginning, we didn’t know how to do all kinds of things, including product management, project management, finances, HR, etc. We had early access to GPT-3, and as we were figuring out how to build the company, we’d start everything by asking AI, “What is X?” and then “How do we do X properly?” For example, we asked questions like “How do you launch a product? What should be the steps in the launch process?” You get a rough step-by-step process, which for a startup was good enough. Obviously, it’s often not correct on the first try, but neither is a human, right? So we’d just iterate naturally from there.
Trying to figure things out by ourselves took days, but with AI and some prompting, we could get rolling in five minutes. We’re still doing this. This week, for example, I asked Perplexity, “How do I write an email inviting someone to Perplexity Pro?” We even tried to use it at times to build our product, but we found AI tooling wasn’t anywhere near good enough when it comes to coding. It could help us write scripts, but if you wanted sustainable code to build a platform on, it didn’t really work. Even today, with the advances and latest models, it still only writes templates. You can’t really design a new long-lived abstraction with it. 2. How many PMs do you have?We have only two full-time PMs, in an organization of 50.
Typical projects we work on only have one or two people on it. The hardest projects have three or four people, max. For example, our podcast is built by one person end to end. He’s a brand designer, but he does audio engineering and he’s doing all kinds of research to figure out how to build the most interactive and interesting podcast. I don’t think a PM has stepped into that process at any point.
We leverage product management most when there’s a really difficult decision that branches into many directions, and for more involved projects. The hardest, and most important, part of the PM’s job is having taste around use cases. With AI, there are way too many possible use cases that you could work on. So the PM has to step in and make a branching qualitative decision based on the data, user research, and so on. For example, a big problem with AI is how you prioritize between more productivity-based use cases versus the engaging chatbot-type use cases. Pretty early on, we decided to focus on the former, but there are still ongoing discussions. We plan to hire one or two more PMs over the next year, but the bar for hiring is going to stay very high. 3. I assume much of your success has been due to hiring well, and keeping a very high bar. What do you most look for when hiring (that maybe others don’t)?Given the pace we are working at, we look foremost for flexibility and initiative. The ability to build constructively in a limited-resource environment (potentially having to wear several hats) is the most important to us. When you take a look at resumes of PMs, a lot of them prioritize helping other people and finding alignment. I believe this becomes less important with the advent of AI. So you don’t necessarily need skills around managing processes or leading people as much. We look for strong ICs with clear quantitative impacts on users rather than within their company. If I see the terms “Agile expert” or “scrum master” in the resume, it’s probably not going to be a great fit. Also, AI allows PMs to do a lot more IC work, especially for data analysis and customer insights. You still need some fundamental knowledge, of course (i.e. math, statistics, a basic grasp of programming), but it’s never been easier to be a truly “technical” PM. We still select for culture fit and being easy to work with, but we’re less looking for people who guide other people’s efforts, because it’s not as necessary with AI. This might change as we get to a certain scale, but at the current scale, there are way more products to build than there are people to work on them. I think in the future, I expect fewer layers of management in the industry in general. And if I had to guess, a technical PM or an engineer with product taste will become the most valuable people at a company over time. 4. Do you structure your teams around products, user types, user journeys, outcomes, or something in between? Has this changed over the years?My goal is to structure teams around minimizing “coordination headwind,” as described by Alex Komoroske in this deck on seeing organizations as slime mold. The rough idea is that coordination costs (caused by uncertainty and disagreements) increase with scale, and adding managers doesn’t improve things. People’s incentives become misaligned. People tend to lie to their manager, who lies to their manager. And if you want to talk to someone in another part of the org, you have to go up two levels and down two levels, asking everyone along the way. Instead, what you want to do is keep the overall goals aligned, and parallelize projects that point toward this goal by sharing reusable guides and processes. Especially with the advance of AI, it’s possible to minimize coordination costs by using AI for “rubber duck debugging” your ideas instead of relying on perfect alignment and consensus. We also keep a “who’s who” list updated in our internal docs, and if you feel the need to reach out to anyone, just do it. This requires a large degree of trust. But even more importantly, with AI, you don’t have to reach out to people as often. Sometimes before asking the question you were going to ask someone else, you could first try spending one minute asking AI to reduce coordination costs and give everyone a reasonable jumping-off point to do it themselves.
5. How far out do you plan in detail, and how has that evolved over the years?Perplexity has existed for less than two years, and things are changing so quickly in AI that it’s hard to commit beyond that. We create quarterly plans. Within quarters, we try to keep plans stable within a product roadmap. The roadmap has a few large projects that everyone is aware of, along with small tasks that we shift around as priorities change. Being nimble is critical as developments in AI often have unforeseeable impact. For example, the rapid developments in open-source models and context length have had downstream impact on the product, roadmap, and overall business. Just recently, Meta released Llama 3 and Mistral released 8x22B; we’re looking into creative ways to use those models in our product. The projects in the product roadmap also need to be flexible because new product development runs in parallel with a technical/model development roadmap. Engineers shift between maintaining existing products and building new products, depending on the week. The technical roadmap tends to grow quickly as we run into limitations of existing systems and accumulate tech debt, but we try to prioritize tech debt that unlocks product improvements. Within a given week, though, plans are fairly stable. Each week we have a kickoff meeting where everyone sets high-level expectations for their week. We have a culture of setting 75% weekly goals: everyone identifies their top priority for the week and tries to hit 75% of that by the end of the week. Just a few bullet points to make sure priorities are clear during the week.
Taking a moment at the beginning of the week to reflect on meta tasks brings clarity and prevents overly reactive or hectic decision-making. Over time, our ability to estimate size and prioritize based on return on investment has also improved. 6. Do you use OKRs in some form?We try to be as rigorous and data-driven as possible in quarterly planning. All objectives are measurable, either in terms of quantifiable thresholds or Boolean “was X completed or not.” Our objectives are very aggressive, and often at the end of the quarter we only end up completing 70% in one direction or another. The remaining 30% helps identify gaps in prioritization and staffing. Underinvestments, for example, in hiring infra engineers become quickly apparent when infrastructural goals aren’t met.
7. How do your product/design review meetings work?After the central objectives and high-level designs are determined, we try to be fairly decentralized in our decision-making. Projects are driven by a single DRI, and execution steps are done in parallel as much as possible. The first step for any project is to break it down into parallel tasks as much as possible to reduce coordination issues. We do this in Linear, and I lead this work along with the PM on the team (or whoever is handling the PM duties). We strive for each task to be self-contained—you should be able to execute on it without blockers. And you will likely have to make some controversial decisions, but you can just work through the controversy later.
At the beginning of each project, there is a quick kickoff for alignment, and afterward, iteration occurs in an asynchronous fashion, without constraints or review processes. When individuals feel ready for feedback on designs, implementation, or final product, they share it in Slack, and other members of the team give honest and constructive feedback. Iteration happens organically as needed, and the product doesn’t get launched until it gains internal traction via dogfooding.
I encourage people to try to work in parallel as much as they can. They should not be waiting for everyone to unblock them. Ideally, you have design, front end, and back end all working at the same time on the same project. And now that we have a business team, all four people could work in parallel, whereas conventionally you might wait for designs or mock-ups to show up first. 8. How do reporting lines work?The teams are currently structured by function (product, R&D, design, business, etc.), and different teams think about different layers of the company and stack. But all energy is directed toward improving the core product. We design objectives that translate to common top-level metrics and improve the user experience holistically. For example, all teams share common top-level metrics while A/B testing within their layer of the stack. Because the product can shift so quickly, we want to avoid political issues where anyone’s identity is bound to any given component of the product. At our current size, we are flat by design, and the reporting structure does not dictate priorities as much as commitments to top-level goals. Our two full-time PMs—one web and one mobile—report to me as head of product. We’ve found that when teams don’t have a PM, team members take on the PM responsibilities, like adjusting scope, making user-facing decisions, and trusting their own taste. 9. You build one of the most beloved and successful products out there. What would you say is unique or central to your approach to product that leads to such success?Central to our approach is to take feedback, both from users and internally, and distill it into a few intuitive products that can work for many customers. We also try to distill the feedback in a way that motivates and informs our team, setting a broad vision but letting individuals control their own decisions about what would best serve the original goal. Our decentralized approach for decisions passes the torch of responsibility, enabling fast-paced iteration without the need for approval processes. Individuals make urgent, locally optimal decisions. Any misalignments are then ironed out quickly afterward. 10. What’s your primary tool for task management, and bug tracking?Linear. For AI products, the line between tasks, bugs, and projects becomes blurred, but we’ve found many concepts in Linear, like Leads, Triage, Sizing, etc., to be extremely important. A favorite feature of mine is auto-archiving—if a task hasn’t been mentioned in a while, chances are it’s not actually important. The primary tool we use to store sources of truth like roadmaps and milestone planning is Notion. We use Notion during development for design docs and RFCs, and afterward for documentation, postmortems, and historical records. Putting thoughts on paper (documenting chain-of-thought) leads to much clearer decision-making, and makes it easier to align async and avoid meetings. Unwrap.ai is a tool we’ve also recently introduced to consolidate, document, and quantify qualitative feedback. Because of the nature of AI, many issues are not always deterministic enough to classify as bugs. Unwrap groups individual pieces of feedback into more concrete themes and areas of improvement. 11. Would you say roadmap ideas primarily come top-down (teams are told what to build) or bottom-up (teams generally come up with the ideas)?High-level objectives and directions come top-down, but a large amount of new ideas are floated bottom-up. We believe strongly that engineering and design should have ownership over ideas and details, especially for an AI product where the constraints are not known until ideas are turned into code and mock-ups. Plenty of brainstorming is going on at all times. We have a dedicated brainstorm channel in Slack, follow-up ideas are collected in Linear, and often polishes go straight to code without anyone asking. The best examples of bottom-up ideas can be seen in Perplexity’s Discovery, Collection, and Sharing experiences. For example, as I shared above, our brand designer Phi builds the Discover Daily podcast and simultaneously makes decisions around the script, ElevenLabs integration, brand, and audio engineering. With AI, it’s impossible to predict use cases until iterations of the product are released. A year ago, we would never have predicted that the Discover experience would eventually be built into a podcast.
12. When people see a company like yours from the outside, it all looks perfect and like you have everything figured out. What are some things that aren’t working well or have been big challenges?Big challenges today revolve around scaling from our current size to the next level, both on the hiring side and in execution and planning. We don’t want to lose our core identity of working in a very flat and collaborative environment. Even small decisions, like how to organize Slack and Linear, can be tough to scale. Trying to stay transparent and scale the number of channels and projects without causing notifications to explode is something we’re currently trying to figure out. 13. What are some fun/unique rituals or traditions you have on the product team or at the company?A lot of features and products at Perplexity were built during one-week (or less) hackathons. Focused sprints to build new features have proved to be the most exciting and memorable times. Our first interactive search prototype, Pro Search (formerly Copilot), was built in a few days, but it has improved over many iterations of polish and fine-tuning.
Thank you, Johnny! Also, a big thank-you to Phi Hoang for helping with visuals. For more, check out Perplexity, and they’re hiring! Have a fulfilling and productive week 🙏 👀 Hiring? Or looking for a new job?I run a white-glove recruiting service specializing in product roles, working with a few select companies at a time. If you’re hiring for senior product roles, apply to work with us: If you’re exploring new opportunities yourself, use the same button above to sign up. We’ll send over personalized opportunities from hand-selected companies if we think there’s a fit. Nobody gets your info until you allow them to, and you can leave anytime. If you’re finding this newsletter valuable, share it with a friend, and consider subscribing if you haven’t already. There are group discounts, gift options, and referral bonuses available. Sincerely, Lenny 👋 Invite your friends and earn rewardsIf you enjoy Lenny's Newsletter, share it with your friends and earn rewards when they subscribe.
|
How Perplexity builds product
Subscribe to:
Post Comments (Atom)
Top 3 UX Design Articles of 2024 to Remember
Based on most subscriptions ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ...
No comments:
Post a Comment