👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. If you’ve been forwarded this email, you can subscribe here. What is Security Engineering? Part 2.A broad overview of the ever-developing security engineering field; a domain that can feel intimidating to some software engineers. With Nielet D'Mello, security engineer at Datadog. Part 2.
This is the second and final part of exploring this important – and, yet, often intimidating! – topic of security engineering. Giving us an overview of this field is Nielet D'Mello: a security engineer at Datadog (previously at Intel and McAfee). In Part 1 we already covered:
In today’s issue, Nielet takes us through:
The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online. With that, it’s over to Nielet. Common security engineering termsAs a brief refresher, we use three terms frequently in this article, so let’s start by defining them:
1. Defining the criticality of a service or systemDo all services and systems need to invest in a security design review? Not necessarily, as the need for a review depends on a service’s or system’s business risk profile. Vulnerabilities will surface as you identify security concerns in a system’s design and architecture. Code reviews and dynamic testing also surface security issues. For critical systems, it’s worth investing in processes like security design reviews. However, how do you decide just how critical a service or system is? Use the dimensions below for a better sense of this:
Business purposeWhat are the primary objectives and functions of the service or system within the context of the organization's business operations? Identify how the service contributes to achieving business goals, generating revenue, or providing stakeholder value. To figure out the risks, it’s essential to know:
Public accessIs the service accessible to external users outside of the organization's network, or the general public? Public access systems offer expanded attack surfaces. For these, you need to assess the potential exposure to security threats and risks associated with providing services over the internet, or other public networks, as these systems are at a much higher risk of automated bot attacks, for example. Custom access controlsAll systems need custom access controls for their data, apps and resources to determine who has access to it and in what circumstances. Role-based access control (RBAC,) or attribute-based access control (ABAC,) are two examples of custom access controls. These have specific access permissions defined for users and identities, and restrictions tailored to the service’s requirements and security needs to ensure confidentiality. The decision to build custom access controls is usually made with the following factors in mind:
Users of the systemWhat different types of users interact with the service? This is key information for defining:
The last one is especially important. Several regulatory frameworks and industry standards mandate the protection of sensitive data through user identification and access controls. Examples of such frameworks include the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA). In these cases, putting these in place is not “just” about making the system secure; it’s about what ensures the system is compliant with privacy and security regulation. Users include:
Deployment environmentsDevelopment, testing, staging, and production environments in which the service operates. Each environment may have different security requirements and configurations. These varying requirements depend on:
For example, a staging environment may have broader internal employee access, meaning it can be accessed by most (if not all) employees. However, the production environment tends to have much stricter access control: only specific employees or groups can access it, and even fewer will have the rights to deploy to it. And while the staging environment is unlikely to have data that is considered confidential customer data: the production environment will! So the production environment will have much more strict data security and monitoring measures deployed on its infrastructure. It’s pretty common for an environment to be a shared infrastructure for various services. When this is the case, robust security controls (like stricter isolation for applications and databases) are even more important! Multi-tenant architectures are a good example for such “shared infrastructure” where stricter security controls are necessary. Data classificationThis refers to labeling data based on sensitivity, confidentiality, and regulatory requirements. Understanding the classification of data helps determine:
2. Scoring a system’s “criticality”It’s helpful to calculate a criticality score for services. For this, I like to assign weights to the security dimensions. Below is a sample of how these scores could look. It’s just an example; simpler than I usually use, and it doesn’t encompass all factors relevant for every system. Just treat it as inspiration: Now we’ve established the basic factors for understanding risk and criticality, we can do some napkin math with criticality scores, based on characteristics:
A simple way to think about a total risk “score” is to add together the weights for each dimension. So, in this case: Total Risk Score = BP + PA + CAC + US + DE + DC. Scoring criticalityLet’s take the example of building a payment system for an e-commerce site. It needs to process a high volume of transactions via credit cards, debit cards, and other payment methods. It also needs to have payment gateway integration, account for fraud prevention, and is subject to PCI DSS compliance. Let’s do the napkin math for this system’s criticality:
We get the total risk score by adding up the dimensions. In this case, it comes to 15 out of a maximum of 18 points (3 + 1 + 2 + 3 + 3 + 3.) This score indicates we are talking about a critical system from a security standpoint. All companies have unique risk-scoring and risk-tracking processes. As a software engineer, you need to figure out what a “high” service risk score means, and at what point you should reach out to the security team at your organization, if there is one. 3. Threat modeling...Subscribe to The Pragmatic Engineer to read the rest.Become a paying subscriber of The Pragmatic Engineer to get access to this post and other subscriber-only content. A subscription gets you:
|
What is Security Engineering? Part 2.
How Perplexity builds product
How Perplexity builds productJohnny Ho, co-founder and chief strategy officer, explains how he organizes his teams like slime mold, uses AI to build their AI company, and much more
👋 Hey, Lenny here! Welcome to this month’s ✨ free edition ✨ of Lenny’s Newsletter. Each week I tackle reader questions about building product, driving growth, and accelerating your career. If you’re not a subscriber, here’s what you missed this month: For $150 a year, get access to these posts and every prior post, along with an invite to a private Slack community with global meetups, a mentor matching program, interview prep support, live AMAs, and more. I guarantee you’ll get 100x the value of a subscription or your money back. Founded less than two years ago, Perplexity has become a many-times-a-day-use product for me, replacing many of my Google searches—and I’m not alone. With fewer than 50 employees, the company has a user base that’s grown to more than (an estimated) 50 million. They’re also generating over $20 million ARR and taking on both Google and OpenAI in the battle for the future of search. Their recent fundraise of $63m values the company at more than $1 billion, and their investors include Nvidia, Jeff Bezos, Andrej Karpathy, Garry Tan, Dylan Field, Elad Gil, Nat Friedman, Daniel Gross, and Naval Ravikant (but sadly not me 😭). Nvidia CEO Jensen Huang said he uses the product “almost every day.” I sat down with Johnny Ho, the company’s co-founder and head of product, to give you an inside look at how Perplexity builds product—which to me feels like what the future of product development will look like for many companies:
For more, check out Perplexity. And they’re hiring! For more stories of how the best product teams operate, don’t miss my deep dives into Linear, Shopify, Figma, Notion, Duolingo, Ramp, Miro, Coda, Gong, and Snowflake. P.S. I’m collaborating with Perplexity on a deep dive into how product managers use Perplexity, and we’d love to hear from you. Fill out this short survey if you use Perplexity regularly, and they’ll reach out to conduct a user interview. How Perplexity builds product
1. How do you use AI tools within Perplexity to build Perplexity?Honestly, at the very beginning, we didn’t know how to do all kinds of things, including product management, project management, finances, HR, etc. We had early access to GPT-3, and as we were figuring out how to build the company, we’d start everything by asking AI, “What is X?” and then “How do we do X properly?” For example, we asked questions like “How do you launch a product? What should be the steps in the launch process?” You get a rough step-by-step process, which for a startup was good enough. Obviously, it’s often not correct on the first try, but neither is a human, right? So we’d just iterate naturally from there.
Trying to figure things out by ourselves took days, but with AI and some prompting, we could get rolling in five minutes. We’re still doing this. This week, for example, I asked Perplexity, “How do I write an email inviting someone to Perplexity Pro?” We even tried to use it at times to build our product, but we found AI tooling wasn’t anywhere near good enough when it comes to coding. It could help us write scripts, but if you wanted sustainable code to build a platform on, it didn’t really work. Even today, with the advances and latest models, it still only writes templates. You can’t really design a new long-lived abstraction with it. 2. How many PMs do you have?We have only two full-time PMs, in an organization of 50.
Typical projects we work on only have one or two people on it. The hardest projects have three or four people, max. For example, our podcast is built by one person end to end. He’s a brand designer, but he does audio engineering and he’s doing all kinds of research to figure out how to build the most interactive and interesting podcast. I don’t think a PM has stepped into that process at any point.
We leverage product management most when there’s a really difficult decision that branches into many directions, and for more involved projects. The hardest, and most important, part of the PM’s job is having taste around use cases. With AI, there are way too many possible use cases that you could work on. So the PM has to step in and make a branching qualitative decision based on the data, user research, and so on. For example, a big problem with AI is how you prioritize between more productivity-based use cases versus the engaging chatbot-type use cases. Pretty early on, we decided to focus on the former, but there are still ongoing discussions. We plan to hire one or two more PMs over the next year, but the bar for hiring is going to stay very high. 3. I assume much of your success has been due to hiring well, and keeping a very high bar. What do you most look for when hiring (that maybe others don’t)?Given the pace we are working at, we look foremost for flexibility and initiative. The ability to build constructively in a limited-resource environment (potentially having to wear several hats) is the most important to us. When you take a look at resumes of PMs, a lot of them prioritize helping other people and finding alignment. I believe this becomes less important with the advent of AI. So you don’t necessarily need skills around managing processes or leading people as much. We look for strong ICs with clear quantitative impacts on users rather than within their company. If I see the terms “Agile expert” or “scrum master” in the resume, it’s probably not going to be a great fit. Also, AI allows PMs to do a lot more IC work, especially for data analysis and customer insights. You still need some fundamental knowledge, of course (i.e. math, statistics, a basic grasp of programming), but it’s never been easier to be a truly “technical” PM. We still select for culture fit and being easy to work with, but we’re less looking for people who guide other people’s efforts, because it’s not as necessary with AI. This might change as we get to a certain scale, but at the current scale, there are way more products to build than there are people to work on them. I think in the future, I expect fewer layers of management in the industry in general. And if I had to guess, a technical PM or an engineer with product taste will become the most valuable people at a company over time. 4. Do you structure your teams around products, user types, user journeys, outcomes, or something in between? Has this changed over the years?My goal is to structure teams around minimizing “coordination headwind,” as described by Alex Komoroske in this deck on seeing organizations as slime mold. The rough idea is that coordination costs (caused by uncertainty and disagreements) increase with scale, and adding managers doesn’t improve things. People’s incentives become misaligned. People tend to lie to their manager, who lies to their manager. And if you want to talk to someone in another part of the org, you have to go up two levels and down two levels, asking everyone along the way. Instead, what you want to do is keep the overall goals aligned, and parallelize projects that point toward this goal by sharing reusable guides and processes. Especially with the advance of AI, it’s possible to minimize coordination costs by using AI for “rubber duck debugging” your ideas instead of relying on perfect alignment and consensus. We also keep a “who’s who” list updated in our internal docs, and if you feel the need to reach out to anyone, just do it. This requires a large degree of trust. But even more importantly, with AI, you don’t have to reach out to people as often. Sometimes before asking the question you were going to ask someone else, you could first try spending one minute asking AI to reduce coordination costs and give everyone a reasonable jumping-off point to do it themselves.
5. How far out do you plan in detail, and how has that evolved over the years?Perplexity has existed for less than two years, and things are changing so quickly in AI that it’s hard to commit beyond that. We create quarterly plans. Within quarters, we try to keep plans stable within a product roadmap. The roadmap has a few large projects that everyone is aware of, along with small tasks that we shift around as priorities change. Being nimble is critical as developments in AI often have unforeseeable impact. For example, the rapid developments in open-source models and context length have had downstream impact on the product, roadmap, and overall business. Just recently, Meta released Llama 3 and Mistral released 8x22B; we’re looking into creative ways to use those models in our product. The projects in the product roadmap also need to be flexible because new product development runs in parallel with a technical/model development roadmap. Engineers shift between maintaining existing products and building new products, depending on the week. The technical roadmap tends to grow quickly as we run into limitations of existing systems and accumulate tech debt, but we try to prioritize tech debt that unlocks product improvements. Within a given week, though, plans are fairly stable. Each week we have a kickoff meeting where everyone sets high-level expectations for their week. We have a culture of setting 75% weekly goals: everyone identifies their top priority for the week and tries to hit 75% of that by the end of the week. Just a few bullet points to make sure priorities are clear during the week.
Taking a moment at the beginning of the week to reflect on meta tasks brings clarity and prevents overly reactive or hectic decision-making. Over time, our ability to estimate size and prioritize based on return on investment has also improved. 6. Do you use OKRs in some form?We try to be as rigorous and data-driven as possible in quarterly planning. All objectives are measurable, either in terms of quantifiable thresholds or Boolean “was X completed or not.” Our objectives are very aggressive, and often at the end of the quarter we only end up completing 70% in one direction or another. The remaining 30% helps identify gaps in prioritization and staffing. Underinvestments, for example, in hiring infra engineers become quickly apparent when infrastructural goals aren’t met.
7. How do your product/design review meetings work?After the central objectives and high-level designs are determined, we try to be fairly decentralized in our decision-making. Projects are driven by a single DRI, and execution steps are done in parallel as much as possible. The first step for any project is to break it down into parallel tasks as much as possible to reduce coordination issues. We do this in Linear, and I lead this work along with the PM on the team (or whoever is handling the PM duties). We strive for each task to be self-contained—you should be able to execute on it without blockers. And you will likely have to make some controversial decisions, but you can just work through the controversy later.
At the beginning of each project, there is a quick kickoff for alignment, and afterward, iteration occurs in an asynchronous fashion, without constraints or review processes. When individuals feel ready for feedback on designs, implementation, or final product, they share it in Slack, and other members of the team give honest and constructive feedback. Iteration happens organically as needed, and the product doesn’t get launched until it gains internal traction via dogfooding.
I encourage people to try to work in parallel as much as they can. They should not be waiting for everyone to unblock them. Ideally, you have design, front end, and back end all working at the same time on the same project. And now that we have a business team, all four people could work in parallel, whereas conventionally you might wait for designs or mock-ups to show up first. 8. How do reporting lines work?The teams are currently structured by function (product, R&D, design, business, etc.), and different teams think about different layers of the company and stack. But all energy is directed toward improving the core product. We design objectives that translate to common top-level metrics and improve the user experience holistically. For example, all teams share common top-level metrics while A/B testing within their layer of the stack. Because the product can shift so quickly, we want to avoid political issues where anyone’s identity is bound to any given component of the product. At our current size, we are flat by design, and the reporting structure does not dictate priorities as much as commitments to top-level goals. Our two full-time PMs—one web and one mobile—report to me as head of product. We’ve found that when teams don’t have a PM, team members take on the PM responsibilities, like adjusting scope, making user-facing decisions, and trusting their own taste. 9. You build one of the most beloved and successful products out there. What would you say is unique or central to your approach to product that leads to such success?Central to our approach is to take feedback, both from users and internally, and distill it into a few intuitive products that can work for many customers. We also try to distill the feedback in a way that motivates and informs our team, setting a broad vision but letting individuals control their own decisions about what would best serve the original goal. Our decentralized approach for decisions passes the torch of responsibility, enabling fast-paced iteration without the need for approval processes. Individuals make urgent, locally optimal decisions. Any misalignments are then ironed out quickly afterward. 10. What’s your primary tool for task management, and bug tracking?Linear. For AI products, the line between tasks, bugs, and projects becomes blurred, but we’ve found many concepts in Linear, like Leads, Triage, Sizing, etc., to be extremely important. A favorite feature of mine is auto-archiving—if a task hasn’t been mentioned in a while, chances are it’s not actually important. The primary tool we use to store sources of truth like roadmaps and milestone planning is Notion. We use Notion during development for design docs and RFCs, and afterward for documentation, postmortems, and historical records. Putting thoughts on paper (documenting chain-of-thought) leads to much clearer decision-making, and makes it easier to align async and avoid meetings. Unwrap.ai is a tool we’ve also recently introduced to consolidate, document, and quantify qualitative feedback. Because of the nature of AI, many issues are not always deterministic enough to classify as bugs. Unwrap groups individual pieces of feedback into more concrete themes and areas of improvement. 11. Would you say roadmap ideas primarily come top-down (teams are told what to build) or bottom-up (teams generally come up with the ideas)?High-level objectives and directions come top-down, but a large amount of new ideas are floated bottom-up. We believe strongly that engineering and design should have ownership over ideas and details, especially for an AI product where the constraints are not known until ideas are turned into code and mock-ups. Plenty of brainstorming is going on at all times. We have a dedicated brainstorm channel in Slack, follow-up ideas are collected in Linear, and often polishes go straight to code without anyone asking. The best examples of bottom-up ideas can be seen in Perplexity’s Discovery, Collection, and Sharing experiences. For example, as I shared above, our brand designer Phi builds the Discover Daily podcast and simultaneously makes decisions around the script, ElevenLabs integration, brand, and audio engineering. With AI, it’s impossible to predict use cases until iterations of the product are released. A year ago, we would never have predicted that the Discover experience would eventually be built into a podcast.
12. When people see a company like yours from the outside, it all looks perfect and like you have everything figured out. What are some things that aren’t working well or have been big challenges?Big challenges today revolve around scaling from our current size to the next level, both on the hiring side and in execution and planning. We don’t want to lose our core identity of working in a very flat and collaborative environment. Even small decisions, like how to organize Slack and Linear, can be tough to scale. Trying to stay transparent and scale the number of channels and projects without causing notifications to explode is something we’re currently trying to figure out. 13. What are some fun/unique rituals or traditions you have on the product team or at the company?A lot of features and products at Perplexity were built during one-week (or less) hackathons. Focused sprints to build new features have proved to be the most exciting and memorable times. Our first interactive search prototype, Pro Search (formerly Copilot), was built in a few days, but it has improved over many iterations of polish and fine-tuning.
Thank you, Johnny! Also, a big thank-you to Phi Hoang for helping with visuals. For more, check out Perplexity, and they’re hiring! Have a fulfilling and productive week 🙏 👀 Hiring? Or looking for a new job?I run a white-glove recruiting service specializing in product roles, working with a few select companies at a time. If you’re hiring for senior product roles, apply to work with us: If you’re exploring new opportunities yourself, use the same button above to sign up. We’ll send over personalized opportunities from hand-selected companies if we think there’s a fit. Nobody gets your info until you allow them to, and you can leave anytime. If you’re finding this newsletter valuable, share it with a friend, and consider subscribing if you haven’t already. There are group discounts, gift options, and referral bonuses available. Sincerely, Lenny 👋 Invite your friends and earn rewardsIf you enjoy Lenny's Newsletter, share it with your friends and earn rewards when they subscribe.
|
