Inside ChatGPT: The fastest growing product in history | Nick Turley (OpenAI)

Lenny Rachitsky:You were a product leader at Dropbox then Instacart now you're the PM of the most consequential product in history

Nick Turley:I didn't know what I would do here because it was a research lab my first task was to fix the blinds or something like that

Lenny Rachitsky:When someone offers you a rocket ship don't ask which seat

Nick Turley:We set out to build a super assistant it was supposed to be a hackathon code base

Lenny Rachitsky:What was it called before

Nick Turley:It was gonna be chat with GPT 3.5 because I really didn't think it was gonna be a successful product

Lenny Rachitsky:And then Sam Altman's just like hey let me tweet about it

Nick Turley:This is a pattern with AI you won't know what to polish until after you ship my dream is that we ship daily

Lenny Rachitsky:By the time people hear this they're gonna have their hands on GPT five

Nick Turley:About 10% of the world population uses it every week with scale comes responsibility it just feels a little more alive bit more human this model has taste

Lenny Rachitsky:Kevin Weil your CPO said to ask you about this principle of is it maximally accelerated

Nick Turley:I just really wanna jump to the punch line why can't we do this now I always felt like part of my role here is just set the pace and the resting heartbeat everyone's always wondering

Lenny Rachitsky:Is chat the future of

Nick Turley:All of this stuff chat was the simplest way to ship at the time I'm baffled by how much it took off I'm even more baffled by how many people have copied

Lenny Rachitsky:ChatGPT is now driving more traffic to my newsletter than Twitter

Nick Turley:That is a type of capability that has been incredibly retentive I've been really excited about what we've been doing in search

Lenny Rachitsky:Can you give us a peek into where this goes long term

Nick Turley:ChatGPT feels a little bit like MS DOS we haven't built Windows yet and it will be obvious once we do

Lenny Rachitsky:Today my guest is Nick Turley Nick is head of ChatGPT at OpenAI he joined the company three years ago when it was still primarily a research lab he helped come up with the idea of ChatGPT and took it from zero to over 700,000,000 weekly active users billions in revenue and arguably the most successful and impactful consumer software product in human history Nick is incredible he's been very much under the radar this is the first major podcast interview that he has ever done and you are in for a treat we talk about all the things including the just launched GPT five a huge thank you to Kevin Weil Claire Vogue George O'Brien Joanne Jing and Peter Ding for suggesting topics for this conversation if you enjoy this podcast don't forget to subscribe and follow it in your favorite podcasting app or YouTube and if you become an annual subscriber of my newsletter you get a year free of a bunch of incredible products including Lovable Replit Bold N Eight N Linear Superhuman Descript Whisperflow Gamma Perplexity Warp Granola Magic Patterns Raycast Chappier D and Mobin check it out at Lenny'snewsletter.com and click bundle with that I bring you Nick Turley this episode is brought to you by Orcus the company behind open source conductor the orchestration platform powering modern enterprise apps and agentic workflows legacy automation tools can't keep pace siloed low code platforms outdated process management and disconnected API tooling fall short in today's event driven AI powered agentic landscape Orcus changes this with Orcus Conductor you gain an agentic orchestration layer that seamlessly connects humans AI agents APIs microservices and data pipelines in real time at enterprise scale visual and code first development built in compliance observability and rock solid reliability ensure workflows evolve dynamically with your needs it's not just about automating tasks it's orchestrating autonomous agents and complex workflows to deliver smarter outcomes faster whether modernizing legacy systems or scaling next gen AI driven apps Orcus accelerates your journey from idea to production learn more and start building at orkus.i0/leni that's 0rkes.i0/leni this episode is brought to you by Vanta and I am very excited to have Christina Casiopo CEO and cofounder of Vanta joining me for this very short conversation

Christina Casiopo:Great to be here big fan of the podcast and the newsletter

Lenny Rachitsky:Vanta is a longtime sponsor of the show but for some of our newer listeners what does Vanta do and who is it for

Christina Casiopo:Sure so we started Vanta in 2018 focused on founders helping them start to build out their security programs and get credit for all of that hard security work with compliance certifications like SOC two or ISO twenty seven zero one today we currently help over 9,000 companies including some startup household names like Atlassian Ramp and Lingchain start and scale their security programs and ultimately build trust by automating compliance centralizing GRC and accelerating security reviews

Lenny Rachitsky:That is awesome I know from experience that these things take a lot of time and a lot of resources and nobody wants to spend time doing this

Christina Casiopo:That is very much our experience but before the company and some extent during it but the idea is with automation with AI with software we are helping customers build trust with prospects and customers in an efficient way and you know our joke we started this compliance company so you don't have to

Lenny Rachitsky:We appreciate you for doing that and you have a special discount for listeners they can get a thousand dollars off Vanta at vanta.com/leni that's vanta.com/leni for $1,000 off Vanta thanks for that Christina

Christina Casiopo:Thank you.

Lenny Rachitsky:Nick thank you so much for joining me and welcome to the podcast.

Nick Turley:Thanks for having me Lenny. It's a busy week but you know we we've been working on this for a while so it also feels really good to get it.

Lenny Rachitsky:So by the time people hear this they're gonna have their hands on GPT five and the newest ChatGPT. What's the simplest way to just understand what this is what it unlocks what people can do with it give us kind of the the pitch.

Nick Turley:I'm so excited about GPT five it I think for most people it's going to feel like a a real step change if you're the average ChatGPT user and we have you know 700,000,000 of them this week we you know you've probably been on GPT four or you know a while you probably don't even think about the model that powers the product and GPT five is is it just feels categorically different I'll talk about a lot of the specifics but you know at the end of the day the vibes are good at least we feel that way we hope that users feel the same and increasingly that is the thing that I think most people notice right they don't look at the academic benchmarks they don't look at evaluations they try the model and and see what it feels like and just on that dimension alone I'm so excited I've been using it for a while but it is also you know the smartest most useful and fastest frontier model that we've ever launched you know on on pure smarts one way to look at that is academic benchmarks on many of the standard ones whether or not it's math or reasoning or you know just raw intelligence this model is state of the art I'm especially excited about its performance on coding whether or not that's SWE bench which is a common benchmark or actually front end coding is really really good as well and that's an area where I I feel like there's a there's a true step change improvement in in in GPT five but really no matter how you sort of measure the smarts it's it's it's quite remarkable and I think people are gonna feel the upgrade especially if they weren't using GPT three already and you know the the second thing beyond smarts is it's just really useful coding is one axis of utility whether or you have coding questions or you're vibe coding an app but he's also a really good writer I write for a living internally externally I just wrote a big blog post that we published Monday and you know this thing is like such an incredible editor and and you know compared to some of the the the the older models it just got it's got taste which I think is really exciting and to me that's like something that is truly useful in in in my day to day and there's other a bunch of other areas like it's it's state of the art on health which is useful when you need it but again the the sort of the thing you can't really express in use cases or even yeah in use cases or or data is sort of the vibe of the model and it just feels a little bit more alive a bit more human in a way that is kind of hard to articulate until you try it so feel good about that and yeah as mentioned it's faster it it thinks too just like GPT three did but you don't have to manually you know tell it to do that it'll just dynamically decide to think when it needs to and when it doesn't need to think it just responds instantly and that ends up feeling quite a bit faster than using GPT three did and then you know maybe the thing that's most exciting is that we're making it available for free and that's like one of those things that I feel like we can uniquely do at OpenAI because you know many companies I think if they have a subscription model like us they would gate it behind their paid plan and for us you know if we can scale it we will and that just feels awesome we did that with GPT four as well so everyone's gonna be able to try GPT five tomorrow hopefully.

Lenny Rachitsky:How long does something like this take like I don't know if there's a simple answer to this but just how long have you guys been working on GPT five?

Nick Turley:We've been working on it for a while you know you can kind of view GPT five as a culmination of a bunch of different efforts you know we had a reasoning tech we had a more classic post screening methodologies and therefore it's really hard to put a beginning on it but but you know it it really is kind of the endpoint of a bunch of different techniques that we've been doing for a while.

Lenny Rachitsky:Can you give us a peek into the vision for where ChatGPT is going GPT in general is going like if you look at on the surface it's just it's been kind of the same idea with a much smarter brain for a long time I'm curious where this goes long term.

Nick Turley:So to to maybe back up a bit now you think of ChatGPT as it's gonna be ubiquitous product again about 10% of the world population uses it every week oh shit you know I think we have like 5,000,000 business customers now it's like you know an established category in its own right but really when we started we set out to build a super assistant that's what we that's how we talked about it at the time in fact the code base that we use is is called SA server we was it's supposed to be a hackathon code base but know things things always turn out a little bit differently and so so yeah in some ways that is still the vision the reason I don't talk about it more than I you know do is because I think assistant is a bit limiting in terms of the mental model we're trying to create you think of this like very personified human thing maybe utilitarian maybe you know and and frankly you know having an assistant is not particularly relatable to most people unless they're like in Silicon Valley and they're a manager or something like that so it's imperfect but like really what you know we envision is is this entity that can help you with any task whether or not that's at home or at work or at school really any context and it's an entity that you know knows what you're trying to achieve so you know unlike ChatGPTs today you don't have to describe your problem in in in minute to detail because it already stands your overarching goals and has context on your life etcetera so you know that's one thing that we're really excited about the sort of inverse of giving it more inputs on your life is giving it more action space so we're really excited to allow it to do over time what a smart empathetic human with a computer could do for you and I think you know the limit of the the types of problems that you can solve for people once you give it access to to tools like that is is very different than what you might be able to do in a chatbot today so you know that's more outputs and I often think okay you know I'm a general intelligence if I what what happened if I you know became Lenny's intern or something and you know I wouldn't be particularly effective despite you know having both of those attributes that I just mentioned and it's because you know I think this idea of building a relationship with this technology is also incredibly important so that's maybe the third piece that I'm excited about is building a product that can truly get to know you over time and you saw us launch some of those things you know with improved memory earlier this year and that's just the beginning of what we're hoping to do so that it really feels like it's your AI so I don't know if super persistent is still the right exact analogy but I think people will just think of it as their AI and I think we can put one in everyone's pocket and help them solve real problems whether or not that's becoming healthy whether or not that's you know starting a business whether or not that's you know just having a second opinion on anything there's so many different problems that you can help with people in their in their daily life and that's what motivates me.

Lenny Rachitsky:So an interesting kind of between the lines that I'm reading here is the vision is for it to be an assistant for people not to replace people it feels like a really important piece of the puzzle maybe just talk about that.

Nick Turley:AI is really scary to people and I understand you know there's decades of movies on AI that have a certain mental model kind of baked in and even if you just look at the technology today once everyone I think has this moment where the AI does something that was really deeply personal to them and you're like kinda thought hey the AI can never do that you know for me was like like weird music theory things where I was like wow this thing actually like understands music better than I do and that's like something I'm passionate about and you know so so it it's naturally scary and I think the thing that's been really important to us for a long time is to build something that feels like it it's helpful to you but you're in the driver's seat and that's even more important as the stuff becomes agentic right like the feeling of being in control and that can be small things like you know we built this way of sort of watching what the AI is doing when it's in agent mode and it's not that like you actually are gonna watch it the whole time but it gives you a mental model and makes you feel in control in the same way that when you're in a Waymo you you get that screen for those of who've tried Waymo you know you can see the other cars it's not like you're gonna actually watch but it gives you the sense that you know how this thing works and what's happening or we you know we always check with you to confirm things it's a little bit annoying but it puts you in the driver's seat which is which is important and for that reason you know we always view technology and the technology that we build as something that amplifies what you're capable of rather than replacing it and that becomes important as the deck gets more powerful.

Lenny Rachitsky:Okay so you mentioned the beginnings of ChatGPT I was reading in a different interview so you joined OpenAI ChatGPT was kind of just this internal experimental project that was basically a way to test GPT 3.5 and then Sam Altman's just like hey let me tweet about it maybe see if people find this interesting yada yada yada it's the most successful consumer product in history I think both in growth rate and users and revenue and just absurd can you give us a glimpse into that early period before it became something everyone's obsessed with.

Nick Turley:Yeah so we had decided that we wanted to do something consumer facing I think you know right around the time that GPT4 finished training and it was actually mainly for a couple of reasons you know we already had a product out there which was our developer product that's actually what I came in to help with initially and you know that has been amazing for the mission in fact it's grown up in how it's the OpenAI platform with I don't know 4,000,000 developers I think but know at the time it was you know early stage and and we were running into some constraints with it because we there was two problems one you couldn't iterate very quickly because every time you would change the model you'd break everyone's app so it was really hard to try things and then the other thing was that it was really hard to learn because the feedback we would get was like the feedback from the end user to the developer to us so it was very disintermediated and we were excited to make fast progress toward towards AGI and it just felt like we needed a more direct relationship with with consumers.

Lenny Rachitsky:So we were trying to figure.

Nick Turley:Out where to start and you know in classic OpenAI fashion especially back then we put together a hackathon of enthusiasts of just hacking on GPT4 to kinda see what awesome stuff we could create and maybe ship to users and everyone's idea had was was some flavor of a super system like they were more specific ideas like we had a meeting bot that would call into meetings and you know the vision was you know maybe we would like help help help it will help you run the meeting over time we had a coding tool which you know full circle now probably ahead of its time and you know the the challenge was that we tested those things but every time we tested these more bespoke ideas people wanted to use it for all this other stuff because it's just a very very generically powerful technology so after a couple of months of prototyping we took that same kinda crew of volunteers and it was truly a volunteer group right we had like someone from the supercomputing team who'd built an iOS team iOS app before we had someone you know on the research team who had written some back end code in their life they they they were all part of this initial ChatGPT team and we decided to ship something open ended because we just wanted a real use case distribution and this is a pattern with AI I think where you know you really have to ship to understand what is even possible and what people want rather than being able to reason about that a priori so ChatGPT came together at the end because we just wanted the learnings as soon as we could and we shipped it right before the holiday thinking we would sort of come back and get the data and then wind it down and obviously that part turned out super differently because people really liked the product as is so I remember sort of going through the motions of oh man dashboard's broken oh wait people are liking it I'm sure it's just you know going viral and and stuff is gonna die down to like oh wow people are retaining but I don't understand why and then eventually we kind of like you know fell into product development mode but it was a little bit by accident.

Lenny Rachitsky:Be like that.

Nick Turley:You know it's just not possible otherwise it's this has been a giant marathon for three years now.

Lenny Rachitsky:Yeah like a sprint marathon.

Nick Turley:Sprint marathon that's right or interval training or something I don't I don't know how to exactly describe the OpenAI launch cadence but you know you you gotta you gotta you know set yourself up in a way that is sustainable even even at if even if this wasn't AI and it didn't have the interesting attributes that I just mentioned I think you you would need to do that but especially with AI it's important to go watch.

Lenny Rachitsky:So on along those lines I talked to a bunch of people that work with you that work at OpenAI Joanne specifically said that urgency and pace are a big part of how you operate that that's just something you find really important to create urgency within the team constantly even when you are the fastest growing product in history growing like crazy talk about just your philosophy on the importance of pace and urgency on teams.

Nick Turley:Well it's nice to hear you say that you know I I spent a lot of two things you know with ChatGPT I you know the when we decided to do it you know we had been prototyping for so long and I was just like you know in ten days we're gonna ship this thing and you know we did so that was like maybe a moment in time thing where I just really wanted to make sure that we go learn something but for you know ever since then I I just spent so much time thinking about why ChatGPT became successful in the first place and I think there was some element of just doing things where you know there was many other companies that had technology in the LM space that just never got shipped and I just felt like you know of all the things we could optimize for learning as fast as possible incredibly important so I just started rallying people around that and that took different forms like a while when we were of that size I just ran this like you know daily release sync and had everyone who was required to make a decision in it and we would just talk about what to do and to pivot from yesterday etcetera obviously at some point that doesn't scale but I always felt like part of my role here obviously was like to think about you know the direction of the product but also to just set the pace and the resting heartbeat for our teams and again this is important anywhere but it's especially important when you know the only way to find out what people like and and and what's valuable is to bring it into the external world so for that reason I think it's become a superpower of OpenAI and I'm glad that Joanne thinks I had some part in that but it it really has taken a village.

Lenny Rachitsky:I love this phrase the resting heart rate of your team that's such a perfect metaphor of just the pace of being equivalent to your resting heart rate.

Nick Turley:I actually learned that at Instacart when I when I showed up there because we were in the pandemic and it was kind of all hands on deck for a while there was this like you know I think there was a company wide stand up because we disbanded all teams we're just trying to keep the site up and for me you know I I'd been used to kind of taking my sweet time and just thinking really hard about things and that's important but I really learned to hustle over there and I think that's come in handy at OpenAI.

Lenny Rachitsky:Okay so along these same lines I asked Kevin Weil your CPO what to ask you and he said to ask you about this principle of is it maximally accelerated talk about that.

Nick Turley:That's funny we have a Slack emoji apparently for this deck because I used to say that now now I try to like paraphrase sometimes I just really wanna jump to the you know to the punchline of like okay why can't we do this now or why can't we do it tomorrow and I think that you know it's a good way to cut through a huge number of blockers with the team and just instill especially if you come from a larger company you know at some point we started hiring people from from you know larger tech companies I think they're used to you know let's check check-in on this in a week or let's you know circle back next quarter to see if we can go on the on on on the plan and I just kind of as a thought exercise I was like people asking like okay if like this was the most important thing and you wanted to truly maximally accelerate it what would you do that doesn't mean that you go do that but it's really good forcing function for understanding what's critical path versus what you know can happen later and I've just always felt like you know execution is incredibly important like these ideas are they're everywhere everyone's talking about you know a personal AI you know you might have seen news on that you know and and and you know I I really think that execution is is is one of the most important things in this space this is a tool so it's funny that that became a meme it's like a little pink Slack emoji that people just put on whatever they're trying to to force the question.

Lenny Rachitsky:I was gonna ask what the emoji was so it's a little pink is there something in there like max it's a.

Nick Turley:Comic Sans emoji that says is this maximally accelerated yeah fuck and.

Lenny Rachitsky:So the kind of the culture there is when someone is working on something the question the push is is this maximally accelerated is there a way we can do this faster is there anything we can unblock.

Nick Turley:Yeah and you know we use that sparingly right and because it it has it needs to be appropriate to the context there there's some things where you don't want to accelerate as as quickly as possible because you you kind of want process and we're very very deliberate on that where your process is a tool and one of the areas where we have an immense amount of process is safety because you know a the stakes are already really high especially with these models you know GPT five pushes the frontier in so many different ways but B you kind of if you believe in the exponential which I do and you know most people who work on this stuff do you have to play practice for a time where you know you really really need the process for sure for sure and that's why I think it's been really important to separate out you know the product development velocity which has to be super high from okay for things like frontier models there actually needs to be a a rigorous process where you red team you work on the system card you get external input and then you put things out with with confidence that it's gone through you know the right safeguards so again it's a nuanced concept but I found it very very useful when we needed and for everything product development you're dead on arrival so it's it's important to get stuff out.

Lenny Rachitsky:We gotta open source this meme so that other teams can build on this approach absolutely so interestingly with ChatGPT and it's not a surprise but not only is it the fastest growing most successful consumer product ever retention is also incredibly high people have shared these stats that one month retention is something like 90% six month retention is something like 80% first of all are these numbers accurate quick can you share that.

Nick Turley:I'm obviously limited on what exactly I can share okay but it is true that our retention numbers are really exciting and that is actually the thing we we look at you know we we don't care at all how much time you spend in the product you know in fact our incentive is just to solve your problem and you know if you really like the product you'll subscribe but you know there's no incentive to keep you in the product for long but we are obviously really really happy if you know over the long run you know a three month period etcetera you're still using this thing and for me this was always the elephant in the room early on it's like hey this may be a really cool product but you know is this really the type of thing that you come back to and it's been incredible to not just see strong retention numbers but to see you know an an an improvement in retention over time even as our cohorts become you know less of an early adopter and more you know the the average person so.

Lenny Rachitsky:Yeah so that but like that note is something that I don't think people truly understand how rare this is yeah when a product the cohort of users comes tries it out and then retention over time goes down and then it comes back up people come back to it a few months later and use it more and that's it's called a smiling curve or smile curve and that's extremely rare.

Nick Turley:Yeah yeah yeah no this this some smiling going on that just on the team and the you know I feel like I have to acknowledge that some of it is is not the product I think people are just getting used to this technology in like a really interesting way where I find and this is why the product needs to evolve too that this idea of delegating to an AI it's not natural to most people it's not like you're going through your life and figuring out what can I delegate like certain sphere of Silicon Valley does that you know because they're in like a self optimization mode and they're trying to delegate everything they can but I think for most people in the world it's actually quite unnatural and you really have to learn okay what what are my goals actually and what could another intelligence help me with and I think that just takes time and people do figure it out once they've had enough time with the product but then of course there's been tons of things that we've done in the product too whether or it's making the core models better whether or not it's you know new capabilities like search and personalization and and all that kind of stuff or you know just standard growth work too which we're starting to do you know that stuff matters too of course.

Lenny Rachitsky:So you might have you might be answering this question already but let me just ask it directly people may look at this and be like okay they're building this kind of layer on top of this godlike intelligence of course it will grow incredibly fast and retention will be incredible what the heck does what are you guys actually doing that sits on top of the model that makes it grow so fast and retain so much is there something that has worked incredibly well does move metrics significantly that you can share.

Nick Turley:I mean one thing we've learned I'll answer that question in a minute but you know the the one thing we've learned with ChatGPT is that there really is no distinction between the model and the product like the model is the product and therefore you need to iterate on it like a product by that I mean is like you know if there's you're obviously you typically start by shipping something very open ended at least if you're OpenAI on that point that's kind of a playbook but then you really have to look at what are people trying to do okay they're trying to write they're trying to code they're trying to get advice they're trying to get recommendations and you need to systematically improve on those use cases and that is pretty similar to product development work obviously the methodology is a bit different but the discovery is is is the same you gotta talk to people you gotta do data science and you gotta try stuff and and get feedback so that's like one chunk of work that we've been very consciously doing is improving the model on the use cases people care about and there's also such thing as vibes as because I'm sure you you know and that's one of the things that I'm excited about in GPT five is that the the vibes are really good so that too is you know we have a a model behavior team and they really focus on you know what is the personality of this model and how you know how does it speak and talk so does that kind of work I would say that's maybe you know a third of the you know retention improvements that we see or so just roughly and then I think another third is is is what I would call sort of product research capabilities they they're research driven for sure they have a research component but they're really new product features or capabilities and like search is one example of that where you know if you remember in the olden days aka like you know maybe twenty months ago or something you would talk to Chattubiti and it'd be like know as of my knowledge cut off you know or you know I can't answer that because that happened too recently or something like that and you know that is a type of capability that has been incredibly retentive and for for good reason it just allows you to do more with the product personalization like this idea of advanced memory where things can really get to know you over time is another example of a capability like that you know I think that's another good chunk and then you know the third stuff is the stuff you would do in any product and those things exist too you know like not having to log in was a huge hit because it removed a ton of the friction and I think we we had this intuition from the beginning but we never got to it because we didn't have enough GPU or you know other other constraint to really really really go do that so you know there's the like kind of traditional product work too so I often think about it sort of as roughly a third a third a third but really you know we're still learning and we're planning to evolve the product a ton which is why I'm sure there's gonna be new levers.

Lenny Rachitsky:You mentioned something that I wanna come back to real quick you said that that it was something like ten days from hackathon to Sam tweeting about ChatGPT being live.

Nick Turley:The you know the hackathon happened much earlier and we were prototyping for a long time but at some point we basically ran out of patience on you know on trying to you know build something more bespoke and again that was mostly because people always wanted to do all this other stuff whenever we tested it so it was ten days from from when we decided we were gonna ship to when we shipped and you know the the research we've been testing for a long time it was kind of an evolution of what we'd called instruction following which was the idea that you know instead of just completing the sentence these models could actually follow your instructions so if you said summarize this it would actually do so and the research had evolved from that into a chat format where we could do it multi turn so that research took way longer than ten days and that kind of baking in the background but the you know the prioritization of this thing was very very fast and you know lots of things didn't make it in like remember we didn't have history which of course was like the you know first user feedback we got the model had a bunch of you know shortcomings and it was so cool to be able to iterate on the model like the thing I just talked about like treating the model as a product was not a thing before chat shipping tip because we would ship it more like hardware where you know we'd there'd be a a release like GPT three and then we would start working on GPT four and these weird giant big spend R and D projects that would take a really long time and you kind of the spec was whatever the spec was and then you'd have to wait another year and ChatGPT really broke that down because we were able to make make iterative improvements to it just like software and really my dream is that it would be amazing if we could just ship daily or even hourly like in software land because you could just fix stuff etcetera but there's of course all kinds of challenges in how you do that while you know keeping the personality intact while like not regressing other capabilities so it's an open open reason field to get there.

Lenny Rachitsky:That's such a good example of isn't maximally accelerated okay we're gonna ship ChatGPT okay ten days ago holy moly we've been talking about ChatGPT clearly it's a kind of a chat interface everyone's always wondering is chat the future of all of this stuff interestingly Kevin Weil had made this really profound point that has always stuck with me when he was on the podcast that chat is a actually a genius interface for building on a super intelligence because it's how we interact with humans of all variety of intelligence it scales from someone at the lower end to the to a super super smart person and so it's really valuable as a way to kind of scale the spectrum maybe just talk about that and just is chat the long term interface for ChatGPT I guess it's called ChatGPT.

Nick Turley:I feel like we should either drop the chat or drop the G GPT at some point because it is a mouthful we're stuck with the name but you know no matter what we do with that you know it it the product will evolve I I think that I agree that there's something profound about natural language like it just really is the most natural form of communicating to humans and therefore it feels important that you should be communicating with your software in natural language I think that's different from chat though I think chat was the simplest way to put something to you know to ship at the time I'm baffled by how much it took off as as a concept I'm even more baffled by how many people have copied the paradigm rather than you know trying out a different way of interacting with AI I'm still hoping that will happen so I think natural language is here to stay but this idea that it has to be a turn by turn chat interaction I think is really limiting and this is one of the reasons I don't love the supersistent analogy even though we you know used to always use it is because if you think that way then you kind of feel like you're talking to a person but you know and G P D five is amazing at at making great front end applications so I I don't see a reason why you wouldn't have you know AIs that you know can can render their own UI in some way and you obviously wanna make that predictable and feel good but it feels limiting to me to think of the end all be all interface as a chatbot it it actually kinda feels dystopian almost where like I don't wanna use all my software through the proxy of some interface like I love being in Figma I love being in you know Google Docs those are all great products to me and they're not chatbots so yes on natural language but no on chat is is where I would describe my my point of view and I'm just hoping in general that we see more sort of consumer innovation on how people interact with AI there's so many possibilities and you just gotta try stuff that's why chat stuck is you know we just did it and people people liked it so I'm hoping that we we see over there and we'll we'll try to do our part.

Lenny Rachitsky:So you mentioned that you kind of like got stuck with his name ChatGPT maybe this was part of the answer but I'm curious just there any accidental decisions you guys made early on that have stuck in have essentially become history changing.

Nick Turley:There there's so many and it's it's funny because you have like no time to think about them and then they end up being super consequential you know the day was one you know we went from chat with gbt three point five to chat gbt the night before slightly better but still really bad

Lenny Rachitsky:What was it called before

Nick Turley:It was gonna be chat with gbt three point five right we we really didn't think it was gonna be a successful product like we were trying to actually be as nerdy as we could about it because that's really what it was it was like you know a research demo not not a product so we didn't think that was bad but you know i i think that in the original release you know making it free was a big deal i i don't think we appreciate that because the gpt 3.5 model was in our api for you know at least six months prior to that i think anyone could have built something like this might not have been quite as good on the modeling side but i think it would have taken off so making it free and putting a nice ui on it very consequential in the way that you take for granted now and this is why i think that a distribution and b the you know the interface are continued continuously important even in in 2025 the paid business which now is it's it's it's a it's a giant business both in you know the consumer space and in the enterprise space the birth of that was just to turn away demand originally like it was not like know we brainstormed oh what is the best monetization model for ai it was really what is what monetization model or what what mechanism would allow us to turn away people who are like you know less serious than the people who are really trying to use it and subscriptions just happened to have that property and it you know grew into a large business yeah i think shipping really kind of funky capabilities before they were polished is another thing where you know that feels like a tactical decision but it became a playbook because we would learn so much like remember when we shipped codeinterpreter we learned so much after we shipped it you know now it's known as i think data analysis and chatgpt or something like that just because we actually got real world use cases back that we could then optimize so i think there's been like a lot of decisions over over time that prove pretty consequential but you know we made them very very quickly as as as we have to so

Lenny Rachitsky:The the $20 a month feels like an important part of this feels like everybody's just doing that now

Nick Turley:And oh that would actually i remember i had this like kind of panic attack we we really needed to launch subscriptions because at the time we we we were taking the product down every time it was a i don't know if you remember we had this like fail whale there's like a e three generated poem on it they're like they had to get this out and they're calling up someone i greatly respect who's like you know incredible at pricing and and know i was like what should i do and like we talked a bunch and i just ran out of time to to incorporate most of that feedback so what i did do is ship a google form to discord with like i think the four questions you're supposed to ask on how to price something i had number one shows yeah exactly yeah it literally had those four questions and i remember distinctly a you know i got a price back and that's kind of how we got to $20 but b the next morning was like a press article on like you won't believe the like four genius questions the chatty bitty team asked to price their it was like if only you knew so there's like something about building in this extreme public where people interpret so much more intentionality into what you're doing than you know might have actually existed at the time but we got with the 20 we're debating you know something slightly higher at the time i often wonder what would have happened because so many other companies ended up copying the $20 price points i'm like do we like erase a bunch of market cap by pricing it this way but ultimately don't care because like the more accessible we can make this stuff the better and i think this is the price point that in western countries has been reasonable to a lot of people in terms of the value that they get back and more importantly we're able to push things down to the free tier semi regularly and we always do that when we can including what you could do by

Lenny Rachitsky:So the survey just to give you the official name the van westin drop survey is how you guys ended up pricing chateapizzi

Nick Turley:It was the top google result this was before chateapizzi had real time information otherwise it could have maybe priced itself but it was discord plus google forum plus a blog post on that methodology that got us there so

Lenny Rachitsky:That is incredible what a fun story this is the survey that rahul vorat superhuman popularized in his first run article

Nick Turley:Yeah yeah yeah that's right that's right yeah definitely don't bring me on here as a pricing expert i think you you you have got better people for that

Lenny Rachitsky:Whether it was right or wrong it is now the fastest growing insane revenue generating business in the world so i wouldn't feel too bad

Nick Turley:No it worked out yeah

Lenny Rachitsky:It worked out and by the way i'm on the 200 a month tier so there's clearly room thank you

Nick Turley:Thank you know that the story of that one is is interesting too because you know originally the purpose of the plus plan was to be able to ship first uptime and then be able to ship capabilities that we couldn't scale to everyone and at some point got so many people in the plus tier that had just lost that property so the re the main reason we came up with the $200 tier is just we had so much incredible research that's actually really really powerful like you know o three pro or you know tomorrow g p t five pro and just having a vehicle of shipping that to people who really really care is exciting even though it kind of violates the standard way a saas page should look it's like a little jarring to see the see the 10x jump so thank you for being a subscriber on that and thank you everyone else who's watching you subscribe to any tier it's it's great

Lenny Rachitsky:I'm just gonna throw a a fishing line into this pond of are there any other stories like this you shared this incredible story of chat with gpt three point five being the original name how you came up with pricing is there anything else enterprise is

Nick Turley:An interesting one too because we've been so much incredible adoption in the enterprise and it's sort of objectively crazy to try to take on building a developer business and a consumer business and a develop and and an enterprise business and and and all at once but you know the story there is in in like month one or or two i it was like very clear that most of the usage was like kinda worky usage actually more than today where you've got so many like kind of consumers on the product and you know it's kinda sort of transcended into pop culture but at the time it was you know writing coding that kind of stuff and we were pretty quickly in know organically in like 90% of fortune 500 companies in a way that i had seen maybe at dropbox back when i you know those were my two jobs ago where we had kind of had a similar story and since then there's been more plg companies but the real reason we did enterprise remember we're debating should we do enterprise or should we launch an ios app because that's how small the team was yeah the reason they did yeah did is we were starting to get banned in companies because they all you know felt you know rightfully or wrongfully that you know the the privacy and deployment story etcetera wasn't there so i was just like man we have to do something we're gonna miss out on a generational opportunity to build a a a a a work product and you know we've literally defined agi as you know outperforming most humans at economically valuable work or i'd probably butchered that but know i think i think that's the way we put it and so it i feel like we had to be present there and it was a fairly you know quick decision at the time but it's grown into an immense business we just hit 5,000,000 business subscribers up from three i think a month or two ago so it is kind of the spin off that it's taking a life of its own that i'm really really excited about for for apps really

Lenny Rachitsky:That is a lot to be handling the platform essentially api the consumer product the fastest growing most successful product in history and also the b two b side which is clearly a massive business do you have any kind of heuristics for how to make these trade offs do all this at once and stay sane and be successful

Nick Turley:That's a good question and yeah first off i don't run the developer stuff anymore we've found someone way more competent with nvidia to do that and he's amazing so i still look after the you know various forms of of of chat but you know luckily you don't have to make make that trade off openai does and i i can get into that too but it keeps me a little bit more sane i will say that there you kinda have to practice in two different ways when you're when you're building on this ai stuff one is sort of working backwards from the model capabilities and that is much more art than science where i think you really need to look at what tech do we have available and what is like the most awesome way to product productize it and if you apply to some sort of pm framework to that i think you would do something horribly wrong because if you have tech that's you know for example g p t five is is really really good at front end coding now like i think we'd that means you gotta reprioritize it you gotta like actually bring that capability to life maybe that's you know making making chateappity better at at vibe coding and rendering you know applications maybe that's more like you know leveraging the taste of the model to make the the ui more expressive there's like a number of things we could do right but you kinda have to replan and reprioritize and that you know is more important than any particular audience segmentation it's really just looking at you know what is the magic thing we have and how do you make it shine voice is a similar thing it wasn't like our customers need voice they're begging for it or something like that it's like wow we figured out a way how to you know to make these things anything in and anything out what is like a creative awesome way to productize that and then we can see what what people do so i think that's one chunk of it but then the other chunk of it really is more like classic product management where you need to listen to customers and then when your customers are really different that can be confusing because you know chatgpt is a very general purpose product we see when you look at end users there's actually an immense amount of overlap in terms of what they want like primitives like projects or you know history search or sharing and collaboration like all all those kind of things they are actually very very present whether or not you're talking to people at work or you're talking to people at home and school they're slightly different mechanics sometimes but they're they're largely similar investments that i think we can get a lot of mileage out of and then there's enterprise specific work that we just have to do like you gotta do hipaa you gotta do soc two you gotta do all those things if you wanna be a serious player and those are just nonnegotiable so it's complex as you correctly identified but it's kind of the the curse of working on a very open ended and powerful technology one analogy that that someone at openeye who really respect sometimes uses is like we're kinda like disney where disney has this like one kind of creative ip which is like their their their content and they have cruises and they have you know theme parks and they have comics and they have all these different things and i think we have amazing models but there's all these different ways that you can prioritize them and we kinda just have to maximize the impact in in all these different ways

Lenny Rachitsky:As you were talking i was thinking about how usually horizontal platforms that are just so general and can do so much take a long time to take off because people don't know what to do with them they're not amazing at anything and this is an amazing counterexample where it took off immediately and everyone figured it out and then over time they figured it out more and more

Nick Turley:But i i think the reason why is because it just went live talk about another consequential decision actually you know we were debating waitlist no waitlist because we just really knew we couldn't scale the engineering systems and you know the fact that there was no waitlist which no openai release had worked like that before yep and it'd be consequential because like you were able to watch what everyone else was doing live i think when you launch these things all at once for everyone there really is a special moment where you can see what other people are doing and learn from that and a lot of that is actually out of product like there's these crazy tiktok posts that go viral and they have like 2,000 use cases in the comments and i go through those in detail because it's it's not like i knew about those use cases either like they're they're very very emergent and i just go through the comments and you know process because there's so much to learn and for that reason i think we get to skip the empty box problem a little bit because you know so much learning is happening out of product as people are watching each other either in iol or or online

Lenny Rachitsky:That is interesting because you you think about airtable you think about notion all these companies they took like years to just build and craft and think and go deep on what it

Nick Turley:Could be it's like the the compare airtable which like know they they had to do templates they had to do like all these kind of things of taking the horizontal product and making it like use case driven oh they compared to the like the instapot which you know there's recipes being shared on everywhere online like there's a kind of this whole ecosystem around it i think we were really lucky with chatgpt that that happened where there's just users sharing use cases with other users everywhere and and therefore i i think you know we we we we we kind of got very lucky by by by you know jumping jumping ahead on on that journey

Lenny Rachitsky:Right and it feels like a core there is sam had a big following and everyone would pay attention to something he launched so that's a really interesting new strategy for launching horizontal product with a huge distribution channel just launch it and see what see what comes up

Nick Turley:Yeah and of course i'm i'm actually really excited to take some of that into the product like i think there's there's we shouldn't you you know rest on the fact that there's so much out of product discovery happening like i actually think for the average consumer it would be amazing if the product did a little bit more work on really exposing to you what is possible i i still feel like chatgpt feels a little bit like ms dos we haven't built windows yet it will be obvious once we do but you know there there there's something that feels a little bit like like imagine ms dos had gone viral and you were just trying to like hack like little conversation starters onto it that might have missed sort of the big picture in terms of how to really communicate affordances and value to people and so i i think there's actually a ton more product work to do in addition to you know just seeing use cases spread

Lenny Rachitsky:Are you able to share just what you think that might look like this windows version of chatgpt I'll let

Nick Turley:You know when we figure it out we're hiring I I think there's so many interesting product problems here

Lenny Rachitsky:Okay got it by the way also love that tiktok was like their feedback channel those

Nick Turley:Comment threads are they're they're just so wild and and and also the love that people have for it like excitement with what you're sharing their product I I I I I kinda feel like it's it's special that people are so excited about to share what they're doing with your product and I don't take that for granted either

Lenny Rachitsky:This episode is brought to you by posthog the product platform your engineers actually want to use posthog has all the tools that founders developers and product teams need like product analytics web analytics session replays heat maps experimentation surveys llm observability air tracking and more everything posthog offers comes with a generous free tier that resets every month more than 90% of customers use posthoc for free you are gonna love working with a team this transparent and technical you'll see engineers landing pull requests for your issues and their support team provides code level assistance when things get tricky posthog lets you have all your data in one place beyond analytics events their data warehouse enables you to sync data from your postgres database stripe hubspot s three and many more sources finally their new ai product analyst max ai helps you get further faster get help building complex queries and setting up your account with an expert who's always standing by sign up today for free at posthog.com/lenny and make sure to tell them lenny sent you that's posth0g.com/lenny how do you find emergent use cases these days I imagine the volume is very high do you have kind of a trick for figuring out oh here's a new thing we should really think about

Nick Turley:Before I built the product team I actually built the data science team because I I was getting frustrated I was talking to as many users as I could in my calendar you know the weeks after chateaputee was just fifteen minute user interview the whole week through and it was usually I stop doing interviews when I like can predict what the next person's gonna say that's how I know I've talked users but it just wasn't happening like I just kept getting new stuff so data is one way out where I think you you know we we have conversation classifiers that without you know us having to look at the conversations allow us to kinda figure out what are people talking about what use cases are taking off etcetera and I think that's very very helpful the qualitative stuff is important for empathy even though you're never gonna get a wrap on like all the use cases people have I still spend a huge amount of my time do doing that and then yeah things like those tiktoks collections of threads I think they're really really useful and it's just fun to watch people talk to each other about the various use cases that they have

Lenny Rachitsky:Is there kind of a a new emergent use case that you're excited about or is there like a really unusual use of chatgpt that you think about that'd be fun to share

Nick Turley:I mentioned this earlier but I had always conceptualized chatgpt as a worky product whether or not you're at home or you're work like I feel like you know helping getting help with your taxes is very similar to you know the types of things you do at work or you know planning a trip is actually very similar to you know planning an event for work so I've always felt like okay this thing is gonna kind of be a productivity tool and I think something has happened I realized you know few months where that has begun to change and I really do think the fact that you have consumers turning to this thing for day to day advice helping them like have better relationships like the seeing like you know people talk about how this thing like you know saved their marriage is like really exciting to me because like they you know process use it to process their own emotions get feedback on their communication style they just have a buddy to talk to about like really difficult things and that comes with a ton of responsibility and work that we have to do to make those things like life advice great but it also is really really important to me because you can't run away from those use cases you have to run towards them and make them awesome and that's part of what we're trying to do so that emergent behavior is really really cool and more broadly I am so excited about education I'm so excited about health like I I think it would really be a waste if we didn't take the opportunity of using chatgpt to really really help people and I think we've just begun to scratch the surface on on that so there's many aspirational use cases that I wanna make happen

Lenny Rachitsky:Along those lines an interesting use case I've recently had I feel like it's gonna be really helpful for couples that are disagreeing about something when they need like a third opinion I just had this recently where my wife's like you can't heat a whole thing that you're gonna only eat part of in the microwave and then put it back in the fridge it's like what's the problem I'll heat it up I'll put it back in the fridge and she's like no that's really dangerous I'm like let's ask and the fact that she so trusts chatgpt now and relies on it throughout the day it's such a valuable third independent party that we can go to

Nick Turley:Yeah yeah totally and and and you know the a lot of those micro interactions talk about like interesting product work right those are microinteractions that are important right did it like definitively weigh in or did it help you guys think through you know that that that disagreement and you know solve it on your own I think those details actually matter a lot and it's where we're spending a bunch of time

Lenny Rachitsky:Along those lines there was this whole launch of the very sycophantic version of chatgpt where it was just you are the best person in the world everything you tell me is amazingly correct are you able to tell us just what happened there

Nick Turley:Yeah we have you know we we have all kinds of collateral online because we really felt like we should overcommunicate on how we discovered it what we did about it etcetera so I encourage people to check that out we'd have a whole retro on on that model release but basically what happened is that we pushed out an update that you know made the model more likely to you know tell you things that sound good in the moment and you know like you're you're totally right you know you you you know should break up with your boyfriend or something like that and you know that's just really dangerous and it's and and we we took it more seriously than you even might expect because again at current technology levels you can kind of laugh about it maybe it's like ah this thing's always complimenting me I thought it was just me I saw all those comments online but you know it actually is is really important to make sure that these models are optimized for the right things and we have an immense I think luxury to have a mission that affords us to really help people a business model that does not incentivize you know maximizing engagement you know or time spent in the product right so it's really important to us that you feel like this product is helping you with your goals within a that's your current goals or even your long term goals and oftentimes you know being extremely complementary with the user isn't actually in in service of that so we instilled new measurement techniques like you know whenever we put these models in contact with reality and we you know learn about a problem we actually go back and make sure we have good metrics for this stuff so you know we measure circumference now every release to make sure we don't regress and can actually improve on that metric g p t five is an improvement which is really exciting for me but we have more work from there and more broadly it caused us to articulate our point of view I actually spent a bunch of time on a blog post that we just published on monday on what we're optimizing chatgpt for and it really is for your you know to to to help you thrive and achieve your goals not to you know keep you in the product and so there was a bunch of good outcomes from from from that incident it's a good example of how contact for the reality is not just important for the use cases but also for learning what to avoid because you would have never discovered this issue purely in a lab unless you actually heard it from the search

Lenny Rachitsky:I am excited to read that blog post then I was gonna ask you this just like yeah I'll get your

Nick Turley:Feedback on it

Lenny Rachitsky:Yeah and yeah I guess is there anything more there just like how you because this tension is so difficult like you know helping people feel supported but not just letting them believe everything they wanna believe is there anything more you can share there just trying to

Nick Turley:Find that middle ground incentives are important there's a famous saying you show me the incentive and I'll show you the outcome Charlie Munger maybe yeah I think that's where it came from right yeah I think that's very very important so I would take a good look at you know our mission our business model the type of product we're trying to build and you know I I I really think that you know chat to meet is a very special product because it I think in vast majority of cases it makes you you leave it feeling better not worse and you've like you know feeling like you're achieving something you're trying trying to do and so I think that those incentives really matter because it helps you reason about okay when there isn't behavior in the wild that's not good was that a bug or was that by design you know and with with sikovitsy I can very much say that to us that's a bug and then on you know the the forward looking work there's so many you know kind of challenging scenarios you get right and you could easily run away from from from from these use cases like you know the like you know you and your wife go into this thing for you know input on a relationship question or like a dispute you could very easily run away if you were totally risk avoidant and say sorry I can't help you with that I think that's what most tech companies do when they hit a certain scale they run away from these use cases and I think it's a lost opportunity to help people so we wanna run towards these use cases by making the model behavior really really great that can mean connecting you with external resources when you're struggling that can mean not directly answering your question but instead giving you a helpful framework you know in the case of like should I break up with my boyfriend chattubiti should probably not answer that question for you but it should help you think through that question in the way that a thoughtful companion would so I think it's really important to do the work be because I think the upside is immense

Lenny Rachitsky:That is a really profound point you're making there that if most companies if their if their users wanna ask them something risky like getting medical advice or should I break up with my partner or what should I do with this big problem I have

Nick Turley:I feel like we would have immense regret if you had a model that was state of the art on healthbench which is you know a g p d five is the state of the art on you know a bunch of these medical benchmarks right and you didn't use that to help people like if you just disable that use case because you wanted to like avoid all possible downside I I think the duty is to make it awesome and to do the work talk to experts figure out how good it really is where it breaks down communicate that and you know I I think this this technology is too important and has too much potential positive impact on people to to run away from from these high stakes excuses

Lenny Rachitsky:And fast forward to today it's saving lives regularly it's probably saving relationships regularly such a consequential decision which I imagine was made early on

Nick Turley:Yeah we're we're just at the beginning of of watching how this people this this this stuff can transform people it's incredibly democratizing if you compare you know your rollout of this with the rollout of the personal computer right you know computers were like so scarce when they first came out and this stuff is ubiquitous in a way where I you you have access to a second opinion on on medical stuff you have access to you know a a relationship buddy you have access to a personal tutor on literally any topic that makes you curious it's really really special that that that we get to do that so

Nick Turley:Unique point in in history

Lenny Rachitsky:Let me zoom out a bit and talk about openai and just product in general so you've worked at traditional let's say traditional product companies dropbox instacart now you're at openai what's what's maybe the most counterintuitive lesson you've learned about building products from your time at openai

Nick Turley:Each time like I always tried to pick the most different maximally different job whenever I made a job change yeah so you know after dropbox I was like craving a real world product because it was just so different than working on saas etcetera and after instacart I was craving on working on something that intellectually was interesting and had you know this kind of like sort of invoke the nerd in me and you know so I've always looked for things that are really different and then once I showed up at these places I tried to understand what makes that place successful like what is truly the thing that they cracked and how we can lean in that into that even more and I think I spent a lot of time thinking about this with openai especially after chatgpt before that you know it was kind of a moot point because we didn't really have much revenue or products or anything that you know like that and there's a you know a few things that that that that come to mind that have driven many decisions one is the empiricism we talked about that a bit the fact that you can only find out by shipping which is why I've maximally leaned into that and that's you know huge part of why we ship so much one of them is that you know amazing ideas come from anywhere the thing about running a research lab is you really don't tell people what to research that's not what you do and we inherited that culture even as we become a research and product company so just letting people do things who have amazing ideas rather than sort of being the the gatekeeper or prioritizer of everything or something like that has been proven you know immensely valuable to us and that's where much of the innovation comes from is empowered smart people on any function really so that was a good inheritance from what I think made openai successful and makes us successful the interdisciplinariness of really making sure that you put research and engineering and design and product together rather than treating them as silos I think that's the thing that has made us successful and that you see come through in every product we ship like if you know we're shipping a feature and it doesn't get two x better as the model gets two x smarter it's probably not a feature we should be shipping you know not always true you know soc two doesn't get better with you know shredder models but you know I think for many of the core capabilities that's a good litmus test so I've always found you really have to lean into why is this place successful and then maximally accelerate that so to speak because it's it's what allows you to turn something that feels like an accident into something that is a repeatable playbook

Lenny Rachitsky:So you talked about this kind of collaboration between researchers and product people and you've been at the beginning of chatgpt from day one to today from zero to 700,000,000 weekly active users not just registered users weekly active users how have you approached building out that team over time

Nick Turley:What are the other inheritances of being in a research lab is that you take recruiting really seriously that's something that you know AI labs know every person matters but many tech companies they go through hypergrowth and they kind of lose their identity they lose their you know their their talent bars they they they just kinda have chaos so we've always had this tendency to run relatively lean so it is a small team that is running ChatGPT I I take inspiration from WhatsApp where like know it was a very small team running a very global scope product and then the more importantly I yeah I you know you have to treat hiring a little bit more like executive recruiting and less like just pure pipelined recruiting where you really need to understand what is the gap you're trying to fill on each team what is the specific skill set and how do you fill it to give you an example you know I'm a product person at heart but sometimes a team doesn't need a product person because like there's already someone doing that role like like know in many cases have a really talented engineering leader who has amazing product sense or we have a researcher who has product ideas and then and my mind they can play that role and maybe we have something else missing instead like maybe we need like a little bit more front end or something like that in other cases maybe what you're missing is an incredible data scientists so I really like to go through every single team and figure out what is the skill sets that that team needs and how do you put it together from principles rather than just assuming hey we're gonna do like you know

Lenny Rachitsky:a bunch of pipeline

Nick Turley:recruiting for all these different roles and then you know people will find a team later so so I think that's always felt really important to me it's the way that you keep your team really small yet super high throughput it also allows you to hire people who I think Keith Keith Rubois calls this like like barrels I think bear barrels and ammunition where he thinks I think I think this comes from him but I the idea being that sort of the throughput of your org depends on how many barrels you have which is like people who can make stuff happen and I think you can hire and then you can add ammunition around them which is like people helping those people and you know I I think that's been really true for our recruiting too where we try to maximize sort of the number of empowered people who can because that's how you have a small team and still get a ton done so there's a couple things and I spend a lot of time on like vibes too with like each team because I think one of the things that is challenging when you try to do research and product together is that the cultures are different people have different backgrounds and I think to make that go super well you need to spend time team building and making sure that people have a huge amount of trust for each other's skill sets feel like they can think across their boundaries like you know I really believe that product is everyone's job for example and and and for that reason the recruiting sort of doesn't stop when you the people are on the door it actually starts because you have to you know start making the teams awesome

Lenny Rachitsky:is there something you do with team building that would be fun to share just like something you do to create a I just

Nick Turley:love whiteboarding with teams like I just like like love getting into a generative mindset it breaks down everything so that's that's the thing that I I I try it's not particularly creative but I find it to be a universal tool where the minute you can get people to stop thinking about you know what's my job versus another person's job and more like you know we're all in a room like trying to crack something together that is incredible

Lenny Rachitsky:you mentioned this idea of first principles this came up actually when I talked to a lot of people about you is this something you're really big on a lot of people talk about first principles most people are like I don't really understand like or they think they're amazing at thinking from first principles is there something you can share of just what it actually looks like to think from first principles maybe an example that comes to mind where you really went to first principles and came up with something unexpected

Nick Turley:yeah this is not something I'd ever say about myself it's nice that someone else would say it but you know it's a mysterious thing I yeah I think you just really gotta get to ground truth on what you're really trying to solve like for example when as I mentioned with the recruiting thing they are not dogmatic that you have to have a product manager and an engineering manager and and a designer or whatever we're just trying to make an awesome team that can ship so in that case first principles means just really understanding what we actually need and what we're missing rather than applying a previously learned process or behavior so you know I think that's a good example another good example of of I think being first principles in this environment is is is you know does this feature need to be polished you know we get a lot of crap for the for for for the model chooser and I own it I've tried to say that every to everyone who will listen you know for those who don't know model chooser it's this like giant drop down in the product that is like literally the anti pattern of any good product traditionally but you know if you are actually reason from scratch is like is it better to wait until you got a polished product or to ship out something raw even if it makes less sense and start learning and getting into people's hands I think a company with a lot of process or a lot of just you know learned behaviors will make one call which is you know we have like a quality environment we ship and that's what we do if your first principle is about it I think you're like you know what we should ship it's embarrassing but that's strictly less bad than you know not getting the feedback you wanted so I think just approaching each scenario from you know from scratch is so important in this space because there is no analogy for what we're building like there's just you can't copy an existing thing there's no you know are we like an Instagram or are we you know a Google or like a like a you know productivity tool or something like that I don't know but you can learn from everywhere but you have to do it from from from scratch and I think that's why that trait tends to make someone effective at OpenAI and it's something we test for in our reviews too

Lenny Rachitsky:so this theme keeps coming up and I think it's just important to highlight something that you keep coming back to which is this trade off of speed and polish and how in this space speed is more important not just to stay ahead but to learn what the hell people actually wanna do with this thing is there anything more that you think people just may be missing about why they need to move so fast in the space of AI

Nick Turley:yeah I mean the boring answer would be oh it's competitive and everyone's in AI and they're trying to you know outcompete each other yeah I think that's that may be true but that's not the reason that I believe this the the reason really is that you're gonna be polishing the wrong things in the space you actually should polish you know things like the model output etcetera but you won't know what to polish until after you ship and I think that is uniquely true in an environment where the properties of your product are emergent and not knowable in advance and I think that many people get that wrong because like the best product people tend to be craftspeople and they have a traditional definition of craft I also think it would be easy to you know use all what I just said as an excuse not to eventually build a great product so I often tell my teams that shipping is just kinda one point on the journey towards awesomeness and you should put pick that point intentionally where it doesn't have to be the end of of of your iteration at all it can be the beginning but you better follow through so we've been doing a bunch of work especially over the last quarter of like really cleaning up the UI of ChatGPT I'm really excited to do the same for the sort of the response layouts and formats next simply because once you know what people are doing there's no excuse to not polish your product it's just really in a world where you don't know yet you might get very distracted so it's situational again you kind of have to be first principles about it but I do think using velocity especially early on as a tool yep actually this has been said about consumer social for example this is it's not the first space where people have said hey you just gotta try 10 things because you're probably gonna be wrong so I I don't think this is you know never existed before as a dynamic either but I do think with AI it's it's it's important to internalize

Lenny Rachitsky:and there's also an element of the models are getting or changing constantly and so you may not even realize what they're capable of I imagine

Nick Turley:totally the models are changing and yeah the the best way to improve them whether or not you're a lab or actually just someone who's doing context engineering or or you know fine tuning a model maybe you need failure cases real failure cases to make these things better the benchmarks are increasingly saturated so really you need real world scenarios where your product or model is not actually doing the thing it was supposed to do and the only way you get that is by shipping because you get back to sort of use case distribution and you can make those things good and and therefore you know it it's actually the best way to then go articulate to your team especially your ML teams what to hill climb on it's like oh you know people are trying to do x and the model's failing in ways y now let's make those things really good

Lenny Rachitsky:this point about failure cases makes me think about something that both Kevin Weil and Mike Krieger shared which is that evals are becoming a huge new skill that product people need to get good at because so much of product building is now evals writing evals is there something there you wanna share

Nick Turley:my entire OpenAI journey has been this journey of rediscovering eternal product wisdom and principles in like slightly new contexts so I remember I I started writing evals before I knew what an eval was because like I was just outlining sort of very clearly specified ideal behavior for various use cases until someone told me hey you should make an eval and I realized there was this entire world of research evaluation benchmarks that had nothing to do with the product that I was trying to make and I was like wow this might be the lingua franca of how to communicate what the product should be doing to people who do AI research and that really clicked for me and at the end of the day it's not that different from the wisdom of you ought to articulate success before you do anything else it's just a new mechanism for doing that but you can do it in a spreadsheet you can you do it anywhere and I really wanna demystify it for people who feel that term like it's not some technical magic that you have to understand it's really just about articulating success in a way that is maximally useful for for training models

Lenny Rachitsky:awesome there's a I have a post coming out soon that gives you a very good how to for PMs of how to write eval

Nick Turley:I would love to read it okay and I hope you just I hope you agree with it with what I just said because I mean they absolutely speak to it yeah

Lenny Rachitsky:yeah and now there's all these tools that make this easier for you totally okay so this this basically backs up this point that this is just a very important skill that product teams and builders need to get good at yeah yeah okay just a few more questions I know you have a lot going on today one is that this trend of ChatGPT being a big driver of growth for traffic to sites for products for example ChatGPT is now driving more traffic to my newsletter than Twitter which completely shocked me I just was looking at my stats I'm like what the hell this is not something I knew was coming so just I guess thoughts on the future of this how much how you think about just ChatGPT driving growth and traffic to products and sites

Nick Turley:I'm really excited about it because you know in the same way that I I find it dystopian to talk to everything through a chatbot I also find it dystopian to you know not have amazing new high quality content out there and for that reason you know I talked a little bit earlier about search and how that solved like a really important user problem early on because you had this like knowledge cut off thing and you suddenly could talk about anything very obvious in retrospect a it wasn't just a user problem right it's an ecosystem problem where like your original ChatGPT it didn't have outlinks it would just you know answer your question and it would keep you in the product and know even if you wanted to keep reading or or go deeper there was no way for us to drive traffic back to the content ecosystem and I've been really excited about what we've been doing in search not just because it gives people more accurate answers because it allows us to surface really high quality content like this podcast to people who wanna see it and of course there's so many interesting questions about well in the sort of Google era you know there was the search engine optimization and there was a clear we understood mechanisms of how to show up and get more traffic so I get a lot of questions from people like what is the equivalent of that the AI era you know if I'm Lenny and I wanna like 10 x the traffic to my podcast you know what do I actually need to do and the truth is we don't have amazing answers there simply because the way to appeal to an AI model ideally is the same way that you would appeal to a real user because the model's supposed to proxy the interest of the user and nothing else at least you know that's how I want our product to work and for that reason you know my advice is super late is like make really high quality content which you know is is is not as actionable as I think people making content would ideally like and I think this is why we have more work to do because maybe there's a better mechanism or protocol that we could come up with but I'm excited this is driving meaningful traffic for you and I hope that you know other other people making great content start to feel this way because again it's a very nice scenario

Lenny Rachitsky:there's two acronyms people have been using for this this specific skill of mhmm AI driven SEO I think one is AEO which is answer engine optimization the other is GEO is that I I don't I forget the G

Nick Turley:one generve yeah I don't know

Lenny Rachitsky:generative yeah AI optimization do you have a favorite of those two are you

Nick Turley:no no I I I try to shy away from these terms unless they become inevitable this is am not entirely sure if if yeah if that should be a concept or not again I think ideally ChatGPT understands your goals and therefore understands what content would be interesting to you and the content creator's job is to to you know share enough information and metadata about that content such that the AI model can make a user aligned decision and therefore I'm I'm not sure if giving this thing a name and you know making a thing is is is what we should be doing or not I'm very eager to learn from folks making content about what this could look like because again we're we're we're still working through

Lenny Rachitsky:along these lines another question people think about is you have GPTs which are kind of these like G P custom GPT apps that you can build to answer very specific use cases there's always this question of you're gonna build kind of like an app store where I can plug in my news my product into ChatGPT monetize that is there stuff there that you could talk about that might be coming someday

Nick Turley:Yeah I started as a philosophy major and took one coding class because I really liked logic and programming most similar was most similar to that and then I fell in love with coding and then eventually computer science and I just kept doing more and more of it but until then I'd never really thought of myself as a technical person so it was kind of a late discovery in my life that I'm very grateful for

Nick Turley:GBTs are cool they're they're kind of ahead of their time in the sense that we built that kind of concept before you could really build very differentiated things at least in the consumer space you know you're like learning GPT is gonna be pretty similar to what the model could already do out of the box so it's mainly like a way of articulating a use case to people but it doesn't have enough tools yet to make something that feels like an app so to speak different in the enterprise by the way we're seeing a ton of adoption of GPTs there because just every single company has very bespoke business processes and and problems etcetera and it's a really really useful tool there they also have unique data that they can hook up to these things that it can retrieve over so we've seen a lot of success there I think the idea is the right one and I and I think we're gonna figure out a good mechanism for it because when you have so much capability packed into AI it feels really powerful to allow people to package that up in ways that have a clear affordance a clear use case and are differentiated from each other I also would love it if you could start a business on ChatGPT like I think there really is a world where you know as this thing hits billion users scale it can get you distribution it can get you you know started on making something in the same way that people built on the internet and you know there was entirely new businesses to be built so I think we'll have more to share there in the future GBT's was an early stab and I'm just excited to evolve the thinking there as the models get good and our reach increases as well

Lenny Rachitsky:What an incredible combination for someone leading this product just it's true

Lenny Rachitsky:amazing that is really cool I'm really excited to see what you guys do there okay co completely different direction something that I know about you is you studied philosophy in college I did computer science and philosophy right a combo

Nick Turley:It is really coming in full circle in a way that I couldn't have predicted like the amount of questions you have to grapple with are truly super interesting and philosophy isn't it's not a traditionally practical skill but it does really teach you to think things through from scratch and to you know articulate a point of view and I think that has come in handy numerous times

Lenny Rachitsky:Is there a specific philosopher or school that has been most handy to you or is there more just a general oh

Nick Turley:There's so many okay I I I wrote my like senior thesis on whether and why rational people can disagree which you know also comes in handy when a lot of people with very different values have opinions on your model behavior or on you know how things should work so I really like you know twentieth century analytical philosophers it's it's kind of nerdy stuff but I don't know if I have a favorite it's too many to count but that's the kind of stuff I like and some of it ends up being quite analytical like you know like let p be this you know this theory of love and let q be you know this other theory of love and then you do some sort of symbolic manipulation so it is just as much a like sort of brain thought exercise as it is or it is much more that than than practical but it it taught me how to think in a way that continues to be pretty valuable

Lenny Rachitsky:Incredible what a cool what a cool comp of skills and and background last question before we get to your very exciting lightning round so you were a product leader at Dropbox then Instacart now you're the PM of arguably the most consequential product in history how did you land in this role what was the story of joining OpenAI and taking on this work

Nick Turley:Every single career decisions I ever made including my first one out of college was just figuring out who who are the smartest people I know that I wanna like hang out with and learn from and can I work with them and I don't know how to vet companies I don't know how to really logically think through you know what space is gonna take off or something like that but I just do feel like I have a sense on people and you know for Dropbox I you know followed like the head teaching assistant for a class that I was TAing and you know for Instacart I followed followed some of the smartest product people I knew and for for OpenAI the person who I recruited who recruited me Joanne I had messaged her about getting off the Dolly waitlist and she said hey only if you interview here so she like kinda turned it into like a reverse recruiting thing and you know initially honestly I didn't know what I would do here because it was a research lab and I was a product person and they said oh you know don't worry we'll figure it out and they were sort of being cagey I thought they were being cagey because it's OpenAI and they can't share anything but they were being cagey because we we actually just didn't know yet at the time so I showed up and I kinda did everything under the sun and it definitely wasn't product you know it was like you know I think my first task was like fix the blinds or something like that and then you know I started sending out NDAs for people because they were needed some op operational help and then you know I started asking wait why am I sending out NDAs it's oh so we could talk to users and I was like talking to users that sounds like the thing I know how to do and I quickly stumbled into doing product work and then eventually you know leading a bunch of product work but it was organic by just you know showing up and doing what had to be done because again the the company I joined was not a product company by any

Lenny Rachitsky:Wow this is such a good example of I don't know if you think of it this way but when someone offers you a seat on a rocket ship don't ask which seat

Nick Turley:Yeah so I didn't know it was a rocket ship I just thought it was I I kinda got nerd sniped is what I would would describe it as mhmm or like you know as I prepared for the conversation to get you know off the Dolly waitlist really you know I I just started you know reading about the space and that you know piqued the like philosophy brain and then also actually the computer science brain was like wait this is cool and then I started reading all the academic papers of that era and you know so so I just it was intellectual itch and and the people but then I stayed for the product opportunity obviously I I you know post chat GVT when that took off realized that you know we'd built a rocket ship where we'd launched it while building it maybe it's a scenario too but I can't say that you know it felt like a hyped job or or anything like that when I applied

Lenny Rachitsky:So kind of a a lesson there is follow as you said follow the smartest people you know there's also just this thread of follow things that are interesting to you just you playing with Dolly led to this opportunity

Nick Turley:Yeah yeah and actually that's something we still test for is is curiosity is like an attribute that we think matters so much more than your ML knowledge you know I'm not making a comment on research hiring I think you do need some ML knowledge I'm afraid but you know on like for product and engineering and design people and you know those kinds of functions I actually think that if you are just curious about the stuff works it doesn't matter at all if you've never done it before in fact if you were to filter for people who've done it before you would have a very narrow filter of very lucky people rather than necessarily the best people you can get so I think we've scaled that certainly what got me here but I think it's actually just generically been a good predictor of success at OpenAI

Lenny Rachitsky:Nick I told you I had a billion I said I had 2,000,000,000 questions to ask you I feel like I've asked a lot I feel like I still have a billion left but I know you told me right after this you have a big GPT five check-in that you gotta

Nick Turley:Get to so we gotta ship we gotta ship

Lenny Rachitsky:Better ship now that this is recorded and we're putting this out

Nick Turley:This is true is this is forcing

Lenny Rachitsky:Function okay so before we get to a very exciting lightning round is there anything else that you want to share leave listeners with think is important to to share

Nick Turley:I try to share a little bit about how I made decisions because I hope to I'm not that far out of school I like relate a lot to people who are coming in the job market who are trying to figure out what to do with their life right now and I feel very confident that if you surround yourself with people that give you energy and if you follow the things you're actually curious about that you're going to be successful in this era so my you know parting advice to folks really is put yourself around good people and do the things you're actually passionate about because in a world where this thing can like you know answer any question asking the right question is very very important and the only way to get you know learn how to do that is is to to you know nurture your own curiosity so I it worked for me and it's the one repeatable thing that I can share everything else is luck

Lenny Rachitsky:And this is counter to what a lot of people are doing right now which is follow the money where can I make the most how do I grow this thing and make a $100,000,000 like all these people that are getting these crazy offers were not planning to make a lot of money doing this

Nick Turley:It's quite interesting to see that stuff play out because I think all these people entered you know school for genuine reasons they were like excited about the space they're researching it they're pursuing knowledge and I'm happy that that's being rewarded and I don't know what the rewards will look like in the future especially in a post AGI world but I just have a feeling that if you if you you know if you if you follow that advice you'll end up okay

Lenny Rachitsky:With that Nick we've reached our very exciting lightning round I've got five questions for you are you ready sure yep what are two or three books that you find yourself recommending most to other people

Nick Turley:In the product space probably things like High Output Management or The Design of Everyday Things or you know those kind of classic type things because I think they're extremely applicable in AI

Lenny Rachitsky:We talked about philosophy I don't know is there a philosophy book you you like here's the one to read if you're gonna

Nick Turley:Oh man like anything by like Rawls and Nozick like I like the political stuff it's I think it's really fun like it's that is the type of thing I recommend I don't think there's a practical reason to to read that stuff but I will nerd out about it with you so at your own peril

Lenny Rachitsky:Do you have a favorite recent movie or TV show you've really enjoyed if you've had time to watch anything

Nick Turley:Think you've gotta do a little bit of sci fi to be in this space you shouldn't copy any of it but i think i think you learn from it so regularly rewatch her and Westworld Severance is Severance was great i think that's the stuff that you know when i have time i'll i'll i'll meddle with

Lenny Rachitsky:That is awesome i love that those are the two of all the sci fi movies those are the ones you resonate most with and find most interesting and valuable

Nick Turley:Yes but that's probably my own limitation so i'm sure there's more to more to discover by

Lenny Rachitsky:The way have you read fire upon the deep that's sci

Nick Turley:Fi no

Lenny Rachitsky:Okay i i don't know if you have time to read this book but it's i think you would love it it's such a oh good okay ai oriented sci fi space opera sort of book

Nick Turley:Great yeah okay it out nice tangent yeah yeah appreciate it

Lenny Rachitsky:Okay is there a favorite do you have a favorite product you recently discovered that you really love

Nick Turley:I actually don't i am like at extreme capacity it's yep it it yeah it's it's it's kind of interesting sometimes like you know api developers ask me it's like hey are you like you know cop gonna copy all of our products there is like i actually just do not have time to to follow-up you know what's going on outside of openai because the pace here is is so so intense so don't have good recs for you i'm afraid

Lenny Rachitsky:That's a really that's a comforting answer i think to a lot of product companies nick has no time to even listen to

Nick Turley:Our stuff

Lenny Rachitsky:Oh man okay do you have a favorite life motto that you find yourself using when things are tough sharing with friends or family that other few people find useful

Nick Turley:Being the average of you know the the people you you spend the most time with is is like a thing that i really internalize and both of my personal life where there's like people who give me energy and who lift me up and make me like a better person my fiance is one of those people but you know there's many people in my life but then there's also just like you know at work there's the equal one and again that's how i've made all the career decisions it's like you know who do i wanna learn from so i apply that principle constantly

Lenny Rachitsky:Final question everybody i talked to told me that you are a very good jazz pianist you have won competitions i think you were planning to do this as your main thing and then you somehow took the side quest

Nick Turley:Yeah i chickened out of that at the very last minute but i was gonna i was gonna go to school for for music and that's still my like hopefully chapter two

Lenny Rachitsky:Wow i love that that might still happen

Nick Turley:Might still happen now i'm like i'm in some some some for fun bands and we will kick from time to time it's like the the one thing i can do when i'm otherwise you know super tired and can't can't can't think anymore because it it balances me out in in good ways but yeah hopefully i'll get to do more of it in the future

Lenny Rachitsky:Is there any analogs between music and your job anything that you you find

Nick Turley:Yeah actually i i feel like i feel like you could think of software development as like you know or being a product person as you could you could be a conductor of an orchestra or you could be in a jazz band and i think of it as a jazz band where i'm like don't believe in the in the idea of everyone having this like set part that they have to play and me like kind of you know telling people when to play i i i love how you know in in jazz or like other forms of improvised music you're kind of riffing off of each other and you listen to what one person played then you like play something back and i i think that great product development is like that in the sense that ideas could come from anywhere it shouldn't be a scripted process you should be like trying stuff out having fun having play in in in in in what you do so i use that analogy a lot for those we for those who like music it tends to resonate

Lenny Rachitsky:Nick i am so thankful that you made time for this i know today is insane today tomorrow is gonna be even more insane for the entire world they have no idea what's coming thank you so much for doing this two final questions where can folks find you if you want them to find you online where can folks find gpt five potentially and then just how can listeners be useful to you

Nick Turley:Just use the product you don't even have to pay it should be your default model starting tomorrow and just use it and don't think about models anymore unless you want to and you're per user in which case you get all the old models so rest assured and useful honestly i i learned so so much from people at large and chatgpt users etcetera so just keep doing your thing i am watching and learning and i appreciate all the feedback so i'm sure after we fix the model chooser you guys will roast me for something else and i'll take it so keep it coming

Lenny Rachitsky:Amazing nick thank you so much for being here

Nick Turley:Thanks for having me lenny

Lenny Rachitsky:And good luck tomorrow thanks bye everyone thank you so much for listening if you found this valuable you can subscribe to the show on apple podcasts spotify or your favorite podcast app also please consider giving us a rating or leaving a review as that really helps other listeners find the podcast you can find all past episodes or learn more about the show at lennyspodcast.com see you in the next episode