Model Is The Product

Nick Turley, head of ChatGPT at OpenAI, shares a framework for how to approach AI product development that challenges traditional product development cycles while maintaining core product principles.

Treat the Model as the Product

• "There really is no distinction between the model and the product - the model is the product"
• Must iterate on the model like a product by systematically improving on observed use cases
• Traditional approach was to ship models like hardware (big releases with long gaps)
• ChatGPT broke this pattern by enabling iterative improvements like software

Ship to Learn, Not Learn to Ship

• "This is a pattern with AI - you won't know what to polish until after you ship"
• "You really have to ship to understand what is even possible and what people want rather than being able to reason about that a priori"
• Empiricism is crucial - you can only find out by shipping
• The only way to find failure cases is through real-world usage
  • "You need real failure cases to make these things better"
  • "Benchmarks are increasingly saturated so really you need real world scenarios"

Maximally Accelerated Development

• Ask: "Is this maximally accelerated?" to cut through blockers
• Forces teams to distinguish between critical path items versus what can happen later
• "Execution is incredibly important... ideas are everywhere"
• Separate product development velocity (which must be super high) from safety processes (which need rigor)

Prioritization Framework

• Work backwards from model capabilities (art more than science)
  • "Look at what tech we have available and what is the most awesome way to productize it"
  • When new capabilities emerge, reprioritize to leverage them
• Listen to customers for common primitives (projects, history, search, sharing)
• Maintain separate track for enterprise-specific requirements (HIPAA, SOC2)

Balancing Speed and Polish

• "Shipping is just one point on the journey towards awesomeness"
• Choose that point intentionally - it doesn't have to be the end of iteration
• "Once you know what people are doing, there's no excuse to not polish your product"
• Use velocity as a tool especially early on, then follow through with refinement

Evaluation and Measurement

• Articulate success criteria before building (similar to traditional product wisdom)
• Create evals to communicate desired product behavior to AI researchers
• "It's not some technical magic - it's really just about articulating success in a way that is maximally useful for training models"
• Implement measurement techniques to track progress on key dimensions
• After discovering issues, create metrics to ensure you don't regress

Team Structure and Collaboration

• Run relatively lean teams (inspired by WhatsApp's small team running global product)
• Treat hiring like executive recruiting - understand specific skill gaps on each team
• Focus on "barrels" (people who can make things happen) over "ammunition" (support)
• Break down silos between research, engineering, design and product
• "Product is everyone's job"
• Create trust across different skill sets and backgrounds
• Use whiteboarding sessions to get teams into a generative mindset

First Principles Thinking

• "Approach each scenario from scratch" rather than applying learned behaviors
• Question whether standard product practices apply to your specific AI context
• "There is no analogy for what we're building - you can't copy an existing thing"
• Learn from everywhere but build from first principles