Polish vs Ship: Reason From Scratch

The "Ship First, Polish Later" Approach to AI Product Development

Nick Turley, head of ChatGPT at OpenAI, reveals a counterintuitive approach to building AI products that prioritizes rapid learning over initial polish. This model challenges traditional product development wisdom by emphasizing that with AI, you must ship to learn what users actually want.

Core Principles of AI Product Development

• You won't know what to polish until after you ship

  • "This is a pattern with AI where you really have to ship to understand what is even possible and what people want rather than being able to reason about that a priori"
  • "You're gonna be polishing the wrong things in the space... you won't know what to polish until after you ship"
  • The properties of AI products are emergent and not knowable in advance

• Treat the model as the product

  • "There really is no distinction between the model and the product like the model is the product"
  • Iterate on the model like a product by systematically improving on discovered use cases
  • "We would ship it more like hardware... these weird giant big spend R&D projects that would take a really long time"
  • ChatGPT broke this pattern by enabling iterative improvements like software

• Maximize velocity as a learning tool

  • "Learning as fast as possible [is] incredibly important"
  • "Is this maximally accelerated?" became a team mantra to challenge blockers
  • "If this was the most important thing and you wanted to truly maximally accelerate it, what would you do?"
  • Use this as a forcing function to distinguish critical path from what can happen later

• Separate product velocity from safety processes

  • "Product development velocity has to be super high"
  • For frontier models, maintain "a rigorous process where you red team, you work on the system card, you get external input"
  • "You have to play practice for a time where you really really need the process"

Implementation Framework

• Ship open-ended products first

  • "If you're OpenAI, on that point that's kind of a playbook"
  • Let users discover use cases rather than prescribing them
  • "ChatGPT came together at the end because we just wanted the learnings as soon as we could"

• Collect real-world failure cases

  • "The best way to improve [models] is you need failure cases, real failure cases"
  • "The benchmarks are increasingly saturated so really you need real world scenarios"
  • Use these to articulate to ML teams what to improve

• Follow through with polish after learning

  • "Shipping is just kinda one point on the journey towards awesomeness"
  • "Once you know what people are doing, there's no excuse to not polish your product"
  • Don't use velocity as an excuse for permanent lack of quality

• Use evals to articulate success criteria

  • Evals are the "lingua franca" to communicate product requirements to AI researchers
  • "It's not that different from the wisdom of you ought to articulate success before you do anything else"
  • "It's not some technical magic... it's really just about articulating success in a way that is maximally useful for training models"

Organizational Approach

• Build interdisciplinary teams

  • "The interdisciplinariness of really making sure that you put research and engineering and design and product together rather than treating them as silos"
  • "If you're shipping a feature and it doesn't get 2x better as the model gets 2x smarter, it's probably not a feature we should be shipping"

• Hire for curiosity over experience

  • "Curiosity is like an attribute that we think matters so much more than your ML knowledge"
  • "If you were to filter for people who've done it before you would have a very narrow filter of very lucky people"
  • Curiosity has been "a good predictor of success at OpenAI"

• Approach teams like jazz bands, not orchestras

  • "I don't believe in the idea of everyone having this like set part that they have to play and me kind of telling people when to play"
  • "In jazz or other forms of improvised music you're kind of riffing off of each other"
  • "Great product development is like that in the sense that ideas could come from anywhere, it shouldn't be a scripted process"