AI & Product

Developing Product Sense in the World of AI

Devika Naik · Feb 2026 · 8 min read

The product sense gap

Every product manager has been there: you're in a roadmap review, an engineer presents an ML model that improves some metric by 12%, and someone asks, "But should we ship it?" The room goes quiet.

This is the product sense gap in AI. Traditional product intuition was built on decades of software patterns - if a user clicks a button, something happens. Cause and effect are visible, testable, and explainable. But AI products break this mental model. Outputs are probabilistic. Edge cases are unpredictable. And the gap between a model that works in a notebook and a product that works in the real world is enormous.

After spending years shipping AI-powered products at Google and building ML-driven features at Deutsche Bank, I've developed a framework for how product managers can build genuine product sense for AI - not just the technical literacy to read a model card, but the intuition to know when an AI feature is ready for users and when it isn't.

1. Think in distributions, not determinism

The single biggest mindset shift for PMs working with AI is moving from deterministic to probabilistic thinking. In traditional software, you can test every path. In AI, you're shipping a system that will be wrong some percentage of the time, by design.

Product sense in AI means developing intuition for:

• Where accuracy matters most - a 95% accurate spam filter is fine; a 95% accurate medical diagnosis tool is not. The same model accuracy means completely different things depending on the cost of being wrong.
• Which errors are recoverable - a bad recommendation on a feed is a mild annoyance. A bad recommendation in a financial product can cost someone their savings. Design for the error, not just the happy path.
• When "good enough" is actually good enough - I've watched teams delay launches by months chasing the last 2% of accuracy, while a competitor shipped at 90% and won the market. The product question is never "how accurate is the model?" - it's "is this accurate enough for users to trust it and come back?"

2. The evaluation trap

Here's something that surprises non-ML PMs: evaluating an AI product is harder than building it. Seriously. You can train a model in a weekend. Figuring out if it actually works for users in production is a months-long problem.

Traditional A/B testing works for "did users click more?" But AI products need a different evaluation lens:

• Offline metrics vs. online behavior - a model that scores well on a holdout set might perform terribly in production because the data distribution shifted. I've seen models with excellent precision/recall metrics that users hated because the errors clustered in visible, embarrassing ways.
• Leading vs. lagging signals - engagement might spike because an AI recommendation is novel and surprising, then crash two weeks later when users realize it's not actually useful. Look for retention and repeat usage, not first-day excitement.
• The silent failure problem - when a traditional feature breaks, users complain. When an AI feature degrades, it just slowly becomes less useful. Nobody files a bug report that says "my recommendations seem slightly less relevant this week." You need monitoring for model drift, not just error rates.

3. Ship the interface, not just the model

One of the most common mistakes I see in AI product development is treating the model as the product. The model is not the product. The interface between the model and the user is the product.

This means:

• Transparency over magic - users trust AI more when they understand why it made a recommendation. At Google, I learned that showing users even a simple "Because you searched for X" alongside an ad increased both trust and engagement. The model was identical; the interface changed everything.
• Graceful degradation - what happens when the model is uncertain? The worst AI products show nothing. The best ones show something useful with appropriate hedging. "We're not sure, but based on similar cases..." is infinitely better than a blank screen or a confidently wrong answer.
• Progressive disclosure of intelligence - don't show users everything the model knows on day one. Start simple, let users pull for more depth when they want it. A stock analysis tool that opens with a simple Buy/Hold/Sell verdict and lets users drill into the evidence is more useful than one that dumps the entire analysis on screen.

4. Build for the loop, not the moment

AI products improve with data. This is the single biggest strategic advantage of AI - and the most underexploited one. Product sense in AI means designing feedback loops from day one.

Every interaction is training data. Every user correction is a label. The product should get smarter with every use, and users should be able to feel it getting smarter.

The best AI product loops I've built follow this pattern:

1. Model generates an output (recommendation, prediction, analysis)
2. User takes an action (clicks, ignores, corrects, overrides)
3. Action becomes signal (positive/negative implicit feedback)
4. Signal improves the model (retraining, fine-tuning, or rule adjustment)
5. User notices improvement (the crucial step most teams skip)

That last step is the difference between an AI product that retains and one that churns. Users need to feel the system learning. This might be as simple as "We noticed you always skip jazz recommendations - we've updated your preferences" or as sophisticated as a visible accuracy score that improves over time.

5. Know when AI is the wrong answer

This might be the most counterintuitive point, but the best AI product sense includes knowing when not to use AI.

I've seen teams spend six months building an ML model for a problem that a SQL query could solve. I've seen recommendation engines built for products with 200 users and 50 items - where a hand-curated list would outperform any model. I've seen AI features added to products purely because investors wanted to hear the word "AI" in the pitch.

AI is the right tool when:

• The problem involves pattern recognition at a scale humans can't handle
• The data is rich, plentiful, and continuously generated
• The cost of errors is manageable and the value of getting it right is high
• The feedback loop is tight enough for the model to improve

AI is the wrong tool when:

• Simple rules or heuristics would get you 80% of the way there
• You don't have enough data to train a meaningful model
• The problem requires explainability that current models can't provide
• The stakes are too high for probabilistic errors

Building the muscle

Product sense isn't a skill you read about and suddenly have. It's a muscle you build through shipping, through watching users interact with your product, and through developing pattern recognition over hundreds of decisions.

For AI products specifically, the muscle comes from:

• Reviewing model outputs daily - not metrics dashboards, actual outputs. Read the recommendations. Look at the predictions. Develop a feel for when the model is being useful versus when it's confidently hallucinating.
• Talking to users about their mental model - not "do you like this feature?" but "what did you expect to see here?" and "when you saw this recommendation, what did you think?" Users' mental models of how the AI works (even when wrong) determine whether they trust it.
• Building things yourself - you don't need to be an ML engineer, but you should be able to fine-tune a model, run an evaluation, and understand why a model makes the predictions it does. The PMs who ship the best AI products are the ones who've gotten their hands dirty with the actual technology.

The world of AI products is still young enough that the playbooks are being written in real time. The PMs who develop genuine product sense for AI - not just technical literacy, but real intuition - will be the ones who define how the next generation of products work.

And that intuition doesn't come from reading papers or attending conferences. It comes from shipping, from failing, and from paying close enough attention to know why.