Frequently Asked Questions (FAQ)¶
1. Which metrics should I track to measure success?¶
The Blueprint uses phase-specific metrics. Use the table below as a starting point and adapt to your project context.
| Phase | Key Metrics | Source |
|---|---|---|
| Discovery & Strategy | Data quality score, feasibility outcome | Evidence Standards |
| Validation (PoV) | Accuracy vs. baseline, latency (p95 = 95th percentile; 95% of requests are faster than this value), cost per prediction | Experiment Ticket, Gate Reviews |
| Development | Test coverage, integration score, Red Line compliance | SDD Pattern |
| Delivery | Adoption rate, user satisfaction (CSAT/NPS), go-live readiness | Handover Checklist |
| Monitoring & Optimisation | Drift indicators, business impact, hallucination rate | Drift Detection, Model Health Review |
| Continuous Improvement | Kaizen velocity, benefits realisation vs. business case | Metrics Dashboards |
Start with three
Choose a maximum of three key metrics per phase. Too many metrics create reporting overhead without insight. Only add metrics when existing ones leave questions unanswered.
2. What kind of estimates do we need in AI projects?¶
Short answer: Story points remain useful for coordination but lose their predictive value for AI-specific experiment work.
Why? AI experiments are non-deterministic: outcomes depend on data quality, model behaviour and unexpected edge cases. Traditional estimation assumes known complexity — with AI, the complexity often only becomes clear in hindsight.
Recommendation:
- Time-boxing instead of estimation for AI experiment work. Use the Experiment Ticket to scope spikes with a fixed end date and decision point (Continue / Pivot / Stop).
- Keep story points for infrastructure, integration and UI work within the same sprint — relative estimation remains valuable there.
- Avoid the trap of retroactively assigning higher points to failed experiments. A failed experiment with valuable insights is not a "bigger ticket" — it is a learned outcome.
See also: Agile Anti-patterns for common pitfalls in AI projects.
Next step: Define your phase-specific metrics and record them in the Project Charter. → See also: Experiment Ticket | Metrics Dashboards