Managers Playbook

This playbook helps engineering managers monitor team performance, identify bottlenecks, improve collaboration, and maximize the ROI of AI coding tools.

Quick Start for Managers

Get Team Onboarded

Ensure all team members have A24Z installed and configured

Set Up Team View

Configure dashboard filters to show team-wide metrics

Establish Baselines

Measure current performance for comparison

Set Goals

Define team OKRs related to AI tool adoption

Monitor and Iterate

Weekly reviews to track progress and identify issues

Key Metrics to Track

1. Team Success Rate

What it is: Average tool success rate across all team members. Why it matters: Indicates overall team proficiency with AI tools. Red flags:

Success rate <80%: Training needed
Wide variance: Inconsistent practices
Declining trend: Tool issues or increasing complexity

Target: >85% team average

2. Team Velocity Impact

What it is: Correlation between AI tool usage and sprint velocity. Why it matters: Quantifies AI tool ROI. How to measure:

Compare velocity before/after AI adoption
Track story points per sprint
Measure time to complete similar tasks

Target: 20-30% velocity increase

3. Adoption Rate

What it is: Percentage of team actively using AI tools. Why it matters: Low adoption = wasted investment. Segments to track:

Active daily users
Occasional users (weekly)
Non-users

Target: >90% active daily usage

4. Tool Usage Distribution

What it is: Which tools are most/least used by the team. Why it matters: Identifies training opportunities and workflow patterns. Actions:

Train on underutilized tools
Share best practices for popular tools
Identify workflow inefficiencies

5. Cost per Developer

What it is: Monthly AI tool costs per team member. Why it matters: Budget management and ROI calculation. Benchmark:

Average: $50-200/developer/month
Heavy users: $200-500/developer/month

Action: Monitor outliers and optimize

Common Goals and How to Achieve Them

Goal: Increase Team Productivity

Baseline Current Performance

Metrics: Sprint velocity, story points completed Action: Record pre-AI tool metrics

Identify Top Performers

Filter: Sort by success rate and tool usage Action: Document their practices

Share Best Practices

Method: Team workshops, pair programming Focus: Successful prompt patterns, tool selection

Measure Improvement

Frequency: Weekly trend analysis Track: Velocity increase, completion rates

Goal: Reduce Onboarding Time

AI tools can accelerate new hire ramp-up: Week 1-2: Learning Phase

Monitor: Tool usage frequency
Goal: Daily active usage
Support: Pair with experienced team members

Week 3-4: Active Usage

Monitor: Success rate >70%
Goal: Completing tasks independently
Support: Review failed executions, provide guidance

Week 5+: Proficiency

Monitor: Success rate >85%
Goal: Contributing at full speed
Measure: Compare to traditional onboarding (typically 3-6 months)

Target: Reduce onboarding from 12 weeks to 6-8 weeks

Goal: Improve Code Quality

Track Quality Metrics:

Bug rate in AI-assisted code vs. traditional
Code review comments per PR
Test coverage

A24Z Correlation:

Higher success rates = fewer bugs
Proper tool usage = better code structure
Team consistency = easier reviews

Actions:

Share code quality standards
Review AI-generated code patterns
Establish team prompt guidelines

Weekly Manager Workflow

Monday: Week Planning (15 minutes)

1. Review last week's metrics
   - Team success rate trend
   - Velocity vs. target
   - Cost per developer

2. Identify focus areas
   - Who needs help?
   - Which practices to share?
   - Any red flags?

3. Set weekly goals
   - Success rate targets
   - Adoption goals
   - Cost optimization

Mid-Week: Check-In (10 minutes)

1. Monitor daily metrics
   - Usage trends
   - Any sudden drops?
   - Cost tracking

2. Address blockers
   - Tool failures
   - Training needs
   - Technical issues

Friday: Retrospective (20 minutes)

1. Review week's performance
   - Did we hit our goals?
   - What worked well?
   - What needs improvement?

2. Plan for next week
   - Adjust targets
   - Schedule training
   - Share learnings

3. Team communication
   - Share wins
   - Recognize top performers
   - Communicate improvements

Team Performance Measurement Framework

DORA Metrics with AI Context

Track how AI tools affect your team’s DORA metrics:

Deployment Frequency

Before AI: 2x per week With AI: 3-4x per week Improvement: 50-100%Track in A24Z:

Velocity increase
Code generation efficiency
Testing time reduction

Lead Time for Changes

Before AI: 3-5 days With AI: 1-2 days Improvement: 40-60%Track in A24Z:

Session duration trends
First-time success rates
Tool usage optimization

Change Failure Rate

Before AI: 15% With AI: 10-12% Improvement: 20-30%Track in A24Z:

Code quality metrics
Test coverage
Review thoroughness

Time to Restore Service

Before AI: 2-4 hours With AI: 30-60 minutes Improvement: 60-75%Track in A24Z:

Debugging session efficiency
Tool usage during incidents
Knowledge sharing

Team Health Indicators

Weekly Team Health Score:

Team Health = (
  (Success Rate × 0.3) +
  (Adoption Rate × 0.2) +
  (Velocity Improvement × 0.3) +
  (Cost Efficiency × 0.1) +
  (Knowledge Sharing × 0.1)
) × 100

Example:
= (0.88 × 0.3) + (0.92 × 0.2) + (0.25 × 0.3) + (0.85 × 0.1) + (0.80 × 0.1)
= 0.264 + 0.184 + 0.075 + 0.085 + 0.080
= 0.688 × 100
= 68.8 (Good)

Scoring:
🟢 Excellent: 80-100
🟡 Good: 60-79
🟠 Fair: 40-59
🔴 Needs Attention: &lt;40

Leading vs Lagging Indicators

Leading Indicators (Predict Future Performance):

Daily active usage rate
Prompt library growth
Knowledge sharing frequency
Training participation
Success rate trends

Use these to: Catch issues early and adjust Lagging Indicators (Show Results):

Sprint velocity
Cost per feature
Bug rates
Time to market
ROI achieved

Use these to: Measure outcomes and report up

Best Practices for Managers

1. Lead by Example

Use AI tools yourself
Share your own metrics
Be transparent about successes and failures

2. Celebrate Progress

Recognize improvements in success rates
Share team wins in standups
Highlight innovative tool usage

3. Provide Support

Regular 1-on-1s to discuss AI tool usage
Pair struggling developers with high performers
Invest in training and workshops

4. Set Clear Expectations

Document team standards:

# Team AI Tool Standards

## Required
- Install A24Z before first sprint
- Maintain &gt;85% success rate
- Attend monthly tool training

## Encouraged
- Share successful prompts in Slack
- Pair program to learn techniques
- Experiment with new tools

## Metrics Review
- Individual metrics reviewed monthly
- Team metrics reviewed weekly
- No punitive measures for learning

5. Foster Experimentation

Allocate time for tool exploration
Encourage sharing of experiments
Don’t penalize failures during learning

Red Flags and Interventions

🚨 Low Team Adoption

Warning: <60% of team using tools daily Root causes:

Lack of training
Tools not valuable for tasks
Technical barriers

Interventions:

Survey team: Why not using?
Provide training sessions
Make adoption easier (automated setup)
Show clear value/ROI

🚨 Declining Success Rates

Warning: Team success rate dropping >10% month-over-month Root causes:

Increasing task complexity
Tool degradation
Knowledge gaps

Interventions:

Analyze failed executions
Provide targeted training
Update prompt templates
Review tool configurations

🚨 High Cost Variance

Warning: Some developers costing 3x+ others Root causes:

Inefficient usage patterns
Different task types
Lack of awareness

Interventions:

Review high-cost sessions
Share optimization techniques
Set budget alerts
Provide cost visibility

🚨 Uneven Performance

Warning: Wide variance in success rates (>30% difference) Root causes:

Skill gaps
Different work types
Inconsistent practices

Interventions:

Pair low/high performers
Standardize practices
Create shared prompt library
Regular knowledge sharing

Measuring ROI

Quantitative Metrics

Productivity:

ROI = (Velocity Increase × Team Size × Avg Salary) / AI Tool Costs

Example:
- 25% velocity increase
- 10 developers
- $150K avg salary
- $1500/month tool costs

ROI = (0.25 × 10 × $150K) / ($1500 × 12)
    = $375K / $18K
    = 20.8x return

Time Savings:

Hours Saved = Avg Task Time Reduction × Tasks per Sprint × Team Size

Example:
- 2 hours saved per task
- 20 tasks per sprint
- 10 developers

Weekly Hours Saved = 2 × 20 × 10 = 400 hours

Qualitative Benefits

Reduced onboarding time
Higher code quality
Better developer satisfaction
Faster feature delivery
Improved team consistency

Communication Templates

Weekly Team Update

# AI Tools Weekly Update - Week of [Date]

## 📊 Team Metrics
- Success Rate: 89% (↑3% from last week)
- Active Users: 9/10 (90%)
- Total Sessions: 287
- Cost: $1,245 ($124/dev)

## 🎉 Wins
- Alice hit 95% success rate!
- Team velocity up 15% this sprint
- Zero tool-related blockers

## 📈 Focus for Next Week
- Onboard Bob to new refactoring tools
- Share Alice's prompt templates
- Cost optimization workshop Thursday

## 💬 Share Your Feedback
Reply with your AI tool wins and challenges!

Monthly Management Report

# AI Tools ROI Report - [Month]

## Executive Summary
Our AI tool investment is delivering 18x ROI with 22% velocity increase.

## Key Metrics
- Adoption: 90% (9/10 engineers)
- Success Rate: 87% avg
- Velocity Impact: +22%
- Cost: $1,450/month ($145/dev)

## Business Impact
- 350 hours saved this month
- $52K in productivity gains
- 3 weeks reduced onboarding time

## Challenges & Solutions
- Challenge: Bob struggling with success rate (65%)
  Solution: Paired with Alice, improving

## Next Month Goals
- Reach 90% team success rate
- Reduce cost per dev by 10%
- Train team on advanced features

Team Training Program

Month 1: Foundations

Week 1: Installation and setup
Week 2: Basic tool usage
Week 3: Prompt engineering basics
Week 4: Team best practices

Month 2: Advanced Usage

Week 1: Advanced prompts
Week 2: Tool combinations
Week 3: Debugging with AI
Week 4: Cost optimization

Month 3: Mastery

Week 1: Custom workflows
Week 2: Team collaboration patterns
Week 3: Performance tuning
Week 4: Sharing & documentation

Ongoing: Monthly Workshops

Tool updates and new features
Team prompt library review
Success story sharing
Q&A and troubleshooting

Coaching and Development

Individual Coaching Based on Metrics

High Success, Low Usage

Profile: Proficient but not adopting fullyMetrics:

Success rate: >85%
Daily usage: <50%
Cost: Low

Coaching Approach:

Ask: “What prevents you from using AI more?”
Understand: May prefer traditional methods
Action: Show time savings data, encourage gradual increase
Goal: Increase usage to 70%+ while maintaining quality

High Usage, Low Success

Profile: Enthusiastic but strugglingMetrics:

Success rate: <75%
Daily usage: >80%
Cost: High

Coaching Approach:

Ask: “What challenges are you facing?”
Understand: May need prompt engineering help
Action: Pair with high performer, share prompt library
Goal: Increase success rate to >85%

Low Success, Low Usage

Profile: Not engaged or facing blockersMetrics:

Success rate: <75%
Daily usage: <50%
Cost: Very low

Coaching Approach:

Ask: “What’s your experience with AI tools?”
Understand: May lack confidence or see no value
Action: 1-on-1 setup session, quick win projects
Goal: Build confidence through small successes

High Success, High Usage

Profile: Power user and potential championMetrics:

Success rate: >90%
Daily usage: >80%
Cost: Optimal

Coaching Approach:

Ask: “What’s working well for you?”
Action: Document their practices, make them a champion
Goal: Share knowledge with team, mentor others

1-on-1 Discussion Framework

Monthly AI Tool Check-in (10 minutes in 1-on-1):

## AI Tools Check-in - [Name] - [Date]

### Metrics Review (3 min)
- Current success rate: [X]% (vs team avg [Y]%)
- Usage frequency: [X] sessions/week
- Notable trends: [observations]

### Wins & Challenges (4 min)
**What's working well?**
- [Developer shares 1-2 wins]

**What's challenging?**
- [Developer shares 1-2 challenges]

### Action Items (3 min)
**Developer commits to:**
- [Action 1]
- [Action 2]

**Manager commits to:**
- [Support action 1]
- [Support action 2]

### Follow-up
- Check-in date: [next 1-on-1]
- Success criteria: [specific goals]

Skill Development Paths

Level 1: Beginner (Weeks 1-4)

Focus: Installation and basic usage
Success Rate: 70-80%
Support: Daily check-ins, pairing
Milestone: First successful AI-assisted feature

Level 2: Intermediate (Weeks 5-12)

Focus: Prompt optimization, workflow integration
Success Rate: 80-85%
Support: Weekly reviews, prompt library
Milestone: Consistently productive with AI

Level 3: Proficient (Weeks 13-24)

Focus: Advanced techniques, team contribution
Success Rate: 85-90%
Support: Monthly check-ins, sharing knowledge
Milestone: Contributing to prompt library

Level 4: Expert (Week 25+)

Focus: Innovation, mentoring others
Success Rate: 90%+
Support: Minimal, self-directed
Milestone: Team champion, thought leader

Next Steps

CTOs Playbook

Learn organization-wide strategies

Engineers Playbook

Understand what engineers see

Team Collaboration

Effective collaboration patterns

Dashboard Guide

Master the metrics dashboard

Team Setup

Configure team access

Onboarding Guide

Complete onboarding process

Playbooks

​Managers Playbook

​Quick Start for Managers

​Key Metrics to Track

​1. Team Success Rate

​2. Team Velocity Impact

​3. Adoption Rate

​4. Tool Usage Distribution

​5. Cost per Developer

​Common Goals and How to Achieve Them

​Goal: Increase Team Productivity

​Goal: Reduce Onboarding Time

​Goal: Improve Code Quality

​Weekly Manager Workflow

​Monday: Week Planning (15 minutes)

​Mid-Week: Check-In (10 minutes)

​Friday: Retrospective (20 minutes)

​Team Performance Measurement Framework

​DORA Metrics with AI Context

Deployment Frequency

Lead Time for Changes

Change Failure Rate

Time to Restore Service

​Team Health Indicators

​Leading vs Lagging Indicators

​Best Practices for Managers

​1. Lead by Example

​2. Celebrate Progress

​3. Provide Support

​4. Set Clear Expectations

​5. Foster Experimentation

​Red Flags and Interventions

​🚨 Low Team Adoption

​🚨 Declining Success Rates

​🚨 High Cost Variance

​🚨 Uneven Performance

​Measuring ROI

​Quantitative Metrics

​Qualitative Benefits

​Communication Templates

​Weekly Team Update

​Monthly Management Report

​Team Training Program

​Month 1: Foundations

​Month 2: Advanced Usage

​Month 3: Mastery

​Ongoing: Monthly Workshops

​Coaching and Development

​Individual Coaching Based on Metrics

​1-on-1 Discussion Framework

​Skill Development Paths

​Next Steps

CTOs Playbook

Engineers Playbook

Team Collaboration

Dashboard Guide

Team Setup

Onboarding Guide

Managers Playbook

Quick Start for Managers

Key Metrics to Track

1. Team Success Rate

2. Team Velocity Impact

3. Adoption Rate

4. Tool Usage Distribution

5. Cost per Developer

Common Goals and How to Achieve Them

Goal: Increase Team Productivity

Goal: Reduce Onboarding Time

Goal: Improve Code Quality

Weekly Manager Workflow

Monday: Week Planning (15 minutes)

Mid-Week: Check-In (10 minutes)

Friday: Retrospective (20 minutes)

Team Performance Measurement Framework

DORA Metrics with AI Context

Team Health Indicators

Leading vs Lagging Indicators

Best Practices for Managers

1. Lead by Example

2. Celebrate Progress

3. Provide Support

4. Set Clear Expectations

5. Foster Experimentation

Red Flags and Interventions

🚨 Low Team Adoption

🚨 Declining Success Rates

🚨 High Cost Variance

🚨 Uneven Performance

Measuring ROI

Quantitative Metrics

Qualitative Benefits

Communication Templates

Weekly Team Update

Monthly Management Report

Team Training Program

Month 1: Foundations

Month 2: Advanced Usage

Month 3: Mastery

Ongoing: Monthly Workshops

Coaching and Development

Individual Coaching Based on Metrics

1-on-1 Discussion Framework

Skill Development Paths

Next Steps