✦ PromptOps 90-Day Learning Path

What PromptOps means

PromptOps is the discipline of managing prompts for LLMs with the same rigor applied to software code. It covers prompt design patterns, versioning and registry management, automated evaluation and regression testing, A/B testing for prompt variants, and deployment pipelines that gate on quality metrics. As prompts become critical application logic, PromptOps prevents regressions and enables systematic improvement.

Who should follow this path

• AI engineers building LLM-powered applications
• ML engineers responsible for prompt quality and reliability
• Product engineers integrating LLM APIs into product features
• Data scientists developing and evaluating prompt strategies
• DevOps engineers extending CI/CD to include prompt pipelines

Prerequisites

• Working experience with at least one LLM API
• Python programming proficiency
• Basic CI/CD pipeline knowledge
• Understanding of software testing concepts
• Familiarity with version control (Git)

The 90-day plan

Daily study recommendation: 2-3 hours/day, six days a week. Consistency beats intensity — block the time in your calendar like a meeting.

Days 1–15: Foundation

• Prompt engineering patterns: zero-shot, few-shot, chain-of-thought, ReAct
• System prompt vs user prompt design
• Temperature, top-p, and sampling parameter effects
• Prompt injection and jailbreak attack awareness
• Prompt structure best practices for different LLM providers

Outcome: Write structured, reliable prompts for three different task types with documented parameter choices.

Days 16–30: Core concepts

• Prompt versioning with Git and semantic versioning principles
• Prompt registries: PromptLayer, LangChain Hub, and custom solutions
• Prompt metadata: version, author, model, use case, evaluation scores
• Prompt templating with Jinja2 and LangChain prompt templates
• Environment promotion: dev/staging/prod prompt versions

Outcome: Set up a versioned prompt registry with metadata and environment promotion workflows.

Days 31–45: Tools and workflows

• Automated prompt evaluation with LangSmith and DeepEval
• LLM-as-judge evaluation patterns
• Regression test suites for prompt changes
• Golden dataset creation and maintenance
• Evaluation metrics: ROUGE, BERTScore, custom task metrics

Outcome: Build an automated evaluation pipeline that gates prompt deployments on quality thresholds.

Days 46–60: Hands-on projects

• A/B testing framework for prompt variants
• Statistical significance in LLM evaluation
• Cost-quality trade-off analysis (prompt optimization)
• Prompt compression techniques (LLMLingua, token reduction)
• Caching strategies for repeated prompt patterns

Outcome: Run an A/B test between two prompt versions with statistically valid results and cost analysis.

Days 61–75: Advanced practices

• CI/CD pipeline integration for prompt changes
• Canary deployment of prompt versions in production
• Rollback procedures for underperforming prompts
• Prompt observability: latency, cost, quality per-version
• Multi-model prompt porting (GPT-4 → Claude → Gemini)

Outcome: Deploy a full CI/CD prompt pipeline with canary rollout, observability, and automated rollback.

Days 76–90: Portfolio, interview & certification prep

• PromptOps portfolio project
• Interview preparation for AI engineer and prompt engineer roles
• PromptOps interview questions
• Metrics: prompt quality scores, cost-per-call, regression rate
• Emerging: agentic prompt chains, structured outputs (JSON mode)

Outcome: Ship a portfolio PromptOps project and be ready for AI engineer interviews.

Weekly outcomes at a glance

Phase	Outcome
Days 1–15	Write structured, reliable prompts for three different task types with documented parameter choices.
Days 16–30	Set up a versioned prompt registry with metadata and environment promotion workflows.
Days 31–45	Build an automated evaluation pipeline that gates prompt deployments on quality thresholds.
Days 46–60	Run an A/B test between two prompt versions with statistically valid results and cost analysis.
Days 61–75	Deploy a full CI/CD prompt pipeline with canary rollout, observability, and automated rollback.
Days 76–90	Ship a portfolio PromptOps project and be ready for AI engineer interviews.

Tools to learn

• PromptLayer
• LangChain
• LangSmith
• DeepEval
• Ragas
• Jinja2
• GitHub Actions
• Langfuse
• OpenAI API
• Anthropic Claude API

Labs to practice

• LLM RAG Application Basics
• CI/CD with GitHub Actions

Mini projects

• Build a prompt registry with semantic versioning, LangSmith evaluations, and GitHub Actions CI gates
• Create an A/B testing framework for prompt variants with statistical significance tracking and cost dashboards
• Implement a prompt canary deployment system that auto-rolls back on quality score degradation

Interview questions to prepare

1. What is PromptOps and why does it matter as LLM applications scale?
2. How do you version control prompts effectively for a team?
3. Explain LLM-as-judge evaluation and its limitations.
4. How do you A/B test two prompt variants with statistical rigor?
5. What is prompt injection and how do you defend against it?
6. How do you optimize a prompt for cost without sacrificing quality?
7. Describe a CI/CD pipeline for prompt changes in production.
8. How do you port a prompt optimized for GPT-4 to a different LLM provider?

Certification suggestions

• DeepLearning.AI Prompt Engineering for Developers — DeepLearning.AI / Coursera
• Hugging Face NLP Course Certificate — Hugging Face
• AWS Certified Machine Learning Specialty — AWS
• Databricks Generative AI Fundamentals — Databricks

Browse the full certification registry for exam details and official links.

Free resources

• OpenAI Prompt Engineering Guide
• LangSmith Documentation
• DeepEval Documentation
• PromptLayer Documentation
• Anthropic Prompt Library

Related roadmaps

• LLMOps Engineer Roadmap
• MLOps Engineer Roadmap

Related tool categories

• LLMOps Tools
• MLOps Tools
• CI/CD Tools
• Testing Tools