Build Production-Ready RAG Systems That Actually Scale
Systematically Improving RAG Applications by Jason Liu is an advanced AI engineering course designed to help engineers, product teams, and AI practitioners move beyond basic Retrieval-Augmented Generation (RAG) demos and build scalable, production-grade retrieval systems.
Instead of relying on trial-and-error experimentation, this program introduces a structured “RAG Flywheel” framework focused on continuous evaluation, optimization, and measurable improvements.
If you want to improve retrieval accuracy, optimize search quality, and build AI systems that perform reliably in production, this course provides a practical, engineering-focused roadmap.
What You’ll Learn
Inside Jason Liu – Systematically Improving RAG Applications, you’ll learn how to:
- Systematically evaluate and improve RAG systems
- Diagnose retrieval quality issues using metrics and evaluation frameworks
- Build multimodal retrieval systems for documents, images, and tables
- Fine-tune rerankers and embeddings for better search performance
- Create synthetic datasets for rapid experimentation
- Route queries intelligently across different retrievers
- Build scalable feedback loops for continuous optimization
- Improve AI search systems using production-grade methodologies
The RAG Flywheel Framework
The course introduces a systematic approach to improving RAG applications through measurable iterations.
With the RAG Flywheel, you’ll learn how to:
✅ Identify retrieval failures using synthetic evaluations
✅ Fine-tune embeddings for measurable performance gains
✅ Collect and use user feedback effectively
✅ Segment queries for targeted optimization
✅ Build multimodal indexing systems
✅ Automatically route queries to the best retriever
This transforms RAG optimization from vague experimentation into structured engineering workflows.
Course Curriculum
The program is organized into multiple weeks covering evaluation, optimization, multimodal retrieval, and query routing systems.
Week 1 – Foundations & Retrieval Optimization
Lecture 0 – Introduction
- Course overview and RAG fundamentals
Lecture 1 – Systematically Improving RAG Applications
- Understanding retrieval bottlenecks
- Building structured optimization workflows
Lecture 2 – Fine-Tuning Rerankers and Embeddings
- Improve retrieval quality with rerankers
- Embedding optimization strategies
- Retrieval performance tuning
Sessions & Tutorials
- Hands-on implementation walkthroughs
- Retrieval evaluation exercises
Week 2 – Product, Feedback & Segmentation
Lecture 3 – Product / UX / Feedback
- Build user feedback loops for retrieval systems
- Improve RAG quality using real user behavior
Lecture 4 – Segmentation
- Segment queries for targeted optimization
- Improve routing and retrieval relevance
Office Hours & Expert Sessions
Topics include:
- AI agent deployment lessons
- Tool-call error handling
- LangChain deployment systems
Week 3 – Multimodal RAG & Routing
Lecture 5 – Multimodal RAG
- Build retrieval systems for:
- Documents
- Images
- Tables
- Structured data
Lecture 6 – Routing
- Intelligent retriever selection
- Query classification systems
- Multi-agent retrieval workflows
Hands-On Engineering Components
The course includes extensive practical implementation resources:
- 12+ hands-on Python notebooks
- Live coding tutorials
- Evaluation datasets and experiments
- Retrieval benchmarking systems
- Real-world production workflows
Advanced Topics Covered
Evaluation & Metrics
- Precision, Recall, and MRR
- Retrieval benchmarking
- Synthetic evaluation pipelines
Hybrid Search Systems
- BM25 + semantic embeddings
- Metadata filtering systems
- Hybrid ranking optimization
Multimodal Retrieval
- Document understanding
- Table indexing
- Image retrieval systems
Advanced Retrieval Architectures
- Cross-encoders
- ColBERT
- SPLADE
- Fine-tuned rerankers
Feedback & Optimization
- User feedback loops
- Continuous improvement systems
- Data flywheel architectures
Bonus Workshops & Expert Sessions
Additional bonus content includes:
- Dynamic AI memory systems
- Text chunking strategies
- Query routing optimization
- Large-scale RAG deployment systems
- Retrieval optimization case studies
- Knowledge graph systems
- Planner and agent evaluation systems
What’s Included
The course includes:
- Live interactive sessions with Jason Liu
- 6 prerecorded lectures
- 6+ office hour Q&A sessions
- 12 hands-on Python notebooks
- Lifetime Slack community access
- Expert speaker library
- $2K+ in cloud and AI credits
Who This Course Is For
This course is ideal for:
- AI engineers building production RAG systems
- Data scientists working with retrieval systems
- Product leaders scaling AI search applications
- Developers implementing LLM applications
- Teams moving from prototypes to production
Prerequisites
To get the most out of this course, students should:
- Understand basic LLM concepts
- Have experience deploying a RAG system
- Be comfortable with technical AI workflows
- Optional: Python experience for hands-on notebooks
About Jason Liu
Jason Liu is a staff machine learning engineer and AI consultant with extensive experience building retrieval and recommendation systems.
He has worked on search and AI systems at companies like Facebook and Stitch Fix and is the creator of the Instructor Python library.
His experience designing large-scale retrieval architectures forms the foundation of this practical, production-focused course.
Why This Course Stands Out
What makes Systematically Improving RAG Applications different is its engineering-first approach:
- Focus on production-ready systems
- Structured optimization methodologies
- Practical implementation over theory
- Covers modern retrieval architectures
- Built around measurable improvements and evaluation
Conclusion
Jason Liu – Systematically Improving RAG Applications provides a complete framework for building scalable, high-performance RAG systems. With structured evaluation methods, multimodal retrieval architectures, and practical optimization workflows, this course helps AI engineers move beyond demos and create reliable production-grade AI applications.
Get Systematically Improving RAG Applications by Jason Liu.
