MLOps with MLflow

A comprehensive MLOps platform for training, deploying, and monitoring machine learning models using MLflow.

Overview

This project provides production-ready ML pipelines for:

  • mlflow-sklearn: Scikit-learn based fraud detection pipeline
  • mlflow-tf: TensorFlow based image classification pipeline
User Guide
Learn how to run ML pipelines and use trained models
Developer Guide
Contribute to the project and extend pipelines
Admin Guide
Deploy and manage infrastructure

Features

  • 🚀 Multiple ML Frameworks: Support for TensorFlow and Scikit-learn
  • 📊 Experiment Tracking: MLflow tracking for all experiments
  • 🔄 Reproducible Pipelines: DVC and Prefect orchestration
  • ☁️ Multi-Cloud Deployment: AWS, Azure, GCP, and local deployments
  • 🐳 Containerized: Docker support for all components

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     MLOps Platform                          │
├─────────────────────────────────────────────────────────────┤
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │  mlflow-tf   │  │mlflow-sklearn│  │ mlflow-spark │      │
│  │  (TensorFlow)│  │(Scikit-learn)│  │   (PySpark)  │      │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘      │
│         │                 │                 │               │
│         └─────────────────┼─────────────────┘               │
│                           │                                  │
│                    ┌──────▼───────┐                         │
│                    │ MLflow Server│                         │
│                    └──────┬───────┘                         │
│                           │                                  │
│         ┌─────────────────┼─────────────────┐               │
│         │                 │                 │               │
│  ┌──────▼──────┐  ┌───────▼──────┐  ┌──────▼──────┐       │
│  │    Local    │  │     AWS      │  │   Azure     │       │
│  │  (asus/tp)  │  │  (us-east-1) │  │   (cloud)   │       │
│  └─────────────┘  └──────────────┘  └─────────────┘       │
└─────────────────────────────────────────────────────────────┘

Getting Started

Prerequisites

  • Python 3.8+
  • Docker (optional)
  • AWS CLI (for cloud deployments)

Quick Start

# Clone the repository
git clone https://bitbucket.org/wilsonify/mlops-with-mlflow.git
cd mlops-with-mlflow

# Install dependencies
pip install -r requirements.txt

# Run sklearn pipeline locally
cd src/mlflow-sklearn
make all

# Or run TensorFlow pipeline
cd src/mlflow-tf
python run_pipeline.py all

Project Structure

mlops-with-mlflow/
├── src/                    # Source code
│   ├── mlflow-sklearn/     # Scikit-learn pipeline
│   ├── mlflow-tf/          # TensorFlow pipeline
│   ├── mlflow-spark/       # Spark pipeline
│   └── *-library/          # Shared libraries
├── deploy/                 # Deployment configurations
│   ├── local/              # Local deployments
│   └── aws/                # AWS deployments
└── docs/                   # Documentation (this site)

Support