Architecture
This document describes the architecture of the MLOps platform.
System Overview
┌─────────────────────────────────────────────────────────────────────┐
│ MLOps Platform │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌───────────┐ │
│ │ src/ │ │ deploy/ │ │ docs/ │ │ tests/ │ │
│ │ Pipelines │ │ Deployments │ │Documentation│ │ Tests │ │
│ └──────┬──────┘ └──────┬──────┘ └─────────────┘ └───────────┘ │
│ │ │ │
│ │ │ │
│ ┌──────▼──────────────────▼──────┐ │
│ │ Shared Libraries │ │
│ │ ┌──────────┐ ┌──────────────┐ │ │
│ │ │doe-lib │ │feature-lib │ │ │
│ │ └──────────┘ └──────────────┘ │ │
│ │ ┌──────────┐ ┌──────────────┐ │ │
│ │ │io-lib │ │metrics-lib │ │ │
│ │ └──────────┘ └──────────────┘ │ │
│ │ ┌──────────┐ │ │
│ │ │plot-lib │ │ │
│ │ └──────────┘ │ │
│ └────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Directory Structure
mlops-with-mlflow/
├── src/ # Source code
│ ├── mlflow-sklearn/ # Scikit-learn pipeline
│ │ ├── mlflow_sklearn/ # Python package
│ │ ├── tests/ # Unit tests
│ │ ├── Dockerfile # Container definition
│ │ └── makefile # Build automation
│ ├── mlflow-tf/ # TensorFlow pipeline
│ │ ├── mlflow_tf/ # Python package
│ │ │ ├── pipeline/ # Pipeline modules
│ │ │ ├── utils/ # Utilities
│ │ │ └── cli/ # CLI interface
│ │ ├── tests/ # Unit tests
│ │ └── configs/ # Configuration files
│ ├── doe-library/ # Design of Experiments
│ ├── feature-library/ # Feature engineering
│ ├── io-library/ # I/O utilities
│ ├── metrics-library/ # Metrics computation
│ └── plot-library/ # Visualization
├── deploy/ # Deployments
│ ├── local/ # Local deployments
│ │ ├── asus/ # Asus machine
│ │ └── thinkpad/ # Thinkpad machine
│ └── aws/ # AWS deployments
│ └── 064592191516/ # AWS account
│ └── us-east-1/ # Region
└── docs/ # Documentation (Hugo)
Pipeline Architecture
Each pipeline follows a standard structure:
mlflow-{framework}/
├── mlflow_{framework}/ # Main package
│ ├── __init__.py
│ ├── pipeline/ # Pipeline components
│ │ ├── base.py # Base pipeline class
│ │ ├── preprocessing.py # Data preprocessing
│ │ ├── training.py # Model training
│ │ ├── evaluation.py # Model evaluation
│ │ └── orchestrator.py # Pipeline orchestration
│ └── utils/ # Utilities
├── tests/ # Tests
├── Dockerfile # Container
├── setup.py # Package setup
└── requirements.txt # Dependencies
Data Flow
┌──────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────┐
│ Raw │───▶│ Preprocessing │───▶│ Training │───▶│ Model │
│ Data │ │ │ │ │ │ Artifact │
└──────────┘ └──────────────┘ └──────────┘ └──────────┘
│ │
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ S3 │ │ MLflow │
│ Bucket │ │ Registry │
└──────────┘ └──────────┘
Design Patterns
Pipeline Pattern
All pipelines inherit from a base class:
from abc import ABC, abstractmethod
class Pipeline(ABC):
def __init__(self, config):
self.config = config
@abstractmethod
def run(self):
"""Execute the pipeline."""
pass
def __enter__(self):
"""Context manager entry."""
mlflow.start_run()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
"""Context manager exit."""
mlflow.end_run()
Configuration Pattern
Configuration uses dataclasses for type safety:
from dataclasses import dataclass
@dataclass
class TrainingConfig:
epochs: int = 10
batch_size: int = 32
learning_rate: float = 0.001
Factory Pattern
Model creation uses factory methods:
def create_model(config: ModelConfig):
if config.framework == "sklearn":
return SklearnModel(config)
elif config.framework == "tensorflow":
return TensorFlowModel(config)
Integration Points
MLflow Integration
- Tracking: All experiments logged to MLflow
- Artifacts: Models stored in artifact store
- Registry: Production models registered
AWS Integration
- S3: Data and artifact storage
- ECR: Container registry
- EC2: Compute instances
CI/CD Integration
- Bitbucket Pipelines: Automated testing
- Docker: Containerized builds
- Terraform: Infrastructure as code