Domain 4 — Monitoring, Maintenance & Security (24%)

Focus: SageMaker Model Monitor, Clarify, drift detection, retraining triggers, IAM, VPC, encryption, and cost optimisation.

---

Why Models Degrade

graph LR
    Deploy[Model Deployed] --> DriftD[Data Drift<br/>Input distribution changes]
    Deploy --> ConceptD[Concept Drift<br/>Relationship changes]
    Deploy --> QualD[Data Quality<br/>Missing / corrupt inputs]
    Deploy --> BiasD[Bias Drift<br/>Fairness changes over time]

    DriftD --> Retrain[Trigger Retraining]
    ConceptD --> Retrain
    QualD --> Alert[Alert + Fix Pipeline]
    BiasD --> Retrain

    style DriftD fill:#fef9c3,stroke:#ca8a04
    style ConceptD fill:#fed7aa,stroke:#ea580c
    style QualD fill:#fce7f3,stroke:#db2777
    style BiasD fill:#f3e8ff,stroke:#9333ea

---

SageMaker Model Monitor

Continuously monitors endpoints for quality issues.

Monitor Types

How It Works

graph LR
    EP[Endpoint<br/>Live Traffic] --> Cap[Data Capture<br/>Inputs + Outputs → S3]
    Cap --> S3[S3<br/>Captured Data]
    S3 --> MM[Model Monitor<br/>Processing Job]
    MM --> Baseline[Compare vs<br/>Baseline]
    Baseline -->|Violation| CW[CloudWatch<br/>Alarm]
    CW -->|Trigger| EB[EventBridge]
    EB --> RT[Retraining<br/>Pipeline]

Setup Steps

1. Enable Data Capture on endpoint → logs requests/responses to S3
2. Create Baseline — run baseline job on training data → generates statistics + constraints
3. Create Monitoring Schedule — runs monitoring jobs periodically (e.g. hourly)
4. Set CloudWatch Alarms on violations
5. EventBridge Rule → trigger retraining pipeline on alarm

# Enable data capture
data_capture_config = DataCaptureConfig(
    enable_capture=True,
    sampling_percentage=100,
    destination_s3_uri='s3://bucket/captures/'
)

# Create baseline
my_monitor = DefaultModelMonitor(role=role)
my_monitor.suggest_baseline(
    baseline_dataset='s3://bucket/training-data/',
    dataset_format=DatasetFormat.csv()
)

# Schedule monitoring
my_monitor.create_monitoring_schedule(
    monitor_schedule_name='my-monitor',
    endpoint_input='my-endpoint',
    statistics=baseline_statistics,
    constraints=baseline_constraints,
    schedule_cron_expression='cron(0 * ? * * *)'  # hourly
)

---

SageMaker Clarify

Detects bias and explains model predictions.

Pre-training Bias (Data Bias)

Run before training to assess if training data is fair.

Metrics:

Post-training Bias (Model Bias)

Run after training to assess if model treats groups differently.

Metrics:

Explainability — SHAP Values

• SHAP (SHapley Additive exPlanations) — assigns importance scores to each feature for each prediction
• Global SHAP — average importance across all predictions
• Local SHAP — importance for a single prediction
• Outputs: feature importance chart, partial dependence plots

from sagemaker import clarify

clarify_processor = clarify.SageMakerClarifyProcessor(
    role=role,
    instance_count=1,
    instance_type='ml.m5.xlarge'
)

explainability_config = clarify.ExplainabilityConfig(
    shap_config=clarify.SHAPConfig(
        baseline=[{'age': 30, 'income': 50000}],
        num_samples=100,
        agg_method='mean_abs'
    )
)

---

Retraining Strategies

Retraining Architecture

Model Monitor Alarm
        ↓
EventBridge Rule
        ↓
SageMaker Pipeline trigger
        ↓
ProcessingStep (new data prep) → TrainingStep → EvaluationStep
        ↓
ConditionStep (metrics > threshold?)
        ↓
RegisterModel → Approve → Deploy (Blue/Green)

---

Security in SageMaker

IAM for ML

graph TD
    Dev[Data Scientist<br/>IAM User/Role] -->|assume| SR[SageMaker<br/>Execution Role]
    SR --> S3[S3<br/>Read training data]
    SR --> ECR[ECR<br/>Pull/Push containers]
    SR --> CW[CloudWatch<br/>Write logs/metrics]
    SR --> KMS[KMS<br/>Encrypt/decrypt]
    SR --> SM2[SageMaker<br/>API calls]

Execution Role best practices:

• Least privilege — only permissions the training job needs
• Separate roles for: training, inference, pipeline execution
• Use IAM conditions to restrict to specific S3 buckets

Encryption

VPC for SageMaker

Run SageMaker workloads inside a VPC for network isolation:

# Training job in VPC
estimator = PyTorch(
    ...
    subnets=['subnet-xxx'],          # private subnets
    security_group_ids=['sg-xxx'],   # security group
)

• Private subnets — no internet access; instances use VPC endpoints to reach S3, ECR, etc.
• VPC Endpoints (PrivateLink) — access AWS services without traversing internet:
  • com.amazonaws.region.sagemaker.api
  • com.amazonaws.region.sagemaker.runtime
  • com.amazonaws.region.s3
  • com.amazonaws.region.ecr.api
• Security Groups — control inbound/outbound traffic for training instances

SageMaker Role-Based Access Control

Data Privacy

• SageMaker Ground Truth — anonymise data before sending to human labellers
• Differential Privacy — add noise to model training (not natively in SM, use TF Privacy)
• Federated Learning — train across distributed data without centralising it

---

Cost Optimisation

Cost Monitoring

• AWS Cost Explorer — track SageMaker spending by service/tag
• AWS Budgets — set alerts when spending exceeds threshold
• Tag everything — project, team, environment tags for cost allocation
• SageMaker Savings Plans — commit to $/hour usage for discounts

---

CloudWatch for ML

Custom Metrics from Training:

# In training script — SageMaker captures these via CloudWatch
print(f"train:loss={loss:.4f}")
print(f"validation:accuracy={accuracy:.4f}")

Regex in Estimator to capture custom metrics:

metric_definitions = [
    {'Name': 'train:loss', 'Regex': 'train:loss=([0-9\\.]+)'},
    {'Name': 'validation:accuracy', 'Regex': 'validation:accuracy=([0-9\\.]+)'}
]

---

Domain 4 — Exam Scenarios