End-to-end data science project — 500,000 synthetic SA banking transactions, PySpark distributed feature engineering, 4-model hyperparameter-tuned comparison, SHAP explainability, business impact analysis in ZAR, and MLflow model registration. Extends the
nedbank-streaming-pipelineproject with offline ML model training.
| Property | Value |
|---|---|
| Dataset | 500,000 SA banking transactions (89MB) |
| Fraud rate | 21.6% (enriched for model training signal) |
| Features | 31 (including 11 PySpark-engineered) |
| Winner | XGBoost |
| Test PR-AUC | 0.7090 (vs random baseline 0.216 — 3.28× better) |
| Test ROC-AUC | 0.8522 |
| Test F1 | 0.6081 (at default threshold 0.50) |
| Optimal threshold | 0.05 (not 0.50 — cost-driven) |
| Recall at optimal | 99.7% — catches nearly all fraud |
| Net financial benefit | R123,121,850 per 75,000 transactions |
| MLflow model | sa-fraud-xgboost-v1 / Version 1 |
500,000 SA Transaction Records (89MB CSV)
│
▼ PySpark 3.5 — 16 partitions
┌──────────────────────────────────────────────────────┐
│ Distributed Feature Engineering │
│ log transforms · cyclical time encoding │
│ composite_risk_score · velocity_amount_interaction │
│ amount_limit_band · age_band · null imputation │
│ StringIndexer → OneHotEncoder (SparkML Pipeline) │
└──────────────────────────────────────────────────────┘
│ 500k × 32 pandas feature matrix
▼
┌──────────────────────────────────────────────────────┐
│ Class Imbalance Handling │
│ SMOTE 0.3 (LR + RF) · scale_pos_weight=3.63 (XGB) │
│ class_weight balanced (MLP) │
│ 70 / 15 / 15 stratified split │
└──────────────────────────────────────────────────────┘
│
▼ RandomizedSearchCV · StratifiedKFold · PR-AUC
┌──────────────────────────────────────────────────────┐
│ 4-Model Shootout │
│ Logistic Regression │ Random Forest │
│ XGBoost │ Neural Network (MLP) │
└──────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────┐
│ Evaluation & Explainability │
│ ROC-AUC · PR-AUC · F1 · Confusion matrices │
│ SHAP global (bar + dependence) · local waterfall │
│ Threshold optimisation · ZAR cost-benefit analysis │
│ MLflow experiment tracking · Model Registry │
└──────────────────────────────────────────────────────┘
# 1. Install dependencies
pip install -r requirements.txt
# 2. Generate 500k transaction dataset
python src/data/generate_dataset.py
# 3. Launch Jupyter and open the notebook
jupyter lab notebooks/fraud_detection_shootout.ipynb
# 4. Run tests
pytest tests/ -v| # | Model | ROC-AUC | PR-AUC ⭐ | F1 | Precision | Recall | Train time |
|---|---|---|---|---|---|---|---|
| 🥇 | XGBoost | 0.8522 | 0.7090 | 0.6081 | 0.5311 | 0.7112 | 15 min 35s |
| 🥈 | Neural Network (MLP) | 0.8447 | 0.6894 | 0.5963 | 0.5169 | 0.7046 | ~5 min |
| 🥉 | Random Forest | 0.8366 | 0.6681 | 0.5802 | 0.4970 | 0.6967 | 2 min 25s |
| 4️⃣ | Logistic Regression | 0.7536 | 0.5059 | 0.4883 | 0.3721 | 0.7102 | 16 min |
Why XGBoost wins: Gradient boosting's sequential error-correction mechanism outperforms bagging (RF) and backpropagation (MLP) on structured tabular fraud data — consistent with Grinsztajn et al. (NeurIPS 2022). The MLP was included deliberately to test and confirm this hypothesis.
# 1. Install dependencies
pip install -r requirements.txt
# 2. Generate 500k transaction dataset (~90 seconds)
python src/data/generate_dataset.py
# 3. Launch Jupyter
jupyter lab notebooks/fraud_detection_shootout.ipynb
# 4. Run all cells top to bottom
# Estimated total runtime: ~45 minutes on a modern laptop
# LR: ~16 min | RF: ~2.5 min | XGBoost: ~15.5 min | MLP: ~5 min
# 5. View MLflow experiment (after Section 9)
mlflow ui # → open http://localhost:5000
# 6. Run tests
pytest tests/ -v
⚠️ Windows users: PySpark requires Java 17 set asJAVA_HOME. Download from adoptium.net (Windows x64 JDK .msi). SetPYSPARK_PYTHONto your Python 3.11 executable path.
| Section | Description | Key output |
|---|---|---|
| 0 | Environment setup | All imports verified |
| 1 | PySpark session & data loading | 500k × 31 schema-validated |
| 2 | Exploratory Data Analysis | 6 visualisations — fraud patterns |
| 3 | Feature engineering (PySpark) | 11 new features on 500k rows |
| 4 | Imbalance handling | SMOTE + class weights |
| 5 | 4-model training & tuning | RandomizedSearchCV per model |
| 6 | Model comparison | ROC/PR curves + confusion matrices |
| 7 | SHAP explainability | Global bar + dependence + local |
| 8 | Business impact | R123M net benefit at threshold 0.05 |
| 9 | MLflow logging | sa-fraud-xgboost-v1 registered |
| 10 | Final summary | Complete shootout scorecard |
Records: 500,000 transactions | Size: ~90MB Fraud rate: 21.6% (enriched — real-world SA banking is 0.1–5%) Features: 31 raw + 11 engineered = 42 columns total Customers: 10,000 synthetic SA banking customers Date range: January 2022 – December 2024
Fraud patterns injected (~21.6% of records):
| Pattern | Description |
|---|---|
| High-value foreign currency | Large USD/EUR/GBP transactions on ZAR accounts |
| Velocity attacks | Same account — multiple transactions in short window |
| High-risk merchants | Gambling, Crypto Exchange, Forex, Pawn Shop, Money Transfer |
| Off-hours ATM | Large withdrawals between 01:00–04:00 |
| New device + large amount | Unrecognised device fingerprint + high spend |
| Geographic velocity | Province switch in under 2 hours |
Intentional dirty data (~5%): negative balances, inconsistent casing, missing values — realistic data quality challenges for the pipeline to handle.
| Feature | Type | Description |
|---|---|---|
log_amount |
Engineered | Log1p transform — compresses right-skewed distribution |
composite_risk_score |
Engineered | Domain-weighted sum of 5 binary fraud flags (SHAP rank: 2nd) |
velocity_amount_interaction |
Engineered | velocity_last_hour × log_amount (SHAP rank: 9th) |
amount_limit_band |
Engineered | Ordinal credit utilisation bucket 0–4 |
hour_sin / hour_cos |
Engineered | Cyclical time encoding — 23:00 close to 00:00 |
age_band |
Engineered | Customer life-stage segmentation |
| Rank | Feature | Mean SHAP | Insight |
|---|---|---|---|
| 1 | credit_score_band_idx |
0.83 | Customer history dominates — who > what |
| 2 | composite_risk_score |
0.53 | Engineered feature validates domain knowledge |
| 3 | merchant_category_idx |
0.30 | Forex/Gambling categories are near-decisive |
| 4 | amount_to_limit_ratio |
0.22 | Over-limit transactions strongly flagged |
| 5 | geo_velocity_anomaly |
0.21 | Province switch in 2hrs — hard evidence |
| 6 | log_amount |
0.21 | Log transform essential — raw amount underperforms |
SHAP revealed
credit_score_band_idxas the dominant feature — raw correlation analysis ranked it outside the top 5. This demonstrates why SHAP is essential for understanding non-linear model behaviour beyond simple correlation.
Cost assumptions (SA banking context):
| Cost type | Amount |
|---|---|
| Average fraud loss | R8,500 per missed transaction |
| False alarm cost | R150 per declined legitimate transaction |
| Analyst review cost | R75 per alert |
| Cost asymmetry ratio | 56.7:1 (fraud loss vs false alarm) |
Threshold optimisation result:
| Threshold | Precision | Recall | Net benefit |
|---|---|---|---|
| 0.50 (default) | 0.531 | 0.711 | ~R75M |
| 0.05 (optimal) | 0.224 | 0.997 | R123,121,850 |
Tiered production response:
| Fraud probability | Action |
|---|---|
| < 0.05 | ✅ Auto-approve |
| 0.05 – 0.30 | 📱 SMS customer confirmation |
| 0.30 – 0.65 | 👤 Analyst queue |
| > 0.65 | 🚫 Auto-block |
| Component | Technology |
|---|---|
| Distributed processing | PySpark 3.5 (16 partitions, 500k rows) |
| ML — tree models | scikit-learn 1.4, XGBoost 2.0 |
| ML — deep learning | TensorFlow 2.x / Keras (MLP — BatchNorm + Dropout + EarlyStopping) |
| Class imbalance | imbalanced-learn (SMOTE) |
| Explainability | SHAP — XGBoost native pred_contribs |
| Experiment tracking | MLflow 2.10 (local + Databricks-ready) |
| Visualisation | matplotlib, seaborn |
| Testing | pytest |
| CI/CD | GitHub Actions |
fraud-detection-ml/
│
├── 📓 notebooks/
│ └── fraud_detection_shootout.ipynb # Main notebook — 10 sections
│
├── 📁 src/
│ └── data/
│ └── generate_dataset.py # 500k synthetic SA transaction generator
│
├── 📁 data/
│ ├── raw/
│ │ └── sa_fraud_transactions.csv # 500k records (generated — not committed)
│ └── models/
│ └── best_fraud_model.pkl # XGBoost + threshold + features (generated)
│
├── 📁 reports/
│ └── figures/ # 11 publication-quality charts (generated)
│
├── 📁 tests/
│ └── test_dataset.py # Dataset validation suite
│
├── 📁 .github/
│ └── workflows/
│ └── ci.yml # GitHub Actions — test + dataset verify
│
├── .gitignore
├── requirements.txt
└── README.md
MIT — feel free to adapt for your own portfolio or organisation.