AFML-style framework: tick ingest, multi-scale bars, labels, CPCV, Dagster/Hydra orchestration, and process-first methodology on FX data.
This project is my private quantitative FX research lab, inspired by practices from Advances in Financial Machine Learning (López de Prado). It covers the full chain: multi-source tick ingestion, canonical schema, bar construction (time, tick, tick-imbalance), causal features, supervised labels, leak-resistant CPCV validation, dual-side primary/meta training, HTML reports, and Streamlit exploration.
The active repository remains private (research configs, Dagster presets, notebooks). A public MIT snapshot of the framework — without alpha or market data — is on GitHub: khonen-git/FinancialMLResearchLab.
After several ad hoc exploration cycles (isolated notebooks, one-off scripts), I consolidated EURUSD work into a single structured repo. The goal was not to “find a signal” quickly, but to build a research infrastructure that supports fast iteration without sacrificing statistical rigor.
Blog posts tagged eurusd-lab document concrete hypotheses tested in this lab (multi-scale pullback, compression/expansion, Tr8dr labels, HMM slope, stochastic event sampling). This page describes the technical ecosystem; methodological details and gate verdicts live in the articles, not here.
The config/ tree composes runtime, dataset, bars, features, labels, models, CV, experiments, and execution policies. Each experiment profile (e.g. dual-side primary/meta) wires model roles to feature and label groups through ConfigService — a single entry point, without scattered Hydra access in pipeline code.
Dagster presets (presets/experiments/) contain Hydra overrides only: a reproducible run launches from the UI or CLI with an explicit YAML file.
Feature, label, bar, and event builders follow a registry + factory + engine pattern:
@register_feature, label builders, EBS presets)This separation lets you add a new builder via YAML + a registered class without touching the Dagster graph.
Ingested ticks (Dukascopy, MT5) normalize to a canonical schema (datetime64[ns] timestamps, mid, spread, UTC session). Composite bars (multi_tick_ti) materialize several samplings from one tick read; auxiliary features align causally onto the primary grid via backward merge_asof.
Swappable backends: pandas, Polars, cuDF/RAPIDS, Numba kernels — selectable per experiment (preferred_dataframe_backend, preferred_compute_backend).
data/processed/; separate MLflow/Dagster artifact stores (see internal DB architecture docs)Supported bar types:
Event-Based Sampling filters bars where a structural rule is confirmed (HA pivots, PBH/PBL pullbacks, HH-X/LL-X breakouts). An event is not a trade signal: it is a causal sampling filter (confirm_index only) that concentrates information for the features + label + model stage.
Dual-side primary / meta pipeline (long and short):
Labels are built on the primary bar grid; CPCV purge horizon follows from that choice.
Default validation profile: cpcv_10_2 (10 folds, 2 test folds). Dagster partitions (path_i) map explicitly to on-disk CPCV paths; HTML report jobs stay separate from partitioned ML jobs to avoid inconsistent materialization.
| Layer | Intent |
|---|---|
| Config contracts | Required params, invalid YAML fixtures |
| Formulas | Independent oracle (pandas/numpy) on synthetic series |
| Causality | No lookahead per builder |
| Parity | pandas / Polars (pilot) |
| Integration | Hydra pipeline, golden parquet, static Dagster assets |
| Leakage | CV purge, dataset alignment |
The public snapshot runs ~950 unit tests (pytest -m unit); the private repo extends the suite (GPU integration, regression, perf) for hundreds of additional CI tests.
Main jobs: prep (ticks → bars → features → labels), CPCV ML primary/meta, HTML reports, final holdout validation. The Dagster instance persists runs, partitions, and lineage; PostgreSQL stores Dagster and MLflow metadata.
Experiment tracking, model metrics (accuracy, F1, AUC, Brier, calibration), dataset artifacts. Clear separation between Dagster and MLflow databases is documented internally.
notebooks_example/) in the public snapshot: ConfigService, triple barrier, feature labResearch follows gates (0→4): statistical existence, measurable detection, OOS predictability, net tradability after costs. Each eurusd-lab blog post reports a verdict per gate — without exposing parameters or alpha results here.
Core principle: do not optimize upstream on PnL; first confirm the phenomenon exists and predicts out-of-sample, before any execution or sizing layer.
Related articles (eurusd-lab series):
| Element | Public MIT snapshot | Private repo |
|---|---|---|
financial_ml framework | Yes | Yes (active) |
config_example/ + symlink | Yes | Prod config/ (~149 YAML) |
| Research Dagster presets | No | Yes |
| Real tick data | No | Yes |
| Alpha / strategies | No | Yes (not published) |
| Support | v1.0.0 snapshot, unmaintained | Active research |
Controlled publication via an internal script; only the framework and tutorial examples ship — never production configs or research notebooks.
Python 3.12+ · micromamba (financial-ml / financial-ml-cpu)
Hydra · Dagster · PostgreSQL · Docker Compose
pandas · Polars · cuDF/RAPIDS · Numba · PyArrow/Parquet
scikit-learn · cuML · MLflow · custom CPCV · auxiliary GARCH
Streamlit · Plotly · HTML templates (built-in viewer)
pytest (unit / integration / perf) · strict markers · golden fixtures · CI
This lab is not a trading product: it is a research infrastructure where methodological rigor comes before chasing a “winning” backtest. The public framework lets you explore AFML-style architecture; research IP stays private.
Python Documentation
Official Python language documentation
NumPy
Numerical computing and multidimensional arrays in Python
pandas
Tabular data manipulation and analysis
scikit-learn
Machine learning library for Python
Tr8dr
Algorithms, models, and markets — HFT, crypto, ML applied to finance