Michi

pip install komichi

Overview

Michi (道, "the path") is a toolbox of independent command-line tools that automate the repetitive implementation work of machine learning — profiling datasets, cleaning them, evaluating models, benchmarking, running experiment grids, reporting — while leaving every judgement call to the person running it.

It works on projects you already have: your CSV, your model.pkl, your repo. There is no project template to adopt, no framework to import, no account to create, and nothing ever leaves your machine. On PyPI the name was taken, so it ships as komichi (小道, "a small path"); the command you type is still michi.

Where Meridian, Polaris, and Zenith are from-scratch implementations of ML systems, Michi is the other half of the craft: the tooling that keeps ML work honest.

Problem

Two failure modes dominate ML tooling. Frameworks want to own your project — adopt the template, import the runtime, restructure your repo. AutoML goes further and makes the judgement calls for you, returning a leaderboard with no indication of whether the gaps on it are real.

Meanwhile the everyday work stays manual and error-prone: spotting the column that leaks the label, checking whether a model beats a dummy baseline, knowing whether model A actually beat model B or just got a friendlier fold split.

The toolbox

Every verb stands alone; use one and ignore the rest.

• inspect — profile a dataset: types, missing values, duplicates, skew, imbalance, correlations, outliers, and leakage suspects, each finding explained. --json for CI diffing, --fail-on high to make it a data-quality gate.
• eval — rigorously evaluate an existing model: metrics with intervals, calibration, baselines, subgroup gaps.
• bench — train and compare models with honest cross-validation, confidence intervals, and significance tests.
• tune, fit, predict, ensemble — hyperparameter search scored on folds the search never saw; stacking and voting that report whether they actually beat the best single model.
• split, diff, threshold, errors — entity- and time-aware splits, data drift detection, decision-cutoff analysis, and the rows behind the score with confident mistakes first.
• clean, apply, export — interactive cleaning and feature engineering that authors a reproducible recipe and exports readable pipeline code.
• sweep, report, ui, plugins — reproducible experiment grids with resume, HTML/Markdown/LaTeX reports, a local read-only viewer, and a plugin surface for your own models.

Design decisions

• A toolbox, not a workflow. Independent verbs over an orchestrated pipeline, so Michi never asks you to restructure a project to use one command.
• Menus, not recommendations. Michi lists options; you choose. Defaults exist for mechanics — folds, seeds — never for judgement.
• Artifacts, not sessions. Every decision becomes a durable, versionable file you own. The interactive console is a skin over the same commands and exports back to plain one-shot invocations.
• Rigor by default. Baselines, confidence intervals, significance tests, and leakage checks are opt-out. Model comparisons use the corrected resampled t-test (Nadeau & Bengio, 2003), because CV folds share training data and a naive test calls noise significant. A dummy baseline is always included, so "is this any good?" always has an answer.
• Entirely local. No server, no account, no telemetry, no network call.

The output reflects this: where most tools would declare a winner, Michi reports that the top three models are statistically indistinguishable at this sample size and that choosing between them on these numbers alone is not supported.

Lessons learned

• Freezing a defect makes it a feature. Before the 1.0 surface freeze, an audit found that bench, tune, and ensemble cross-validated without honouring the --group flag — Michi contradicting its own advice on exactly the pattern that advice exists for. On a dataset whose label is a property of the customer, that overstated balanced accuracy by 26 points (0.836 vs an honest 0.580). Fixed before the freeze, not after.
• Some bugs are invisible to tests. Two flags were declared and never wired, so they silently did nothing — undetectable by a normal suite, because there is nothing to assert against a feature that was never connected. The fix was an audit that walks every command's signature and asserts each parameter is referenced in its own body; that audit is now itself a test.
• Guarantees nobody knows about buy nothing. The Python API had been frozen since 1.0 and never documented as something to use.

Engineering

Python 3.11–3.13, mypy --strict, Ruff lint and format, and pytest — all four gates run in CI on Linux, macOS, and Windows. From 1.0 the artifact schemas, CLI surface, and plugin contract are frozen under semantic versioning (ADR-0002), and a regression test reads committed 1.0 artifacts on every CI run, so an artifact Michi writes today stays readable. That is enforced, not promised.

Roadmap

Growth happens at the edges — in plugins — rather than in the core. The frozen public surface is the point: stability is the feature.