https://ethicalos.org/ https://www.wired.com/story/ethical-os/

Whether humans are directly using machine learning classifiers as tools, or are deploying models within other products, a vital concern remains: if the users do not trust a model or a prediction, they will not use it. It is important to differentiate between two different (but related) definitions of trust: (1) trusting a prediction, i.e. whether a user trusts an individual prediction sufficiently to take some action based on it, and (2) trusting a model, i.e. whether the user trusts a model to behave in reasonable ways if deployed.

• https://arxiv.org/pdf/1602.04938.pdf

There are many steps involved from prepping data, choosing algorithms, building, training, and deploying models … and iterating over and over again.

• Enterprise AI Guide, AWS

However, the harsh reality is that without a reasonable understanding of how machine learning models or the data science pipeline works, real-world projects rarely succeed.

• https://towardsdatascience.com/human-interpretable-machine-learning-part-1-the-need-and-importance-of-model-interpretation-2ed758f5f476

In this paper, we argue that ML systems have a special capacity for incurring technical debt, because they have all of the maintenance problems of traditional code plus an additional set of ML-specific issues. This debt may be difficult to detect because it exists at the system level rather than the code level. Traditional abstractions and boundaries may be subtly corrupted or invalidated by the fact that data influences ML system behavior. Typical methods for paying down code level technical debt are not sufficient to address ML-specific technical debt at the system level.

• https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

Quality depends not just on code, but also on data, tuning, regular updates, and retraining.

• https://www.oreilly.com/ideas/specialized-tools-for-machine-learning-development-and-model-governance-are-becoming-essential

Second, the idea that you can audit and understand decision-making in existing systems or organisations is true in theory but flawed in practice. It is not at all easy to audit how a decision is taken in a large organisation.

• https://www.ben-evans.com/benedictevans/2019/4/15/notes-on-ai-bias

• Model Analysis
• ML Pipelines
• Interfaces: Preconditions and Postconditions
• Workflow Management
• Baselines and QA
• Deployments and Versioning
• Causality

• Seaborn
• LIME https://homes.cs.washington.edu/~marcotcr/aaai18.pdf
• Yellowbrick http://joss.theoj.org/papers/10.21105/joss.01075

• https://christophm.github.io/interpretable-ml-book/interpretability.html
• https://www.brighttalk.com/webcast/16463/351228
• https://towardsdatascience.com/human-interpretable-machine-learning-part-1-the-need-and-importance-of-model-interpretation-2ed758f5f476
• https://arxiv.org/pdf/1811.01439.pdf

• http://interpretable.ml/

• 2019 machine learning pipelines — the operational pipeline this oversight pass attaches to.
• Machine learning design patterns — has the surrounding pattern catalog (drift detection, retraining triggers).
• Fine-tuning ML models — interpretability shifts considerably when "the model" is an LLM rather than a tabular classifier.
• AWS DeepLens — companion 2019 note on the edge-deployment side of the same workflow.

The interpretability stack consolidated faster than the original note expected. SHAP became the default tabular-model explainer (Lundberg's TreeExplainer / DeepExplainer / KernelExplainer covering most cases) and PyTorch shipped Captum in 2019, which absorbed integrated gradients, LayerWise Relevance, and most attribution-method research into a single library. Fairness consolidated into two toolkits: Fairlearn (originated at Microsoft Research, now community-maintained) and the University of Chicago's Aequitas; both ship group fairness metrics, mitigation algorithms, and audit reports out of the box. The harder regulatory frame arrived: the EU AI Act entered force on 1 August 2024 and introduced impact assessments and conformity requirements for high-risk systems (the categorisation that covers most fintech credit and insurance use cases). The frontier moved to LLMs: attribution methods that work for tabular and CNN models do not trivially extend to autoregressive text generation, and Anthropic's 2024 sparse-autoencoder and dictionary-learning research is the first practical mechanistic-interpretability toolkit to scale to a frontier model — see Scaling Monosemanticity (May 2024). The 2019 note's framing of "trust the prediction" vs "trust the model" still holds; the 2026 question is "trust which feature in the model".