Transform high-cardinality categorical variables into fixed-size representations using hashing. Useful when vocabulary is incomplete or too large.

import hashlib

def hash_feature(value, num_buckets=1000):
    hash_value = int(hashlib.md5(value.encode()).hexdigest(), 16)
    return hash_value % num_buckets

Learn dense, low-dimensional representations of sparse, high-dimensional data. Essential for text, user IDs, and categorical features.

Combine multiple features to capture non-linear interactions without adding model complexity.

Handle inputs from different modalities (text, images, structured data) by learning separate representations and combining them.

Transform a problem type to leverage different ML approaches (e.g., regression to classification, or vice versa).

Handle cases where instances can belong to multiple classes simultaneously.

Combine multiple models to improve predictions through averaging, voting, or stacking.

Chain models where each addresses a specific subtask or confidence level.

Add an explicit "unknown" or "uncertain" class to handle ambiguous cases.

Intentionally overfit on small datasets or specific cases (e.g., lookup tables, memorization).

Save model state periodically during training for recovery and analysis.

Leverage pre-trained models and adapt them to new tasks with less data.

Parallelize training across multiple devices or machines.

Systematically search the hyperparameter space using grid search, random search, or Bayesian optimization.

Package models as stateless functions for scalable, reliable serving.

Process predictions in bulk for offline workloads.

Monitor model performance continuously in production.

Separate candidate generation from ranking for efficiency at scale.

Track predictions through the system with unique identifiers.

Encapsulate feature engineering logic for consistent application in training and serving.

Ensure deterministic train/validation/test splits.

Handle schema evolution between training data and serving inputs.

Process streaming data in windows for consistent feature computation.

Orchestrate ML workflows as directed acyclic graphs (DAGs).

Centralize feature computation and serving for consistency.

Track model versions with their associated code, data, and hyperparameters.

Establish baseline performance using simple rules or heuristics.

Provide interpretable outputs alongside predictions.

Evaluate model performance across different demographic groups.

• Document ingestion and chunking pipeline
• Embedding model for semantic encoding
• Vector database for similarity search
• Retrieval strategy (dense, sparse, or hybrid)
• LLM for response generation with retrieved context

• HyDE (Hypothetical Document Embeddings): Generate hypothetical answers to improve retrieval
• RAPTOR: Recursive abstractive processing for hierarchical retrieval
• Corrective RAG: Verify and refine retrieved documents
• Self-RAG: Model decides when retrieval is needed
• Graph RAG: Combine knowledge graphs with vector retrieval

Provide examples in the prompt to guide model behavior without fine-tuning.

Classify the sentiment of movie reviews.

Review: "The acting was superb and the plot kept me engaged."
Sentiment: Positive

Review: "Waste of two hours. Predictable and boring."
Sentiment: Negative

Review: "{{user_review}}"
Sentiment:

Encourage step-by-step reasoning for complex tasks.

Q: If a train travels 120 miles in 2 hours, what is its speed?

Let me think step by step:
1. Speed = Distance / Time
2. Distance = 120 miles
3. Time = 2 hours
4. Speed = 120 / 2 = 60 miles per hour

A: 60 miles per hour

Interleave reasoning traces with actions (tool use, retrieval).

Sample multiple reasoning paths and select the most consistent answer.

Explore multiple reasoning branches for complex problem-solving.

Constrain LLM output to specific formats (JSON, XML) for reliable parsing.

from pydantic import BaseModel
from openai import OpenAI

class MovieReview(BaseModel):
    sentiment: str
    confidence: float
    key_phrases: list[str]

client = OpenAI()
response = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[{"role": "user", "content": review_text}],
    response_format=MovieReview
)

Adapt large models with minimal trainable parameters.

• LoRA (Low-Rank Adaptation): Add low-rank decomposition matrices to attention layers
• QLoRA: Combine LoRA with quantization for memory efficiency
• Prefix Tuning: Prepend learnable tokens to input
• Adapter Layers: Insert small trainable modules between frozen layers

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b")

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none"
)

peft_model = get_peft_model(model, lora_config)
# Only ~0.1% of parameters are trainable

Fine-tune on instruction-following datasets to improve task generalization.

Align model outputs with human preferences using reward models.

Simpler alternative to RLHF that directly optimizes on preference data.

Use a smaller draft model to propose tokens, verified by the larger model in parallel.

Dynamically batch requests to maximize GPU utilization.

Efficiently manage key-value caches for transformer attention.

Reduce model precision (INT8, INT4) for faster inference with minimal quality loss.

• Online and offline serving
• Point-in-time correctness for training
• Feature versioning and lineage
• Feature discovery and reuse
• Real-time feature computation

• Feast (open source)
• Tecton
• Databricks Feature Store
• Amazon SageMaker Feature Store
• Vertex AI Feature Store

from feast import FeatureStore

store = FeatureStore(repo_path=".")

# Get training data with point-in-time join
training_df = store.get_historical_features(
    entity_df=entity_df,
    features=[
        "user_features:age",
        "user_features:purchase_count_30d",
        "item_features:price",
    ]
).to_df()

# Get online features for serving
online_features = store.get_online_features(
    features=["user_features:purchase_count_30d"],
    entity_rows=[{"user_id": 123}]
).to_dict()

• Model versioning and lineage
• Stage transitions (staging, production, archived)
• Model metadata and documentation
• Approval workflows
• Integration with CI/CD

• MLflow Model Registry
• Weights & Biases
• Neptune
• Amazon SageMaker Model Registry
• Vertex AI Model Registry

• Shadow Mode: New model receives traffic but responses are not used
• Canary Deployment: Gradually increase traffic to new model
• Multi-Armed Bandit: Dynamically allocate traffic based on performance

Monitor input feature distributions for changes from training data.

Track changes in model output distributions.

Monitor business metrics and model accuracy over time.

from evidently import Report
from evidently.metric_preset import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=train_df, current_data=prod_df)
report.save_html("drift_report.html")

Use embeddings to find similar past queries and return cached responses.

Convert text to dense vectors for semantic similarity search.

from openai import OpenAI

client = OpenAI()

def get_embedding(text: str, model="text-embedding-3-small") -> list[float]:
    response = client.embeddings.create(input=text, model=model)
    return response.data[0].embedding

Embed images and text into a shared vector space (CLIP, ImageBind).

• Fixed-size chunks: Simple but may break semantic units
• Semantic chunking: Split on natural boundaries (paragraphs, sections)
• Recursive chunking: Hierarchical splitting with overlap
• Late chunking: Context-aware chunking using LLMs

• HNSW (Hierarchical Navigable Small World): Fast approximate search
• IVF (Inverted File Index): Cluster-based search
• PQ (Product Quantization): Compressed vectors for memory efficiency

Combine dense vector search with sparse keyword search (BM25).

from qdrant_client import QdrantClient
from qdrant_client.models import SearchRequest, NamedVector

client = QdrantClient("localhost", port=6333)

# Hybrid search combining dense and sparse vectors
results = client.search(
    collection_name="documents",
    query_vector=NamedVector(
        name="dense",
        vector=dense_embedding
    ),
    query_filter=None,
    limit=10
)

LLMs invoke external tools and APIs to accomplish tasks.

Multiple specialized agents collaborate on complex tasks.

Agents create plans, execute steps, and reflect on results.

Strategies for working with 100K+ token context windows.

Implement short-term and long-term memory for conversations.

Use LLMs to evaluate outputs of other LLMs.

Systematically test models for vulnerabilities and failures.