-   Name: SmallCon
-   Date: December 11, 2024
-   Focus: Applied AI and LLMs in Production
-   Format: In-person


# 👥 Session Details

-   Time: 15:19
-   Type: Panel
-   Speaker(s):
    -   Travis Addair, CTO and Co-founder, Predibase
    -   Daniel Han, Head of AI Engineering, Harvey
    -   Moasati, CTO, Converza
    -   Atten Samuel, CTO and Co-founder, Galileo
    -   Abhishek, Senior Staff Engineer, New Bank
-   Session Goal: Discuss the shift from LLM experimentation to production-grade systems and real-world applications.


# 💡 Key Technical Insights

Production Challenges:

-   Scaling costs with large prompts
-   Limitations of fine-tuning large models
-   Need for modular complex workflows
-   Gap between controlled and real-world environments

Evaluation and Quality:

-   Traditional NLP metrics insufficient for long-form text
-   Limitations of automated LLM Judge techniques at scale
-   Critical role of human-in-the-loop evaluation
-   Importance of continuous feedback loops

Implementation Strategy:

-   Gradual release process emphasis
-   Focus on smaller, task-specific models
-   Cost and throughput optimization
-   Modularization of complex workflows

Risk Mitigation:

-   Building user confidence in high-stakes industries
-   Addressing unpredictable hallucinations
-   Bias mitigation through human evaluation
-   Quality assurance through feedback loops

The panel highlighted the growing sophistication in LLM deployment practices, with particular emphasis on the role of human feedback in ensuring quality and reliability in production systems. The discussion underscored a clear trend toward smaller, specialized models with robust evaluation frameworks.