AIware Bootcamp - Mini 2025

Registration

How is JetBrains leveraging FMware to improve the software development experience?

Danny Dig

This talk will present compelling services and applications featuring JetBrains AI Assistant that power software engineering tasks in the IDE to improve developer productivity.

Performance engineering for AIware

Boyuan Chen and Haoxiang Zhang

This session will outline the various performance engineering challenges as one optimizes the performance of an AIware for production. Topics covered include:

• Transformer decoding 101
• Single model serving challenges and the state-of-the-art (e.g., vllm)
• End-to-end performance engineering challenges in the AIware development lifecycle
• Latency considerations for different cognitive architectures
• Multi-model pipeline serving challenges and vision
• Curated SPE challenges for AIware
• Semantic caching and FM routers

Break

Please note that refreshments will not be provided by the bootcamp during the breaks. Participants are kindly encouraged to make their own arrangements for snacks and beverages as needed.

Evaluating AIware

Justina Lin

This session will detail the challenges and techniques that can be used to evaluate an AIware. Covered topics include:

• Overview of evaluating an AIware - Importance
• Evaluation primitives - Evaluation with ad hoc vibe checks, benchmarks, manually curated datasets, trace data, data splits, and repetitions
• Evaluation metrics overview
• Evaluating individual components of AIware - Agents, RAG, etc.
• Testing AIware - Unit tests, summary evaluations, response evaluations, regression testing, backtesting
• AI as judge - Overview, benefits, and costs

AIware Observability

Ben Rombaut

The session will discuss AIware observability challenges. Covered topics include:

• AIware Ops overview
• AIware observability overview - What is it, why is it important, and how is it different for AIware
• The importance of semantics vs raw observability
• Existing observability tools and platforms – strengths and weaknesses
• Evaluation and guardrails - Validating FM responses, comparing models on golden sets, measuring response time, tracking and altering observability metrics

Lunch

Lunch will not be provided by the Bootcamp. The participants are encouraged to explore the diverse food scene of Ottawa.

Agentic architectures and workflows

Keheliya Gallaba

This session will discuss what an agent is and the different cognitive architectures that an Agent could use to execute tasks. Covered topics include:

• What is an Agent (e.g., perception, Brain, Action, Environment)?
• Different types of agentic memory
• How does an Agent use tools (Toolformer, TALM)
• How to enable an Agent to plan and reason - Introduction to theory of mind
• Cognitive architectures - Multi-layered nature (recursion, multi vs single agent architectures)
• Agent patterns - Chains, routers, graphs
• Control mechanisms - Static vs dynamic
• Patterns of multi-agent collaboration
• Self-reflection and multi-agent hacks

Development Platforms for Agentic Software

Gustavo Oliva

This session will give an overview of the various Agentic platforms that one can use to build AIware. Covered topics include:

• Multi-agent frameworks/platforms (Introduction, Comparison, their abstractions). Examples include Autogen, CrewAI, LangGraph
• Agent memory representations and abstractions
• Long-term agents - Agents evolving over time, End-user feedback used for personalization
• Standardization efforts e.g., Agent protocol

The Data Flywheel for FMware

Gopi Krishnan Rajbahadur

This session will go over the different types of data used throughout the lifecycle of an FMware. Topics covered include:

• Data flywheel - An overview of the different types of data used for different types of alignment, including:
• Pre-training data
• Fine-tuning data
• Preference data
• User-feedback data
• Crowd-sourced data
• Synthetic data
• Synthetic data generation techniques and data alignment (Microsoft Phi models)
• QA for Knowledge distillation - Data quality and knowledge quality enhancing processes like deduplication, information metrics, removing data and knowledge clones
• Contextualized and goal-oriented documentation to enhance knowledge quality
• Operationalizing user-feedback data - Leveraging thumbs up/down, Google DIDACT
• BigCode project - A successful application of the data flywheel
• Crowdsourced knowledge curation - Instructlab
• Importance of multi-modal training data - How code is used and generating different types of data with domain-specific rules

Break

Please note that refreshments will not be provided by the bootcamp during the breaks. Participants are kindly encouraged to make their own arrangements for snacks and beverages as needed.

Balancing Cost and Quality in FMware

Kirill Vasilevski

This session will provide an in-depth exploration of Foundation Model (FM) routing for FMware, focusing on balancing quality and inference cost. Covered topics include:

• An introduction to FM routing, where requests are routed to FMs of varying sizes and capabilities
• A survey of existing routing methods that rely on data-driven learning to make optimal routing decisions
• The challenges posed by existing approaches, such as reliance on curated data, complex computations, and the evolution of weaker FMs
• The introduction of Real-time Adaptive Routing (RAR), a novel approach that continuously adapts FM routing decisions using guided in-context learning
• How RAR reduces dependence on stronger, more expensive FMs while maintaining high response quality
• The intra-domain generalization benefits of RAR’s guided learning approach in enhancing weaker FMs

Challenges in Productionizing AIware

Gopi Krishnan Rajbahadur

This session will cover the different types of challenges that industry typically encounters in the different stages of the AIware development lifecycle that make productionizing AIware challenging. Covered topics include:

• An overview of the challenges across the AIware development lifecycle
• Testing challenges
• Observability-related challenges
• Controlled-execution-related challenges
• Resource-aware QA challenges
• Feedback integration challenges
• Built-in quality assurance challenges
• Other overarching challenges

FMArts platform - An FMware lifecycle engineering platform

Dayi Lin

This session will showcase the FMArts platform, the world's only lifecycle engineering platform for creating FMware. We will discuss how software engineering principles can be leveraged in the FMware context and highlight the challenges involved. The session will feature a demo application built with FMArts.

Closing remarks and future initiatives

Ahmed E. Hassan

Closing remarks and discussion on future initiatives.