AIware Leadership
Bootcamp 2024

Registration and Morning coffee break

Prompt Engineering

Filipe Roseiro Cogo

This session will offer a comprehensive overview of prompt engineering techniques and best practices to build a successful FMware. Covered topics include:

• Basics of prompting - How to talk to a FM
• Prompting patterns
• Prompt components
• Prompt structuring
• Prompt decoding strategies
• Fragility of prompts - Manual prompt-tuning lifecycle, prompt formatting, context window sensitivity, few-shot ordering, and golden labels
• Prompt anti-patterns (e.g. god prompts) - How to decompose a prompt effectively for success
• Prompt output structuring and prompt debugging
• Compiling prompts for success - prompt optimization, prompt tuning, FM-based prompt mutation and evolution, DSLs for prompt optimization, Intent-based prompt calibration
• Common prompting pitfalls
• Leveraging prompt ecosystems - Introduction to prompt stores and Reddit discussions

Break

RAG Engineering

Keheliya Gallaba

This session will cover the steps involved in building a robust RAG (Retrieval Augmented Generation) pipeline for an AIware app. The covered topics include:

• What is RAG? - An overview of Indexing, retrieval, and generation
• An overview of query translation approaches - Multi-query, RAG-fusion, Least-to-most, step-back prompting, HyDE
• Query routing strategies - Logical routing and semantic routing
• Query construction strategies
• Advanced indexing techniques for effective RAG - Multi-representation indexing, RAPTOR, ColBERT
• Graph RAGs
• RAG Re-rankers
• Large context challenges: Needle in a haystack, counting stars, data movement and caching/pinning of context in the cloud
• Context vs built-in prioritization
• A reference architecture for enterprise-grade production-ready RAGware

Lunch

Alignment Engineering

Gopi Krishnan Rajbahadur

This session will cover different ways of aligning a Foundation Model (FM). Covered topics include:

• A general overview of FM pre-training
• Fine-tuning - Types of used data: supervised fine-tuning, PEFT, Prompt tuning, soft prompts, Adapter tuning, AdapterHub, Selective finetuning (e.g. BiFit), Reparametrization-based fine tuning (LoRA)
• Preference tuning - RLHF, DPO, RLSF
• Constitutional AI
• Overview of curriculum learning
• Advanced curriculum generation techniques and benefits, e.g., how curriculum learning is used in the Microsoft Phi and IBM Granite models

Break

Improving FMware Output Quality

Prem Devabu

This session will explore the role of calibration in evaluating the output quality of Foundation Models (FMs) for code generation and its impact on decision-making. Covered topics include:

• The need for reliable confidence measures in FM-generated code to ensure rational decision-making and reduce the risk of using incorrect outputs
• How calibration can assess the trustworthiness of FM-generated code and guide the level of review needed before use
• Unique challenges in calibrating generative models for code generation, compared to traditional classification models
• A framework for evaluating the calibration of FM-generated code across different tasks, correctness criteria, and datasets
• Key findings on the poor calibration of FM-generated code and the risks this poses for developers
• The use of techniques like Platt scaling to improve FM output calibration and its applicability in the context of code generation
• Practical insights into improving FM output quality for code generation, enabling better decision-making and fostering more reliable AI-driven software engineering practices

How is JetBrains leveraging FMware to improve the software development experience?

Danny Dig

This talk will present compelling services and applications featuring JetBrains AI Assistant that power software engineering tasks in the IDE to improve developer productivity.

Registration and morning coffee break

Research experiences and best practices building AIware at JetBrains

Danny Dig

This talk presents cutting-edge research on combining the creativity of LLM-based recommendations with the safety and reliability of static and dynamic analysis in the JetBrains IDEs to provide an order of magnitude improvements over previous state-of-the-art solutions for refactoring.

Towards AI-Native Software Engineering (SE 3.0)

Ahmed E. Hassan and Gustavo Oliva

This talk will discuss a vision of an AI-native approach to software engineering, which is characterized by an intent-first, conversation-oriented development where humans and AI collaborate seamlessly. Covered topics include:

• A critical analysis of AI-assisted/augmented SE 2.0
• Autonomous software engineers (e.g., Devin AI)
• Our vision of AI-native SE and its technology stack
• Challenges and open questions in the path of realizing our vision:
• Challenge 1: Speeding up human-AI alignment
• Challenge 2: Improving the efficiency of code synthesis
• Challenge 3: Improving runtime performance
• Challenge 4: Improving FM’s understanding of code and SE
• Challenge 5: Eliminating the need for prompt engineering

Break

Agentic architectures and workflows

Keheliya Gallaba

This session will discuss what an agent is and the different cognitive architectures that an Agent could use to execute tasks. Covered topics include:

• What is an Agent (e.g., perception, Brain, Action, Environment)?
• Different types of agentic memory
• How does an Agent use tools (Toolformer, TALM)
• How to enable an Agent to plan and reason - Introduction to theory of mind
• Cognitive architectures - Multi-layered nature (recursion, multi vs single agent architectures)
• Agent patterns - Chains, routers, graphs
• Control mechanisms - Static vs dynamic
• Patterns of multi-agent collaboration
• Self-reflection and multi-agent hacks

Lunch

Agentic Development Platforms

Gustavo Oliva

This session will give an overview of the various Agentic platforms that one can use to build AIware. Covered topics include:

• Multi-agent frameworks/platforms (Introduction, Comparison, their abstractions). Examples include Autogen, CrewAI, LangGraph
• Agent memory representations and abstractions
• Long-term agents - Agents evolving over time, End-user feedback used for personalization
• Standardization efforts e.g., Agent protocol

Break

FMwareBOM - Towards enabling transparency, traceability and compliance with next generation FMware BOMs

Gopi Krishnan Rajbahadur

This talk will highlight the current state of AIBOM and ongoing efforts on FMwareBOM and the potential it holds for enabling compliance, transparency, and traceability in the FMware era.

Responsible AIware Engineering

Qinghua Lu

This session will cover the challenges and emerging trends in responsible AI engineering in the context of AIware systems. Covered topics include:

• The shift towards AI-as-Software (AIware) with foundation models embedding business logic within parameters and weights
• The integration of business functions into models, replacing traditional business code logic
• The implications of encapsulating logic within a single black box model
• Growing concerns about responsible AI and AI safety with autonomous AIware
• The importance of compound AI systems that combine foundation models with other components for quality and risk control
• A system-level engineering approach to ensure responsible AI development in foundation model-based and multi-agent AI systems

Quality Assurance, Governance, and Responsible AIware Development

Qinghua Lu, Bram Adams, Prem Devanbu, Gopi Krishnan Rajbahadur, and Zhen Ming (Jack) Jiang. Moderator: TBD

This panel will discuss the interplay between quality assurance, governance structures, and responsible AIware development practices. It will explore key topics such as compliance frameworks, governance strategies, and emerging legislation that shape the development and deployment of AI systems. The discussion will also address responsible AI, emphasizing ethical considerations, transparency, and risk mitigation in AI-powered systems. Panelists will share insights on balancing innovation with regulatory compliance, ensuring high-quality and trustworthy AIware, and managing the challenges of accountability in AIware.

Registration and Morning Coffee Break

FMware - Hands on

Instructor team

We will be building a few AIware apps which we will make more complex as we progress through the bootcamp. Every hands-on will progressively build on the concepts of that day to make the day more comprehensive. This hands-on session will guide the participants who will be encouraged to add more features to the app and submit it to the AIware conference challenge. The hands-on session will methodically walk the attendees through:

• Setting up a FM locally
• Accessing a cloud FM
• Setting up a basic Vector store
• Building a basic AIware app
• Picking the right FM for their FMware
• Preventing data contamination
• Evaluating your FMware

Break

FMware - Hands on - continued

Instructor team

This session will be a continuation of the previous session, covering the following topics:

• Prompting strategies
• Prompt anti-patterns
• Debugging a prompt
• Prompt compilers - DsPY
• Fine-tuning an FM with LoRa - Data preparation (including quality)
• Preference tuning
• RAG 101
• Best RAG practices
• Progressively building an AIware app with basic cognitive architecture
• A multi-agent collaborative architecture
• Simple memory implementation

FMArts platform - An FMware lifecycle engineering platform

Dayi Lin

This session will showcase the FMArts platform, the world's only lifecycle engineering platform for creating FMware. We will discuss how software engineering principles can be leveraged in the FMware context and highlight the challenges involved. The session will feature a demo application built with FMArts.

Lunch

Challenges in Productionizing AIware

Ahmed E. Hassan and Gopi Krishnan Rajbahadur

This session will cover the different types of challenges that industry typically encounters in the different stages of the AIware development lifecycle that make productionizing AIware challenging. Covered topics include:

• An overview of the challenges across the AIware development lifecycle
• Testing challenges
• Observability-related challenges
• Controlled-execution-related challenges
• Resource-aware QA challenges
• Feedback integration challenges
• Built-in quality assurance challenges
• Other overarching challenges

Balancing Cost and Quality in FMware

Kirill Vasilevski

This session will provide an in-depth exploration of Foundation Model (FM) routing for FMware, focusing on balancing quality and inference cost. Covered topics include:

• An introduction to FM routing, where requests are routed to FMs of varying sizes and capabilities
• A survey of existing routing methods that rely on data-driven learning to make optimal routing decisions
• The challenges posed by existing approaches, such as reliance on curated data, complex computations, and the evolution of weaker FMs
• The introduction of Real-time Adaptive Routing (RAR), a novel approach that continuously adapts FM routing decisions using guided in-context learning
• How RAR reduces dependence on stronger, more expensive FMs while maintaining high response quality
• The intra-domain generalization benefits of RAR’s guided learning approach in enhancing weaker FMs

Break

Evaluating AIware

Justina Lin

This session will detail the challenges and techniques that can be used to evaluate an AIware. Covered topics include:

• Overview of evaluating an AIware - Importance
• Evaluation primitives - Evaluation with ad hoc vibe checks, benchmarks, manually curated datasets, trace data, data splits, and repetitions
• Evaluation metrics overview
• Evaluating individual components of AIware - Agents, RAG, etc.
• Testing AIware - Unit tests, summary evaluations, response evaluations, regression testing, backtesting
• AI as judge - Overview, benefits, and costs

Compiler.Next

Filipe Roseiro Cogo

This session will cover the challenges and future directions of AI-native software systems in the context of Software Engineering 3.0 (SE 3.0). Covered topics include:

• An overview of Compiler.next, a novel search-based compiler designed for AI-native software evolution
• How Compiler.next generates software from human-written intents through search-based optimization
• The dynamic optimization of cognitive architectures, foundation models, and system parameters
• The balance between accuracy, cost, and latency in AI-powered software generation
• Lowering the technical barriers for non-experts to enable scalable, adaptable AI-driven software systems
• A roadmap to address core challenges in intent compilation, including programming constructs, search heuristics, and compiler interoperability
• The vision for fully automated, search-driven software development that fosters faster innovation in AI-powered systems

Registration and morning coffee break

Performance engineering for AIware

Boyuan Chen and Haoxiang Zhang

This session will outline the various performance engineering challenges as one optimizes the performance of an AIware for production. Topics covered include:

• Transformer decoding 101
• Single model serving challenges and the state-of-the-art (e.g., vllm)
• End-to-end performance engineering challenges in the AIware development lifecycle
• Latency considerations for different cognitive architectures
• Multi-model pipeline serving challenges and vision
• Curated SPE challenges for AIware
• Semantic caching and FM routers

Break

AIware Observability

Ben Rombaut

The session will discuss AIware observability challenges. Covered topics include:

• AIware Ops overview
• AIware observability overview - What is it, why is it important, and how is it different for AIware
• The importance of semantics vs raw observability
• Existing observability tools and platforms – strengths and weaknesses
• Evaluation and guardrails - Validating FM responses, comparing models on golden sets, measuring response time, tracking and altering observability metrics

Experiences designing AI-assisted coding tools (like CodeCompose and SQLCompose) at Meta

Peter Rigby

This session will cover various challenges encountered during the development and deployment of AI-assisted code authoring solutions at industry scale. Covered topics include:

• An overview of the design and implementation of CodeCompose, Meta's internal copilot
• How Meta achieved multi-line AI-assisted code generation
• The challenges and solutions behind SQLCompose, a copilot specifically tuned for SQL
• Fine-tuning AI models for industry-specific applications
• The deployment process of AI-assisted tools
• Evaluation methods, including mixed-methods approaches for large-scale AI code generation systems

Lunch

Leading the AIware Revolution - Ask Me Anything!

Ahmed E. Hassan, Mike Godfrey, Mei, Peter Rigby, and Danny Dig. Moderator: TBD

This panel will explore how effective leadership is shaping the AIware era, with a focus on guiding innovation, research, and strategic decisions in a rapidly evolving landscape. We'll touch on the role of leaders in bridging industry and academia, balancing the pace of applied AI with foundational research, and navigating the development of open models. Whether you’re leading a team or aspiring to, this is your chance to ask experienced leaders about the challenges and opportunities in driving the future of AI-powered software.

Break

Showcases of AIware for SE innovations and demos of developed AIware by camp attendees

This session will showcase works from leading experts in the field, and selected AIware projects that have been developed will be presented.

Closing remarks and future initiatives

Ahmed E. Hassan

Closing remarks and discussion on future initiatives.