AIware Leadership
Bootcamp 2024

Registration and Morning coffee break

Prompt Engineering

Filipe Roseiro Cogo

This session will offer a comprehensive overview of prompt engineering techniques and best practices to build a successful FMware. Covered topics include:

• Basics of prompting - How to talk to a FM
• Prompting patterns
• Prompt components
• Prompt structuring
• Prompt decoding strategies
• Fragility of prompts - Manual prompt-tuning lifecycle, prompt formatting, context window sensitivity, few-shot ordering, and golden labels
• Prompt anti-patterns (e.g. god prompts) - How to decompose a prompt effectively for success
• Prompt output structuring and prompt debugging
• Compiling prompts for success - prompt optimization, prompt tuning, FM-based prompt mutation and evolution, DSLs for prompt optimization, Intent-based prompt calibration
• Common prompting pitfalls
• Leveraging prompt ecosystems - Introduction to prompt stores and Reddit discussions

Break

RAG Engineering

Keheliya Gallaba

This session will cover the steps involved in building a robust RAG (Retrieval Augmented Generation) pipeline for an AIware app. The covered topics include:

• What is RAG? - An overview of Indexing, retrieval, and generation
• An overview of query translation approaches - Multi-query, RAG-fusion, Least-to-most, step-back prompting, HyDE
• Query routing strategies - Logical routing and semantic routing
• Query construction strategies
• Advanced indexing techniques for effective RAG - Multi-representation indexing, RAPTOR, ColBERT
• Graph RAGs
• RAG Re-rankers
• Large context challenges: Needle in a haystack, counting stars, data movement and caching/pinning of context in the cloud
• Context vs built-in prioritization
• A reference architecture for enterprise-grade production-ready RAGware

Lunch

Lunch will be provided for all participants.

Alignment Engineering

Gopi Krishnan Rajbahadur

This session will cover different ways of aligning a Foundation Model (FM). Covered topics include:

• A general overview of FM pre-training
• Fine-tuning - Types of used data: supervised fine-tuning, PEFT, Prompt tuning, soft prompts, Adapter tuning, AdapterHub, Selective finetuning (e.g. BiFit), Reparametrization-based fine tuning (LoRA)
• Preference tuning - RLHF, DPO, RLSF
• Constitutional AI
• Overview of curriculum learning
• Advanced curriculum generation techniques and benefits, e.g., how curriculum learning is used in the Microsoft Phi and IBM Granite models

Break

Improving FMware Output Quality

Prem Devanbu

This session will explore the role of calibration in evaluating the output quality of Foundation Models (FMs) for code generation and its impact on decision-making. Covered topics include:

• The need for reliable confidence measures in FM-generated code to ensure rational decision-making and reduce the risk of using incorrect outputs
• How calibration can assess the trustworthiness of FM-generated code and guide the level of review needed before use
• Unique challenges in calibrating generative models for code generation, compared to traditional classification models
• A framework for evaluating the calibration of FM-generated code across different tasks, correctness criteria, and datasets
• Key findings on the poor calibration of FM-generated code and the risks this poses for developers
• The use of techniques like Platt scaling to improve FM output calibration and its applicability in the context of code generation
• Practical insights into improving FM output quality for code generation, enabling better decision-making and fostering more reliable AI-driven software engineering practices

How is JetBrains leveraging FMware to improve the software development experience?

Danny Dig

This talk will present compelling services and applications featuring JetBrains AI Assistant that power software engineering tasks in the IDE to improve developer productivity.

Registration and morning coffee break

The Data Flywheel for FMware

Gopi Krishnan Rajbahadur

This session will go over the different types of data used throughout the lifecycle of an FMware. Topics covered include:

• Data flywheel - An overview of the different types of data used for different types of alignment, including:
• Pre-training data
• Fine-tuning data
• Preference data
• User-feedback data
• Crowd-sourced data
• Synthetic data
• Synthetic data generation techniques and data alignment (Microsoft Phi models)
• QA for Knowledge distillation - Data quality and knowledge quality enhancing processes like deduplication, information metrics, removing data and knowledge clones
• Contextualized and goal-oriented documentation to enhance knowledge quality
• Operationalizing user-feedback data - Leveraging thumbs up/down, Google DIDACT
• BigCode project - A successful application of the data flywheel
• Crowdsourced knowledge curation - Instructlab
• Importance of multi-modal training data - How code is used and generating different types of data with domain-specific rules

Measuring and Curating Code LLM Supply Chains

Audris Mockus

This session covers the data supply chain in Large Language Models (LLMs), focusing on:

• Pre-training data sources: Understanding the origins and diversity of data used.
• Fine-tuning and adapter models:
• Specialized dataset adaptation, cost reduction through quantization, and task-specific fine-tuning.
• Data versioning and leakage:
• Managing dataset versions, data reuse, and avoiding training-test overlap.
• Training dependencies:
• Impact of parameters on models and ensuring consistency.
• Model integration into software systems and feedback loops:
• Code created by models as new training data, forming continuous learning cycles.
• Documentation and metadata challenges:
• Current practices and gaps, particularly in tracking data provenance.
• Reconstructing the supply chain to address:
• Data vulnerabilities, adversarial risks, and other threats.

Break

Agentic architectures and workflows

Keheliya Gallaba

This session will discuss what an agent is and the different cognitive architectures that an Agent could use to execute tasks. Covered topics include:

• What is an Agent (e.g., perception, Brain, Action, Environment)?
• Different types of agentic memory
• How does an Agent use tools (Toolformer, TALM)
• How to enable an Agent to plan and reason - Introduction to theory of mind
• Cognitive architectures - Multi-layered nature (recursion, multi vs single agent architectures)
• Agent patterns - Chains, routers, graphs
• Control mechanisms - Static vs dynamic
• Patterns of multi-agent collaboration
• Self-reflection and multi-agent hacks

Lunch

Lunch will be provided for all participants.

Development Platforms for Agentic Software

Gustavo Oliva

This session will give an overview of the various Agentic platforms that one can use to build AIware. Covered topics include:

• Multi-agent frameworks/platforms (Introduction, Comparison, their abstractions). Examples include Autogen, CrewAI, LangGraph
• Agent memory representations and abstractions
• Long-term agents - Agents evolving over time, End-user feedback used for personalization
• Standardization efforts e.g., Agent protocol

Break

FMwareBOM - Towards enabling transparency, traceability and compliance with next generation FMware BOMs

Gopi Krishnan Rajbahadur

This talk will highlight the current state of AIBOM and ongoing efforts on FMwareBOM and the potential it holds for enabling compliance, transparency, and traceability in the FMware era.

Responsible AIware Engineering

Qinghua Lu

This session will cover the challenges and emerging trends in responsible AI engineering in the context of AIware systems. Covered topics include:

• The shift towards AI-as-Software (AIware) with foundation models embedding business logic within parameters and weights
• The integration of business functions into models, replacing traditional business code logic
• The implications of encapsulating logic within a single black box model
• Growing concerns about responsible AI and AI safety with autonomous AIware
• The importance of compound AI systems that combine foundation models with other components for quality and risk control
• A system-level engineering approach to ensure responsible AI development in foundation model-based and multi-agent AI systems

Quality Assurance, Governance, and Responsible AIware Development

Qinghua Lu, Bram Adams, Prem Devanbu, Gopi Krishnan Rajbahadur, Audris Mockus and Zhen Ming (Jack) Jiang. Moderator: Gustavo Oliva

This panel will discuss the interplay between quality assurance, governance structures, and responsible AIware development practices. It will explore key topics such as compliance frameworks, governance strategies, and emerging legislation that shape the development and deployment of AI systems. The discussion will also address responsible AI, emphasizing ethical considerations, transparency, and risk mitigation in AI-powered systems. Panelists will share insights on balancing innovation with regulatory compliance, ensuring high-quality and trustworthy AIware, and managing the challenges of accountability in AIware.

Dinner 2 - Aanch

Enjoy a delightful dining experience at Aanch, a modern Indian restaurant featured in the Michelin Guide. Savor expertly crafted dishes in a stylish and inviting atmosphere, perfect for networking and relaxation.

Location: 259 Wellington Street West, Toronto, ON M5V 3E4, Canada

For your convenience, here is the location on Google Maps:

Registration and Morning Coffee Break

FMware - Hands on

Instructor team

We will be building a few AIware apps which we will make more complex as we progress through the bootcamp. Every hands-on will progressively build on the concepts of that day to make the day more comprehensive. This hands-on session will guide the participants who will be encouraged to add more features to the app and submit it to the AIware conference challenge. The hands-on session will methodically walk the attendees through:

• Setting up a FM locally
• Accessing a cloud FM
• Setting up a basic Vector store
• Building a basic AIware app
• Picking the right FM for their FMware
• Preventing data contamination
• Evaluating your FMware

Break

FMware - Hands on - continued

Instructor team

This session will be a continuation of the previous session, covering the following topics:

• Prompting strategies
• Prompt anti-patterns
• Debugging a prompt
• Prompt compilers - DsPY
• Fine-tuning an FM with LoRa - Data preparation (including quality)
• Preference tuning
• RAG 101
• Best RAG practices
• Progressively building an AIware app with basic cognitive architecture
• A multi-agent collaborative architecture
• Simple memory implementation

FMArts platform - An FMware lifecycle engineering platform

Dayi Lin

This session will showcase the FMArts platform, the world's only lifecycle engineering platform for creating FMware. We will discuss how software engineering principles can be leveraged in the FMware context and highlight the challenges involved. The session will feature a demo application built with FMArts.

Lunch

Lunch will be provided for all participants.

Challenges in Productionizing AIware

Ahmed E. Hassan and Gopi Krishnan Rajbahadur

This session will cover the different types of challenges that industry typically encounters in the different stages of the AIware development lifecycle that make productionizing AIware challenging. Covered topics include:

• An overview of the challenges across the AIware development lifecycle
• Testing challenges
• Observability-related challenges
• Controlled-execution-related challenges
• Resource-aware QA challenges
• Feedback integration challenges
• Built-in quality assurance challenges
• Other overarching challenges

Balancing Cost and Quality in FMware

Kirill Vasilevski

This session will provide an in-depth exploration of Foundation Model (FM) routing for FMware, focusing on balancing quality and inference cost. Covered topics include:

• An introduction to FM routing, where requests are routed to FMs of varying sizes and capabilities
• A survey of existing routing methods that rely on data-driven learning to make optimal routing decisions
• The challenges posed by existing approaches, such as reliance on curated data, complex computations, and the evolution of weaker FMs
• The introduction of Real-time Adaptive Routing (RAR), a novel approach that continuously adapts FM routing decisions using guided in-context learning
• How RAR reduces dependence on stronger, more expensive FMs while maintaining high response quality
• The intra-domain generalization benefits of RAR’s guided learning approach in enhancing weaker FMs

Break

Evaluating AIware

Justina Lin

This session will detail the challenges and techniques that can be used to evaluate an AIware. Covered topics include:

• Overview of evaluating an AIware - Importance
• Evaluation primitives - Evaluation with ad hoc vibe checks, benchmarks, manually curated datasets, trace data, data splits, and repetitions
• Evaluation metrics overview
• Evaluating individual components of AIware - Agents, RAG, etc.
• Testing AIware - Unit tests, summary evaluations, response evaluations, regression testing, backtesting
• AI as judge - Overview, benefits, and costs

From Prompt Enhancement to Fine-Tuning: Improving Automatic SE Tasks with LLMs

Song Wang

This talk will explore how LLMs can improve software development tasks through two approaches: prompt enhancement and fine-tuning. Covered topics include:

• ClarifyGPT: A prompt intention clarification framework that improves code generation by enabling LLMs to identify and resolve ambiguous requirements.
• EPiC (Evolutionary Prompt Engineering for Code): Employs a lightweight evolutionary algorithm to enhance original prompts for generating higher-quality code.
• Benefits and limitations of Retrieval Augmented Generation (RAG) for unit test generation.
• Fine-tuning smaller LLMs like CodeT5 to outperform larger models (e.g., GPT-4) in test generation through domain adaptation.

Registration and morning coffee break

Performance engineering for AIware

Boyuan Chen and Haoxiang Zhang

This session will outline the various performance engineering challenges as one optimizes the performance of an AIware for production. Topics covered include:

• Transformer decoding 101
• Single model serving challenges and the state-of-the-art (e.g., vllm)
• End-to-end performance engineering challenges in the AIware development lifecycle
• Latency considerations for different cognitive architectures
• Multi-model pipeline serving challenges and vision
• Curated SPE challenges for AIware
• Semantic caching and FM routers

Break

AIware Observability

Ben Rombaut

The session will discuss AIware observability challenges. Covered topics include:

• AIware Ops overview
• AIware observability overview - What is it, why is it important, and how is it different for AIware
• The importance of semantics vs raw observability
• Existing observability tools and platforms – strengths and weaknesses
• Evaluation and guardrails - Validating FM responses, comparing models on golden sets, measuring response time, tracking and altering observability metrics

Experiences designing AI-assisted coding tools (like CodeCompose and SQLCompose) at Meta

Peter Rigby

This session will cover various challenges encountered during the development and deployment of AI-assisted code authoring solutions at industry scale. Covered topics include:

• An overview of the design and implementation of CodeCompose, Meta's internal copilot
• How Meta achieved multi-line AI-assisted code generation
• The challenges and solutions behind SQLCompose, a copilot specifically tuned for SQL
• Fine-tuning AI models for industry-specific applications
• The deployment process of AI-assisted tools
• Evaluation methods, including mixed-methods approaches for large-scale AI code generation systems

Lunch

Lunch will be provided for all participants.

Leading the AIware Revolution - Ask Me Anything!

Ahmed E. Hassan, Mike Godfrey, Meiyappan Nagappan, Peter Rigby, Dayi Lin and Danny Dig. Moderator: Gustavo Oliva

This panel will explore how effective leadership is shaping the AIware era, with a focus on guiding innovation, research, and strategic decisions in a rapidly evolving landscape. We'll touch on the role of leaders in bridging industry and academia, balancing the pace of applied AI with foundational research, and navigating the development of open models. Whether you’re leading a team or aspiring to, this is your chance to ask experienced leaders about the challenges and opportunities in driving the future of AI-powered software.

Break

Showcases of AIware for SE innovations and demos of developed AIware by camp attendees

This session will showcase works from leading experts in the field, and selected AIware projects that have been developed will be presented.

Closing remarks and future initiatives

Ahmed E. Hassan

Closing remarks and discussion on future initiatives.