AI Copilot System

Code completion and chat in the editor

Design an AI coding assistant: ghost-text completions that appear while the developer types, and a chat that answers questions about their code, with each customer's code kept to that customer.

Overview

An AI copilot lives in the editor and does two jobs. Inline completion shows grey 'ghost text' at the cursor while the developer types; Tab accepts it. Chat answers a question about the code in a side panel. They share models and context code, but they are different products: completions must be fast and cheap because most are never used, while a chat answer can take seconds and must be right.

The Language Server Protocol (LSP), the standard editors use to talk to language tools, added an inline completion request in version 3.18. Its description fits this problem: the request is sent either when the user asks or automatically while typing, and an item may complete a large piece of code that is not yet syntactically valid, which is why editors show it inline in lower contrast instead of in a list.

The quality of a completion depends mostly on what goes into the prompt. GitHub wrote in 2023 that Copilot builds each prompt by selecting, ranking and packing snippets from the current file and other sources, and that two prompt changes raised acceptance: reading all open editor tabs (5% relative) and fill-in-the-middle, which gives the model the code after the cursor too (10% relative).

Questions to ask first

Completions, chat, or both? — Both; completions carry most of the traffic
How many users? — 1 million developers active on a workday
How fast must a completion appear? — About 300 ms after the developer pauses (a target we chose)
Where does context come from? — The editor: current file, open tabs, imports, recent edits
Business customers? — Yes: their code must never reach another customer, and they may want a customized model
Do we train the base model? — No: we serve a small model for completions and a larger one for chat

What you will be able to do

Lay out a latency budget for an inline completion and say which parts you can shrink.
Assemble context for a prompt within a token budget, including fill-in-the-middle.
Design debounce and cancellation so the GPUs do not serve completions nobody sees.
Define acceptance rate, explain why it is the metric, and compute it from telemetry.
Keep one customer's code out of another customer's results, including with customized models.

Key concepts

Inline completion
Fill-in-the-middle
Context assembly
Debounce and cancel
Acceptance rate
Model routing
Continuous batching
Tenant isolation
LoRA adapters
Public-code filter

Functional requirements

Suggest inline completions (single line or a block) as the developer types or asks
Answer chat questions about selected code, the open file and the repository
Build prompts from the cursor's prefix and suffix, open tabs, imports and recent edits
Cancel requests the developer has typed past
Record which suggestions were shown, accepted, partly kept or dismissed
Let an organization customize suggestions with its own code, visible only to it
Block or label suggestions that match public code, per the organization's policy

Non-functional requirements

Inline completion: first ghost text about 300 ms after the pause, p90 (a target we chose)
Chat: first token within about 1 s, answer streamed (a target we chose)
No customer's code, prompts or suggestions visible to another customer
Graceful degradation: when overloaded, drop completions before chat, and never block typing
Cost scales with requests shown, not keystrokes

Interview questions

How do you make inline completions fast enough?
How do you choose the context for a completion?
What metric tells you the copilot is good?
How do you keep one customer's code from appearing for another?
Why fill-in-the-middle?
How do you handle cancellation end to end?
How would you add a per-customer model?