Code Generation System

Inline Code Completion and Chat in the Editor

Design an AI coding assistant for the editor: gray inline suggestions that appear as a developer pauses, plus a chat panel, with the right context from the project, fast enough to feel instant and safe for company code.

Overview

An AI code assistant lives in the editor. As the developer types and pauses, it shows a gray inline suggestion (ghost text) that Tab accepts. A chat panel answers questions about the code and writes larger pieces. This problem is the system behind both.

Inline completion is the hard part. It fires thousands of times a day per developer, each answer is useful for only a moment, and the model must fit new code between what is above the cursor and what is below it. Chat is closer to a general language-model service, covered in the ChatGPT / LLM System topic; here it shares the context and safety layers.

Questions to ask first

Which features? — Inline completion first; chat second
How many developers? — 1 million a day, mostly in working hours
How fast must suggestions appear? — Under about 500 ms after the developer pauses (our target)
Where does context come from? — The open file, other open files, and files the current one imports; indexed on the developer's machine
Can code be stored on the server? — No: prompts are used to answer and then discarded; telemetry holds no code
Enterprise needs? — Exclude chosen files; block suggestions that match public code; audit settings per organization

What you will be able to do

Explain fill-in-the-middle and why a model must be trained for it.
Choose and pack context into a token budget, nearest code first.
Keep the request count sane with debounce and cancellation, and size the serving fleet.
Filter suggestions for secrets, public-code matches and excluded files.
Measure quality online with acceptance rate and offline with pass@k.

Key concepts

Fill-in-the-middle (FIM)
Context packing
Debounce and cancellation
Small model for inline, large for chat
Streaming and prefix caching
Acceptance rate
pass@k
Public-code matching
Content exclusion

Functional requirements

Show an inline suggestion at the cursor after the developer pauses, using code before and after the cursor
Accept all or part of a suggestion; dismiss by typing on
Use context from other files in the project
Chat about selected code, with answers streamed
Honor organization settings: excluded files, blocking suggestions that match public code
Record whether each shown suggestion was accepted, without recording code

Non-functional requirements

Inline suggestion visible within about 500 ms of a pause at p90 (a target we chose)
A stale request is canceled on the next keystroke and stops using GPU time
No code retained on the server after the answer is sent
When the service is slow or down, the editor works normally with no suggestions
Region-local serving so network time stays small

Interview questions

How does fill-in-the-middle work, and why does it matter?
What context do you send, and how do you fit it in the budget?
How do you make suggestions feel instant?
How many GPUs, roughly, and what drives the number?
How do you measure whether suggestions are good?
How do you handle licensing and company privacy concerns?