Load balancing distributes work among suitable instances. Health, connection reuse and uneven request cost affect effectiveness.
Before you start
You should understand API requests, storage and basic capacity estimates. Begin with a concrete user action and its correctness requirement. Draw data flow and failure boundaries before selecting infrastructure; a technology name by itself does not explain why a design meets the requirement.
The practical goal is to reason through this situation: Two servers with equal request counts can have very different CPU load if requests vary in cost. Read the walkthrough first, then try the interview exercise before opening its answer. The important part is explaining the decision and its consequences, rather than remembering a definition alone.
Step-by-step walkthrough
Step 1: Measure request cost
Equal request counts do not guarantee equal computational load.
Step 2: Choose eligible capacity
Health and readiness determine which instances should receive work.
Step 3: Bound overload
Deadlines and admission limits prevent indefinite queue growth.
Worked scenario
Two servers with equal request counts can have very different CPU load if requests vary in cost.
One server handles ten cheap reads while another handles ten expensive exports. Round-robin balances counts but not work. Long-lived connections and sticky sessions can further skew distribution. Select a policy matching observable capacity and avoid sending more traffic to dependencies that are already saturated.
Common mistake
Round-robin alone does not solve overload or dependency saturation.
Verify the behavior
Compare per-instance load and tail latency with mixed request costs.
Interview exercise
Handle long-running requests.
Answer and reasoning
Choose an informed balancing and capacity policy, set deadlines and reject excess work rather than building unbounded queues.
Continue learning
Compare the scenario with the System Design interview questions and test your understanding with the System Design MCQs. For terminology and implementation details, consult the reference material.