The cover illustrates a fictional store, not a deployed system or real test result.
Ready Parts adds a second server, but some customers are logged out, and recently uploaded product images cannot be found. This hypothetical event shows why doubling machines is insufficient. Work can move to another machine only if required state is not trapped on the first one and shared services can handle the extra load.
I would add servers after removing unnecessary work and measuring the bottleneck. If the database is waiting on locks, more app instances may only create a longer line. This episode separates a larger machine, more machines, and the readiness to distribute work. Distribution is not every business’s end goal.
Bigger machines and more machines solve different problems
Scaling up adds resources to a machine or service, such as CPU or memory. It may be simpler for an existing system, but has limits and a resize process to check. Some systems must pause or move work. A bigger machine does not fix a query that reads too much data or a request waiting on an outside service.
Scaling out adds service units and distributes work among them. It can expand some layers and reduce dependence on one machine, but adds coordination, state handoffs, error tracking, and version management. With modest load and a small team, a simpler managed service or a right-sized upgrade may suit better than building several components.
Stateless: required request state is not tied to one app machine; business data does not disappear. State still lives in databases, file stores, or a designed session service, each with appropriate correctness, availability, and access control.
Move shared state off the app machine
For the fictional store, authentication sessions might live in a shared store with expiration and revocation rules, or use verifiable tokens under the system design. Tokens do not solve every revocation or permission-change case by themselves. Account for logout, theft, and role changes; do not include secrets just because users cannot see them on screen.
Product files that must persist should not live only on an app’s temporary disk. Use file storage with clear access and lifecycle rules. Test that new instances can read authorized files and that private files do not become public URLs. Separate storage adds a service to pay for and maintain, so include it in cost rather than treating “stateless” as free.
Scheduled jobs are another easy trap. If every instance sends the same emails or closes the same cycle, work may happen twice. Give a job one owner or use an appropriate coordination mechanism, and make business outcomes tolerate a repeated start. “There is normally only one server” stops being a safeguard once there are several.
Do not keep required state on one machine
Sessions, files, transactions, and duplicate-prevention keys
Check connection quotas and availability
A load balancer needs to know which instance can accept work
A load balancer selects a destination for a request; it cannot know every business correctness condition. Checking that a process is alive differs from checking that it can serve work. A newly started instance may still be loading data or connecting to services. Separate liveness from readiness according to the platform.
A health check should not run heavy queries or call several services on every probe until it creates its own load. Also beware of removing every instance at once when a shared service briefly falters. Choose checks and timing to match graceful degradation: the catalog might remain available while order confirmation is paused, with an honest status shown.
When removing an instance or rolling out a new version, stop new assignments and let existing work finish within a defined bound, or move it with a method that tolerates interruption. An order submitted during deployment should not silently vanish. Customers need a way to learn the result and retry without creating a duplicate. The queue and idempotency episode goes deeper.
| Component | Risk when adding instances | Evidence to request |
|---|---|---|
| Sessions and rights | Inconsistent state, logouts, or stale permissions | Switch instances, log out, and change rights with expected results |
| Files | New instance cannot find a file or opens it too broadly | Upload, read, delete, and check access across instances |
| Scheduled jobs | Repeat once per instance | Outcome records and tolerance of repeated starts |
| Database | Connections surge and waits lengthen | Total connection budget and pool-wait time |
| Deployment | Pending work or incompatible data across versions | Old/new version coexistence test and rollback path |
More app instances may also mean more connections
Suppose each instance can open a database pool of at most 20 connections. Expanding from two to 10 instances raises the ceiling for this app layer from 40 to 200, before background workers, administrators, or an overlapping old/new deployment. This calculates a hypothetical ceiling, not a recommended database setting or a claim that all connections stay in use.
If the database cannot handle that many, additional app instances may reduce processing time in one layer while increasing waits in another. Budget connections across the whole system, cap queue and pool waits, reserve room for administration and incidents, and test at the maximum allowed instance count, not just a normal day’s count.
Outside services have limits too: request counts, email rates, or partner API connections. More of our servers do not raise their quotas. Set timeouts against the request’s total time budget, limit retries, and show an honest outcome when one part is slow or unavailable. Do not let every layer retry its full allotment and multiply the load.
Autoscaling has a delay before new capacity is ready
Autoscaling changes resource count or size by signals and policy; it is not spare capacity that appears instantly. There is metric delay, instance start time, and warmup. Platform documentation: consider warmup and choose a metric related to load per unit.
If CPU stays low because every request waits on the database, CPU-only scaling may not help. Scaling by request count may also send more work to the same bottleneck. I would use several signals to diagnose the problem, then choose a controller whose behavior we understand, with minimum and maximum counts and a response for hitting the limit.
Predictable bursts may justify adding capacity in advance. Unexpected bursts need risk-based headroom and a way to limit excess work. The cost of that reserve is not always waste; it can cover requests while a new instance warms. But the amount needs a reason and a check for overprovisioning.
Signal and policy trigger
Prepare code, connections, and data
Pass readiness checks
Test more than the number of instances added
Test how much more work completes successfully within the latency and error targets. “We can run ten instances” is not an acceptance result without usable throughput. Separate causes of less-than-proportional gain, such as lock contention, networks, serial work, or a saturated shared service.
Test one instance disappearing under load, scaling down afterward, and deployment while orders arrive. Stopping one machine in a lab does not prove resilience against every failure, but it tests a stated assumption. Write the result’s scope clearly so a budget approver does not mistake it for readiness at every level.
The team must still be able to operate the larger system
As instance counts change, the team must still find traces and logs for one transaction. Configuration, code versions, and secrets should be delivered through auditable, revocable methods rather than hand-edits on individual machines. Extra instances with differing behavior create intermittent faults that are harder to trace than a small, well-understood system.
Alert when the maximum instance count is reached or new capacity fails to start, not only when CPU is high. Track the time from a scale-down command until the provider actually stops billing for resources. A lower count on the dashboard does not prove all costs disappear immediately; disks, IPs, file storage, or service minimums may remain.
If the team still spends much time fixing individual machines, making deployment repeatable and status visible may pay more than adding orchestration immediately. Choose VMs, a managed app service, or another platform based on actual load and skills. Kubernetes is one option with its own operating duties, not a certificate that the site can now grow.
App-only ceiling: 40 connections
App-only ceiling: 200 connections
I would add instances when that layer can divide work, state can move, shared services have headroom, and the system can handle the wait for new capacity. If any condition fails, fix that constraint before multiplying machines.
The handoff to the database episode should show the maximum app count, connection budget, work needing shared state, and graphs that separate waiting from processing. A business owner need not memorize every service name, but should know which queue the extra machine is shortening and which shared bottleneck remains.
Read the sources and limits of the example
The following sources explain the concepts and limitations and were checked on October 18, 2026. The fictional store numbers and choices are my synthesis, not provider prices, test results, or guarantees. Check the current version, plan, and terms before applying them.