Suppose a database comes back online after an incident, but some transactions entered just before it stopped are missing. Who checks what was lost, and how does the business keep working meanwhile? That question carries more weight than a failed request for a test database. Real work and data that cannot simply be recreated are now involved.
In the earlier article (part three) I separated a limited service expansion trial from permission to run production work. Now assume the same product team asks to put an application database for real operations on the platform. That request needs a fresh decision, even if provisioning looks the same and the earlier trial worked.
The team, service, and incident in this article continue the hypothetical case. They are not test or production results from any organization. I use this case to explain that before accepting production work, we need agreed limits on harm and evidence that people and systems can respond within them. "Production" thus refers to specified work and conditions, not a certification label for the entire platform.
Start with what the business must be able to keep doing
Before discussing which machines host the database, I would ask the work owner what the application does, when it cannot stop, whether work can continue if it can read but not write, and which lost data could not be recreated. The answers change recovery design, service cost, and staffing.
Then write a service agreement that users and operators understand the same way: which usage and workload are supported, when incidents are handled, where to report them, who may limit use, and who decides that business work can resume. If the work needs after-hours response but the plan has only occasional help when someone is free, close that gap before promising a launch.
The database is online. Who confirms the business can resume?
- Business work owner
- Agrees on tolerable impact, temporary work methods, and conditions for resuming operations
- Application team
- Checks reads and writes, transaction correctness, and work left to process
- Platform service operators
- Handle incidents, coordinate recovery, manage permissions, and hand over evidence between shifts
- Underlying provider
- Carries the duties in the purchased service terms; check who restores which parts and what limits apply
The agreement should also say what the standard service covers and when more discussion is required. Sensitive data, specific retention rules, or a system involving several parties may need different handling from the currently supported database. Each new exception needs someone to carry the work and a budget for it.
Measure work users can complete
Google's Implementing SLOs explains that service indicators should reflect user experience and objectives should be agreed with those affected. An SLI is the measure chosen; an SLO is its target over a defined period.
For the hypothetical service, I would measure successful database requests separately from an application's ability to read and write as agreed. A healthy provisioning screen can coexist with users unable to save work. Define which requests count, where measurement happens, which failures count, and the time window. Do not use the provider's availability figure as a direct measure of the application.
An SLO is an objective for operating and deciding about a service. Setting one does not automatically create a contractual guarantee or promise that the system will never stop. I would not invent a percentage for this hypothetical service before knowing the work, the impact, and demonstrated capacity. If the business needs more than the team can provide, discuss alternatives or added budget before launch.
Choose how far back recovery may go and how long it may take
Two more terms need to be tied to actual impact: recovery time objective (RTO), the longest target outage until service is restored, and recovery point objective (RPO), the maximum target amount of recent data loss measured in time. AWS Well-Architected guidance on recovery objectives says to derive them from business needs and test feasibility, including dependencies.
In this hypothetical case, the work owner helps choose how long the work can stop and how much recent data can be lost. Some jobs may tolerate a longer recovery but cannot lose saved transactions. Different needs change design and cost; no single pair of numbers fits every system.
How far back can data be recovered, and when can work resume?
- Last recoverable data pointCheck backups, ongoing logs, and required keys together to see the actual point of recovery.
- Time of incidentMeasure the gap from the recoverable point to the incident and the transactions that may be missing, then compare it with the agreed RPO.
- Restore and check dependent systemsRestore data, reconnect access and network paths, check the application, and handle queued or duplicate transactions.
- Confirm service can resumeMeasure outage until work is usable under the agreed definition, then compare with the agreed RTO.
"Backup completed" does not tell us whether restoration works. I would test recovery in an isolated area with controlled access, starting with data created for testing, and check that the restored application can read and write. If real data is needed for a test, assess and approve access separately for that data's scope.
Time the whole agreed process, including detection, contacting the responsible person, obtaining access, restoring the database, and checking business work. Do not time only the restore command and present that as the business's entire wait. A small data set's result does not establish the same timing for a different production volume and workload.
See incidents that require action, and change systems with a recovery path
Dependencies may include networks, identity systems, storage, and external services. Testing only a healthy database leaves many questions. I would simulate incidents important to this case in an authorized isolated area, such as an application unable to connect or an identity service outage. What do users see? Where is work queued? Can the responder continue the investigation?
Monitoring needs to help answer those questions, from failed requests to related causes, without recording unnecessary secrets. Alerts should reach someone who owns a response and can act. Google's Alerting on SLOs focuses on events that materially affect service objectives. For the hypothetical team, I would check whether responders know where to look, what they can do themselves, and whom to call for help.
A system change also needs a data plan. Rolling an application back may help with bad code, but if the new version changed the database schema or discarded data, rolling code back does not restore the data. Check compatibility of old code with the new schema, rehearse migration, and agree when to stop, fix forward, or restore data, including what happens to records written after the change.
Before accepting real data, separate access for tests, production, operators, and application accounts. Check that forbidden access is denied and that revocation takes effect. Follow the same boundaries through backups, logs, and recovery areas. Different screen or database names alone do not prove data is isolated.
Use evidence to decide when to move the work
A review before accepting production should reflect the service's architecture, risks, and capacity. The Production Readiness Review concept in Evolving the SRE Engagement Model helps identify reliability checks before taking on operations. I would adapt it into a discussion among the work owner, application team, and service operators, with an owner for every open item.
The next table is a fictional team record to show how evidence should change the decision. Its missing evidence is invented for teaching, not a finding about a running platform.
| Decision area | Evidence in the hypothetical case | Next work and owner |
|---|---|---|
| Recovery | A backup-success report exists, but no test has shown that the application works with restored data. | Service operators and the application team rehearse recovery, check data and queued work, then compare with RTO/RPO agreed by the owner. |
| Incident response | Someone responds during business hours, but the proposed work also needs after-hours coverage. | The budget owner and service operators agree on staffing, a backup responder, and cost before accepting this scope. |
| Version changes | Application rollback exists, but the team has not checked what happens when a new version changes data structure. | The application team and service operators test compatibility and data repair before allowing such a change. |
| Access and data | Test and production accounts are separate in the design, but denial and revocation have not been tested. | Access operators and data owners check the actual use paths, including backups and recovery areas. |
| Decision this round | Recovery and incident-coverage gaps still affect the requested work scope. | Do not move production yet. Rehearse in an isolated area, close material gaps, then have the work owner and operators review together. |
Once evidence supports the agreed scope, begin transition with work whose impact can be limited. Define stop conditions, decision makers, and data handling during a stop. Say what to observe and when to review. One recovery drill may answer the case it tested, but different incidents, data volumes, and later changes still need checking.
Reserve capacity for routine operations too: headroom for expected load, updates, recovery drills, access reviews, and incident coverage. A budget that pays only for the database and none of this time sends the responsibility back to the same overloaded person.
For the table's example, I would not set a production migration date until I see recovery results and incident ownership is agreed. Before approval, I want to know what work will begin, what data may enter, who responds, and what evidence shows the team can carry that work's risks. If one answer is missing, name the task and owner before revisiting the decision.
Source and scope
This article uses SLI/SLO, alerting, and pre-operations review concepts from Google's linked SRE books and AWS's linked recovery-objective guidance. The scenario, roles, diagram, table, and decision proposal are my application to a hypothetical case, not joint requirements or certification criteria from those sources. A real service still needs direct review of the provider's terms, data, and work risks.