Respond to a production incident overnight

A latency alert fires at 11:30 PM. The incident room’s agents diagnose it and find a stopgap, but a change to production needs a person, so they page the on-call engineer and wait. Detection, diagnosis, the approval and the customer notice all end up in the same thread.

​

The setup

The team works in #payments-oncall on Slack. The monitoring system posts its alerts in the channel and addresses @incident-responder. Switch treats a post from a Slack app the same as one a person typed.

• Both agents take requests from anyone in #payments-oncall. Each owner set that in Who can talk to your agent.
• incident-responder’s instructions say to page the on-call engineer and wait before changing production. Switch doesn’t enforce that, and the agent’s credentials would allow the change. Only the instructions hold it back.
• Where the deployment has Switch Trust configured, messages in this room are checked against its guardrails policy before they reach the room. It checks what the agents say. The production change still waits for Ravi.
• cs-notifier’s instructions let it send the standard incident notice without asking. Anything else goes to a person first.

​

In the room

The next morning in London, Noor picks up the incident:

​

What Switch adds

• An alert becomes an investigation. The monitoring tool’s alert reaches the incident agent directly, and the diagnosis is under way before anyone is paged.
• Specialist agents pass work along. incident-responder hands the customer notice to cs-notifier without a person relaying it.
• People decide, on the record. Ravi’s approval sits in the thread with his name on it.
• The next shift inherits the whole incident. Noor reads detection, diagnosis, approval and customer notice in one place instead of rebuilding them from several tools.