At 3.10 a.m. the pager goes off for the fourth time this month, and it is the same engineer's phone. Arjun, a name we use only to make the story easier to follow, has taken most of the late pages for a year because he picks up fast and does not complain. Last week he accepted an offer from a cloud services firm. His manager is shocked. The rota sheet would not have been.
What the pattern looks like in the logs
- One or two engineers show most of the acknowledged pages, while others show none.
- Pages to a secondary are rare because the primary always answers.
- Swap requests are agreed informally on chat and never reach the rota.
- Alert noise is high: flapping disk alarms and duplicate tickets wake people for non-events.
- Comp-off is promised after a bad night, but nobody tracks whether it was taken.
This is quiet unfairness. It builds a hero, and heroes burn out or get poached.
What it costs a managed services firm
A tired primary misses a P1 alert or makes a risky change at dawn. The client sees a longer outage and a slow RCA. The firm sees an SLA credit and a tense QBR. Meanwhile, the engineers who never get paged lose their sharpness on the very systems they are paid to know. When the hero leaves, nobody else has ever carried the pager, and the first night without him is worse than any outage before.
Why the pattern repeats
- 1
The rota is built on trust, not data
Managers assign the person who answers fastest, because it works. That punishes reliability.
- 2
Allowance rules differ by site
Standby pay at one client desk, none at another. Engineers calculate which duty is worth taking and avoid the rest.
- 3
Nobody sees the burden
Without a monthly count of pages, night wake-ups and weekend hours, a manager's memory does the job, and memory favours the loudest.
- 4
Skill gaps feed the loop
Only a few people know a given stack well enough to take its pages, so they are always on.
A fix that fits in one quarter
- 1
Count first
Pull three months of PagerDuty-style page logs, Zabbix alerts or ticket timestamps. Tally by person: pages acknowledged, night hours, weekend hours, mean time to acknowledge. Show the table to the team.
- 2
Set a primary and secondary rotation
Publish it a month ahead, one week per person, with a named secondary. The secondary is paged only if the primary does not acknowledge within a stated window.
- 3
Write one allowance rule
State a standby amount per on-call day and a call-out rate for each page that needs hands on keyboard. Apply it at every site. Our pay design page explains how to build such a scheme.
- 4
Add a swap rule
A swap is valid when both names and the dates are entered in the rota tool before it takes effect, and when the manager sees it.
- 5
Cut the noise
Review the top ten alerts by volume. Retire duplicate tickets, tune SNMP thresholds, add auto-resolve and silence maintenance windows. A calmer pager is a retention tool.
- 6
Give recovery time
A page that runs past 2 a.m. earns a late start the next day, logged by the manager. No heroics.
Where the noise comes from
Walk through last month's alerts with the NOC wallboard open. Disk-space warnings that clear by themselves, SNMP traps from a flapping switch port, a nightly backup job that fails on a locked file and a certificate expiry email that arrives at midnight are the usual suspects. Each one wakes a person for work that could have waited, or that a script could have handled. Assign each noisy alert an owner and a fix date. Cut the top ten and the pager quietens by itself.
Make the junior ramp safe
Put a new on-call engineer through three stages. First, shadow: sit on the call with the primary and listen. Second, reverse shadow: take the page while the senior watches. Third, solo with a named backup who has agreed to be woken. Allow nobody into the primary seat before the runbooks for their stacks exist. Our companion note on covering absence in managed services deals with the same skill grid from the leave side.
Review it every month
- Publish a one-page report: pages per person, nights disturbed, swaps made, comp-off taken.
- Meet for half an hour with the team lead and one engineer from each shift.
- Flag anyone above the team average for two months in a row and rebalance next month.
- Include the figures in the manager's own review, so fairness is part of the job.
Pair this with the leave and shift process and the performance management page for managed service providers. More patterns for the sector sit on the IT infrastructure and managed services hub.
