Why “monitoring is set up” isn’t the same as “monitoring works”
“Monitoring is enabled” can be true while the reality is dangerously different: the agent stops checking in, notifications quietly fail, an old on-call number still routes, or alerts arrive but no one is accountable to acknowledge them.
For Manhattan SMBs, this risk is amplified by after-hours building access, doorman policies, freight-elevator schedules, and the fact that many incidents require coordination with vendors, landlords, and ISPs—not just technical troubleshooting.
What this 2‑hour acceptance test proves (and what it doesn’t)
This go-live drill is designed to validate the *end-to-end chain*: device/service detects an issue → monitoring platform generates an alert → notifications reach the right humans → the alert is acknowledged within an agreed window → escalation triggers if not → the client can see evidence afterward.
It does not replace deeper technical remediation work (patch baselines, backup restores, security hardening). It simply prevents a common failure mode: a “managed” environment where nobody actually gets the message when something breaks.
Who should participate (minimum viable room)
You need one MSP engineer running the monitoring console, one MSP on-call contact (or dispatcher) receiving alerts, and one client-side point person who can confirm what was seen and when.
If your environment has strict access rules, include a building contact path (security desk/doorman/management office) and make it explicit whether after-hours access requires pre-authorization.
The ground rules: define pass/fail before you start
Agree on a simple scorecard with time stamps. The “test” is only credible if the expectations are written down first.
Define:
- Which systems are “in scope” (a small but representative set is fine)
- Which alerts count (availability, disk capacity, UPS, WAN)
- Notification targets (email, SMS, phone call, app push)
- Acknowledgement SLA for the drill (for example: acknowledge within X minutes)
- Escalation behavior (who is next, and after how long)
What evidence should be saved (signoff artifacts)
Ask for artifacts that a non-technical owner can understand later. Evidence should show both *that* an alert triggered and *who* responded.
Artifacts can include:
- Ticket creation with timestamps
- Screenshot/export of alert details (device, metric, threshold)
- Screenshot/export of notification rule / recipient list
- Call/SMS logs or on-call app acknowledgement record
- Short post-drill summary listing each scenario, expected result, actual result, and any fixes
The 2‑hour “Prove the Pager Works” drill (scripted and time-boxed)
This structure keeps the session controlled: short setup, four simulations, then a debrief and remediation list. The specific scenarios below are chosen because they are common, easy to reproduce safely, and good at surfacing silent failures.
Step 1: Pre-flight (15 minutes)
Confirm you’re testing the real alert paths, not a demo channel or an engineer’s personal inbox.
Validate:
- The MSP’s on-call contact method for *after-hours* (not just business hours)
- The client escalation contact(s) and alternates
- The monitoring “heartbeat” mechanism (how it detects an agent is dead)
- Ticketing linkage (alerts generate tickets with correct priority)
Step 2: Run four controlled simulations (80 minutes)
Each simulation should follow the same mini-format: start time → trigger → expected alert → who receives it → acknowledgement time → escalation if missed → what the client sees.
Keep each simulation to ~20 minutes: 5 minutes to trigger, 5–10 minutes to observe delivery and acknowledgement, 5 minutes to reset.
Simulation A: Silent failure drill — monitoring agent offline
Purpose: prove you’ll know when monitoring stops monitoring.
How to simulate (safely): stop the monitoring agent/service on one test machine or disconnect it from the network.
Verify:
- An “agent offline/no heartbeat” alert triggers within the expected window
- It routes to on-call (not just the project engineer)
- A ticket is created with the right asset name and site
- Acknowledgement happens within the agreed time
- The alert includes enough context to act (device, last check-in, IP/site)
Simulation B: UPS on battery (power event signal)
Purpose: prove you’ll get actionable warning before a shutdown, and that the alert isn’t buried as “informational.”
How to simulate: if you have a UPS management interface, temporarily move the UPS input to a controlled test condition only if safe and approved. If you can’t safely simulate battery mode, simulate the monitoring signal by temporarily disabling UPS communication to produce a meaningful alert.
Verify:
- Alert severity/priority matches your business impact expectations
- Notification includes location (closet/floor/suite) and runtime estimate if available
- The client escalation path is clear if physical intervention is needed
Simulation C: Disk space nearing full (capacity risk)
Purpose: prove threshold alerts are set to warn *before* outage and that they don’t spam at the wrong level.
How to simulate: create a large temporary file on a non-critical volume or adjust a threshold temporarily on a test device.
Verify:
- Two thresholds behave correctly (warning and critical) if you use them
- Alerts are deduplicated (no endless repeats)
- The ticket captures the volume/drive and % free
- Acknowledgement and notes show a plan (cleanup/expansion) rather than “closed—no action”
Simulation D: ISP flap / WAN instability (connectivity issue)
Purpose: prove you can distinguish “site down,” “ISP shaky,” and “single device offline,” and that escalation includes who can approve ISP calls.
How to simulate: briefly disconnect WAN on a firewall in a controlled window, or simulate by disabling an uplink interface if your network team is present and approves.
Verify:
- The alert clearly identifies the site and the upstream dependency
- On-call receives it quickly via the agreed method
- The escalation path includes the right ISP account-holder contact (client-side)
- The client can see a record of the event for later vendor conversations
Step 3: Debrief and fix list (25 minutes)
End the session with a short, written “acceptance memo.” This prevents the drill from becoming a one-off meeting with no operational change.
The memo should include:
- Each scenario and pass/fail result
- Acknowledgement times and escalation behavior
- Any misrouted alerts (wrong person, wrong channel)
- Any missing context (unclear device names, no location data)
- A remediation list with an owner and due date
Client-facing visibility: what you should be able to see afterward
Even if you never log into the MSP’s tools daily, you should be able to review what happened.
Ask for a client-friendly view of:
- The tickets created from the drill (with timestamps)
- The alert history for the tested devices/services
- The on-call acknowledgement record (who, when)
- The contact and escalation list used during the test
A practical acceptance scorecard you can reuse
Use a simple table (even in a shared document) with objective fields. This keeps everyone aligned and avoids “it seemed fine.”
Recommended columns:
- Scenario name
- Trigger time
- Alert generated time
- First notification received time
- Acknowledged by (name/role)
- Acknowledged time
- Escalation triggered? (Y/N) and when
- Ticket ID
- Evidence link (screenshot/export)
- Pass/Fail and notes
- Confirm on-call route(s) for after-hours: SMS/phone/app and backup channel
- Confirm escalation ladder: primary, secondary, and “cannot reach anyone” procedure
- Pick 3–5 representative devices/services for the drill (not everything)
- Run the four simulations (agent offline, UPS/power signal, disk capacity, WAN flap)
- Save artifacts: tickets, alert screenshots/exports, acknowledgement logs
- End with a written acceptance memo + remediation list
Common ways this goes wrong (and how to prevent it)
The most frequent failure is testing with the “project team” rather than the actual operations rotation. Make sure the people who will hold the pager next month are the ones receiving and acknowledging alerts today.
Another failure is ambiguous asset naming—alerts arrive, but nobody knows which “SERVER-01” it is or where it lives. Require a location-aware naming standard (suite/floor/closet) for anything that might require hands-on work.
A third failure is treating notification delivery as “someone saw it eventually.” Your drill should measure acknowledgement time and confirm escalation triggers when the first contact does not respond.
Key Takeaways
- Monitoring isn’t real until you’ve tested alert delivery, acknowledgement, and escalation end-to-end.
- Include a *silent failure* scenario (agent offline) to prove you’ll know when monitoring breaks.
- Time-box the drill to two hours, capture evidence, and produce a written acceptance memo.
- NYC after-hours constraints (access rules, vendor handoffs) should be part of the test plan.
Frequently Asked Questions
Do we need to run this after-hours to be realistic?
Not necessarily, but you should test the *after-hours routing* (the real on-call rotation and escalation ladder). Many teams run the drill late afternoon and confirm that the same on-call path would trigger at night.
What if we can’t safely simulate a UPS battery event or WAN outage?
Don’t force risky tests. Substitute a safe signal simulation (for example, temporarily breaking UPS monitoring communication, or disabling a secondary uplink in a controlled way) and focus on the acceptance goal: does the right alert reach the right people with the right context?
How many endpoints should be included in the drill?
A small, representative set is better than “everything.” Pick a server/workstation, a network edge device, and anything that represents a single point of failure (firewall, core switch, key line-of-business host).
What acknowledgement time should we require?
Set something that matches your risk tolerance and coverage model. The key is to define it in advance and confirm escalation works if it’s missed—because the escalation behavior is often the real point of failure.
What deliverables should we insist on from the MSP?
At minimum: the scorecard, ticket IDs, alert screenshots/exports, acknowledgement logs, and a short acceptance memo with any fixes and due dates. If you can’t audit the outcome later, the drill didn’t happen in a meaningful way.
Take the Next Step
If you’re switching providers—or you’re not fully confident your current monitoring will actually wake someone up—schedule a go-live acceptance session and require proof.
Your Expert Tech can help you run a structured 2‑hour monitoring + escalation acceptance test, document the results, and turn any gaps into an actionable remediation list. Contact us to set a time window and we’ll provide a drill script you can use for signoff.