Device Fleet Monitoring and Health
Device fleet monitoring guide: what to track, dashboards and alerting, heartbeats, automated recovery, and capacity planning for phone farms.
Device fleet monitoring is the practice of continuously tracking each device's availability, health, and session state so problems surface as alerts and common failures self-heal. A fleet without monitoring degrades silently; this page covers what to measure, how to alert, and how to automate recovery.
- Heartbeats are the backbone — a device that stops reporting is a problem regardless of the underlying cause.
- Effective availability requires a healthy automation session, not just power and network connectivity.
- Build automated recovery (reconnect, restart app, reboot) before manual firefighting.
- Alert on trends, not just thresholds — a slow battery-health decline matters as much as a hard offline.
- Health data feeds capacity planning: know real available headroom before scheduling work.
What to monitor
Treat each device as a small host emitting telemetry. The essential signals:
Availability (online/offline)
The most basic signal: is the device present and reachable? For Android, this maps to ADB device state (device, offline, unauthorized, or absent). For iOS, it's the host-agent connection to the device. Track state transitions, not just current state — flapping between online and offline points to cable, power, or hub problems (see USB hubs and charging at scale).
Battery and thermal health
- Battery % — for duty-cycling and to catch devices that won't hold charge.
- Battery temperature — sustained high temp signals thermal or charging issues.
- Battery health/capacity — a slow-moving metric that predicts end-of-life.
- Charging state — is a device that should be charging actually charging?
Set a hard alert on abnormal battery temperature and any device that reports charging but never gains charge. These can indicate a failing or swelling cell — investigate physically and remove the device if you find bulging or excessive heat.
Storage
Automation generates logs, screen recordings, and app data that fill devices over time. Low free space causes install failures and flaky sessions. Alert on a free-space floor and reclaim space automatically where possible.
Connectivity
- Network reachability — Wi-Fi/cellular up, can reach the endpoints jobs need.
- Throughput/latency if your workload is network-sensitive.
- IP/connectivity state relevant to your setup.
Session and ADB/host status
Beyond "is it online," track whether the automation session is healthy: is the ADB connection authorized, is the Appium/WebDriverAgent session alive, is the device responsive to input? A device can be powered and online yet have a wedged session that silently fails every job.
On Android this session state is visible directly from adb devices (device, unauthorized, offline). On iOS there's no equivalent single-command view — health comes from whether the macOS host still holds a live connection to the device and whether the WebDriverAgent process running on-device is still responding, so iOS monitoring typically leans more on the host agent's own reporting than on a device-side query.
App crashes and errors
Capture crash signals (Android logcat/ANRs, iOS crash logs) and job-level error rates. A device that suddenly starts failing a specific app flow needs attention even if every hardware metric looks green.
Monitoring matrix
| Signal | Alert on | |
|---|---|---|
| Availability | ADB state / host agent connection | Offline, flapping |
| Battery % | dumpsys battery / iOS API | Below/above duty-cycle band |
| Battery temp | Battery telemetry | High temperature |
| Battery health | OEM/iOS capacity reading | Slow decline, low capacity |
| Storage | Filesystem stats | Below free-space floor |
| Session status | ADB auth, Appium session | Unauthorized, dead session |
| App crashes | logcat, crash logs | Crash/ANR spikes |
Heartbeats
A heartbeat is a periodic "I'm alive and here's my state" report from each device (or its host agent) to a central collector. Design principles:
- Push, don't just poll where possible — devices report on an interval, and missing heartbeats are themselves the alert.
- Include a compact state payload (battery, temp, storage, session status) so one message covers most signals.
- Choose an interval that balances freshness against overhead — often tens of seconds to a couple of minutes.
- Treat N consecutive missed heartbeats as offline to avoid alerting on single blips.
Heartbeats turn "we didn't notice device 43 died three days ago" into an immediate, actionable signal.
Dashboards and alerting
Collect telemetry into a time-series store and visualize it. You don't need exotic tooling — a standard metrics stack (a time-series database plus a dashboarding layer) works well.
Dashboards should answer at a glance:
- How many devices are online right now vs total?
- Which devices are unhealthy, and why?
- Fleet-wide trends: battery health distribution, crash rates, storage pressure.
Alerting should distinguish severity:
- Page-worthy: many devices offline at once (likely power/hub/network), abnormal battery temperature.
- Ticket-worthy: a single stuck device, storage nearing floor, rising crash rate.
- Trend: gradual battery-health decline flagging replacement planning.
Alert on rates and trends, not only instantaneous thresholds — a fleet degrades gradually, and the early signal is usually a slope, not a cliff.
Automated recovery
The goal is to resolve the common failures without a human. Build an escalation ladder and try the cheapest fix first:
- Reconnect the session — re-establish ADB/Appium on Android, or the host-to-device connection and WebDriverAgent on iOS, before anything heavier.
- Restart the app — clear a wedged app state.
- Reboot the device — resolves many stuck states (offline ADB or a dropped iOS host connection, memory pressure, flaky sessions).
- Power-cycle via smart PDU/USB switch if a soft reboot doesn't recover it.
- Quarantine and page a human — after the ladder fails, remove the device from scheduling and open a ticket.
iOS recovery has one extra wrinkle: WebDriverAgent's signed build can expire or need reinstalling independent of anything wrong with the phone, so a "dead session" alert on iOS is worth checking against provisioning-profile validity before assuming a hardware problem.
Guard automated recovery so it doesn't mask real problems: if a device needs rebooting every hour, that's a signal to investigate, not to keep silently rebooting. Log every recovery action and alert on recovery frequency.
Recovery pairs tightly with scheduling — a self-healing device should re-enter the work pool automatically once healthy. See scheduling and orchestration for how monitoring state gates job assignment.
Capacity planning
Monitoring data tells you your real capacity, which is always less than your device count. Use it to:
- Compute effective availability — devices that are online, healthy, and session-ready, not merely powered.
- Track utilization to know how much headroom exists before you need more devices or hosts — for an iOS tier this includes macOS host capacity, since each host can only drive so many concurrent WebDriverAgent sessions before it becomes the bottleneck rather than the phones.
- Correlate failure rates with load to find the point where throughput starts costing reliability.
- Feed replacement planning from battery-health trends so you buy ahead of failures.
Putting it together
A healthy monitoring setup looks like: every device (or host agent) emits heartbeats with a compact health payload; telemetry lands in a time-series store; dashboards show live online-count and health distribution; alerts separate fleet-wide incidents from single-device tickets; an automated recovery ladder handles the routine failures and quarantines the stubborn ones; and capacity planning runs off effective availability, not raw device count.
Building and maintaining all of this is itself an ongoing engineering commitment, not a one-time setup — which is a large part of why teams weighing build vs. rent often move this whole layer onto a managed provider instead of running it in-house.