Large fleets (100k–700k devices)
The defaults suit a few thousand devices checking in every minute. For a large fleet, the load is set mostly by how often devices check in and how often they refresh posture. 3.7.0 adds the settings below and spreads load by itself:
- Start and interval jitter: each agent waits a random 0–60 seconds after start and varies every interval by ±10%, so a reboot wave or service restart does not arrive as one spike.
- Heartbeats: routine heartbeats that change nothing are no longer audited.
- Posture reads: they come from a one-row-per-device-and-question table instead of the message history.
- Cleanup: runs in small batches with a time budget.
Recommended settings
App setting (Fleet__…) | Default | 700k devices | Why |
|---|---|---|---|
PollIntervalSeconds | 60 | 300–600 | How long an idle device waits between check-ins. 60 s ≈ 170 requests/s per 10k devices; 600 s at 700k ≈ 1,200 requests/s |
ActivePollIntervalSeconds | 15 | 15–30 | Used only while a device has work queued, so answers still arrive quickly |
PostureRefreshIntervalMinutes | 240 | 1440 | Each refresh is 7 questions per device (8 with posture custom queries) |
MaxInFlightRequestsPerTenant | 1000 | 50000–100000 | Caps queued and leased work per tenant. At 1000, a posture refresh of 700k devices takes days to drain |
CompletedMessageRetentionDays | 30 | 7 | Posture no longer reads message history, so it can be kept short |
AuditRetentionDays | 400 | your policy | Audit events older than this are pruned |
AuditRoutineHeartbeats | false | false | Audit every heartbeat (true) or only changes (false) |
MaxPruneBatchesPerPass, PruneTimeBudgetSeconds | 50, 20 | default | Bounds each cleanup pass so it never holds long locks |
AuditPruneBatchSize, AttestationChallengePruneBatchSize | 5000 | default | Rows deleted per batch |
MaxAgentsPerQuery | 100 | 10000 (the maximum) | Most devices one ad-hoc query may reach, for every target type. A query that would reach more is refused (HW-QUERY-001), never sampled. Target a tag to stay under it |
Recommended sizes
Starting points for the -Size choices in the deploy script.
How they were worked out
- Request rates are devices ÷ interval for heartbeats and polls, plus posture answers (7 per device per refresh).
- Each request's database cost comes from the bench below (about 1.2–1.6 ms per query at 600k devices; a heartbeat or poll is 2–4 queries, a result about 3 plus its writes).
- The servers and database are sized so the estimate sits at or below about 60% busy, with room for portal use and the daily snapshot.
This is a capacity model built on workstation measurements, not a load test in Azure. Watch Azure SQL CPU/DTU % and Log IO % and the App Service CPU and HTTP 429 metrics for the first week and adjust.
| Devices | -Size | Heartbeat / poll | Requests / s | Azure SQL | API (App Service) | Portal | Posture refresh | MaxInFlightRequestsPerTenant |
|---|---|---|---|---|---|---|---|---|
| up to 1,000 | Small | 300 s / 60 s | ≤ 20 | Standard S1 (20 DTU) | B1, 1 instance | same plan | 240 min | 1,000 (default) |
| 25,000 | Medium | 300 s / 120 s | ≈ 300 | Standard S4 (200 DTU) | P2v3, 1 instance | same plan | 240 min | 10,000 |
| 150,000 | Large | 300 s / 300 s | ≈ 1,000 | General Purpose 4 vCores | P3v3, 2 instances | same plan | 720 min | 50,000 |
| 300,000 | between Large and XLarge | 600 s / 600 s | ≈ 1,050 | General Purpose 8 vCores or Hyperscale 8 | P3v3, 2–3 instances | P1v3 | 1440 min | 75,000 |
| 600,000 | XLarge | 600 s / 600 s | ≈ 2,050 | Hyperscale 8 vCores | P3v3, 3 instances (24 cores) | P1v3, 2 instances | 1440 min | 100,000 |
Correction to earlier versions of this page: the old table counted only polls. Heartbeats are separate requests (HeartbeatIntervalSeconds, default 300), so at 600,000 devices they alone are 1,000–2,000 requests/s.
At the defaults (heartbeat 300 s, poll 60 s) 600,000 devices would send about 12,000 requests/s. That is why Large and XLarge lengthen both intervals, and why the default tenant limit (120,000/min) is raised to 600,000/min for XLarge.
Assumptions that matter:
- About 1–5 ms of API CPU per request (JSON, EF Core, token checks; XLarge's 24 cores cover the high end with one server down).
- Database CPU about 0.5–1 ms per request.
- One tenant.
- Devices spread evenly in time (agents already add ±10% jitter).
At the 600k tier, the whole-fleet scan and the device-list sorts are the heaviest work; schedule them off-hours. If your devices run extra checks (App Control evidence, firewall rounds, self-tests), raise one step.
- Rate limits.
Fleet__RateLimiting__TenantRequestsPerMinute(default 120,000, about 2,000 requests/s) caps a whole tenant: keep it at least twice the check-in rate × 60.Fleet__RateLimiting__CallerRequestsPerMinute(default 600) caps one operator or one device. Devices are separate callers only with device certificates or client certificates; with the shared agent registration every device is one caller, which suits small pilots only; use device identity above a few hundred devices. - Database growth. Plan about 2–3 KB per device for the device row and latest posture, plus finished requests for
CompletedMessageRetentionDaysand audit forAuditRetentionDays. 700k devices with 7-day request retention and heartbeat auditing off fit comfortably under 100 GB. - Keep views to a slice. Use tags or saved views in the portal: the posture report judges up to 10 pages of 5,000 devices live, each read from where the last ended (a cursor), and the Devices report shows 500 rows. Whole-fleet numbers come from the daily snapshot; whole-fleet files come from scheduled exports.
- Background reporting. The daily snapshot and each whole-fleet export read every device twice in pages of
ScanBatchSize; memory stays flat, but at 700k each takes about 10–11 minutes on the bench and loads SQL for that time. SetFleet__Reporting__SnapshotHourUtcto a quiet hour, keep whole-fleet export schedules few (they run one after another), and prefer tag-scoped exports for daily files. - Region and zones. Put SQL and both web apps in one region; turn on zone redundancy for SQL (GP/BC/Hyperscale) and App Service (Premium v3, 3+ instances) for production.
Ad-hoc query or posture? An ad-hoc query asks devices now and is capped at MaxAgentsPerQuery devices (at most 10,000); use it for a tag, a site or an investigation. For a question about every device, mark the custom query Include in posture: each device then answers it on its own posture schedule and the result appears on Posture for the whole fleet, with no fan-out limit.
Measured
Latest run, 600,000 devices: 4.2 million posture answers and 2 million finished requests, seeded in 28 minutes on SQL Server Express (LocalDB, about 4 cores and a small buffer pool) on an 8-core workstation.
The 600k column comes from that run. Express has far less memory than the Azure tiers above, so cold-cache and sort timings are pessimistic and vary run to run; the per-query check-in costs are the numbers that carry over. The check-in rows for that column were not re-run and show the 700k run.
Measured with tools\Highwind.Fleet.ScaleBench (the real service code against SQL Server LocalDB on one developer workstation, median of repeated runs; 700k devices, 4.9 million posture answers, 3 million finished requests).
The paths every device hits on every check-in stay at a few milliseconds at 700k, so check-in load is bounded by request rate, not table size. Portal views over the whole fleet take seconds; filter by tag or search by name to keep them fast. Azure SQL will differ; run the bench on a dedicated database at your production tier to size it.
| Operation | 20k devices | 700k devices | 600k devices (later run) |
|---|---|---|---|
| Device check-in: lease lookup (per poll) | 3 | 3.1 | 1.3 (700k run) |
| Device check-in: posture due check (per poll) | 3 | 2.6 | 1.2 (700k run) |
| Device check-in: tenant in-flight count | 30 | 2.0 | 1.2 (700k run) |
| Posture answer upsert (per result) | 3.5 | 5.1 | 1.6 (700k run) |
| Device list, first page (name order) | 28 | 191 | 114 |
| Device list, name prefix search | 7 | 11 | 1,384 (cold cache) |
| Device list, one tag | 105 | 1,396 | 1,064 |
| Device list, operator limited to 2 tags | 84 | 1,013 | 2,338 |
| Device list, sorted by last seen | 114 | 4,810 | 1,776 |
Device list, page at device 600,000 (offset, API without after) | 16 (deep page) | 4,423 | 1,864 |
Device list, page at device 600,000 (cursor, after; the portal's Next) | — | 127 | 155 |
| Posture report, first 5,000 devices (35,000 answers, 13 MB) | ~1,100 | 886 | 772 |
| Posture report, 5,000 devices from device 600,000 (offset) | — | 6,959 | 3,079 |
| Posture report, 5,000 devices from device 600,000 (cursor; the portal's page 2–10) | — | 521 | 800 |
| Tag list for the baseline page | 30 | 615 | — |
| Query fan-out, 100 devices | 73–220 | 123 | — |
| Query fan-out, 10,000 devices | — | 7,391 | — |
| Whole-fleet daily snapshot (two streaming passes, 73 tags) | — | ≈ 640 s, 71 MB heap | ≈ 736 s, 126 MB heap |
| Whole-fleet scheduled export, devices CSV (700,000 rows) | — | ≈ 650 s | — |
dotnet run -c Release --project tools\Highwind.Fleet.ScaleBench -- --agents 700000 --connection "Server=tcp:<server>;Database=HighwindScaleBench;Authentication=Active Directory Default"Seeding deletes and recreates the whole database, so the bench refuses any database whose name does not contain Bench. Give it a dedicated database on the same Azure SQL tier as production; never production itself.
Protecting the service
The API is an internet-facing HTTPS endpoint that every device reaches. Highwind defends the application layer itself; protection against volumetric attacks and network-level controls come from Azure, and you choose how much you need.
Built in
- Nothing anonymous does work. Every route needs an Entra token (operators, agents) or a device-signed request (device certificates). The only anonymous routes are health and the device-attestation challenge, which are rate limited to 20 per minute per IP.
- Rate limits with
429. Per tenant (120,000 requests per minute, configurable) and per caller (600 per minute, configurable); per caller also heartbeats and check-ins 20, requests 60, queries 10, model egress 20 and enrollment 10 per minute. A limited device simply tries again at its next interval. - Bounded work. Request bodies are size-capped, ad-hoc queries are capped at
MaxAgentsPerQuery, in-flight work is capped per tenant, and the device never accepts anything but its allowlisted read-only operations. - No inbound path to devices. Devices only ever call out; the service cannot connect to them, so a compromised service cannot reach into the estate.
- Platform DDoS protection. Every Azure public endpoint, including App Service, has Azure’s infrastructure DDoS protection at no cost. It absorbs large network-layer floods but is not tuned to your app.
Recommended for production
| Control | What it adds | When |
|---|---|---|
| Azure Front Door Premium + WAF in front of the API and portal | Global edge, managed OWASP rules, bot protection, per-IP rate limiting before traffic reaches App Service, geo filtering. Lock App Service to Front Door with an access restriction on the AzureFrontDoor.Backend service tag and the X-Azure-FDID header | Any fleet reachable from the internet; strongly advised above 10k devices. In Azure Government use Front Door or Application Gateway WAF v2 where available in your region |
| Application Gateway WAF v2 (regional alternative) | WAF and rate limiting inside your virtual network; pairs with private endpoints | When traffic must stay regional or private |
| Azure DDoS Network Protection | Adaptive tuning, attack telemetry, rapid response, cost protection for the virtual network that holds the gateway | Regulated or high-value fleets |
| Private endpoints for Azure SQL and Key Vault; public network access off | The database and CA key are unreachable from the internet | Always for production |
| App Service access restrictions | Only your gateway (or your networks) can reach the API; turn off the SCM site’s public access | Always for production |
| Conditional Access on the portal and API registrations | Compliant device, MFA and location for operators; operators are where a real attacker would aim | Always |
| Alerts | Azure Monitor alerts on HTTP 429 and 5xx rates, SQL CPU, and audit events such as AgentRegistrationDenied and DeviceAttestationRejected spikes | Always |
Proxies on the device side
The agent uses the .NET HTTP stack. It honours the machine-wide HTTPS_PROXY, HTTP_PROXY and NO_PROXY environment variables and the proxy settings of the account it runs as (LocalSystem); it does not read netsh winhttp settings. To send it through a proxy, set the variable machine-wide and restart the service:
[Environment]::SetEnvironmentVariable("HTTPS_PROXY", "http://proxy.contoso.com:8080", "Machine")
Restart-Service HighwindFleetAgentAllow the API host and the Entra sign-in host (login.microsoftonline.com or login.microsoftonline.us). With device certificates, requests are signed inside the message, so a TLS-inspecting proxy that the device trusts does not break them; if you use the older client-certificate (mTLS) mode, exempt the API host from TLS inspection.