Alerts, metrics and the API¶
New in the next release
This is on the dev and qa builds now, and in the next release.
Alerts¶
| Alert | Key | Tab | Level | When |
|---|---|---|---|---|
| Capacity runway short | capacity_runway |
Capacity | warning, critical | A cluster, datastore, pool, backup store or volume is forecast to reach its threshold within the warning runway (90 days unless set): a warning. Within the critical runway (30 days), or over its threshold already: critical |
| Platform license expiring, expired or over-used | platform_license |
Virtualization | warning, critical | A vCenter license expires within 30 days (warning), has expired (critical), or has more assigned than it holds (warning). The evaluation license never alerts |
| PDU near its rating | pdu_load |
Power and servers | warning, critical | A PDU's draw passes 80% of its rating (warning), or 95% (critical) |
Hosts never raise runway alerts: a cluster balances them. A backup store whose retention is still filling says in the alert that the date may be early.
They're ordinary alerts: on Needs attention, counted on their tab, in
the alert history, and sent through your notification rules, which can name
the tab (capacity) or the key, such as NOTIFY_RULES=capacity_runway=bad.
Maintenance mode holds them like any other.
Metrics¶
On /metrics, and in VictoriaMetrics, with the usual infra_ prefix.
| Metric | Labels | What it is |
|---|---|---|
infra_capacity_used_ratio |
source, kind, name, resource |
Use of usable capacity on the latest day: CPU and memory at their 95th percentile, disk its average, allocation against its limit |
infra_capacity_threshold_ratio |
the same | The threshold the forecast is for |
infra_capacity_runway_days |
the same | Days until the threshold; 0 when over it. Absent when not rising, not within a year, or too little history |
infra_pdu_up |
source, pdu |
1 when the PDU answers |
infra_pdu_current_amperes, infra_pdu_rated_amperes, infra_pdu_power_watts, infra_pdu_load_ratio |
source, pdu |
The PDU's draw, rating, power and load |
infra_pdu_outlet_on, infra_pdu_outlet_current_amperes, infra_pdu_outlet_power_watts |
source, pdu, outlet |
Each outlet |
infra_ipam_subnet_free_addresses, infra_ipam_subnet_size_addresses |
source, subnet |
Free and usable addresses, as the IPAM records them |
infra_dcim_rack_free_units |
source, rack |
Free rack units, from NetBox |
kind is cluster, host, datastore, pool, backup or volume.
resource is cpu, mem, disk, alloc_cpu, alloc_mem or
alloc_disk.
A Grafana panel of the shortest runways:
bottomk(10, infra_capacity_runway_days)
Capacity metrics need a license; the others are on every edition.
The API¶
Read with the same session or token as the dashboard. The capacity ones need a license and answer 403 without one.
| Method and path | Returns |
|---|---|
GET /api/capacity |
Every entity with its resources and forecasts, the reclaim list, storage growth, the summary's status, and the thresholds, runway, ratios and VM size in force |
GET /api/capacity/history?key=<key>&resource=<resource> |
One resource's daily history, for the chart |
GET /api/capacity/vms?key=<cluster key> |
A cluster's VMs, busiest first, for moving VMs |
GET /api/capacity/expand?key=<key>&disks=<n>&tb=<size> |
What n disks of that size add to a pool or volume, and its layout. n from 1 to 200 |
GET /api/capacity/readiness?key=<cluster key> |
The rack check for one more host |
GET /api/capacity/summary?month=YYYY-MM |
The monthly summary page. Add &download=1 to save it as a file |
GET /api/capacity/settings |
Settings, with what's built in and what's set in the stack (admins) |
POST /api/capacity/settings |
Save thresholds, spare hosts and the summary's settings (admins) |
POST /api/capacity/summary/send?month=YYYY-MM |
Email a month's summary now; last month without month (admins) |
GET /api/plant |
Reserved ports and racks (admins) |
POST /api/plant/port |
Reserve or free a port: {"device": "<MAC>", "port": 6, "note": "…", "release": false} (admins) |
POST /api/plant/racks |
Replace the racks: {"racks": [{"name": "R1", "free_u": 3, "host_u": 1, "holds": ["esx01"], "ups": "rack-ups"}]} (admins) |
A key is source|kind|name, such as vcsa|cluster|Prod or
tank|volume|tank, as /api/capacity gives it. URL-encode the | as
%7C.
Every change through the admin endpoints is in the audit log:
capacity_settings_changed, capacity_summary_sent, port_reserved and
racks_changed.
The files on the volume¶
Everything is kept in files beside the outage archive, on Ferrule's own
volume (/var/lib/ferrule), so it's in backups and survives
updates.
| File | What's in it | Kept |
|---|---|---|
capacity/capacity-YYYY-MM.jsonl |
One line per thing per day: averages, 95th percentiles, peaks, capacity, allocation | Two years |
capacity/vm-YYYY-MM.jsonl |
One line per VM per day, for reclaim and moving VMs | About four months |
capacity/today.json |
The day so far, saved every ten minutes so a restart loses little | Until midnight |
capacity/seeded |
When history was first read back from old inventories, so it's done once | Always |
capacity/summary-sent |
The last month the summary was emailed | Always |
Reserved ports, racks and capacity settings are in the settings store, not here.