How forecasts work¶
New in the next release
This is on the dev and qa builds now, and in the next release.
This page is for anyone who needs to trust, or explain, a date.
What's recorded¶
Every poll of every source adds a reading to today's summary for each cluster, host, datastore, pool, backup store and volume: CPU and memory in use, disk used, and the capacity of each.
At midnight (local time, from TZ) the day is closed into one line:
- for CPU and memory, the day's average, 95th percentile and peak;
- for disk, the day's average (disk doesn't swing within a day);
- the capacity that day, and for clusters the size of the largest host and how many hosts there were;
- from the newest inventory, the allocation: vCPUs and memory given to VMs, and disk provisioned on each datastore.
Daily summaries are kept for two years, in files on Ferrule's volume (see the files). That's independent of how long VictoriaMetrics keeps its own history.
Why the 95th percentile?
A backup job that pins the CPU for ten minutes isn't a capacity problem. A cluster that's busy all afternoon is. The 95th percentile ignores the short spike and keeps the long one.
History from before¶
On its first start, Ferrule reads back every inventory it already holds and fills in disk and allocation history from them, as far back as they go. CPU and memory start from the first poll, because an inventory doesn't record load. The History column says how many days came from inventories.
Fitting the trend¶
For each resource, Ferrule takes the days since its capacity last changed, up to the last 90:
- Start after the last change of capacity. If usable capacity moved by more than 2% from one day to the next (a host added, a datastore grown, a host kept spare), the forecast starts from that day. The old trend described different hardware.
- Wait for three weeks. Fewer than 21 days and it says how many more it needs, never a guess.
- Fit a line that ignores odd days. The slope is the median of the slopes between every pair of days at least a week apart (the Theil–Sen method). One odd day, such as a restore, a test VM or a bad reading, barely moves it, where an ordinary average would be pulled off course.
- Plan for the busiest weekday. With four weeks or more, Ferrule works out how far above the trend each weekday usually sits, and adds the busiest. The date is when your busiest day crosses the threshold, and a Monday peak is never mistaken for growth.
- Give a range. The likely earliest and latest dates come from the steeper and gentler slopes among those pairs (the 90th and 10th percentiles). A steady trend gives a narrow range, and a noisy one a wide range. When the gentle end never reaches the threshold within two years, the range is open-ended.
- Look a year ahead. Past 365 days it says not within a year.
The date is recomputed for any threshold you type on the tab without refitting, so trying "70%" is instant.
Usable capacity¶
Forecasts are against usable capacity, which can be less than what's installed.
Keeping hosts spare (N+1)¶
A four-host cluster at 80% may not survive losing a host. Set the cluster to
keep one spare (on Settings, or CAPACITY_HA)
and its usable CPU and memory become the cluster less its largest host. The
forecast then dates when you lose the ability to absorb a host failure,
which comes well before "full". N+2 takes away the two largest.
Ceph¶
A Ceph pool is planned to 85%, Ceph's own "nearfull" warning; Ceph stops writes at 95%, so full is never 100%.
With four or more hosts, usable capacity also leaves room to heal: what's left once a lost host's data is re-replicated onto the rest, that is the raw capacity × (hosts − 1) / hosts. With three hosts there's nowhere to heal to under three replicas, so only the 85% applies.
Ceph's capacity is raw space, as Ceph reports it, so data written at three replicas uses three times its size.
Allocation and thin provisioning¶
The allocation rows are against limits you set, not against hardware:
| Row | Usable is | Default ratio |
|---|---|---|
| vCPUs allocated | Cores × ratio | 4 vCPUs per core |
| Memory allocated | Host memory × ratio | 1× (no overcommit) |
| Disk provisioned | Datastore capacity × ratio | 1.5× |
Thin-provisioned disks use less than their size, so a datastore can be 60% full and 140% provisioned. The Disk row forecasts real use; the Disk provisioned row shows the overcommit beside it: how much would be needed if every thin disk filled.
What counts¶
- Powered-off VMs count in allocation (their vCPUs and memory are reserved for them) but not in demand.
- Templates count in neither.
- Hosts in maintenance or not answering still count in capacity: they're your hardware, coming back.
- vCenter reservations, limits and HA admission control aren't read yet. If you rely on admission control, set the spare hosts on Settings to match.
Backup stores¶
A backup store grows while its retention fills, then levels off. A straight line through that would forecast growth forever. When a PBS datastore's retention is still filling, its row and any runway alert say the date may be early. See Storage and backups.
How sure is it?¶
- The range is the honest answer. A wide range says the trend is noisy.
- The History column says how much is behind it.
- A forecast can't know what you're about to do. Planned projects belong in What if.