increase() invents dataPrometheus only sees your counter at scrape points. To report a clean total over a window, it draws a line through the samples it has — and then projects that line out to the window's edges, into time it never observed. Drag the window below and watch the phantom appear.
When you ask for increase(cost_total[60s]), Prometheus runs its extrapolatedRate algorithm. The first three steps are sound. The fourth is a deliberate, documented guess.
Only the scrape points that actually fall within the range are considered — nothing from before rangeStart or after rangeEnd.
If the counter dropped (a pod restarted), Prometheus adds the pre-reset value back so the series looks monotonic. This part is genuinely helpful.
Rise over run, using only the two real endpoints it observed. This is exactly what xincrease() stops at.
The first sample almost never lands exactly on rangeStart, nor the last on rangeEnd. So Prometheus extends the slope into those gaps — counting increase in time it never scraped. That extension is the amber phantom in the chart.
The reasoning is reasonable: a window of real width 60s usually only contains ~57s of observed span, because the first sample sits a little inside the edge. Extrapolating "fills in" that missing slice so a graph of a steady counter reads smoothly instead of wobbling with scrape jitter. The Prometheus maintainers cap it (it won't extend more than ~half a scrape interval past either end), so it's bounded — but never zero.
For dashboards that watch trends, this is invisible and fine. For a number you put next to a currency symbol, it means increase() reads slightly high, and — because the padding is per-window — the total shifts when you change the bucket size. That's the interval-sensitivity you've already seen on the cost panels.
VictoriaMetrics ships xincrease() (and xrate()) precisely for this. Same first three steps — samples, reset-stitching, slope — but it stops at step 3. The result is exactly last observed − first observed within the window. No projection, no phantom, and the total stops drifting when you re-bin.
Extrapolates to window edges. Great for rates and trend graphs; over-reports totals and reacts to bucket size. There is no flag to turn the extrapolation off — the maintainers have declined to make it tunable.
Reports only the change between observed samples. Stable across window sizes, closer to true totals. Requires running VictoriaMetrics (or its engine) as the query layer — it's not available in vanilla Prometheus.
On the Bifrost panels you run sum(increase(bifrost_success_requests_total[$__interval])). 3 requests arrive on a brand-new label combo — the counter series didn't exist before them. At dashboard resolution the raw data looks like null, null, null, 3, 3, 3… — the series seems to be born already at 3. Yet the panel draws a bar of 1 at one aggregation interval and a bar of 2 at another, never 3. Two effects gang up here: the birth of the series is invisible to increase(), and $__interval chops whatever is visible into buckets. The amber bars below are exactly what your bar chart drew.
At 15s (= one scrape) you get two bars of +1 — the only deltas Prometheus ever saw — summing to 2. Nudge the alignment and every bucket traps a single sample: all bars vanish and the burst reads 0. Widen the interval and the deltas merge into one bar of +2, sometimes inflated toward 3 by the zero-point extrapolation. Flip on $__rate_interval: the splitting stops — but notice the result is an estimate hovering near 3, not a guaranteed count.
scrape interval (e.g. 15s) is your real data resolution — the atoms are the deltas between scrapes. Min step (min_interval, the datasource "Min step") is a floor Grafana clamps the step up to. $__interval = time_range ÷ max_data_points, floored by Min step and rounded — the value actually substituted into [$__interval].
In increase(metric[$__interval]) the same number picks the range-vector window (which scrape deltas get grouped) and the evaluation step (how far apart buckets sit). Widen the dashboard time range and $__interval grows — so your buckets silently get wider and the answer moves.
Each bucket computes last − first over the samples inside it. The two visible deltas (1→2 and 2→3) land wherever the edges happen to fall: one interval keeps them in separate buckets (bars of 1), another merges them (a bar of 2), and an unlucky alignment traps single samples — a bucket with fewer than two samples returns no data at all.
The first sample arrived already at 1 — there is no earlier sample to diff against, and Prometheus does not assume counters start at 0 (prometheus#1673, open since 2016). That opening increment is invisible to plain sample math. Only extrapolation's zero-point guess — a counter this small can't have been running long — sometimes pads part of it back. That's why the bars sum to 2, occasionally ~3, but never reliably 3.
Worth being blunt about one thing: if the first stored sample really were 3 with nothing before it, a flat series has no delta — increase() would return 0 (or no data) everywhere, and you'd never have seen bars of 1 and 2 at all. Those bars prove the scraper caught the counter on the way up (1, then 2). Grafana panels sample at the dashboard step, so short-lived intermediate points are easy to miss; run bifrost_success_requests_total{...}[10m] as an instant query in Explore to see the actual scrapes.
increase() never interpolates between your data points; it never makes up intermediate counter readings. What you're seeing is the combination of three effects, and it helps to name them apart:
The first sample of a new series has no predecessor, so whatever the counter already held when it appeared never enters any increase(). Sparse, label-churny metrics — every new virtual_key × model combo, every restarted pod — pay this tax on each new series. This is why your bars sum to 2, not 3.
Which scrape deltas land in the same [$__interval] window. This is what flips the panel between 1 and 2: pure grouping, driven entirely by window width and alignment.
The "phantom" from Part 1: the first→last slope is stretched toward the window edges. Its zero-point rule can even hand back part of the lost birth — that's how a bucket can report 2 when only a +1 delta was visible, and why values aren't clean integers.
All three fall out of one fact: increase() is really rate() × window — a smoothed rate estimate scaled up to the window, not a literal tally of events. Ask a rate question and it's perfect. Ask "how many requests, exactly" over a sparse burst on a newborn series and it will lie to you politely.
Concretely, on the Bifrost panels:
$__interval → $__rate_intervalGrafana computes it as max($__interval + scrape_interval, 4 × scrape_interval), where scrape_interval is your query's Min step if set, otherwise the datasource's scrape interval. The window always spans ≥ 4 scrapes, so visible deltas can't be chopped between buckets no matter how you zoom. It does not resurrect the lost birth — that it can only estimate.
This floors $__interval (= max(time_range ÷ max_data_points, min_step)) so the step never drops below your data resolution — no more empty buckets from a step finer than the scrape.
The clean fix for the birth problem lives in Bifrost, not PromQL: register the counter at 0 as soon as a virtual key / provider / model combo becomes active, before its first request. Then the first request is a visible 0→1 delta and nothing is lost. (Prometheus's created-timestamp feature does this natively, but it's still opt-in.)
increase() at allAs Part 1 said: a scraped counter through increase() is a rate tool. For an exact, invoice-grade count of a sparse burst, source it from the raw request/usage data, not from a re-bucketed range vector.
increase(x[$__interval]) answers "roughly how fast, over whatever window the dashboard happens to pick" — not "how many". Your burst created the series (its opening increment is invisible), $__interval split the two visible deltas into different bars — 1 here, 2 there — and extrapolation seasoned the numbers. Use $__rate_interval, emit 0 when a series is born, and count anything invoice-grade from source data.