Latency-Aware Cost Models for Serverless Inference at the Network Edge: Derivation and Multi-Platform Validation

1.Introduction

Serverless platforms are an attractive substrate for inference at the network edge because they remove capacity planning at hundreds of locations where traffic is individually sparse and collectively significant. They also make cost difficult to reason about. Billing couples request count, execution duration, memory reservation and, implicitly, the platform's residency behaviour, which the operator does not control and cannot directly observe.

Existing cloud inference cost models are built for the opposite regime. They assume that instances are warm, that utilisation is high enough for residency to amortise, and that arrival rates are stable enough for average-case reasoning. At the edge, a location may see four requests per second, arriving in bursts, against a cold-start penalty that is a substantial fraction of the useful work. Under those conditions average-case models mis-rank configurations, and we observed operators making placement decisions on figures that were wrong by a factor approaching two.

We contribute: (i) an analytical cost model for edge serverless inference that treats cold-start probability as an explicit function of arrival process and residency policy; (ii) a derivation of the cost-minimising batch window under a p95 latency constraint; (iii) multi-platform empirical validation over 41.6 million billed requests; and (iv) a three-parameter closed-form approximation intended for use in capacity and placement planning.

2.Background and Related Work

Cold-start behaviour has been characterised extensively for general serverless workloads, establishing that initialisation latency is heavy-tailed and strongly dependent on image size and runtime. Inference workloads differ in that initialisation includes model weight loading, which shifts the tail further and makes residency economically decisive rather than merely a latency concern.

Batching is well understood as a throughput-latency trade in server-class inference, where batch sizes are chosen against fixed accelerator capacity. The edge setting inverts the constraint: batch size is limited by arrival sparsity long before it is limited by hardware, so the relevant question is the batch window in time rather than the batch size in requests. Our derivation is formulated accordingly.

Prior cost modelling for serverless has focused on function placement and data transfer. We treat those as fixed and isolate the inference-specific interaction between residency and batching, which our measurements identify as the dominant term in sparse deployments.

3.The Cost Model

We model arrivals at a location as a Markov-modulated Poisson process with two states, capturing the bursty diurnal pattern observed at all 14 sites. Expected cost per completed request decomposes into a billed-duration term, a memory-reservation term and a cold-start term whose weight is the steady-state probability that an arriving request finds no warm instance. That probability is derived from the arrival process and the platform's idle-eviction interval, which we estimate per platform by probing rather than assuming a published value.

Introducing a batch window of length w delays each request by at most w, raises the mean number of requests served per initialisation, and therefore reduces both the amortised cold-start term and the per-request reservation term. It also raises p95 latency by an amount we bound analytically. Minimising expected cost subject to a p95 constraint yields a batch window that is increasing in cold-start cost and decreasing in arrival rate, with a closed-form optimum under the single-state approximation.

The full model requires the arrival process parameters, eviction interval, initialisation-duration distribution and per-request service time. For planning use we derive a reduced form requiring only mean arrival rate, mean initialisation duration and mean service time, and we report its error against the full model in Section 5.

4.Experimental Setup

Platforms and locations. We deployed the same quantised transformer model to three commercial serverless platforms at 14 edge locations spanning four continents, chosen to span two orders of magnitude in traffic. Deployments ran continuously for four weeks, serving 41.6 million production-mirrored requests.

Instrumentation. Each request was tagged with initialisation state, queue delay, service duration and billed duration as reported by the platform. Invoices were reconciled independently against our own accounting by two authors; discrepancies above 1% were investigated and, in two cases, traced to platform-side rounding which we then modelled explicitly.

Configuration sweep. Separately from the production mirror, we swept 1,320 configurations varying arrival rate (0.5–120 rps), batch window (0–200 ms) and memory reservation, with three repetitions per cell, to measure model accuracy away from observed operating points.

5.Validation Results

Accuracy. Mean absolute percentage error between predicted and billed cost was 5.4%, 6.0% and 5.7% on the three platforms. Error was largest at the sparsest locations (9.1% in the lowest traffic quartile), where cold-start probability approaches one and small errors in the eviction-interval estimate propagate strongly.

Batching. At measured arrival rates, windows of 40–80 ms reduced per-token cost by 22% (95% CI 19–25%) against no batching, with an 8% throughput reduction and 31 ms added p95 latency. Beyond 120 ms, additional cost savings were under 3% while p95 latency rose steeply, giving a clear operating region.

Residency. Cold starts accounted for 34% of spend below 4 rps and under 6% above 40 rps. Residency policy is therefore the dominant optimisation only for sparse locations; above roughly 25 rps in our deployments, batching and reservation sizing dominate and residency tuning is close to noise.

Reduced form. The three-parameter approximation stayed within 11% of the full model across the sweep and within 7% in the traffic range covering 90% of our observed locations, which we consider adequate for placement decisions where alternatives differ by more than that margin.

6.Discussion

The practical consequence is a decision rule: measure arrival rate per location first, and only then choose an optimisation target. Sparse locations should be consolidated or given explicit warm capacity; dense locations should be tuned on batch window and memory reservation, where our model and its reduced form give usable predictions.

The result also cautions against portfolio-level averaging. Operators in our study computed a single cost per request across all locations and used it for placement; because the cold-start term is non-linear in arrival rate, that average systematically underestimated the cost of adding sparse locations and thus encouraged over-expansion of the edge footprint.

7.Limitations and Future Work

We validated on three platforms with one model architecture and one quantisation level; initialisation duration scales with weight size, so the residency term will differ for larger models even though the structure of the model does not. Commercial pricing changed once during our window and we re-fitted accordingly; predictions are therefore period-specific in magnitude.

Our arrival model uses two states, which fits diurnal bursts well but not event-driven spikes such as broadcast traffic. Extending to a higher-order modulating chain, and to multi-model co-residency where several models compete for warm capacity at one location, are the two extensions we consider most valuable.

8.Conclusion

A cost model that represents cold-start probability as a consequence of arrival process and residency policy predicted commercial edge inference billing within 6% across three platforms and 41.6 million requests. It identifies modest time-based batching as a 22% cost reduction at acceptable latency, and shows that residency tuning matters only below roughly 25 requests per second. The reduced three-parameter form is accurate enough for placement planning from data operators already hold.

9.References

1. [1] Haddad, J. (2025). Cold starts revisited: initialisation tails in inference workloads. Proceedings of SoCC, 311–324.
2. [2] Osei, P., Tan, M. & Rahman, A. (2024). Residency policies for serverless inference. ACM Transactions on Computer Systems 42(3), 1–27.
3. [3] Tan, M. (2026). Time-window batching under sparse arrivals. JIPCET 2(4), 91–118.
4. [4] Rahman, A. (2026). Evaluation cost models for multi-step inference workloads. JIPCET 1(1), 12–29.
5. [5] Marsh, S. (2025). Markov-modulated arrivals in edge deployments. Performance Evaluation 168, 102–121.
6. [6] Whitfield, B. & Russo, G. (2025). Reconciling platform billing with request-level accounting. Proceedings of ICPE, 74–86.