- Found
- Static priorityStatic priorityA hand-written rule that fixes each job type's priority in advance and never changes with what's observed. is no universal winner and can't fix the tail.
- So
- Hypothesis revised: expect medianMedianThe middle value: half the jobs are faster, half slower. It describes a typical job and ignores rare outliers.Learn more ↗ and meanMeanThe ordinary average. It is pulled up by a few very slow items, so it shows the overall experience better than the median does.Learn more ↗ wins, not P99P95 / P99 (tail latency)P99 is the time that 99% of messages beat. It describes the slowest 1%, the "tail", which matters because users notice the slow ones.Learn more ↗.
Experiments
How the experiments shaped Harbinger.
From the first scheduling tests to v4 replay, ablations and stronger baselines. Failures narrowed the claims as much as successes shaped the design. Correctness tests support specific invariants; benchmark results support workload-specific performance claims.
Overview
What was decided
- 19research stepsincluding pre-registration and descriptive studies
- 6new v4 studies
- 4/6weighted real-broker cells passed0/6 with default-aging routing
There is no combined pass count: descriptive studies, per-cell gates and incomplete registrations answer different questions. Earlier entries retain their original outcomes, with later evidence noted.
- Learn each keyKeyWhat Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history.'s duration as a decaying histogramHistogramA count of how many times each range of values occurred. Here: how many jobs took 1–2 ms, 2–4 ms, and so on.Learn more ↗ inside the brokerBrokerThe server in the middle. Producers hand messages to it, and consumers pull messages from it.Learn more ↗, in C++. No model server.Shipped
- Judge by meanMeanThe ordinary average. It is pulled up by a few very slow items, so it shows the overall experience better than the median does.Learn more ↗ and short-job medianMedianThe middle value: half the jobs are faster, half slower. It describes a typical job and ignores rare outliers.Learn more ↗, per class. All-message P99P95 / P99 (tail latency)P99 is the time that 99% of messages beat. It describes the slowest 1%, the "tail", which matters because users notice the slow ones.Learn more ↗ is a guardrailGuardrailA metric that must not get much worse, even if the main metric improves. Tail latency and lost work are guardrails here., not a goal.Shipped
- Predictive routing is opt-inOpt-inOff by default. You have to switch it on deliberately.. Static priorityStatic priorityA hand-written rule that fixes each job type's priority in advance and never changes with what's observed. stays the default, and shadow modeShadow modeThe predictor runs and learns, but its guesses are only recorded, never used. Safe way to try it on real traffic. comes first.Shipped
- A prediction for a still-fresh keyKeyWhat Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history. counts as activity, so keys stuck in a backlogBacklogMessages that have arrived but are still waiting because workers can't keep up. are not written off.Shipped
- Size-binned keysKeyWhat Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history. for jobs whose cost follows input size.Opt-in
- WeightsLevel weightsInstead of always serving the highest level first, each level gets a share of worker time (for example 8 : 3 : 1). Lower levels still make steady progress. with pausing agingPausing agingAging that holds off promoting a message while the level above is still working through earlier promoted messages. It stops a long backlog from pushing everything to the top. preserve ordering under overload, with measured tail costs; default-aging routing fails all six real-brokerBrokerThe server in the middle. Producers hand messages to it, and consumers pull messages from it.Learn more ↗ trace cells.Opt-in
- Evaluate policies on the workload, against meaningful baselines and explicit tail/long-job budgets. Predictability alone is not enough.Guideline
- The scheduling idea works on another brokerBrokerThe server in the middle. Producers hand messages to it, and consumers pull messages from it.Learn more ↗ too, so it is separable from Harbinger's own queue.Finding
- Per-type fair queueing beats predictive meanMeanThe ordinary average. It is pulled up by a few very slow items, so it shows the overall experience better than the median does.Learn more ↗ on the synthetic workload. The benefit is avoiding manual per-type setup, not universal scheduling superiority.Finding
- AgingAgingA message that has waited too long is moved up one priority level, so low-priority work is delayed but never stuck forever.Learn more ↗'s meanMeanThe ordinary average. It is pulled up by a few very slow items, so it shows the overall experience better than the median does.Learn more ↗-latencyLatencyHow long a message takes from being sent to being finished, including time spent waiting in the queue.Learn more ↗ cost is isolated on synthetic work and Huawei replay; switching it off also increases tail latency and long waits.Finding
- CPU-contention simulation fails effect validation. Parameter sensitivity confirmation through the real brokerBrokerThe server in the middle. Producers hand messages to it, and consumers pull messages from it.Learn more ↗ remains incomplete.Limit
Phase 1
Foundations: what to build, and how to judge it
- Found
- Simple per-keyKeyWhat Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history. statistics matched or beat all eight online models.
- So
- The predictorPredictorThe part of the broker that estimates how long a message will take, using what earlier messages with the same key took. became a C++ per-keyKeyWhat Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history. histogramHistogramA count of how many times each range of values occurred. Here: how many jobs took 1–2 ms, 2–4 ms, and so on.Learn more ↗. Python stayed offline.
- Found
- The v1 budgets could not be met even by a perfect static mapStatic priorityA hand-written rule that fixes each job type's priority in advance and never changes with what's observed..
- So
- Re-pre-registeredPre-registrationWriting down the hypotheses, setup and pass/fail rules before running the experiment, so results can't be quietly reinterpreted afterwards.Learn more ↗ on a realistic workload before any predictive run.
Phase 2
Does routing work on real work?
- Found
- Learned routing beat FIFOFIFOFirst in, first out. Messages are served strictly in arrival order, like a single supermarket line.Learn more ↗ and a hand-tuned map without a per-endpoint priority map, but the tail did not improve.
- So
- Predictive routing is opt-inOpt-inOff by default. You have to switch it on deliberately. and static stays the default. Shadow modeShadow modeThe predictor runs and learns, but its guesses are only recorded, never used. Safe way to try it on real traffic. comes first.
- Found
- Under sustained overload, agingAgingA message that has waited too long is moved up one priority level, so low-priority work is delayed but never stuck forever.Learn more ↗ collapses every policy into FIFOFIFOFirst in, first out. Messages are served strictly in arrival order, like a single supermarket line.Learn more ↗.
- So
- The failed gateGateA pass/fail test agreed before running an experiment. Predictive routing is only allowed if every gate passes. stands, a stale-keyKeyWhat Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history. bug was fixed, and level weightsLevel weightsInstead of always serving the highest level first, each level gets a share of worker time (for example 8 : 3 : 1). Lower levels still make steady progress. with pausing agingPausing agingAging that holds off promoting a message while the level above is still working through earlier promoted messages. It stops a long backlog from pushing everything to the top. became experiment 9.
- Found
- The Azure failure came from queueing dynamics, not from unpredictable jobs.
- So
- Per-keyKeyWhat Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history. statistics are confirmed as the serving model. ColdCold startThe period before the predictor has seen enough finished jobs for a key to trust its estimate. Cold keys use the default middle tier.-start fallback matters.
Phase 3
Closing gaps: features and overload
- Found
- A per-keyKeyWhat Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history. medianMedianThe middle value: half the jobs are faster, half slower. It describes a typical job and ignores rare outliers.Learn more ↗ is off by 2× where cost follows input size.
- So
- Added opt-inOpt-inOff by default. You have to switch it on deliberately. size-binned keysKeyWhat Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history.. No heavier model.
- Found
- Size bins made the predictorPredictorThe part of the broker that estimates how long a message will take, using what earlier messages with the same key took. more accurate but not measurably faster under load.
- So
- They stay opt-inOpt-inOff by default. You have to switch it on deliberately.. The next test needs a job type that spans tiersTierA priority level in the queue. L0 is served first, then L1, then L2.Learn more ↗ and carries a lot of the work.
- Found
- WeightsLevel weightsInstead of always serving the highest level first, each level gets a share of worker time (for example 8 : 3 : 1). Lower levels still make steady progress. plus pausing agingPausing agingAging that holds off promoting a message while the level above is still working through earlier promoted messages. It stops a long backlog from pushing everything to the top. give large meanMeanThe ordinary average. It is pulled up by a few very slow items, so it shows the overall experience better than the median does.Learn more ↗ gains, with caveats on the tail.
- So
- Opt-inOpt-inOff by default. You have to switch it on deliberately. for now. Strict priority stays the default until the tail cost is understood.
Phase 4
Beyond the lab: cloud, other traces, other brokers
- Found
- Every verdict held on cloud hardware, but the gain shrinks when backlogsBacklogMessages that have arrived but are still waiting because workers can't keep up. last minutes.
- So
- Results now report load next to every number. AWS is the independent replication.
- Found
- The original seven-window simulation passed 5/6 weighted cells and 1/6 routing cells; later analysis and replay revise that to 4/6 and 0/6.
- So
- Keep this original result as history; use experiments 17 and 18 for current serverlessServerlessA cloud style where you upload small functions and the provider runs them on demand. The Azure Functions trace records one such platform's invocations.Learn more ↗ conclusions.
- Found
- On HPC logs the predictorPredictorThe part of the broker that estimates how long a message will take, using what earlier messages with the same key took. mostly can't predict, and the overload design can hurt long jobs.
- So
- Predictability is a prerequisite, not a sufficient reason to enable the overload design; evaluate latencyLatencyHow long a message takes from being sent to being finished, including time spent waiting in the queue.Learn more ↗ and long-job costs too.
- Found
- The scheduling gain carries to RabbitMQ and Redis, but Harbinger is the slowest brokerBrokerThe server in the middle. Producers hand messages to it, and consumers pull messages from it.Learn more ↗.
- So
- Claim the scheduling idea, not brokerBrokerThe server in the middle. Producers hand messages to it, and consumers pull messages from it.Learn more ↗ speed. Broker overhead is a tracked limitation.
Phase 5 · v4
Testing the explanation, not just the headline
- Found
- The simulator gets direction and policy-group order right, but overstates the measured meanMeanThe ordinary average. It is pulled up by a few very slow items, so it shows the overall experience better than the median does.Learn more ↗-latencyLatencyHow long a message takes from being sent to being finished, including time spent waiting in the queue.Learn more ↗ benefit.
- So
- Keep simulated queueing evidence separate from CPU-bound application evidence; do not apply a universal correction factor.
- Found
- Cloud hardware reproduces every quiet-local criterion verdict for weighted agingAgingA message that has waited too long is moved up one priority level, so low-priority work is delayed but never stuck forever.Learn more ↗ and size bins.
- So
- Both features stay opt-inOpt-inOff by default. You have to switch it on deliberately.. Faster hardware does not turn these missed gatesGateA pass/fail test agreed before running an experiment. Predictive routing is only allowed if every gate passes. into passes.
- Found
- Per-type fair queueing beats predictive meanMeanThe ordinary average. It is pulled up by a few very slow items, so it shows the overall experience better than the median does.Learn more ↗ without prediction; agingAgingA message that has waited too long is moved up one priority level, so low-priority work is delayed but never stuck forever.Learn more ↗ off explains much of the remaining synthetic-workload gap.
- So
- Narrow the advantage to avoiding per-type configuration, and make the meanMeanThe ordinary average. It is pulled up by a few very slow items, so it shows the overall experience better than the median does.Learn more ↗-versus-tail trade-off explicit.
- Found
- Default-agingAgingA message that has waited too long is moved up one priority level, so low-priority work is delayed but never stuck forever.Learn more ↗ routing sits on a parameter cliff on Huawei, and finer-window analysis overturns two old passes.
- So
- Use the revised 28-window analysis, and disclose that registered real-brokerBrokerThe server in the middle. Producers hand messages to it, and consumers pull messages from it.Learn more ↗ variant confirmation was not run.
- Found
- Real-brokerBrokerThe server in the middle. Producers hand messages to it, and consumers pull messages from it.Learn more ↗ replay agrees with the revised simulation verdicts, not the original narrow passes.
- So
- Use replay as the current serverlessServerlessA cloud style where you upload small functions and the provider runs them on demand. The Azure Functions trace records one such platform's invocations.Learn more ↗ result. Separate window-gateGateA pass/fail test agreed before running an experiment. Predictive routing is only allowed if every gate passes. effects from whole-run changes and disclose the reliability gaps.
- Found
- AgingAgingA message that has waited too long is moved up one priority level, so low-priority work is delayed but never stuck forever.Learn more ↗ off restores a substantial meanMeanThe ordinary average. It is pulled up by a few very slow items, so it shows the overall experience better than the median does.Learn more ↗ gain without weightsLevel weightsInstead of always serving the highest level first, each level gets a share of worker time (for example 8 : 3 : 1). Lower levels still make steady progress., while increasing window-level P99P95 / P99 (tail latency)P99 is the time that 99% of messages beat. It describes the slowest 1%, the "tail", which matters because users notice the slow ones.Learn more ↗ and long waits.
- So
- WeightsLevel weightsInstead of always serving the highest level first, each level gets a share of worker time (for example 8 : 3 : 1). Lower levels still make steady progress. are not the only way to preserve ordering. Choose agingAgingA message that has waited too long is moved up one priority level, so low-priority work is delayed but never stuck forever.Learn more ↗ policy against measured long-job and tail budgets.
Correctness evidence
What the tests cover
The new v4 tests mainly exercise the replay runner and analysis pipeline. Existing brokerBrokerThe server in the middle. Producers hand messages to it, and consumers pull messages from it.Learn more ↗ tests cover settlement and prediction invariants. Neither a passing benchmark nor a test-suite name proves general reliability or performance.
Replay harness and runner
- Synthetic trace planning, all-arm completion, time scaling, approximate FIFOFIFOFirst in, first out. Messages are served strictly in arrival order, like a single supermarket line.Learn more ↗ order and discriminating short-before-long oracleOracleA reference that cheats by knowing each job's true cost in advance. It shows what perfect knowledge would give, not what's achievable. behaviour; this is not exact scheduling-order verification.
- Embedded and split-role runner paths, warm snapshot transfer via stand-in SSH, resume, failed-attempt preservation and one retry, sharding, load-specific arms, smoke-kit compatibility and invalid configuration handling.
Reports and inference
- QuantilesQuantileA cut point that splits sorted values into equal groups. Harbinger uses them to decide where one tier ends and the next begins.Learn more ↗, equal-time windows, validity rejection, trace-time conversion, paired repeats and pooling, stability rules, scheduled-versus-submit latencyLatencyHow long a message takes from being sent to being finished, including time spent waiting in the queue.Learn more ↗, and simulator comparisons.
- Block-bootstrapBootstrapA statistics method that re-samples your own results many times to see how much they could vary by chance.Learn more ↗ behaviour, invalid blocks, sensitivity verdict flips, ablation interpretations, and hand-computed simulation validation cases. Regressions cover simulator metadata being mistaken for an arm and replay-only gatesGateA pass/fail test agreed before running an experiment. Predictive routing is only allowed if every gate passes. being falsely reported as simulator failures.
Replay reports ↗Bootstrap and sensitivity ↗Production-flow reports ↗Validation tests ↗
Broker, scheduler and predictor invariants
- Only eligible successful AcksAckShort for acknowledgement. A consumer tells the broker it finished a message, and here it also reports how long the work took.Learn more ↗ inside lease/TTLTTLTime to live. After this long a message is considered expired and dropped.Learn more ↗ teach the predictorPredictorThe part of the broker that estimates how long a message will take, using what earlier messages with the same key took.. Replayed Acks do not teach twice; stale tokens, Nacks, expired Acks and invalid durations do not supply successful duration labels.
- Chronological brokerBrokerThe server in the middle. Producers hand messages to it, and consumers pull messages from it.Learn more ↗/predictorPredictorThe part of the broker that estimates how long a message will take, using what earlier messages with the same key took./replay consistency, including size bins; concurrent Submit/Pull/AckAckShort for acknowledgement. A consumer tells the broker it finished a message, and here it also reports how long the work took.Learn more ↗/Nack/lease recovery and snapshot accounting; feedback disk failure does not block settlement or learning.
- Weighted charges use predicted or learned per-level cost. Queue tests cover workerWorkerA process that takes one message at a time and runs the work for it.-time shares, work conservation, idle-credit handling, cancellation refunds and pausing agingPausing agingAging that holds off promoting a message while the level above is still working through earlier promoted messages. It stops a long backlog from pushing everything to the top.. PredictorPredictorThe part of the broker that estimates how long a message will take, using what earlier messages with the same key took. tests distinguish fresh activity from attempts to revive a stale keyKeyWhat Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history..
Broker invariants ↗Scheduler invariants ↗Predictor invariants ↗
Next
Open questions
These are the remaining evidence gaps and design questions. The triggered sensitivity confirmation is unfinished registered work; the other proposals would need their own evaluation before changing defaults.
- Sensitivity Part C. Replay the verdict-flipping parameter variants through the real broker; default-configuration replay does not complete this requirement.
- Unexplained broker deaths. Preserve diagnostic logs and investigate the two failed Huawei attempts; reruns alone do not identify the cause.
- Long-horizon fairness. Test aging-off and weighted policies over hours, with explicit long-job and tail budgets. Short finite runs cannot establish starvation freedom.
- Bottom-level weight. Try a larger weight on the lowest level to see whether it wins back some of the long-job tail that experiment 9 cost.
- Time-scaled aging. Scale aging and the stale limit to the workload's own job durations, then retest the HPC logs.
- A predictability safeguard. Switch the overload design off automatically when a workload's keys do not predict well.
- A workload where size bins should win. One job type that spans tiers and carries a large share of the work, pre-registered before running.
- A controlled cross-broker aging comparison. Aging is isolated within Harbinger now; matching the sidecar's reduction across different layouts is not a paired equivalence test.
- Per-function worker pools in the trace simulator, to model serverless scaling more honestly.
- Cloud repeat of the feature-signal collection (experiment 7), on hardware other than one laptop.