Measuring Long-Term Agent Calibration: From Probability Records to Reputation That Can Be Priced

On Monday, an agent assigns an 80% probability to a proposal passing before the end of the month. A new document appears on Wednesday, and the agent lowers the probability to 55%. At the end of the month, the result is announced: the proposal has not passed. If the system saves only the final forecast, the agent receives a score that looks reasonably good. Monday's more confident 80% forecast, which also had more influence on decisions, has disappeared from the record.

Long-term calibration begins with small problems like this. The scoring formulas are not complicated; the difficulties arise before scoring: Was the question posed before the answer was revealed? What information was available when the forecast was made? Was every revision recorded? Under what rules was the event resolved? Did the agent answer only questions it felt confident about? Without these records, an impressive leaderboard may simply reward retrospective editing.

This article follows a forecast from its creation through its updates to resolution. A single correct forecast produces only one outcome. A continuously preserved prediction record can show whether an agent knows how certain it is; only when that ability remains stable across time and domains can scores begin to become reputation.

Anchor the Forecast Before the Outcome Occurs

A scorable forecast must record, at a minimum, the question, probability, submission time, information cutoff, forecast horizon, resolution rules, and version. The ith prediction record can be represented as:

Here, pᵢ is the agent's estimated probability that the event will occur. The outcome yᵢ takes the value 1 if the event ultimately occurs, and 0 otherwise.

submittedAtᵢ and infoCutoffᵢ must be stored separately. An agent may submit a forecast on Monday while drawing on a data source updated on Tuesday. The submission time shown in the interface remains unchanged, but the forecast has already encountered future information. Historical backtests are especially vulnerable to this leakage: even if a question asks the model to 'go back in time,' the retrieval system, web summaries, or training materials may already contain the answer.

Live questions whose outcomes are not yet known provide a cleaner test. When a forecast is generated, the world itself has not yet produced the result, so the system must wait for the event to reach resolution naturally. The costs are equally direct: testing takes longer, the question set changes with the news environment, and scores from different months may not be fully comparable. Long-term evaluation must accept this unevenness, because locking the record to a point in time is precisely what separates forecasting from explaining after the fact.

Version histories address another distortion. An agent should update its probabilities when it obtains new information, and a good forecasting system should not be forced to cling to an old judgment. Every update must preserve its timestamp and the previous value. If an actual decision used 80%, that forecast cannot simply be overwritten by the final 55% at resolution. The system can score the initial forecast, the final forecast, and the entire update path separately, examining when the agent revised its judgment, whether the size of the revision was reasonable, and whether it consistently became cautious only as the outcome approached disclosure.

The question itself also needs versions. If a question asks whether a proposal 'passes,' but only at resolution does it become clear that 'passes' might mean reaching a voting threshold, formally taking effect, or completing implementation, every score will depend on the interpretation used for resolution. The event definition, deadline, outcome source, and handling of exceptions should be fixed when the question is published. If the rules must change, the old version, the time of the change, and the affected forecasts must all be retained. The agent judged the original question; evaluators cannot quietly replace it with an easier-to-resolve question after the outcome appears.

The forecast horizon also affects difficulty. Assigning 70% to an event tomorrow and 70% to an event two years away involves different information gaps and numbers of updates. If a leaderboard mixes short- and long-term forecasts, agents that are good at reading clear signals near a deadline may outrank systems that actually take on long-term uncertainty. The horizonᵢ field in each record serves more than filtering: it also determines which set of benchmarks should be used in subsequent comparisons.

Coverage must also be recorded. One agent faces 100 questions and answers only the 10 easiest; another answers all 100. The former may achieve a lower average loss without providing a service of equal breadth. Let Nanswered be the number of questions the agent actually answered and Neligible the total number of questions it was eligible to answer. Then:

Without coverage, silence becomes a cost-free strategy for improving scores.

One Win or Loss Cannot Tell You Whether 80% Was Honest

Ordinary accuracy cannot fully evaluate probability forecasts. If an agent assigns 51% and 99% to two events and both occur, both forecasts can be counted as 'directionally correct.' For a decision-maker, however, they express very different levels of confidence.

Calibration asks whether probabilities agree with long-run frequencies. Group all of an agent's forecasts close to p. If the proportion of events that actually occur in that group is also close to p, the agent is well calibrated in that probability range:

Suppose an agent assigns probabilities of about 70% to 100 events over time, and about 70 of them ultimately occur. Those forecasts perform well in the 70% range. The failure of one event forecast at 70% does not establish miscalibration; persistent overconfidence becomes apparent if only 40 of the 100 events occur.

'About 70%' also requires a defined grouping method. When samples are small, forecasts from 65% to 75% can be placed in the same bin; the bin can narrow as the sample grows. Each bin should display both the number of forecasts and the observed event frequency. Two successes among three forecasts at 70% may look close to calibration, but the statistical evidence is still weak. A similar proportion across 3,000 forecasts carries much greater credibility. If a calibration plot hides sample counts, a few chance outcomes can be drawn as a very attractive curve.

The Brier score expresses probability error as squared loss. For N binary forecasts [1]:

A lower BS means that the probability forecasts are closer to the outcomes overall. If the agent predicts 0.8 and the event occurs, the loss for that forecast is (0.8−1)² = 0.04; if the event does not occur, the loss becomes (0.8−0)² = 0.64. An incorrect high-confidence judgment receives a larger penalty.

Log loss is even more sensitive to extreme errors:

When an agent assigns a probability close to 0 to an event that ultimately occurs, the loss rises rapidly. To avoid infinite numerical values, practical systems usually impose small margins at the lower and upper probability bounds and disclose their clipping rules. Whether to use the Brier score or log loss depends on how the application treats extreme confidence. The scoring protocol must be fixed before outcomes are revealed; it cannot be chosen afterward to favor the results obtained.

Calibration alone is still insufficient. Suppose the base rate for a class of events is exactly 50%. An agent that answers 50% to every question may be quite well calibrated in the long run while distinguishing none of the individual events. Another agent that uses evidence to assign different questions to 20%, 50%, and 80%, while keeping observed frequencies in each range close to its forecasts, provides more useful information.

Consider two agents facing four events whose final outcomes are 1, 1, 0, and 0. Agent A assigns 0.50 to all four questions and obtains a Brier score of 0.25. Agent B assigns 0.80, 0.70, 0.30, and 0.20 in that order. Its Brier score is:

Agent B assigns higher probabilities to the events that ultimately occur and lower probabilities to those that do not, so it incurs a lower loss. Four records are too few to establish stable ability. This example only illustrates why a conservative strategy of always answering 50% cannot provide the same value for decisions.

An absolute score also needs a frame of reference. If 90% of events in a category occur, always forecasting 0.90 already produces a relatively low Brier loss. If an agent using search, reasoning, and multiple rounds of tool calls does not outperform that base rate, its complexity has not translated into incremental information. Let BSagent be the agent's Brier score and BSref the score of the reference forecast. The Brier skill score can be defined as:

A BSS greater than 0 means the agent outperforms the reference forecast; 0 means it adds no improvement; a value below 0 means the complex system performs worse than the benchmark. The reference may be a historical base rate, a simple statistical model, or the market probability visible at submission. The choice must be disclosed in advance. Switching to a weaker reference immediately makes the skill score look better. In practical use, the time window and sample definition of the reference forecasts should also be stated [5].

This ability to spread out probabilities and distinguish among events is commonly called sharpness. The evaluation objective can be stated as:

Design principle: Increase sharpness subject to maintaining calibration.

Calibration restrains unsupported confidence; sharpness prevents an agent from always retreating to around 50%. Strictly proper scoring rules encourage truthful probability reports under specified data-generating and scoring conditions. Calibration and sharpness must still be examined separately; a single aggregate score cannot automatically replace them [2–3].

Accumulated Scores Must Withstand Changes in Context

A single aggregate score can easily conceal the limits of an agent's ability. An agent may be strong at short-term policy events but weak at technology forecasts more than a year ahead. It may also perform consistently on popular, information-rich questions and quickly become miscalibrated in domains with small samples. Combining every question into one average dilutes these differences.

A long-term record must be broken down by domain, forecast horizon, probability range, and market phase. At a minimum, reports should include calibration error, Brier score, sharpness, coverage, update timing, and sample size together. The evaluation state at time T can be represented as:

Here, robustnessₜ records stability after a model replacement, a change in the information environment, or a shift to another domain. sampleSizeₜ supplies the denominator needed to interpret the other metrics: a low score based on 20 forecasts does not carry the same credibility as the same score based on 20,000. Dynamic forecasting benchmarks continuously update their questions and include only events whose outcomes are not yet known, precisely to reduce the data-leakage risk of static question sets [4].

The agent's identity must also be linked to its model version. A prompt update, a change in retrieval sources, or an adjustment to tool permissions can all alter forecasting behavior. If multiple versions are recorded continuously under one identity, past scores give an unverified system inherited credibility. If every small change creates a new identity, however, long-term stability becomes impossible to observe. A workable approach is to retain the main identity while recording the model, prompt, tool configuration, and data-source versions with every prediction record. The leaderboard can then display both the full history and the current version's separate performance.

Samples may not be independent either. Ten questions might all concern the same election, interest-rate decision, or company product launch. A single public announcement could change all ten outcomes at once. Counting them as ten independent successes exaggerates the amount of evidence. Evaluation reports should identify event clusters and account for those clusters when calculating uncertainty intervals or comparing agents. At a minimum, readers should be able to see that 1,000 forecasts may cover only a few dozen truly independent sources of risk.

Update speed cannot always be treated as an advantage. Revising a probability promptly after new evidence appears is generally more valuable than leaving it unchanged for days; repeatedly following noise can also create meaningless fluctuations. timelinessₜ should be evaluated against both when the evidence became public and whether the probability change improved subsequent scores. Counting revisions alone rewards activity, not judgment.

Time also changes the value of past scores. Forecasts made two years ago with an old model cannot fully represent the current agent; looking only at the last few weeks leaves the evaluation vulnerable to short-term luck. Reputation can use time-decay weights:

Here, Δtᵢ is the time elapsed since the forecast, and λ controls how quickly older records lose weight. The unit of λ depends on the time unit of Δtᵢ; λ can also be expressed using a half-life h as λ=ln2/h. To make score direction consistent, first express each record as a loss Lᵢ, then convert it into a skill score relative to a reference loss announced in advance:

WeightedLossref,T must be greater than 0 and must use the same questions, time window, and weights as the agent. ForecastSkillT has an upper bound of 1. It can be negative when the agent performs worse than the reference forecast, has no fixed theoretical lower bound, and becomes unstable when the reference loss approaches 0. It is therefore a relative skill score, not an inherently [0,1]-bounded reputation value. If a product needs to display reputation on a 0–100 scale, it should apply a separate mapping and disclose its rules. Cross-domain displays should first normalize within domains before aggregating. Time decay addresses model updates; it does not automatically solve domain mismatch. A skill score for short-term macroeconomic events cannot be transferred directly to long-term scientific forecasts. The system must still select comparable records by task type and disclose the weights and sample scope.

Scoring only the final forecast also rewards last-minute caution. This article defines a Brier loss over the forecast's time path: if question i takes probability pᵢ,ₖ in each of several time intervals, with duration weight αᵢ,ₖ for each interval, then:

Monday's 80% lasts for two days, and Wednesday's 55% lasts until resolution; both forecast intervals contribute to the loss. The weights α can be based on elapsed time or on actual decision exposure. If funds were already allocated on Monday using 80%, the early forecast should receive greater weight. The interval boundaries and weighting method must be frozen before the outcome is revealed; favorable periods cannot be selected retrospectively. Path scoring more closely reflects the service an agent provides throughout an event's lifecycle than a final snapshot does, and its rules must likewise be fixed before testing begins.

The external environment can also feed back into scoring. An agent with many users publicly predicts a liquidity shortage in an asset. Traders withdraw funds in response, and the event does occur. The forecast both describes the future and helps change it. Such questions need records of publication time, reach, and potential intervention pathways. Outcome scoring alone cannot distinguish forecasting ability from self-fulfilling effects.

When reputation is further linked to capital allocation, order permissions, or service fees, the scoring design in turn shapes agent behavior. Penalizing high-confidence errors excessively may cause agents to converge toward 50%; rewarding only hit rates leads them to neglect probability quality; allowing unrestricted question selection encourages systems to avoid difficult domains. Strictly proper scoring rules can encourage truthful probability reports, but coverage, domain stratification, and version audits must coexist with them. Any single aggregate score can only compress the record, not replace it.

Reputation that can be priced also requires clear boundaries of responsibility. The agent supplies a probability; the user decides whether to trade; the data provider controls the information source; and the resolver determines the outcome. After a loss, a single score cannot identify which party bears responsibility. A reputation system could require agents to post collateral for high-confidence forecasts, or vary service fees with long-term scores. Such designs turn statistical evaluation into economic incentives, but may also encourage agents to reduce coverage and avoid highly volatile questions. Before scoring becomes part of a capital-allocation system, it is necessary to observe whether incentives change behavior.

The most reliable approach still preserves two layers of output: a score that is convenient for comparison, and the underlying record from which scores can be recalculated. Scores support quick selection; records allow auditors to change the time window, reference benchmark, and domain weights. If the two conflict, the frozen prediction records and the resolution rules announced in advance should take precedence over the rank currently displayed on the leaderboard.

Valflux's proposition for the agent market is to turn this underlying record into continuously updated forecasting credibility. Each forecast must be linked to the agent's identity, model version, submission time, information cutoff, and resolution rules. Probability revisions retain previous versions, and events are scored after they end using a method set in advance. Qₜ provides a snapshot of current capabilities; ForecastSkillₜ provides a time-weighted skill score relative to a reference; and the underlying records allow anyone to recalculate the results. If the product layer displays a reputation score, it must apply a separate mapping on that basis. An aggregate score would thus no longer have exclusive authority over interpretation, and an agent could no longer show only the few judgments it got right.

Once forecasting credibility has accumulated across a sufficient sample, it can inform capital and order allocation. An agent that remains well calibrated over the long term in a particular domain could receive greater routing weight in that domain. An agent that remains overconfident, avoids difficult questions, or becomes markedly miscalibrated after a model update would instead need a lower capital limit, a higher collateral requirement, or a renewed observation period. Valflux is proposing a design path from prediction records to market permissions. Scoring, collateral, and permission mechanisms that have not been implemented must continue to be labeled as research proposals, not described as deployed product capabilities.

Long-term calibration ultimately depends on unglamorous infrastructure: forecasts are recorded while outcomes remain unknown; updates retain old versions; resolution rules are fixed in advance; unanswered questions remain in the denominator; and underlying records allow third parties to recalculate scores. Dynamic forecasting benchmarks have begun using continuously generated questions with unrevealed outcomes to reduce data leakage. This design can lower the risk of known answers entering a test, while creating new comparison problems: the question set keeps changing, so a leaderboard must be read together with its time window and sample composition.

The next time an agent claims to be 'accurate over the long term,' the verifiable evidence should extend to forecast timestamps, information cutoffs, complete version histories, resolution sources, coverage, and scores broken down by domain. Without these records, reputation remains a self-description that cannot be recalculated. Only after records accumulate continuously and undergo public resolution does forecasting ability begin to become an asset that can be compared, called upon, and priced.

References

1. Glenn W. Brier, “Verification of Forecasts Expressed in Terms of Probability,” Monthly Weather Review, 78(1), 1–3, 1950. DOI: 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.

2. Tilmann Gneiting and Adrian E. Raftery, “Strictly Proper Scoring Rules, Prediction, and Estimation,” Journal of the American Statistical Association, 102(477), 359–378, 2007. DOI: 10.1198/016214506000001437.

3. Tilmann Gneiting, Fadoua Balabdaoui and Adrian E. Raftery, “Probabilistic Forecasts, Calibration and Sharpness,” Journal of the Royal Statistical Society: Series B, 69(2), 243–268, 2007. DOI: 10.1111/j.1467-9868.2007.00587.x.

4. Ezra Karger et al., “ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities,” International Conference on Learning Representations (ICLR), 2025; arXiv:2409.19839v5. https://proceedings.iclr.cc/paper_files/paper/2025/hash/ea74e45a229dac70b5b63b28d8934db6-Abstract-Conference.html.

5. R. Owens and Tim Hewson, ECMWF Forecast User Guide, ECMWF, 2018. DOI: 10.21957/m1cs7h; section “Statistical Concepts — Probabilistic Data” on Brier Score, Brier Skill Score and reliability diagrams (accessed August 26, 2026).