✗ Mechanism failed 2026

Constant-sum ordinal preference loss

Implementation & benchmark of arXiv:2608.23859 — Ranking by points and ordinal models

Usefulness6/10
Difficulty4/10
Novelty6/10

Source paper: Ranking by points and ordinal models arXiv:2608.23859 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use a constant-sum point vector to encode ordered pairwise outcomes and train a neural scorer with an adjacent-categories ordinal likelihood whose slope parameters are tied to those points. The accumulated point score is then a theoretically motivated compressed statistic for repeated comparisons, rather than an arbitrary regression target or one-hot label.

Formulas

$$k^{\mathrm{a}}_{y;i,j}:=k^{\mathrm{h}}_{L-1-y;j,i}$$
$$p_y+p_{L-1-y}=C,\qquad y=0,\ldots,L-1$$
$$\log\frac{\Pr(Y=y\mid i,j)}{\Pr(Y=y-1\mid i,j)}=a_y+p_y\bigl(f_\theta(x_i)-f_\theta(x_j)+b_{ij}\bigr),\qquad y=1,\ldots,L-1$$
$$r_0=1,\quad r_y=\exp\!\left(\sum_{t=1}^{y}[a_t+p_t\delta_{ij}]\right),\quad \Pr(Y=y\mid i,j)=\frac{r_y}{\sum_{q=0}^{L-1}r_q},\quad \delta_{ij}=f_\theta(x_i)-f_\theta(x_j)+b_{ij}$$

Mathematical statement

The paper defines the away-side count by k^{\mathrm{a}}_{y;i,j}:=k^{\mathrm{h}}_{L-1-y;j,i}, where L is the number of ordered outcome levels, y\in\{0,\ldots,L-1\} is the outcome from player i's perspective, and k^{\mathrm{h}}_{y;i,j} counts matches hosted by i against j with outcome y. Its central sufficiency result states that a point total is sufficient for the observations if and only if the points are constant-sum, p_y+p_{L-1-y}=C for every outcome y, and the data follow an adjacent-categories ordinal model with the score points as slope parameters. A concrete neural adaptation uses a scalar skill difference \delta_{ij}=f_ heta(x_i)-f_ heta(x_j)+b_{ij}, where f_ heta is the neural scorer and b_{ij} is an optional context offset. Define adjacent log-odds by \log[p(Y=y)/p(Y=y-1)]=a_y+p_y\delta_{ij} for y=1,\ldots,L-1. Here p_y is the point value for outcome y, a_y is a learned category intercept, and C is the fixed total assigned to complementary outcomes. Recover normalized probabilities using r_0=1, r_y=\exp(\sum_{t=1}^{y}(a_t+p_t\delta_{ij})), and p(Y=y)=r_y/\sum_{q=0}^{L-1}r_q. The transferable property is that repeated observations can be aggregated as S_i=\sum_m p_{y_m}; under the stated model this score preserves the information needed for ranking skill parameters.

Implementation notes

Integrate this method at the output loss for a model that predicts a scalar utility, such as a text preference model, recommender embedding model, graph-node scorer, or learned ranking function. Given two examples x_i and x_j, compute z_i=f_theta(x_i) and z_j=f_theta(x_j), then form d=z_i-z_j+b_ij. Choose L ordinal outcome levels and initialize a point vector p satisfying p_y+p_{L-1-y}=C. Enforce the constraint by parameterization: for each complementary pair, set p_y=C/2+v_y and p_{L-1-y}=C/2-v_y; for the middle category when L is odd, set p_{(L-1)/2}=C/2. For each adjacent pair, compute inc[y]=a[y]+p[y]*d. Set log_r[0]=0 and recursively compute log_r[y]=log_r[y-1]+inc[y]. Apply log_softmax to log_r and minimize the negative log probability of the observed ordinal label. Pseudocode: z_i=f(x_i); z_j=f(x_j); d=z_i-z_j+b; log_r=[0]; for y in 1..L-1: log_r.append(log_r[-1]+a[y]+p[y]*d); loss=-log_softmax(log_r)[label]. Learn theta, a, v, and context parameters jointly. The mathematical quantities are the constant-sum constraint and adjacent log-odds recursion; estimate all neural scores and intercepts empirically by gradient descent. For a first cheap experiment, train a small MLP on MovieLens pairwise preferences converted into three ordered labels, comparing this loss against one-hot cross-entropy, standard cumulative-link ordinal regression, and Bradley-Terry loss. Also create repeated comparisons with balanced and skewed pair frequencies. Aggregate each item's observations with S_i=sum_m p_{y_m}, then measure held-out ordinal log-likelihood, calibration, Kendall tau of rankings, and performance versus the number of labels per pair. Success means improved ranking and calibration at equal training examples, especially for repeated or imbalanced comparisons, while balanced schedules should reduce the gap to ordinary baselines.

Verification

Mechanism failed

Stage 1 · Toy mechanism gate: Failed ✗

Stage 2 · Mechanism transferred to benchmark: Not tested

Stage 2 · Practical benchmark result: Not run

Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Implemented the supplied adjacent-categories loss with constant-sum point parameterization, plus a paper-faithful category-logit AC variant and multiclass cross-entropy control. The recursion is numerically exact, but the supplied formula's equal-point-total likelihood gap changes from 0.25 to 1.45 as skill difference changes, so constant-sum points are not sufficient under that formulation. The paper-faithful AC model gives the expected skill-independent gap and only a negligible learning improvement over CE, so the supplied idea does not show the claimed effect.

Agent confidence
9/10
Baseline
Multiclass CE: mean held-out NLL 0.84228, accuracy 0.6332, pair-ranking Kendall tau 0.824 mean, with high seed instability.
Idea
Supplied adjacent recursion: mean held-out NLL 0.85051, accuracy 0.6283, pair-ranking Kendall tau 0.8039. Adjacent-log-odds error was 4.4e-16, but the equal-score likelihood gap changed by 1.20 across skill differences. Paper-faithful constant-sum AC: mean NLL 0.84215, accuracy 0.6321, tau 0.8238; equal-score gap change was 0.0.

Limitations: Only a small synthetic paired-comparison experiment was run. MovieLens, real repeated schedules, calibration, explicit score aggregation, FLOPs, and larger-scale generalization were not tested. The supplied adjacent formula and the paper's category-logit AC parameterization are mathematically different, so the paper-faithful results should not be attributed directly to the supplied formula.

How to run: python3 experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Artifacts

Implementation overview ⬇ Download all as ZIP 3 files · code, reports and structured results