# Fano-Calibrated Multi-User Watermark Budget

- ID: 838
- Canonical URL: https://synthcore.org/idea/838/fano-calibrated-multi-user-watermark-budget
- API JSON: https://synthcore.org/api/idea/838.json
- API Markdown: https://synthcore.org/api/idea/838.md
- Verification status: unverified
- Source: [arXiv:2607.13003](https://arxiv.org/abs/2607.13003)
- Category: training
- Solves: accuracy, scalability, stability
- ML areas: transformer, loss, fine-tuning, regularization
- Math tags: information-theory, probability, statistics, optimization
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Use the paper's attribution converse to calibrate watermark strength and sequence length for a registry of N users, rather than tuning detection and attribution thresholds independently. A dual controller allocates a per-token information and KL budget so that the learned key information approaches the minimum required for reliable attribution, avoiding both underpowered marks and unnecessarily visible perturbations.

## Mathematical statement

Let U be a uniformly distributed user identity over N enrolled keys, X be the generated sequence of length n, and \(P_e\) be the probability that an attribution rule outputs the wrong user. Under the paper's per-token information cap \(\Delta\), Theorem 3.2 gives \(I(U;X)\leq n\Delta\) and the Fano lower bound \(P_e\geq 1-(n\Delta+\log 2)/\log N\). Therefore achieving target attribution error \(\epsilon\) requires approximately \(n\Delta\geq(1-\epsilon)\log N-\log 2\). The paper also gives, for key K and any reference natural distribution Q, \(I(K;X)\leq\mathbb{E}_K\operatorname{KL}(P_{X\mid K}\|Q)\), where \(P_{X\mid K}\) is the keyed generator distribution. Set Q to the unwatermarked generator distribution \(p_{\mathrm{nat}}\); the sequence KL can be estimated as the sum of teacher-forced per-token KLs. The adaptation uses a dual variable to enforce a total KL budget while maximizing a variational lower bound on \(I(U;X)\).

## Key formulas

- $$P_e\ \geq\ 1-\frac{n\Delta+\log 2}{\log N},\qquad\text{so reliable attribution requires }n\Delta\gtrsim\log N.$$
- $$I(K;X)\ \leq\ \mathbb{E}_{K}\,\operatorname{KL}\!\left(P_{X\mid K}\,\|\,Q\right).$$
- $$\widehat{I}(U;X)=\frac{1}{B}\sum_{b=1}^{B}\left[\log q_\phi(u_b\mid x_b)-\log p_U(u_b)\right],\qquad p_U(u)=1/N.$$
- $$\min_\theta\ \mathcal{L}_{\mathrm{LM}}(\theta)-\beta\widehat I(U;X)+\lambda\left(\widehat D_{\mathrm{KL}}-n\Delta_{\mathrm{target}}\right)^2_+,\qquad \Delta_{\mathrm{target}}=\frac{(1-\epsilon)\log N-\log 2}{n}.$$

## Implementation notes

Modify the watermarking logit-bias or adapter-training component of an autoregressive transformer. Maintain an unwatermarked frozen reference model p_nat and a keyed model p_theta(.|prefix,u), where u is sampled uniformly from a registry of N user IDs. For every training example, compute the per-token KL between the keyed and reference next-token distributions and sum it over n positions: D_KL_hat = sum_t KL(p_theta(.|x_<t,u) || p_nat(.|x_<t)). Train a separate attribution decoder q_phi(u|x_1:n), and estimate key information using the variational classification expression in the third formula. The generator objective minimizes language-model loss and maximizes this estimated key information, while a hinge or augmented-Lagrangian term limits total KL to n times Delta_target. Set Delta_target from the Fano expression using a chosen target error epsilon; for example, with N=1024, n=128, and epsilon=0.05, the initial budget is roughly ((0.95 log 1024)-log 2)/128. Use an adaptive multiplier lambda: increase it when measured KL exceeds the budget and decrease it when below. Alternate q_phi and generator updates, and evaluate on fresh prompts and keys not seen in the minibatch. Compare against fixed-strength watermarking at matched perplexity. Measure top-1 attribution error as N grows, key confusion entropy, detection error without the key, and average token KL. The falsifiable prediction is that calibrated training reaches the target attribution error with lower average KL than fixed-strength baselines, and that error sharply degrades when n Delta falls below the Fano threshold. Because the bound is a converse rather than an achievability theorem, report the gap between empirical required information and the lower bound.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.