# Information-Profile Watermark Shaping

- ID: 837
- Canonical URL: https://synthcore.org/idea/837/information-profile-watermark-shaping
- API JSON: https://synthcore.org/api/idea/837.json
- API Markdown: https://synthcore.org/api/idea/837.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.13003](https://arxiv.org/abs/2607.13003)
- Category: regularization
- Solves: accuracy, stability, generalization
- ML areas: transformer, loss, regularization, fine-tuning
- Math tags: information-theory, probability, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Train a generative watermark so that its information about the payload is deliberately distributed across positions or overlapping windows instead of being concentrated in a few easily cropped tokens. The objective uses the paper's conditional information profile and the footprint-resolution lower bound to select the smallest carrier support compatible with a target crop size, while preserving generation quality outside that support.

## Mathematical statement

The paper defines the information profile of a payload S encoded in a generated sequence X=(X_1,...,X_n) as \(\nu(t)=I(S;X_t\mid X_{<t})\), where \(X_{<t}=(X_1,...,X_{t-1})\), and the total payload information is \(\|\nu\|_1=\sum_t\nu(t)=I(S;X)\). For a carrier support \(R\subseteq\{1,...,n\}\), let \(|R|\) be its number of active positions. Theorem 3.12 states that any w-crop-robust scheme with local dilation radius r satisfies \(|R|w\geq(n-w+1)/(2r+1)\), hence \(|R|w=\Omega(n)\) for bounded r; for a post-hoc embedding scheme in class \(\mathcal{E}(m)\), \(|R|\leq m\), giving \(mw=\Omega(n)\). We estimate \(\nu(t)\) with a variational decoder \(q_\phi\): \(\widehat\nu_t=\mathbb{E}[\log q_\phi(S\mid X_{\leq t})-\log q_\phi(S\mid X_{<t})]\), where \(X_{\leq t}=(X_1,...,X_t)\).

## Key formulas

- $$\nu(t):=I(S;X_t\mid X_{<t}),\qquad I(S;X)=\sum_{t=1}^{n}\nu(t)=\lVert\nu\rVert_1.$$
- $$|R|\,w\ \geq\ \frac{n-w+1}{2r+1},\qquad\text{and for }\mathcal{E}(m):\quad m\,w=\Omega(n).$$
- $$\widehat\nu_t=\frac{1}{B}\sum_{b=1}^{B}\left[\log q_\phi(s_b\mid x_{b,\leq t})-\log q_\phi(s_b\mid x_{b,<t})\right].$$
- $$\mathcal{L}=\mathcal{L}_{\mathrm{NLL}}+\lambda_{\mathrm{quality}}\mathcal{L}_{\mathrm{KL}}+\lambda_{\mathrm{profile}}\sum_{j=1}^{n-w+1}\left(\tau-\sum_{t=j}^{j+w-1}\widehat\nu_t\right)_+^2+\lambda_{\mathrm{support}}\sum_{t=1}^{n}\operatorname{HardConcrete}(a_t).$$

## Implementation notes

Integrate this into a transformer watermarking fine-tuning setup. The generator receives a secret payload s and produces tokens autoregressively; add a lightweight decoder q_phi that predicts s from prefixes. Introduce a trainable gate a_t in [0,1] on the watermark perturbation to the token logits, for example logits = base_logits + a_t * watermark_logits. For each minibatch of sequences, compute decoder log probabilities for q_phi(s|x_<t) and q_phi(s|x_<=t), then estimate the incremental profile with the difference shown in the third formula. Choose a crop length w, robustness radius r, and target per-window information tau. Add the hinge penalty over all length-w windows so every possible retained crop contains at least tau total estimated information. Add an L0-style Hard-Concrete penalty on a_t to minimize active carrier positions, but set a lower-bound target m_min = ceil((n-w+1)/((2r+1)w)) so sparsity cannot violate the theorem's necessary scaling. The quality term should be the KL between watermarked and unwatermarked next-token distributions, optionally combined with standard language-model NLL. Alternate generator and decoder updates, because q_phi must remain useful as the profile estimator. Estimate empirical support as positions with a_t>0.5 and independently test decoder mutual information on held-out payloads; the theorem supplies only a necessary scaling law, while tau and r are engineering choices. Start with GPT-2-small on C4 or WikiText-103, 128-token sequences, and compare uniform green-list watermarking and unconstrained keyed watermark fine-tuning. Evaluate payload recovery after deleting every contiguous crop of length w, generation perplexity, active-token fraction, and detection rate. Success is higher worst-case crop recovery at equal KL/perplexity, or the same recovery with fewer active positions.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a reproducible binary-channel MVP for information-profile watermark shaping. Exact enumeration confirmed the footprint-resolution lower bound in all tested cases; spreading equal total information across positions improved worst-case 6-token crop information from 0 to 1.062009 bits, and independent-carrier information scaled exactly linearly with carrier count and BSC capacity. The mechanism manifested clearly in the toy setting, but no practical transformer-generation quality win was established.

### Mechanism check

- Verdict: Built a reproducible binary-channel MVP for information-profile watermark shaping. Exact enumeration confirmed the footprint-resolution lower bound in all tested cases; spreading equal total information across positions improved worst-case 6-token crop information from 0 to 1.062009 bits, and independent-carrier information scaled exactly linearly with carrier count and BSC capacity. The mechanism manifested clearly in the toy setting, but no practical transformer-generation quality win was established.
- Confidence: 8/10
- Limitations: This is only a toy binary-channel study. It does not test GPT-style generation, NLL/perplexity or KL quality, learned generator/decoder updates, Hard-Concrete gates, detection, payload generalization, correlated natural text, actual token cropping, GPU performance, or a practical watermarking baseline.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.8081
- Paired wins: 3/8
- Benchmark verdict: no measurable effect

## Artifacts

- [REPORT.md](https://synthcore.org/code/341/REPORT.md)
- [bench_report.json](https://synthcore.org/code/341/bench_report.json)
- [bench_run.py](https://synthcore.org/code/341/bench_run.py)
- [experiment.py](https://synthcore.org/code/341/experiment.py)
- [payload_track.py](https://synthcore.org/code/341/payload_track.py)
- [report.md](https://synthcore.org/code/341/report.md)
- [report_bench_2026-08-31T014706.md](https://synthcore.org/code/341/report_bench_2026-08-31T014706.md)
- [results.json](https://synthcore.org/code/341/results.json)
- [Download all files as ZIP](https://synthcore.org/download/341)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.