Information-Profile Watermark Shaping

Integrate this into a transformer watermarking fine-tuning setup. The generator receives a secret payload s and produces tokens autoregressively; add a lightweight decoder q_phi that predicts s from prefixes. Introduce a trainable gate a_t in [0,1] on the watermark perturbation to the token logits, for example logits = base_logits + a_t * watermark_logits. For each minibatch of sequences, compute decoder log probabilities for q_phi(s|x_<t) and q_phi(s|x_<=t), then estimate the incremental profile with the difference shown in the third formula. Choose a crop length w, robustness radius r, and target per-window information tau. Add the hinge penalty over all length-w windows so every possible retained crop contains at least tau total estimated information. Add an L0-style Hard-Concrete penalty on a_t to minimize active carrier positions, but set a lower-bound target m_min = ceil((n-w+1)/((2r+1)w)) so sparsity cannot violate the theorem's necessary scaling. The quality term should be the KL between watermarked and unwatermarked next-token distributions, optionally combined with standard language-model NLL. Alternate generator and decoder updates, because q_phi must remain useful as the profile estimator. Estimate empirical support as positions with a_t>0.5 and independently test decoder mutual information on held-out payloads; the theorem supplies only a necessary scaling law, while tau and r are engineering choices. Start with GPT-2-small on C4 or WikiText-103, 128-token sequences, and compare uniform green-list watermarking and unconstrained keyed watermark fine-tuning. Evaluate payload recovery after deleting every contiguous crop of length w, generation perplexity, active-token fraction, and detection rate. Success is higher worst-case crop recovery at equal KL/perplexity, or the same recovery with fewer active positions.