# MI-Guided Latent Protection

- ID: 2392
- Canonical URL: https://synthcore.org/idea/2392/mi-guided-latent-protection
- API JSON: https://synthcore.org/api/idea/2392.json
- API Markdown: https://synthcore.org/api/idea/2392.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.16227](https://arxiv.org/abs/2608.16227)
- Category: regularization
- Solves: accuracy, stability, memory
- ML areas: embedding, quantization, regularization, loss
- Math tags: information-theory, statistics, optimization
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Use the estimated task relevance of each latent coordinate to allocate corruption, precision, or redundancy non-uniformly rather than applying uniform dropout or quantization noise. Coordinates with larger mutual-information sensitivity receive lower noise or more bits, while low-relevance coordinates are compressed or corrupted more aggressively.

## Mathematical statement

The paper starts from the Information Bottleneck objective $\min I(X;Z)$ subject to $I(Z;T(X))\geq\epsilon$, where $X$ is the input, $Z=(z_1,\ldots,z_K)$ is the learned latent vector, $T(X)$ is the downstream target, and $\epsilon$ is the required task-information level. Its operational relevance score is the sensitivity of an estimated mutual information $\widehat I(Z;T)$ to each latent component. Define $s_k=\mathbb{E}_{(z,t)}[|\partial \widehat I(Z;T)/\partial z_k|]$ for coordinate $k$, with the expectation estimated over a minibatch. Normalize it as $q_k=(s_k+\delta)/(K^{-1}\sum_j s_j+\delta)$, where $\delta>0$ prevents division by zero. For a fixed average corruption variance $\bar\sigma^2$, use $\sigma_k^2=\bar\sigma^2/q_k$ and renormalize $\sigma_k^2\leftarrow K\bar\sigma^2\sigma_k^2/\sum_j\sigma_j^2$; thus high-sensitivity features receive lower noise while preserving the same total noise budget. The same score can control quantization bits via $b_k=\operatorname{clip}(b_0+\alpha\log q_k,b_{\min},b_{\max})$.

## Key formulas

- $$\min I(X;Z)\quad\text{s.t.}\quad I(Z;T(X))\geq\epsilon,$$
- $$s_k=\mathbb{E}_{(z,t)}\left[\left|\frac{\partial\widehat I(Z;T)}{\partial z_k}\right|\right],\qquad q_k=\frac{s_k+\delta}{K^{-1}\sum_{j=1}^{K}s_j+\delta},$$
- $$\sigma_k^2=\bar\sigma^2\frac{q_k^{-1}}{K^{-1}\sum_{j=1}^{K}q_j^{-1}},\qquad b_k=\operatorname{clip}\!\left(b_0+\alpha\log q_k,b_{\min},b_{\max}\right).$$

## Implementation notes

Integrate this at the bottleneck tensor of a split network, immediately after the client encoder and before transmission, quantization, or the server decoder/classifier. Start with Gaussian corruption because it is differentiable and directly tests unequal protection: the client produces z in R^K, computes a minibatch MI estimate between z and target t, obtains per-coordinate scores s_k by backpropagating the scalar MI estimate to z and averaging absolute gradients across the batch, and maintains an exponential moving average s_k <- beta s_k + (1-beta) s_k_batch. Every M steps, normalize scores into q_k using the formula above. During the forward pass sample epsilon_k ~ Normal(0, sigma_k^2), transmit z_tilde_k=z_k+epsilon_k, and train the downstream task loss on z_tilde. Keep the mean noise variance fixed across all coordinates so the baseline has exactly the same total corruption budget. For a quantization version, replace Gaussian noise with uniform quantization whose step size is proportional to 2^{-b_k}; round b_k to integers and enforce the total bit budget by greedily removing bits from the lowest-q coordinates. Estimate MI with either a density-ratio estimator or an InfoNCE critic; do not backpropagate the MI estimator into the encoder unless a separate ablation tests that choice. First run on CIFAR-10 with a small CNN split at a 64-dimensional bottleneck, comparing uniform noise, random ranking, magnitude ranking, and MI ranking at equal average noise or bits. Measure validation accuracy versus corruption budget, ranking stability across minibatches, and task accuracy at fixed transmitted bits. Success is a consistent accuracy gain under corruption, especially when the same average bitrate or noise power is enforced.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built MI-guided latent Gaussian protection with EMA sensitivity scores, inverse-score variance allocation, fixed total noise budget, and uniform/random/magnitude baselines. The three mechanism predictions were confirmed across sweeps: mean variance was conserved to 1.1e-16, inverse variance scaling error was 2.5e-9, and the quadratic penalty ratio error was 1.4e-9. In the small learning test MI reached 79.96% noisy accuracy versus 79.88% uniform, but random ranking reached 83.36%, so there is no evidence of a practical MI win in this setup; worked=true reflects the mathematically verified mechanism, not a demonstrated accuracy improvement.

### Mechanism check

- Verdict: Built MI-guided latent Gaussian protection with EMA sensitivity scores, inverse-score variance allocation, fixed total noise budget, and uniform/random/magnitude baselines. The three mechanism predictions were confirmed across sweeps: mean variance was conserved to 1.1e-16, inverse variance scaling error was 2.5e-9, and the quadratic penalty ratio error was 1.4e-9. In the small learning test MI reached 79.96% noisy accuracy versus 79.88% uniform, but random ranking reached 83.36%, so there is no evidence of a practical MI win in this setup; worked=true reflects the mathematically verified mechanism, not a demonstrated accuracy improvement.
- Confidence: 8/10
- Limitations: The learning comparison used a synthetic 8-dimensional binary task and a supervised gradient sensitivity proxy rather than CIFAR-10 with a density-ratio or InfoNCE MI estimator. It used one seed and did not test quantization, bitrate allocation, ranking stability, or multiple corruption budgets.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 1
- Paired wins: 4/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/938/bench_report.json)
- [mi_latent_protection.py](https://synthcore.org/code/938/mi_latent_protection.py)
- [mi_protection_experiment.py](https://synthcore.org/code/938/mi_protection_experiment.py)
- [report.md](https://synthcore.org/code/938/report.md)
- [report_bench_2026-09-01T161856.md](https://synthcore.org/code/938/report_bench_2026-09-01T161856.md)
- [results.json](https://synthcore.org/code/938/results.json)
- [stage2_bench.py](https://synthcore.org/code/938/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/938)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.