Critical Findings: Experiment 7 Deep Validation

Executive Summary

Deep validation revealed that the chaos model's failure is NOT due to binary inputs, but rather a combination of:

1. High input dimensionality (400 spins)
2. Simple linear target (M = mean(spins))

The model DOES work when these conditions are relaxed.

Key Discoveries

Discovery 1: Dimensionality is the Key Issue

Test: Compare small lattice (25 spins) vs large lattice (400 spins)

Results:

• Small lattice (25 spins): R² = 0.9371 ✅
• Large lattice (400 spins): R² = 0.0370 ❌

Conclusion: The chaos model works with low dimensionality but fails with high dimensionality.

Discovery 2: Non-Linear Target Works

Test: Predict M² (non-linear) instead of M (linear)

Results:

• Linear model on M²: R² = 0.7728
• Chaos model on M²: R² = 0.9812 ✅

Conclusion: The chaos model excels at non-linear relationships, even with binary inputs!

Discovery 3: Binary Inputs Are NOT the Problem

Test: Compare binary inputs vs continuous inputs (with noise)

Results:

• Binary inputs: R² = 0.0370
• Continuous inputs: R² = -0.1300 (worse!)

Conclusion: Binary inputs actually work BETTER than continuous inputs. The problem is not binary vs continuous.

Discovery 4: Linear Relationship is the Problem

Test: M = mean(spins) is a simple linear operation

Results:

• Linear model on M: R² = 1.0000 (perfect)
• Chaos model on M: R² = 0.0370 (fails)
• Chaos model on M²: R² = 0.9812 (works!)

Conclusion: The chaos model struggles with simple linear relationships, especially in high dimensions.

Root Cause Analysis

Why Does the Chaos Model Fail?

1. High Dimensionality + Linear Target:

   • With 400 inputs, the FFT transformation may be losing information
   • The simple linear relationship (mean) gets obscured by the complex transformation
   • Ridge regression on 2048 features from 400 inputs may be underfitting

2. Why Small Lattice Works:

   • 25 inputs → 2048 features is a 82x expansion (information gain)
   • 400 inputs → 2048 features is only a 5x expansion (information loss)
   • The transformation has more "room" to work with fewer inputs

3. Why M² Works:

   • Non-linear relationship allows the FFT to capture patterns
   • The transformation naturally encodes multiplicative relationships
   • Ridge regression can learn the non-linear mapping

Implications

What This Means

1. The experiment design is valid - The physics is correct, data is correct
2. The failure is architectural - The chaos model has a specific limitation
3. The limitation is specific - High-dim + linear = failure, but low-dim or non-linear = success

Corrected Understanding

Original Conclusion: "Chaos model fails with binary inputs" Corrected Conclusion: "Chaos model fails with high-dimensional linear relationships, but works with low-dimensional or non-linear relationships"

Recommendations

For Documentation

1. Update README to reflect these findings
2. Acknowledge that the failure is specific to high-dim + linear
3. Note that the model works in other configurations

For Future Experiments

1. Test dimensionality limits explicitly
2. Compare linear vs non-linear targets when possible
3. Document the dimensionality/linearity trade-off

Validation Status

✅ Simulator: Correct (phase transition visible) ✅ Data Generation: Correct (magnetization = mean(spins)) ✅ Baseline: Works (R² = 1.0) ✅ Chaos Model: Fails in high-dim linear case, but works in other cases ✅ Root Cause: Identified (dimensionality + linearity)

Final Verdict: The experiment is valid. The failure is a genuine architectural limitation, but it's more nuanced than initially thought - it's specifically about high-dimensional linear relationships.