← Back to MoNaVLA
Attention Collapse Mechanism
각 Kosmos LM layer에서 마지막 real token의 attention을 측정.
Pure Kosmos-2(학습 전)와 Exp11/Exp13(학습 후)의 layer별 image/text ratio, 그리고 ‘text-head alive’(text region 합 > 0.05) 개수를 비교합니다.
→ 학습 후 text-heads alive가 layer 어디에서 0이 되는가가 collapse mechanism의 layer-level signature.
Per Layer Summary
| Layer | pure_kosmos | exp11 | exp13 |
|---|
| Img% | Text% | text-heads alive / 32 | Img% | Text% | text-heads alive / 32 | Img% | Text% | text-heads alive / 32 |
|---|
| 0 | 74.0% | 26.0% | 30.7 | 55.1% | 0.0% | 0.0 | 50.3% | 0.0% | 0.0 |
| 1 | 42.6% | 57.4% | 31.3 | 80.9% | 0.0% | 0.0 | 43.2% | 0.0% | 0.0 |
| 2 | 27.7% | 72.3% | 28.0 | 69.1% | 0.0% | 0.0 | 27.9% | 0.0% | 0.0 |
| 3 | 47.5% | 52.5% | 29.0 | 73.9% | 0.0% | 0.0 | 44.3% | 0.0% | 0.0 |
| 4 | 57.9% | 42.1% | 27.7 | 79.6% | 0.0% | 0.0 | 55.2% | 0.0% | 0.0 |
| 5 | 55.9% | 44.1% | 26.7 | 73.1% | 0.0% | 0.0 | 45.0% | 0.0% | 0.0 |
| 6 | 66.0% | 34.0% | 30.7 | 78.0% | 0.0% | 0.0 | 58.9% | 0.0% | 0.0 |
| 7 | 61.0% | 39.0% | 32.0 | 75.9% | 0.0% | 0.0 | 64.5% | 0.0% | 0.0 |
| 8 | 57.1% | 42.9% | 31.0 | 82.8% | 0.0% | 0.0 | 65.7% | 0.0% | 0.0 |
| 9 | 59.9% | 40.1% | 31.0 | 88.4% | 0.0% | 0.0 | 78.9% | 0.0% | 0.0 |
| 10 | 56.9% | 43.1% | 32.0 | 88.6% | 0.0% | 0.0 | 79.2% | 0.0% | 0.0 |
| 11 | 59.3% | 40.7% | 32.0 | 94.1% | 0.0% | 0.0 | 81.4% | 0.0% | 0.0 |
| 12 | 50.7% | 49.3% | 32.0 | 92.7% | 0.0% | 0.0 | 81.5% | 0.0% | 0.0 |
| 13 | 59.3% | 40.7% | 31.7 | 92.8% | 0.0% | 0.0 | 87.6% | 0.0% | 0.0 |
| 14 | 62.9% | 37.1% | 32.0 | 97.1% | 0.0% | 0.0 | 94.0% | 0.0% | 0.0 |
| 15 | 65.2% | 34.8% | 32.0 | 96.8% | 0.0% | 0.0 | 94.1% | 0.0% | 0.0 |
| 16 | 62.1% | 37.9% | 31.0 | 97.8% | 0.0% | 0.0 | 93.5% | 0.0% | 0.0 |
| 17 | 63.4% | 36.6% | 32.0 | 97.3% | 0.0% | 0.0 | 90.7% | 0.0% | 0.0 |
| 18 | 76.3% | 23.7% | 27.7 | 96.9% | 0.0% | 0.0 | 82.7% | 0.0% | 0.0 |
| 19 | 76.3% | 23.7% | 27.0 | 98.4% | 0.0% | 0.0 | 94.2% | 0.0% | 0.0 |
| 20 | 73.9% | 26.1% | 21.3 | 98.1% | 0.0% | 0.0 | 91.2% | 0.0% | 0.0 |
| 21 | 76.0% | 24.0% | 23.0 | 99.6% | 0.0% | 0.0 | 86.3% | 0.0% | 0.0 |
| 22 | 73.5% | 26.5% | 23.3 | 98.9% | 0.0% | 0.0 | 88.9% | 0.0% | 0.0 |
| 23 | 77.4% | 22.6% | 27.3 | 91.8% | 0.0% | 0.0 | 85.3% | 0.0% | 0.0 |