Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activation
1. Paper overview
In one sentence: Modality subspace steering makes Omni-LLM outputs more responsive to missing sensory inputs, but the reported gains in sensitivity do not consistently preserve answer accuracy. e02e03e07e10
| At a glance | What to know |
|---|---|
| Research problem | Source description A correct multimodal answer can conceal reliance on textual shortcuts. The paper asks whether removing an indispensable modality changes the decision. Its argument depends on a strong dataset condition: humans must perform better with all modalities than with any proper subset, for every sample and evaluator. CausalMSBench is designed to enforce this condition, but the detailed construction audit is deferred to unavailable supplementary material. e02e08e16 |
| Core mechanism | |
| A key reported result | CausalMSBench question answering with Qwen3-Omni-30B-A3B-Instruct: MSA: 65.15% accuracy; 30.58 average CMS. Accuracy (%) and CMS (paper’s displayed scale). 617-example evaluation; full-modality accuracy and CMS averaged over three omission protocols. Unsteered: 57.33% accuracy; 26.40 CMS. Accuracy gain: 7.82 percentage points. The strongest full-input accuracy result. Qwen2.5-Omni-7B also improves, from 49.35% to 55.70%, with CMS 31.79 to 52.36. No uncertainty is reported. e08e09e18 |
| Reading caution | Reader analysis CMS is a sensitivity score, not a calibrated grounding probability. From Eq. (3), nondegenerate LAD can span 0–2, so Eq. (7) is not intrinsically bounded by one. Positive rescaling of centered logits leaves LAD unchanged. Figure 5 also shows high sensitivity with only 8.79% accuracy. e03e04e14 |
Core contributions
Figure 2. A diagnostic pipeline followed by an intervention before the language head. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the large panels from left to right. The first makes modality necessity a prerequisite; the second enumerates full input and three omission conditions. The third separates unchanged answer choices from movements of the option-logit vector. In the purple panel, follow the two distinct paths: anchor hidden states form SVD bases, while a runtime hidden state is projected onto those bases and its projection norms are sorted. The plus sign boosts the weakest projection; the minus sign suppresses the strongest. Equations (8)–(13) on page 5 specify the feature centering and right-singular-vector basis that the schematic abbreviates. e02e03e05e06e07e17
What it supports. MSA changes the representation passed to an existing language head, using a basis estimated from separate samples. The drawing supports the information flow and correction signs; it does not depict new parameter training or an action controller. The final red text states the authors’ intended diagnostic outcome, which must be checked against the experiments.
Where the evidence stops. The schematic labels bases U and repeats H_visual on the audio-colored matrix. The method instead uses top right singular vectors V-hat. Follow the equations for implementation; retain these source notation discrepancies. Neither the figure nor Eq. (13) gives a separate activation-gap trigger.
2. Motivation
2.1 The problem and the proposed response
A correct multimodal answer can conceal reliance on textual shortcuts. The paper asks whether removing an indispensable modality changes the decision. Its argument depends on a strong dataset condition: humans must perform better with all modalities than with any proper subset, for every sample and evaluator. CausalMSBench is designed to enforce this condition, but the detailed construction audit is deferred to unavailable supplementary material. e02e08e16
2.2 What this reading follows
Imagine an audio-visual question whose answer requires both what happened and what was heard. If a model gives the same answer after losing one stream, accuracy alone cannot tell us how its decision changed internally. This paper pairs answer retention with the angular movement of option logits, then proposes MSA to rebalance modality contributions before the final language head. The illustrations trace that argument from diagnostic assumptions to SVD projections and controlled comparisons. The key reading discipline is to keep responsiveness and correctness separate: the reported intervention raises CMS across all seven models, yet full-input accuracy falls for five of them. e02e03e07e10
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Not assigned. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-work categories fit a multimodal-backbone diagnostic, metric and inference intervention. The WAM architecture, prediction paradigm and quadrant are not applicable: this paper supplies neither joint future/action prediction nor inverse dynamics or an executed action model. e03e07e17
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Establish necessity before interpreting an unchanged answer
The paper’s first step is a dataset assumption, not a model intervention. With visual, audio and text modalities available, Eq. (1) requires a positive gap between human accuracy on complete input and the best accuracy under any proper subset. This is meant to exclude examples solvable from language priors or redundant sensory streams. Only under that premise does removing a modality become an informative test of dependence. Next distinguish the two measurements: ARR asks whether the top option survives, while LAD measures cosine distance between centered option-logit vectors. The Figure 3 examples show why both are needed. Reader interpretation: CMS is useful for diagnosing responsiveness, but neither an unchanged correct choice nor a large logit rotation alone proves the quality of perceptual grounding. The missing construction supplement limits verification of the starting premise. e02e03e04e12e16
5.2 Separate subspace preparation from runtime correction
MSA’s preparation phase collects final hidden states from separate single-modality anchor inputs. For each stream, the authors stack these states as an N-by-d_model matrix, subtract the feature-wise mean across examples and perform SVD. The top k right singular vectors define the projection basis; Figure 2’s U labels should therefore be read alongside the more precise equations. At runtime, the current hidden state is projected into each modality’s basis, and projection norms select the weakest and strongest streams. Equation (13) adds part of the former and subtracts part of the latter before the language head predicts the option. No gradient update is involved. Reader interpretation: these are subspaces estimated from controlled inputs, not proven independent semantic compartments; overlap could make a correction affect several modality contributions simultaneously. e05e06e07e08
5.3 Read the intervention as an accuracy–sensitivity tradeoff
Table 1 shows why CMS must be read alongside correctness: all seven models gain sensitivity, but five lose accuracy. Figure 5 reinforces this warning with accuracy of 8.79% at lambda 0.1. The uncropped Figure 6 comparison is equally instructive. At default lambda 0.5, Random reaches 43.00% accuracy and 48.92 CMS, exceeding MSA’s 40.07% and 45.58. At lambda 0.9, MSA instead retains more accuracy, 44.63% versus 38.44%, but has lower CMS, 40.10 versus 61.06. Page 7’s prose explicitly supplies 38.44%; the plot partly obscures that label. Reader interpretation: these controls establish changes in decision behavior with setting-dependent utility, without isolating useful grounding from instability. The authors attribute some control accuracy to shortcuts, but neither the two scores nor the chart identifies why individual answers are correct. Stronger evidence would show that responsiveness tracks indispensable content while preserving correct full-input answers. e09e10e14e15e17e19
5.4 Training and inference
During training
MSA has no optimization loss, gradient-training stage or model-weight update. Preparation uses 500 randomly selected SocialOmni examples disjoint from evaluation, with default rank k=64. The paper reports hidden-state dimensions of 2,000–4,000. These are source-reported settings, not independently checked checkpoint properties. e06e08
During inference
The default balancing factor is 0.5. Runtime projection uses h directly in Eq. (10), without subtracting the anchor mean. Increasing lambda shifts the balance toward boosting and away from suppression; it is not simply an increasing overall intervention magnitude. e07e14
5.5 Implementation flow
- Compare full and incomplete input
Evaluate identical questions with all modalities, without vision, without audio, and without both. Restrict language-head logits to answer-option tokens. ARR records whether the argmax changes, regardless of correctness; LAD compares the directions of option logits after subtracting their within-option mean. e03e08
- Estimate modality subspaces
For each modality, stack anchor final hidden states into an activation matrix. Subtract the feature-wise mean across samples, as Eq. (8) specifies, then retain the top right singular vectors. Although the prose says ‘sample-wise,’ the equation defines centering across samples for each feature. e06
- Measure and rebalance activation
Project the runtime hidden state onto each basis and rank the projection norms. Add a weighted weakest projection and subtract a weighted strongest projection; the middle modality receives no explicit update. Equation (13) supplies no separate activation-gap threshold. Bases are individually orthonormal; mutual orthogonality across modalities is not established. e06e07
- Decode the corrected state
Feed the altered state to the existing language-model head for final-choice prediction. This is a representation intervention in question answering, with no future-world prediction, action extraction, environmental feedback or executed control policy. e07e17
6. Experiments & results
MSA intervenes in an Omni-LLM’s final hidden state using modality-specific SVD subspaces. Its companion benchmark and CMS metric test whether answers and option logits react when audio or vision disappears. All seven tested variants gain sensitivity, but only two gain accuracy; the central tradeoff is between responsiveness and correctness, not demonstrated robot-control reliability.
Five legible original crops are presented. Figure 6 was inspected on page 7, but its high-lambda Random CMS bar overlaps the printed accuracy label, obscuring the final digit. It is therefore not selected as a standalone crop. The complete steering-control result remains in the base report and third walkthrough; Random’s 38.44% accuracy is explicitly supported by the adjoining page 7 prose. The original graphic has not been retouched. e15e19
6.1 Read the original evidence
Table 1. Sensitivity rises in every row; accuracy gains concentrate in two Qwen variants. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the two Normal columns: Acc. and Acc.(MSA) compare full-modality answer accuracy in percent. The middle groups describe unsteered results after removing vision, audio or both. At the right, Avg. Perturb. averages the three omission settings; its paired CMS columns compare sensitivity without and with MSA. Parenthesized downward changes belong to average accuracy, not CMS. Dark, medium and light red highlight first, second and third ranked entries within columns. Compare each metric within its own scale: the displayed CMS scores are not percentages of correctly answered questions or calibrated probabilities of grounding. e04e08e09e10e11e18
What it supports. Qwen3-Omni-30B-A3B-Instruct improves from 57.33% to 65.15% accuracy, a 7.82-point gain, while average CMS rises from 26.40 to 30.58. Across the complete table, however, accuracy improves in only two of seven rows. OmniVinci’s drop from 35.34% to 22.80% accompanies a CMS increase from 23.12 to 34.55.
Where the evidence stops. No uncertainty is reported. The adjoining prose calls Interactive-Omni-4B less misaligned than MiniCPM-4.5, but their CMS values are 39.29 and 41.74 respectively, contradicting the stated higher-CMS interpretation. The table’s values are preserved here.
Figure 3. The same correct option can conceal a changed pattern of relative option logits. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each perturbation banner before its red LAD label. Panel (a) asks about the position of a humming sound’s source and removes both vision and audio; panel (b) asks where a speaking character is located and removes vision. The caption and page 7 case study identify Interactive-Omni-4B and state that both predictions remain correct and unchanged. Those facts are not recoverable from the stills alone. ARR therefore records retention for both examples, while the annotated LAD values describe a separate change in the centered option-logit direction. The images supply context, not the complete scoring trace. e03e12
What it supports. These cases establish why answer retention and logit geometry are complementary measurements. Retention is 1.0 in both, but LAD is 1.34 and 0.85. A preserved top option therefore does not imply that every relative option preference stayed fixed, even when the output answer is still correct.
Where the evidence stops. The crop does not show answer options, raw logits or the actual predicted answer strings. It supports the reported examples, not independent recomputation of LAD or a claim about how frequent this pattern is across the benchmark.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CausalMSBench question answering with Qwen3-Omni-30B-A3B-Instruct 617-example evaluation; full-modality accuracy and CMS averaged over three omission protocols. | MSA: 65.15% accuracy; 30.58 average CMS. Accuracy (%) and CMS (paper’s displayed scale) | Unsteered: 57.33% accuracy; 26.40 CMS. Accuracy gain: 7.82 percentage points. The strongest full-input accuracy result. Qwen2.5-Omni-7B also improves, from 49.35% to 55.70%, with CMS 31.79 to 52.36. No uncertainty is reported. e08e09e18 |
| Cross-model sensitivity–accuracy tradeoff Same CausalMSBench table, all seven model variants. | CMS improves in seven of seven variants; accuracy improves in two and declines in five. Full-input accuracy (%) and average CMS | OmniVinci: accuracy 35.34% to 22.80%, CMS 23.12 to 34.55. Interactive-Omni-4B: accuracy 47.88% to 40.07%, CMS 39.29 to 45.58. Reader analysis: these decreases are substantial counterexamples to treating increased sensitivity as universal task improvement. e10 |
| Detecting distribution changes behind retained answers Two Interactive-Omni-4B examples, removing both sensory modalities or vision alone. | Both retain correct answers with retention 1.0; LAD is 1.34 and 0.85 respectively. Answer retention and LAD | A discrete retention-only diagnostic would treat both as invariant. LAD detects differences that ARR misses; selected examples do not establish population frequency. e12 |
| MSA versus steering controls Figure 6: default lambda=0.5 left of the dashed divider and comparison at 0.9 to its right; blue bars use the left accuracy axis and red hatched bars the right CMS axis. The caption does not explicitly identify the model. | At 0.5: MSA 40.07 / 45.58; Random 43.00 / 48.92; Equal 47.72 / 41.00; Reverse 49.92 / 38.66. At 0.9: MSA 44.63 / 40.10; Random 38.44 / 61.06. The adjoining prose explicitly supplies 38.44%, whose final plotted digit is obscured. Accuracy (%) / CMS | Unsteered baseline: 47.88 / 39.29. Random exceeds default MSA on both displayed metrics. At 0.9, MSA retains more accuracy but has less sensitivity; the controls do not establish uniform dominance. Equal boosts all modalities; Reverse boosts the dominant and suppresses the weak modality. The discussion attributes some control accuracy to shortcuts, but the chart does not measure why individual answers are correct. e15e18e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. More anchor samples change the subspace estimate, but the measured response is not monotonic. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The zero tick is explicitly the baseline without MSA; the remaining ticks count samples used to build the activation matrices. Follow blue circles against the left accuracy axis and red squares against the right CMS axis. Their intersections are not equality of scores because the scales differ. The settings section identifies SocialOmni as the anchor source and gives the standard choice of 500 examples and rank 64. Read the irregular intermediate points before accepting the discussion’s stability explanation: accuracy and sensitivity do not move together, and neither curve supplies direct measurements of singular-vector stability or construction cost. e06e08e13e18
What it supports. The paper selects 500 anchors as a compromise between representation quality and preparation cost. The plotted comparison supports sensitivity to anchor count, but it does not show 500 as the highest-accuracy setting or a monotonic approach to a stable optimum. Larger anchor sets remain an empirical design choice.
Where the evidence stops. No repeated samples, error bars or measured memory/runtime costs accompany the sweep. The figure caption does not name the model, so the apparent agreement with another model’s table baseline should not be promoted into verified model identity.
Figure 5. A highly sensitive model can be a poor question-answering model. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the blue left axis for accuracy and the red right axis for CMS. The marked 0.5 point is the authors’ default balance. At small positive lambda, Eq. (13) heavily subtracts the strongest projection while adding little of the weakest; increasing lambda reverses that balance. Thus the horizontal label ‘Intervention Strength’ should not be read as a uniform scaling of the whole update. Also separate the first ‘0 (w/o)’ point from literal lambda zero: the plot labels it an unsteered baseline, but the equation would still perform suppression at that numerical value. e07e14e18
What it supports. At lambda 0.1, the reported accuracy is only 8.79% despite a large CMS increase. At 0.5, accuracy recovers to 40.07%. This is direct evidence for reading the two metrics together: maximizing perturbation responsiveness alone can select a severely degraded answerer rather than a reliably grounded model.
Where the evidence stops. The source does not reconcile its baseline label with Eq. (13). A reproduction must implement a separate bypass and report literal lambda zero independently. The sweep also lacks uncertainty and explicit model identification in its caption.
7. Analysis & limitations
7.1 What the evidence leaves open
CMS is a sensitivity score, not a calibrated grounding probability. From Eq. (3), nondegenerate LAD can span 0–2, so Eq. (7) is not intrinsically bounded by one. Positive rescaling of centered logits leaves LAD unchanged. Figure 5 also shows high sensitivity with only 8.79% accuracy. e03e04e14
The dependency premise lacks its promised detailed human-validation evidence in this PDF. Delta and gamma are introduced as thresholds without numerical settings here. Missing supplements prevent auditing filtering, annotator agreement and individual-sample necessity. e02e16
Figure 2 calls the bases U and duplicates a visual matrix label, whereas the method specifies right singular vectors V-hat. The results prose also calls Interactive-Omni-4B less misaligned than MiniCPM-4.5 despite its lower CMS. This report follows the equations and table while preserving both discrepancies. e05e11
The ablations show no repeated-run uncertainty and do not explicitly name their model. N=500 is justified as a cost compromise, but construction costs are not measured. Figure 4 does not show monotonic improvement with more anchors. e13e18
7.2 Questions for discussion
- How much of a CMS increase reflects useful sensory dependence rather than destabilized decisions?
- Would modality subspace overlap or different anchor domains change which stream is labeled strongest?
8. Reproducibility audit
8.1 Requirements and known gaps
A reconstruction requires the 617 evaluation examples, disjoint SocialOmni anchors, matching checkpoints, option-token extraction and hooks before the language head. Missing details include exact prompts, preprocessing, seed choices, hardware/software configuration, zero-norm LAD handling, and how MSA treats absent modalities during omission evaluation. Supplement-dependent dataset construction remains a prerequisite. Random’s sampling rule and complete variant equations are also unspecified in the supplied text; additional strategies are deferred to the missing supplement. e03e07e08e15e16e18
Proposed checks: separately implement a true intervention bypass and literal lambda=0, then repeat the anchor and balancing sweeps across seeds while reporting accuracy, ARR and LAD separately. Compare MSA to matched random steering and test whether sensitivity gains retain correctness. These would test mechanism specificity and stability; they are not completed reproductions. e07e13e14e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does modality-directed steering beat matched generic disruption?
Reader-proposed check, not performed: on the same held-out CausalMSBench examples and fixed anchor bases, compare an explicit no-intervention bypass, MSA, Reverse, and a fully specified random-direction control whose update norm is matched per example. Repeat at lambda 0.5 and 0.9 over several random seeds. Record full-input accuracy and each omission condition’s ARR and LAD separately, including transitions from correct to incorrect answers. Useful modality steering should improve sensitivity while preserving more correct answers than the matched controls. If generic perturbations reproduce the CMS gain with equal or better accuracy, the claim that the selected modality subspaces provide a special grounding benefit would be weakened. e03e04e07e10e15e18
Check 2: Are the anchor bases stable, and is the baseline implemented correctly?
Reader-proposed check, not performed: preserve the disjoint SocialOmni/evaluation separation and rank 64, then rebuild bases from repeated anchor draws at N=125, 250, 500 and 1000. Implement Eq. (8)’s feature-wise centering and right singular vectors explicitly. For each draw, evaluate both a true bypass and literal lambda zero, which Eq. (13) predicts will still suppress the strongest projection. Measure variation in projector outputs, accuracy and separate CMS components, together with construction time and peak memory. Stable gains across draws would support the claimed adequacy of 500 anchors; large variance or a materially different literal-zero baseline would reveal sensitivity hidden by the published single curves. Exact prompts and preprocessing must first be resolved. e06e07e08e13e14e18
8.3 Reading coverage
Visual audit: All nine supplied PDF pages were rendered and visually inspected: title, author and version block plus Figure 1 on page 1; conceptual assumptions on pages 2–3; Figure 2 and metric equations on page 4; all subspace, inference and benchmark settings on page 5; Table 1, Figure 3 and remaining settings on page 6; case discussion, Figures 4–6 and their supporting prose on page 7; references ending the artifact on pages 8–9. All five final crops, comprising Figures 2–5 and Table 1, were separately viewed at their helper-produced dimensions. Figure 6 and its previous crop were also inspected: its original high-lambda Random bar obscures the last digit of the accuracy label. Figure 6 is not a final visual; the full comparison is retained in the report and walkthrough using the readable labels and supporting page 7 prose. Figure 2’s basis and matrix-label discrepancies, Table 1’s conflicting prose ranking, and Figure 5’s bypass-versus-zero ambiguity are disclosed. No appendix appears in this PDF; separate supplements, code and external resources remain outside this reading. All numerical, method and proposed-check source premises are supported on the declared pages.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Introduction (pp. 1–2)
- Related Work: all three subsections (pp. 2–3)
- Method: Multi-Modal Dependency and Causal Modality Sensitivity (pp. 3–5)
- Method: Modality Subspace Activation, Eqs. (8)–(13) (p. 5)
- Experiment: Experimental Settings and Overall Results (pp. 5–6)
- Case Study; The Analysis and Discussion of MSA; Conclusion (p. 7)
- References (pp. 8–9)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: the title and all five authors match the catalog. The inspected margin identifies arXiv:2608.14655v1, 31 Jul 2026. A separate 2027 AAAI copyright line is preserved as printed, without inferring publication or a later edition. No revision comparison was supplied (e01).
- Acquisition omission: text extraction does not reconstruct figure images. This reading inspected all nine PDF pages, Figures 1–6 and Table 1, resolving that visual-access gap for the supplied artifact.
- Acquisition omission: separate supplemental material availability has not been fully verified. No supplement was supplied; promised dataset-construction details and extra steering variants remain outside the reading. The PDF itself contains no appendix.
- All four supplied text chunks were read individually and completely. Code and external resources were not inspected; no experiments were reproduced.
- Figure 6 was visually read, but an original bar overlaps the last digit of Random’s high-lambda accuracy label. The exact 38.44% result is independently printed in the adjacent page 7 prose and retained; this figure is not among the five final crops.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block, arXiv margin and copyright footnote
The exact title and five authors match the catalog. The PDF prints arXiv:2608.14655v1 [cs.LG], 31 Jul 2026, and a separate 2027 AAAI copyright notice. Affiliations are Xiamen University, Shanghai Artificial Intelligence Laboratory and Tencent YouTu Laboratory.
Go to primary source ↓e02PDF pp. 1–3, Introduction; Method / Multi-Modal Dependency, Eq. (1), and Causal Modality Sensitivity, Eq. (2)
The diagnostic argument requires indispensable modalities and reduced language-prior shortcuts. Equation (1) requires a positive human accuracy gap for every sample and evaluator between full input and the best proper modality subset; Eq. (2) describes low model sensitivity.
Go to primary source ↓e03PDF p. 4, Causal Modality Sensitivity, Eqs. (3)–(6)
LAD is one minus cosine similarity between mean-centered option-logit vectors under full and omitted modalities. ARR averages equality of option-logit argmax predictions, whether correct or incorrect.
Go to primary source ↓e04PDF p. 5, Eq. (7)
CMS averages one minus ARR and LAD. The formula does not normalize LAD by its maximum or define a zero-vector convention.
Go to primary source ↓e05PDF p. 4, Figure 2 and caption; p. 5, Eqs. (8)–(13)
The figure flows from dependency assumptions to omission diagnosis and MSA. It labels subspace bases U and duplicates the visual label on an audio-colored activation matrix; the equations instead specify top right singular vectors V-hat. The figure and Eq. (13) agree on adding the weak projection and subtracting the strong projection.
Go to primary source ↓e06PDF p. 5, Modality Subspace Activation / Subspace Construction, Eqs. (8)–(9)
Single-modality final hidden states form an N by d_model activation matrix. Equation (8) subtracts the feature-wise mean across anchor samples; Eq. (9) defines SVD and the following text retains the top k right singular vectors. Anchors must be disjoint from evaluation data.
Go to primary source ↓e07PDF p. 5, Inference-Time Activation Strength Perception and Dual-Direction Geometric Regulation, Eqs. (10)–(13)
The runtime state is projected onto each modality basis. Projection L2 norms determine weakest and strongest modalities. The corrected state adds lambda times the weakest projection and subtracts one minus lambda times the strongest, then enters the language-model head.
Go to primary source ↓e08PDF pp. 5–6, Experimental Settings / CausalMSBench, Subspace Construction Samples Source and Other Settings
CausalMSBench has 617 samples from JointAVBench, DailyOmni and newly collected examples. The authors select 500 random SocialOmni anchors, use k=64 and evaluate seven model variants under No-Visual, No-Audio and No-Visual&Audio conditions. The stated hidden-state dimensions span 2,000–4,000.
Go to primary source ↓e09PDF p. 6, Table 1, Qwen3-Omni-30B-A3B-Instruct and Qwen2.5-Omni-7B rows, Normal and Avg. Perturb. columns
Qwen3 full-input accuracy changes from 57.33 to 65.15 and average CMS from 26.40 to 30.58 with MSA. Qwen2.5-Omni-7B accuracy changes from 49.35 to 55.70 and CMS from 31.79 to 52.36.
Go to primary source ↓e10PDF p. 6, Table 1, all rows; Overall Results, final paragraph
Average CMS increases in all seven rows, while full-input accuracy improves in two and falls in five. OmniVinci accuracy is 35.34 without and 22.80 with MSA; CMS is 23.12 and 34.55. Interactive-Omni-4B accuracy is 47.88 and 40.07; CMS is 39.29 and 45.58.
Go to primary source ↓e11PDF p. 6, Table 1, MiniCPM-4.5 and Interactive-Omni-4B rows; Overall Results, second paragraph
MiniCPM-4.5 average CMS is 41.74 versus Interactive-Omni-4B 39.29. Their average accuracy drops are 9.60 and 11.13 percentage points. The prose nevertheless says Interactive-Omni-4B has lower misalignment, conflicting with the paper’s higher-CMS interpretation.
Go to primary source ↓e12PDF p. 6, Figure 3 and caption; p. 7, Case Study
Two Interactive-Omni-4B examples retain correct answers, with retention 1.0, after No-Visual&Audio and No-Visual perturbations respectively. Their displayed LAD values are 1.34 and 0.85. The figure shows queries and frames, but not option-logit vectors.
Go to primary source ↓e13PDF p. 7, Figure 4 and Sample Scale Sensitivity in Activation Matrix Construction
The anchor sweep uses N=125, 250, 500 and 1000 plus an unsteered baseline labeled N=0. Accuracy and CMS vary nonmonotonically. The authors select 500 for sample/representation efficiency and discuss larger construction costs without numerical cost measurements.
Go to primary source ↓e14PDF p. 7, Figure 5 and Sensitivity Analysis on Intervention Strength; p. 5, Eq. (13)
The lambda sweep shows 8.79% accuracy at 0.1 and 40.07% at 0.5; the authors choose 0.5 as a tradeoff. The plotted zero point is labeled without intervention, whereas literal lambda=0 in Eq. (13) still subtracts the strongest projection.
Go to primary source ↓e15PDF p. 7, Figure 6 and Different Enhancement Strategies
At default lambda=0.5, MSA has accuracy/CMS 40.07/45.58, Random 43.00/48.92, Equal 47.72/41.00 and Reverse 49.92/38.66. At lambda=0.9, Random is 38.44/61.06 and MSA 44.63/40.10. Equal enhances all modalities; Reverse boosts the dominant and suppresses the weak modality. The adjacent prose explicitly states Random’s 38.44% accuracy and 61.06 CMS at lambda=0.9, independently of the overlapping chart label.
Go to primary source ↓e16PDF p. 5, CausalMSBench paragraph; p. 7, Different Enhancement Strategies; pp. 8–9, References
Dataset construction and additional steering variants are deferred to supplementary material. The supplied nine-page PDF ends with references and contains no appendix or supplement.
Go to primary source ↓e17PDF p. 7, Conclusion; pp. 5–6, Experimental Settings and Table 1
The authors claim MSA reduces shortcut reliance by enhancing causal modality sensitivity. The reported experiments concern multimodal question answering, with no world-state rollout, robot action execution or closed-loop control evaluation.
Go to primary source ↓e18PDF pp. 5–7, Experimental Settings, Table 1 and Figures 4–6
The experimental account gives anchor source/count and rank, but does not supply hardware, software versions, exact prompts or seeds, preprocessing specifications, repeated-run uncertainty or numerical construction/latency costs. Figures 4–6 do not name the evaluated model in their captions.
Go to primary source ↓e19PDF p. 7, Figure 6, Random at lambda=0.9; Different Enhancement Strategies, right-column continuation
The high-lambda Random red CMS bar overlaps the end of its blue accuracy label. The adjoining prose explicitly reports Random accuracy of 38.44% and CMS of 61.06, and MSA accuracy of 44.63%; the MSA CMS label 40.10 is readable in the figure.
Go to primary source ↓8.5 Primary sources
Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activation ↗
PDF · 5,675 extracted words
Source fingerprint
0c17d381aaa9ec695e8001d292a98b37fa9ebf92596a6c169b88b399c0519729