Academic Journal
View in EDS

Depth-based visual capture of sound in virtual reality is driven by perceived distance.

Bibliographic Details
Title: Depth-based visual capture of sound in virtual reality is driven by perceived distance.
Authors: McCracken MK; Department of Psychology, University of Utah, 380 S 1530 E, Salt Lake City, UT, 84112, USA. maggie.mccracken@psych.utah.edu., Rowland ME; Department of Psychology, University of Utah, 380 S 1530 E, Salt Lake City, UT, 84112, USA., Finney HC; Department of Psychology, University of Utah, 380 S 1530 E, Salt Lake City, UT, 84112, USA., Stefanucci JK; Department of Psychology, University of Utah, 380 S 1530 E, Salt Lake City, UT, 84112, USA., Creem-Regehr SH; Department of Psychology, University of Utah, 380 S 1530 E, Salt Lake City, UT, 84112, USA.
Source: Attention, perception & psychophysics [Atten Percept Psychophys] 2026 Aug 31; Vol. 88 (7). Date of Electronic Publication: 2026 Aug 31.
Publication Type: Journal Article
Language: English
Journal Info: Publisher: Springer Country of Publication: United States NLM ID: 101495384 Publication Model: Electronic Cited Medium: Internet ISSN: 1943-393X (Electronic) Linking ISSN: 19433921 NLM ISO Abbreviation: Atten Percept Psychophys Subsets: MEDLINE
Imprint Name(s): Publication: 2011- : New York : Springer
Original Publication: Austin, Tex. : Psychonomic Society
MeSH Terms: Distance Perception*/physiology , Sound Localization*/physiology , Depth Perception*/physiology , Visual Perception*/physiology , Virtual Reality*, Humans ; Female ; Male ; Adult ; Young Adult ; Cues
Abstract: Visual information typically dominates auditory cues in spatial localization, particularly in depth, leading to the proximity image effect (PIE) in which displaced sounds are perceived closer to plausible visual objects than they actually are (visual capture). It is unclear whether fundamental properties of audiovisual integration generalize to virtual environments where spatial perception is systematically distorted. Across two experiments, we tested whether real-world sound sources are visually captured by virtual objects in immersive virtual reality (VR). Participants completed a perceptual-matching task in which they indicated when a real-world sound source aligned with the depth of a virtual object. In Experiment 1, visual capture reliably occurred; however, the effect was opposite to real-world findings, with stronger capture happening when the sound originated in front of the visual object rather than behind it. Experiment 2 replicated this pattern and revealed significant visual distance underestimation measured with a blind-walking task. When judgments of auditory-visual co-location were interpreted relative to the participants' perceived visual distance rather than the actual physical distance, the expected real-world symmetry reemerged. These results demonstrate that core principles of multisensory integration are preserved in VR, but VR-specific biases, such as the visual underestimation of distance, may alter the direction of the established effects.
(© 2026. The Psychonomic Society, Inc.)
Competing Interests: Declarations. Conflicts of interest: All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or nonfinancial interest in the subject matter or materials discussed in this manuscript. Ethics approval: The study protocol was approved by the University of Utah Institutional Review Board. Consent to participate: Written informed consent was obtained for all participants. Consent to publication: Participants signed informed consent regarding publishing their data.
References: Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14(3), 257–262. https://doi.org/10.1016/j.cub.2004.01.029. (PMID: 10.1016/j.cub.2004.01.02914761661)
Bertelson, P., & Radeau, M. (1981). Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Perception & Psychophysics, 29(6), 578–584. https://doi.org/10.3758/BF03207374. (PMID: 10.3758/BF03207374)
Bertelson, P., Vroomen, J., De Gelder, B., & Driver, J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception & Psychophysics, 62(2), 321–332. https://doi.org/10.3758/BF03205552. (PMID: 10.3758/BF03205552)
Bonath, B., Noesselt, T., Martinez, A., Mishra, J., Schwiecker, K., Heinze, H.-J., & Hillyard, S. A. (2007). Neural basis of the ventriloquist illusion. Current Biology, 17(19), 1697–1703. https://doi.org/10.1016/j.cub.2007.08.050. (PMID: 10.1016/j.cub.2007.08.05017884498)
Bruns, P., & Röder, B. (2015). Sensory recalibration integrates information from the immediate and the cumulative past. Scientific Reports, 5(1), 12739. https://doi.org/10.1038/srep12739. (PMID: 10.1038/srep12739262380894523860)
Buck, L., Paris, R., & Bodenheimer, B. (2021). Distance compression in the HTC Vive Pro: A quick revisitation of resolution. Frontiers in Virtual Reality, 2, 28667. https://doi.org/10.3389/frvir.2021.728667. (PMID: 10.3389/frvir.2021.728667)
Calcagno, E. R., Abregú, E. L., Eguía, M. C., & Vergara, R. (2012). The role of vision in auditory distance perception. Perception (London, England), 41(2), 175–192. https://doi.org/10.1121/1.4799881. (PMID: 10.1121/1.4799881)
Champely, S. (2023). Pwr: Basic functions for power analysis [R Package Version 1.3-0]. https://github.com/heliosdrm/pwr . Accessed 31 Jan 2026.
Creem-Regehr, S. H., Stefanucci, J. K., & Bodenheimer, B. (2023). Perceiving distance in virtual reality: Theoretical insights from contemporary technologies. Philosophical Transactions of the Royal Society B: Biological Sciences, 378(1869), 20210456. https://doi.org/10.1098/rstb.2021.0456. (PMID: 10.1098/rstb.2021.0456)
Creem-Regehr, S. H., Stefanucci, J. K., & Thompson, W. B. (2015). Perceiving absolute scale in virtual environments: How theory and application have mutually informed the role of body-based perception. In B. H. Ross (Ed.), Psychology of learning and motivation (vol. 62, pp. 195–224). Elsevier. https://doi.org/10.1016/bs.plm.2014.09.006. (PMID: 10.1016/bs.plm.2014.09.006)
Creem-Regehr, S. H., Stefanucci, J. K., Thompson, W. B., Nash, N., & McCardell, M. (2015). Egocentric distance perception in the Oculus Rift (DK2). Proceedings of the ACM SIGGRAPH Symposium on Applied Perception (pp. 47–50). ACM. https://doi.org/10.1145/2804408.2804422. (PMID: 10.1145/2804408.2804422)
Cutting, J. E., & Vishton, P. M. (1995). Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth. In W. Epstein & S. Rogers (Eds.), Perception of space and motion (pp. 69–117). Elsevier. https://doi.org/10.1016/B978-012240530-3/50005-5. (PMID: 10.1016/B978-012240530-3/50005-5)
Delong, P., & Noppeney, U. (2021). Semantic and spatial congruency mould audiovisual integration depending on perceptual awareness. Scientific Reports, 11(1), 10832. https://doi.org/10.1038/s41598-021-90183-w. (PMID: 10.1038/s41598-021-90183-w340353588149651)
Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 5. https://doi.org/10.1038/415429a. (PMID: 10.1038/415429a)
Frangenberg, N., Waldow, K., & Fuhrmann, A. (2021). Investigating the influence of sound source visualization on the ventriloquism effect in an auralized virtual reality environment. IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 609–610). IEEE. https://doi.org/10.1109/VRW52623.2021.00187. (PMID: 10.1109/VRW52623.2021.00187)
Gardner, M. B. (1968). Proximity image effect in sound localization. The Journal of the Acoustical Society of America, 43(1), 163–163. https://doi.org/10.1121/1.1910747. (PMID: 10.1121/1.19107475636394)
Huisman, T., Ahrens, A., & MacDonald, E. (2021). Ambisonics sound source localization with varying amount of visual information in virtual reality. Frontiers in Virtual Reality, 2, 722321. https://doi.org/10.3389/frvir.2021.722321. (PMID: 10.3389/frvir.2021.722321)
Huisman, T., Dau, T., Piechowiak, T., & MacDonald, E. (2022). The ventriloquist effect is not consistently affected by stimulus realism. Journal of Perceptual Imaging, 5, 1–10. https://doi.org/10.2352/J.Percept.Imaging.2022.5.000404. (PMID: 10.2352/J.Percept.Imaging.2022.5.000404)
Interrante, V., Ries, B., & Anderson, L. (2006). Distance perception in immersive virtual environments, revisited. IEEE virtual reality conference (VR 2006) (pp. 3–10). IEEE. https://doi.org/10.1109/VR.2006.52. (PMID: 10.1109/VR.2006.52)
Kelly, J. W. (2022). Distance perception in virtual reality: A meta-analysis of the effect of head-mounted display characteristics. IEEE Transactions on Visualization and Computer Graphics (pp. 1–13). IEEE. https://doi.org/10.1109/TVCG.2022.3196606. (PMID: 10.1109/TVCG.2022.3196606)
Loomis, J. M., Da Silva, J. A., Fujita, N., & Fukusima, S. S. (1992). Visual space perception and visually directed action. Journal of Experimental Psychology: Human Perception and Performance, 18(4), 906. (PMID: 1431754)
Loomis, J. M., Da Silva, J. A., Fujita, N., & Fukusima, S. S. (1992). Visual space perception and visually directed action. Journal of Experimental Psychology: Human Perception and Performance, 18(4), 906–921. https://doi.org/10.1037/0096-1523.18.4.906. (PMID: 10.1037/0096-1523.18.4.9061431754)
Lorenzi, C., Gatehouse, S., & Lever, C. (1999). Sound localization in noise in normal-hearing listeners. The Journal of the Acoustical Society of America, 105(3), 1810–1820. https://doi.org/10.1121/1.426719. (PMID: 10.1121/1.42671910089604)
Melo, M., Goncalves, G., Monteiro, P., Coelho, H., Vasconcelos-Raposo, J., & Bessa, M. (2022). Do multisensory stimuli benefit the virtual reality experience? A systematic review. IEEE Transactions on Visualization and Computer Graphics, 28(2), 1428–1442. https://doi.org/10.1109/TVCG.2020.3010088. (PMID: 10.1109/TVCG.2020.301008832746276)
Mendonça, C., Mandelli, P., & Pulkki, V. (2016). Modeling the perception of audiovisual distance: Bayesian causal inference and other models (C. F. Altmann, Ed.). PLOS ONE, 11(12), e0165391. https://doi.org/10.1371/journal.pone.0165391. (PMID: 10.1371/journal.pone.0165391279599195154506)
Mershon, D., Desaulniers, D., Amerson, T., & Kiefer, S. (1980). Visual capture in auditory distance perception: Proximity image effect reconsidered. Journal of Auditory Research, 20(2), 1129–136.
Odegaard, B., Wozny, D. R., & Shams, L. (2015). Biases in visual, auditory, and audiovisual perception of space (A. A. Faisal, Ed.). PLOS Computational Biology, 11(12), e1004649. https://doi.org/10.1371/journal.pcbi.1004649. (PMID: 10.1371/journal.pcbi.1004649266463124672909)
Parise, C., & Spence, C. (2008). Synesthetic congruency modulates the temporal ventriloquism effect. Neuroscience Letters, 442(3), 257–261. https://doi.org/10.1016/j.neulet.2008.07.010. (PMID: 10.1016/j.neulet.2008.07.01018638522)
Recanzone, G. H. (2009). Interactions of auditory and visual stimuli in space and time. Hearing Research, 258(1/2), 89–99. https://doi.org/10.1016/j.heares.2009.04.009. (PMID: 10.1016/j.heares.2009.04.009193933062787663)
Renner, R. S., Velichkovsky, B. M., & Helmert, J. R. (2013). The perception of egocentric distances in virtual environments—A review. ACM Computing Surveys, 46(2), 1–40. https://doi.org/10.1145/2543581.2543590. (PMID: 10.1145/2543581.2543590)
Rieser, J. J., Ashmead, D. H., Talor, C. R., & Youngquist, G. A. (1990). Visual perception and the guidance of locomotion without vision to previously seen targets. Perception, 19(5), 675–689. https://doi.org/10.1068/p190675. (PMID: 10.1068/p1906752103000)
Rohe, T., & Noppeney, U. (2015). Cortical hierarchies perform Bayesian causal inference in multisensory perception (C. Kayser, Ed.). PLOS Biology, 13(2), e1002073. https://doi.org/10.1371/journal.pbio.1002073. (PMID: 10.1371/journal.pbio.1002073257103284339735)
Roßkopf, S., Mühlberger, A., Stärz, F., De Par, S. V., Blau, M., & Kroczek, L. O. (2025). Impact of Visual virtual scene and localization task on auditory distance perception in virtual reality. IEEE Transactions on Visualization and Computer Graphics (pp. 1–11). IEEE. https://doi.org/10.1109/TVCG.2025.3549855. (PMID: 10.1109/TVCG.2025.3549855)
Roßkopf, S., Mühlberger, A., Stärz, F., Van De Par, S., Blau, M., & Kroczek, L. O. (2025b). Impact of visual virtual scene and localization task on auditory distance perception in virtual reality. IEEE Transactions on Visualization and Computer Graphics. https://doi.org/10.1109/TVCG.2025.3549855.
Slutsky, D. A., & Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect. NeuroReport, 12(1), 7–10. https://doi.org/10.1097/00001756-200101220-00009. (PMID: 10.1097/00001756-200101220-0000911201094)
Thompson, W. B., Willemsen, P., Gooch, A. A., Creem-Regehr, S. H., Loomis, J. M., & Beall, A. C. (2004). Does the quality of the computer graphics matter when judging distances in visually immersive environments? Presence, 13(5), 560–571. https://doi.org/10.1162/1054746042545292. (PMID: 10.1162/1054746042545292)
Vroomen, J., Bertelson, P., & de Gelder, B. (1998). A visual influence in the discrimination of auditory location. In D. Burnham, J. Robert-Ribes, & E. Vatikiotos-Bateson (Eds.), Proceedings of the International Conference on Auditory-Visual Speech Processing AVSP’98 (pp. 131–135). Causal Production.
Zahorik, P. (2001). Estimating sound source distance with and without vision. Optometry and Vision Science, 78(5), 270–275. https://doi.org/10.1097/00006324-200105000-00009. (PMID: 10.1097/00006324-200105000-0000911384003)
Zahorik, P. (2022). Asymmetric visual capture of virtual sound sources in the distance dimension. Frontiers in Neuroscience, 16, 958577. https://doi.org/10.3389/fnins.2022.958577. (PMID: 10.3389/fnins.2022.958577361176379475063)
Grant Information: 2139322 National Science Foundation Graduate Research Fellowship Program
Contributed Indexing: Keywords: 3D perception: Cue integration; Multisensory Processing; Spatial localization
Entry Date(s): Date Created: 20260831 Date Completed: 20260901 Latest Revision: 20260831
Update Code: 20260901
DOI: 10.3758/s13414-026-03291-y
PMID: 42675284
Database: MEDLINE
Description
ISSN:1943-393X
DOI:10.3758/s13414-026-03291-y