Academic Journal
Segment anything in pathology images with natural language.
| Τίτλος: | Segment anything in pathology images with natural language. |
|---|---|
| Συγγραφείς: | Chen Z; Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China., Hou J; Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China., Lin L; School of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China., Wang Y; Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China., Bie Y; Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China., Wang X; Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China.; Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China., Zhou Y; Tencent AI Platform Department, Shenzhen, China., Li D; Department of Pathology, Nanfang Hospital, Southern Medical University, Guangzhou, China.; Department of Pathology, School of Basic Medical Sciences, Southern Medical University, Guangzhou, China., Tan H; Department of Anatomical and Cellular Pathology, The Chinese University of Hong Kong, Hong Kong SAR, China., Liang L; Department of Pathology, Nanfang Hospital, Southern Medical University, Guangzhou, China.; Department of Pathology, School of Basic Medical Sciences, Southern Medical University, Guangzhou, China., Chan RCK; Department of Anatomical and Cellular Pathology, The Chinese University of Hong Kong, Hong Kong SAR, China., Chen H; Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China. jhc@ust.hk.; Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China. jhc@ust.hk.; Division of Life Science, The Hong Kong University of Science and Technology, Hong Kong SAR, China. jhc@ust.hk.; HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, The Hong Kong University of Science and Technology, Futian, Shenzhen, China. jhc@ust.hk.; State Key Laboratory of Nervous System Disorders, The Hong Kong University of Science and Technology, Hong Kong SAR, China. jhc@ust.hk. |
| Πηγή: | Nature computational science [Nat Comput Sci] 2026 Sep; Vol. 6 (9), pp. 1024-1038. Date of Electronic Publication: 2026 Sep 10. |
| Τύπος έκδοσης: | Journal Article |
| Γλώσσα: | English |
| Στοιχεία περιοδικού: | Publisher: Springer Nature Country of Publication: United States NLM ID: 101775476 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 2662-8457 (Electronic) Linking ISSN: 26628457 NLM ISO Abbreviation: Nat Comput Sci Subsets: MEDLINE |
| Imprint Name(s): | Original Publication: [New York, N.Y.] : Springer Nature, [2021]- |
| Ιατρικοί όροι (MeSH): | Image Processing, Computer-Assisted*/methods , Image Interpretation, Computer-Assisted*/methods , Natural Language Processing*, Breast Neoplasms/diagnostic imaging ; Breast Neoplasms/pathology ; Humans ; Female ; Algorithms ; Databases, Factual |
| Περίληψη: | Segmenting tissues and cells in pathology images enables quantitative analysis but usually requires task-specific models or repeated spatial prompts. Here we show PathSegmentor, a foundation model that uses natural language descriptions to segment structures across anatomical regions and spatial scales. We assembled PathSeg from 21 public datasets, comprising 275,200 image-mask-label triples organized into a 3-level hierarchy of anatomical region, histological structure and object type. A single PathSegmentor model achieved the highest overall performance across 16 internal datasets and generalized to external public and clinical cohorts. Its text prompts reduced the need to identify every object with points or boxes and remained robust to variations in wording. We further used its predicted structures to explain breast cancer classification models through object-level perturbation and activation maps. These results establish a unified framework for flexible pathology segmentation with potential utility for clinically interpretable image analysis. (© 2026. The Author(s), under exclusive licence to Springer Nature America, Inc.) |
| Competing Interests: | Competing interests: The authors declare no competing interests. |
| References: | Xing, F. & Yang, L. Robust nucleus/cell detection and segmentation in digital pathology and microscopy images: a comprehensive review. IEEE Rev. Biomed. Eng. 9, 234–263 (2016). (PMID: 10.1109/RBME.2016.2515127) Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J. & Maier-Hein, K. H. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18, 203–211 (2021). (PMID: 10.1038/s41592-020-01008-z) Zhang, J., Ma, K., Kapse, S., Saltz, J., Vakalopoulou, M., Prasanna, P. & Samaras, D. SAM-Path: a segment anything model for semantic segmentation in digital pathology. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 Workshops (eds Celebi, M. E. et al.) 161–170 (Springer, 2023). Hörst, F. et al. CellViT++: energy-efficient and adaptive cell segmentation and classification using foundation models. Comput. Methods Programs Biomed. 277, 109206 (2026). (PMID: 10.1016/j.cmpb.2025.109206) Kirillov, A. et al. Segment anything. In Proc. IEEE/CVF International Conference on Computer Vision 4015–4026 (IEEE, 2023). Ma, J. et al. Segment anything in medical images. Nat. Commun. 15, 654 (2024). (PMID: 10.1038/s41467-024-44824-z) Cheng, J. et al. SAM-Med2D. Preprint at https://doi.org/10.48550/arXiv.2308.16184 (2023). Zhao, T. et al. A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities. Nat. Methods 22, 166–176 (2025). (PMID: 10.1038/s41592-024-02499-w) Archit, A. et al. Segment anything for microscopy. Nat. Methods 22, 579–591 (2025). Du, Y., Bai, F., Huang, T. & Zhao, B. Segvol: universal and interactive volumetric medical image segmentation. Adv. Neural Inf. Process. Syst. 37, 110746–110783 (2024). (PMID: 10.52202/079017-3516) Zhao, Z. et al. One model to rule them all: towards universal segmentation for medical images with text prompts. Preprint at https://doi.org/10.48550/arXiv.2312.17183 (2023). Wang, A., Islam, M., Xu, M., Zhang, Y. & Ren, H. SAM meets robotic surgery: an empirical study on generalization, robustness and adaptation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 Workshops (eds Celebi, M. E. et al.) 234–244 (Springer, 2023). Wang, C. et al. Seganypath: a foundation model for multi-resolution stain-variant and multi-task pathology image segmentation. IEEE Trans. Med. Imaging 44, 3924–3937 (2025). (PMID: 10.1109/TMI.2024.3501352) Zou, X. et al. Segment everything everywhere all at once. Adv. Neural Inf. Process. Syst. 36, 19769–19782 (2023). (PMID: 10.52202/075280-0868) Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. Encoder–decoder with atrous separable convolution for semantic image segmentation. In Proc. European Conference on Computer Vision (eds Ferrari, V. et al) 833–851 (Springer, 2018). Cox, E. A method of assigning numerical and percentage values to the degree of roundness of sand grains. J. Paleontol. 1, 179–183 (1927). Network, T. C. G. A. Comprehensive molecular portraits of human breast tumors. Nature 490, 61–70 (2012). (PMID: 10.1038/nature11412) Velden, B. H., Kuijf, H. J., Gilhuijs, K. G. & Viergever, M. A. Explainable artificial intelligence (XAI) in deep learning-based medical image analysis. Med. Image Anal. 79, 102470 (2022). (PMID: 10.1016/j.media.2022.102470) Hou, J. et al. Self-explainable AI for medical image analysis: a survey and new outlooks. Preprint at https://doi.org/10.48550/arXiv.2410.02331 (2024). Hou, J., Xu, J. & Chen, H. Concept-attention whitening for interpretable skin lesion diagnosis. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 (eds Linguraru, M. G. et al.) 113–123 (Springer, 2024). Diao, J. A. et al. Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes. Nat. Commun. 12, 1613 (2021). (PMID: 10.1038/s41467-021-21896-9) Kludt, C. et al. Next-generation lung cancer pathology: development and validation of diagnostic and prognostic algorithms. Cell Rep. Med. 5, 101697 (2024). (PMID: 10.1016/j.xcrm.2024.101697) Rosen, P. P. Rosen’s Breast Pathology (Lippincott Williams & Wilkins, 2001). Zhou, Z.-H. & Zhang, M.-L. Neural networks for multi-instance learning. In Proceedings of the International Conference on Intelligent Information Technology (eds Shi, Z. Z. & He, Q.) 455–459 (People’s Posts and Telecommunications Publishing House, 2002). Petsiuk, V., Das, A. & Saenko, K. RISE: randomized input sampling for explanation of black-box models. In British Machine Vision Conference 2018 151 (BMVA Press, 2018). Ribeiro, M. T., Singh, S. & Guestrin, C. “Why should I trust you?”: explaining the predictions of any classifier. In Proc. 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (eds Krishnapuram, B. et al.) 1135–1144 (ACM, 2016). Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. & Torralba, A. Learning deep features for discriminative localization. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 2921–2929 (IEEE, 2016). Pai, S. et al. Foundation model for cancer imaging biomarkers. Nat. Mach. Intell. 6, 354–367 (2024). (PMID: 10.1038/s42256-024-00807-9) Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019). (PMID: 10.1038/s42256-019-0048-x) Budd, S., Robinson, E. C. & Kainz, B. A survey on active learning and human-in-the-loop deep learning for medical image analysis. Med. Image Anal. 71, 102062 (2021). (PMID: 10.1016/j.media.2021.102062) Li, M., Li, S., Zhang, X. & Zhang, L. UniVS: unified and universal video segmentation with prompts as queries. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 3227–3238 (IEEE, 2024). Yang, J., Li, C., Dai, X. & Gao, J. Focal modulation networks. Adv. Neural Inf. Process. Syst. 35, 4203–4217 (2022). (PMID: 10.52202/068431-0304) Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthc. 3, 1–23 (2021). (PMID: 10.1145/3458754) Zou, X. et al. Generalized decoding for pixel, image, and language. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 15116–15127 (IEEE, 2023). Li, J., Li, D., Savarese, S. & Hoi, S. BLIP-2: bootstrapping language–image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning (eds Krause, A. et al.) 19730–19742 (PMLR, 2023). Sellergren, A. et al. MedGemma technical report. Preprint at https://doi.org/10.48550/arXiv.2507.05201 (2025). Gadermayr, M. & Tschuchnig, M. Multiple instance learning for digital pathology: a review of the state-of-the-art, limitations and future potential. Comput. Med. Imaging Graph. 112, 102337 (2024). (PMID: 10.1016/j.compmedimag.2024.102337) Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations (ICLR, 2019). Paszke, A. et al. PyTorch: an imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32, 8024–8035 (2019). Lu, M. Y. et al. A visual-language foundation model for computational pathology. Nat. Med. 30, 863–874 (2024). (PMID: 10.1038/s41591-024-02856-4) Ilse, M., Tomczak, J. & Welling, M. Attention-based deep multiple instance learning. In Proceedings of the 35th International Conference on Machine Learning (eds Dy, J. & Krause, A.) 2127–2136 (PMLR, 2018). Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5, 555–570 (2021). (PMID: 10.1038/s41551-020-00682-w) Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) (ICLR, 2015). Christgen, M. et al. Lobular breast cancer: clinical, molecular and morphological characteristics. Pathol. Res. Pract. 212, 583–597 (2016). (PMID: 10.1016/j.prp.2016.05.002) Milletari, F., Navab, N. & Ahmadi, S.-A. V-net: fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV) 565–571 (IEEE, 2016). Chen, Z. PathSegmentor v1.0.0. Zenodo https://doi.org/10.5281/zenodo.21800277 (2026). Amgad, M. et al. Structured crowdsourcing enables convolutional segmentation of histology images. Bioinformatics 35, 3461–3467 (2019). (PMID: 10.1093/bioinformatics/btz083) Han, C. et al. WSSS4LUAD: grand challenge on weakly-supervised tissue semantic segmentation for lung adenocarcinoma. Preprint at https://doi.org/10.48550/arXiv.2204.06455 (2022). Naylor, P., Laé, M., Reyal, F. & Walter, T. Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE Trans. Med. Imaging 38, 448–459 (2019). (PMID: 10.1109/TMI.2018.2865709) Graham, S. et al. CoNIC Challenge: pushing the frontiers of nuclear detection, segmentation, classification and counting. Med. Image Anal. 92, 103047 (2024). (PMID: 10.1016/j.media.2023.103047) |
| Grant Information: | 62202403 National Natural Science Foundation of China (National Science Foundation of China) |
| Entry Date(s): | Date Created: 20260910 Date Completed: 20260922 Latest Revision: 20260922 |
| Update Code: | 20260922 |
| DOI: | 10.1038/s43588-026-01042-5 |
| PMID: | 42722894 |
| Βάση Δεδομένων: | MEDLINE |
καταχωρήστε σχόλιο πρώτοι!