- [Resource Hub](/) - [Publication](/?resource-type=publication#library) - MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models # MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models Balezo G, Trullo R, Pla Planas A, Decenciere E, Walter T. arXiv . Posted May 15, 2025. arXiv [2505.10294](https://arxiv.org/abs/2505.10294). How to cite ### AMA Balezo G, Trullo R, Pla Planas A, Decenciere E, Walter T. MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models. arXiv . Posted May 15, 2025. arXiv:2505.10294 ### APA Balezo, G., Trullo, R., Pla Planas, A., Decenciere, E., & Walter, T. (2025). MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models. arXiv . https://arxiv.org/abs/2505.10294 ### BibTeX @misc{balezo2025miphei, title = {MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models}, author = {Balezo, Guillaume and Trullo, Roger and Pla Planas, Albert and Decenciere, Etienne and Walter, Thomas}, year = {2025}, eprint = {2505.10294}, archivePrefix = {arXiv}, primaryClass = {eess.IV} } A routine H&E slide shows tissue structure, but it cannot tell you which proteins mark which cells. Multiplex immunofluorescence can read many markers at once on the same section, yet its staining, instruments, and cost keep it off most slides. This group trained MIPHEI-ViT, a model built on a vision-transformer pathology foundation encoder, to predict marker signals and single-cell type calls directly from an ordinary H&E image. They trained and tested it on the publicly released Orion colorectal-cancer dataset, which pairs H&E with multiplex immunofluorescence from the same tissue, then checked it on five more independent datasets. The result points toward reading marker-level, cell-type-aware signal from H&E slides that already exist, with one honest limit: the model learns everything it knows from real multiplex data. [Read publication at arXiv](https://arxiv.org/abs/2505.10294) ## Key findings - MIPHEI-ViT calls single-cell types from H&E alone. On the OrionCRC test set it reached F1 scores of 0.93 for Pan-CK, 0.83 for alpha-SMA and 0.68 for CD3e. - Training and evaluation ran on the public OrionCRC dataset. It supplied 41 colorectal-cancer whole-slide images pairing H&E with 18-channel immunofluorescence on the same tissue sections, split into 37 training, 2 validation and 2 test slides. - The H&E-trained model generalized to 5 independent external datasets. Beyond OrionCRC, it was validated on HEMIT, PathoCell, IMMUcan, Lizard and PanNuke, each carrying domain shifts in mIF technology and H&E appearance. ## How this study used Orion data “All models are trained on H&E-mIF images from OrionCRC extracted as 256x256 pixel tiles at 0.5 mpp.” — Balezo et al., arXiv (2025), Training Setup, “DataConfiguration” Provenance: this study used the publicly available Orion CRC dataset (Lin et al., Nature Cancer 2023); the authors did not run a RareCyte instrument. Disclosure: RareCyte is listed as an author affiliation on the publication cited above. Disclosure: RareCyte is named in the competing-interests statement of the publication cited above. ## Why it matters for Orion users If you are weighing Orion for spatial work, look at what its public data was asked to do here. The authors ran no instrument of their own. They trained a foundation-model pipeline entirely on the OrionCRC dataset, and that data was strong enough to teach a model to call cell types from H&E alone. What makes Orion data usable for a problem like this is how it is made. Each colorectal-cancer slide carries H&E and an 18-channel immunofluorescence readout registered to the same cells, produced in a single staining and imaging round rather than from serial sections. Because the morphology and the markers come from one piece of tissue, a model can learn the mapping between them cell for cell, and 41 whole-slide images gave the study its training and test material. For your own work, that same-section registration is the payoff: when your H&E and your marker readout share the same coordinates, either one can feed tools built for the other. [Back to all resources](/#library) ## Related resources [Article](https://www.selectscience.net/article/advanced-spatial-biology-sheds-new-light-on-therapeutic-research) Orion SelectScience ### Advanced spatial biology sheds new light on therapeutic research [Read Article](https://www.selectscience.net/article/advanced-spatial-biology-sheds-new-light-on-therapeutic-research) [Article](https://www.selectscience.net/editorial-articles/bringing-spatial-biology-to-the-clinic-a-new-lens-on-cancer-biology/?&artID=58147) Orion SelectScience ### Bringing spatial biology to the clinic: “A new lens on cancer biology” [Read Article](https://www.selectscience.net/editorial-articles/bringing-spatial-biology-to-the-clinic-a-new-lens-on-cancer-biology/?&artID=58147) [Article](https://www.selectscience.net/article/how-to-unlock-the-complexity-of-tumor-microenvironments) Orion SelectScience ### How to unlock the complexity of tumor microenvironments [Read Article](https://www.selectscience.net/article/how-to-unlock-the-complexity-of-tumor-microenvironments) [Brochure](https://go.pardot.com/l/455452/2020-07-15/8p7453/455452/1606951069teslfZQe/BR21_101_Orion_Brochure_201202.pdf) Orion ### Orion Spatial Proteomics Platform [View Brochure](https://go.pardot.com/l/455452/2020-07-15/8p7453/455452/1606951069teslfZQe/BR21_101_Orion_Brochure_201202.pdf) [Spec Sheet](https://pardot.rarecyte.com/l/455452/2026-02-24/965v3w/455452/1771949067AeuR6syb/Orion_HT_spec_sheet.pdf) Orion ### Orion HT [View Spec Sheet](https://pardot.rarecyte.com/l/455452/2026-02-24/965v3w/455452/1771949067AeuR6syb/Orion_HT_spec_sheet.pdf) [Spec Sheet](https://pardot.rarecyte.com/l/455452/2025-10-31/961nyb/455452/1761933957WEJWJyeN/2025.07.14___SS___Orion__LE_.pdf) Orion ### Orion LE [View Spec Sheet](https://pardot.rarecyte.com/l/455452/2025-10-31/961nyb/455452/1761933957WEJWJyeN/2025.07.14___SS___Orion__LE_.pdf) [Spec Sheet](https://go.pardot.com/l/455452/2020-07-15/8p742r/455452/1607104572nfrlJouR/DF_24_101_201203_Orion_Datafile.pdf) Orion ### Orion [View Spec Sheet](https://go.pardot.com/l/455452/2020-07-15/8p742r/455452/1607104572nfrlJouR/DF_24_101_201203_Orion_Datafile.pdf) [Research Spotlight](/resources/spatial-profiling-immune-correlates-mss-crc/) Orion 2026 ### Spatial Profiling Uncovers Immune Correlates of Durable Response in Phase 1/2 MSS Colorectal Cancer Trial [Read Spotlight](/resources/spatial-profiling-immune-correlates-mss-crc/) [Publication](/resources/copanlisib-nivolumab-microsatellite-stable-colorectal/) Orion Nature Communications 2026 ### Copanlisib in combination with nivolumab for microsatellite stable colorectal cancer: a phase 1/2 trial [Read More](/resources/copanlisib-nivolumab-microsatellite-stable-colorectal/) [Publication](https://doi.org/10.1158/2159-8290.CD-26-0171) Orion Cancer Discovery 2026 ### Spatial Integration of Protein and Chromosomal States Reveals Early Copy-Number Changes and Genotype-Associated Immune Neighborhoods in Serous Ovarian Cancer Evolution [View Publication](https://doi.org/10.1158/2159-8290.CD-26-0171) [Publication](https://www.nature.com/articles/s41467-026-74933-w) Orion Nature Communications 2026 ### Multiple human transgenes prolong survival of triple-carbohydrate knockout porcine kidney xenografts in nonhuman primates [View Publication](https://www.nature.com/articles/s41467-026-74933-w) [Video](https://www.youtube.com/watch?v=KOiToZIGN0g) Orion 2026 ### Dr. Tanjina Kader on mapping spatial proteomics and chromosomal changes on the same tissue section [Watch Video](https://www.youtube.com/watch?v=KOiToZIGN0g)