Chantal Pellegrini

About Me

I am a PhD candidate in Computer Science at the Technical University of Munich with Prof. Nassir Navab and previously interned at Apple, focusing on deep learning based verification of Siri Visual Responses. My research focuses on domain-specific vision–language models for medical environments, from fine-grained image understanding, collaborative radiology assistance, and operating room understanding to multimodal patient modeling and forecasting from Electronic Health Records.

CV: Download (PDF)
Location: Munich, Germany

Email: chantal.pellegrini(at)tum.de

Experience

Apple — AI/ML Intern

06/2025 – 09/2025

New York City, USA

Development of AI-based error detection for Siri Visual responses
Investigated both zero-shot use of foundation models as well as fine-tuning based approaches
Deployed final tool as web server and integrated in CI/CD pipeline for automated checks

Ph.D. Candidate

11/2022 – tentatively 11/2026

Technical University of Munich, Munich, Germany

Multimodal Medical Foundation Models: Worked on large vision-language models combining visual information, structured representations, and language models for interactive conversation and reporting in radiology and operating room (OR) understanding (Best Paper Runner-Up at MICCAI 2024).
Reinforcement Learning for LLMs: Reinforcement learning for calibrated confidence expression in large language models in factual question answering and for sequential clinical decision making, learning in LLM-simulated clinical environments.
Longitudinal Sequence Modeling with LLMs: Learning from multi-modal patient information for modeling and forecasting patient pathways during hospital stays.
Medical Image Understanding: Developed methods for fine-grained and interpretable medical image understanding, with a focus on explainable zero-shot X-ray diagnosis and structured radiology reporting via hierarchical visual question answering.
Teaching and supervision: Teaching courses in deep learning for medical imaging and supervising bachelor’s and master’s theses as well as practical courses.

Student Researcher

02/2020 – 08/2020

Itestra GmbH, Munich, Germany

Analysis of the usefulness of source code comments using classical ML and deep learning-based NLP
Development of a Visual Studio Code plugin to show analysed comment usefulness while coding

Software Developer

02/2019 – 01/2020

Qualicen GmbH, Munich, Germany

Backend development for automated requirements analysis
Development of an automated testing framework for the company’s software

Featured Publications

Rewarding Doubt: A Reinforcement Learning Approach to Confidence Calibration of Large Language Models

Accepted at ICLR 2026

Bani-Harouni, D.*, Pellegrini, C.*, Stangel, P., Özsoy, E., Keicher, M., & Navab, N.

Project Page Paper Code

From EHRs to Patient Pathways: Scalable Modeling of Longitudinal Health Trajectories with LLMs

under review

Pellegrini, C., Özsoy, E., Bani-Harouni, D., Keicher, M., & Navab, N.

Project Page arXiv Code

RaDialog: Large Vision-Language Models for X-Ray Reporting and Dialog-Driven Assistance

Medical Imaging with Deep Learning (MIDL), 2025

Pellegrini, C., Özsoy, E., Busam, B., Wiestler, B., Navab, N., & Keicher, M.

Project Page Publisher arXiv Poster Code

Oracle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling

MICCAI 2024 (Best Paper Runner-Up) · arXiv:2404.07031

Özsoy, E.*, Pellegrini, C.*, Keicher, M., & Navab, N.

Project Page arXiv Publisher Poster Code

Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosis

MICCAI 2023 · arXiv:2303.13391

Pellegrini, C., Keicher, M., Özsoy, E., Jiraskova, P., Braren, R., & Navab, N.

Project Page arXiv Publisher Poster Code

Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reporting

MICCAI 2023 · arXiv:2307.05766

Pellegrini, C., Keicher, M., Özsoy, E., & Navab, N.

Project Page arXiv Publisher Poster Code

Unsupervised Pre-training of Graph Transformers on Patient Population Graphs

Medical Image Analysis 89, 2023 · arXiv:2207.10603

Pellegrini, C., Navab, N., & Kazi, A.

Project Page arXiv Publisher Code

Additional Publications

D. Bani-Harouni, C. Pellegrini, E. Özsoy, M. Keicher, N. Navab. Language Agents for Hypothesis-driven Clinical Decision Making with Reinforcement Learning. Accepted at ICLR 2026. Paper
E. Özsoy, F. Holm, C. Pellegrini, T. Czempiel, M. Saleh, N. Navab, B. Busam. Location-free Scene Graph Generation. CVPR Workshops, 2025. Paper
E. Özsoy, C. Pellegrini, T. Czempiel, F. Tristram, K. Yuan, D. Bani-Harouni, et al. MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments. CVPR, 2025. Paper
E. Özsoy, C. Pellegrini, D. Bani-Harouni, K. Yuan, M. Keicher, N. Navab. ORQA: A Benchmark and Foundation Model for Holistic Operating Room Modeling. under review, 2025. Paper
E. Özsoy, T. Czempiel, F. Holm, C. Pellegrini, N. Navab. Labrad-or: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms. MICCAI, 2023. Paper
I. Ezhov, T. Mot, S. Shit, J. Lipkova, J.C. Paetzold, F. Kofler, C. Pellegrini, et al. Geometry-aware Neural Solver for Fast Bayesian Calibration of Brain Tumor Models. IEEE Transactions on Medical Imaging 41(5), 2021. Paper
C. Pellegrini, E. Özsoy, M. Wintergerst, G. Groh. Exploiting Food Embeddings for Ingredient Substitution. HEALTHINF, 2021. Paper

Chantal Pellegrini