New GSCo Framework Significantly Improves Diagnostic Accuracy and Enables Low-Cost AI Model Development

HKUST Develops Novel AI Framework Enabling Efficient Collaboration between Generalist and Specialist Models for Disease Diagnosis

Prof. Chen Hao (center), Assistant Professor of the Department of Computer Science and Engineering and the Department of Chemical and Biological Engineering and Director of the Collaborative Center for Medical and Engineering Innovation at HKUST, together with co-first authors Mr. HE Sunan (left) and Mr. NIE Yuxiang (right), both PhD students from the Department of Computer Science and Engineering, in collaboration with an international team, have developed the novel medical AI framework GSCo.
Prof. Chen Hao (center), Assistant Professor of the Department of Computer Science and Engineering and the Department of Chemical and Biological Engineering and Director of the Collaborative Center for Medical and Engineering Innovation at HKUST, together with co-first authors Mr. HE Sunan (left) and Mr. NIE Yuxiang (right), both PhD students from the Department of Computer Science and Engineering, in collaboration with an international team, have developed the novel medical AI framework GSCo. [Download Photo]

A research team from The Hong Kong University of Science and Technology (HKUST) has developed a novel AI framework, the Generalist–Specialist Collaboration (GSCo), which enables a generalist foundation model and specialist models to collaborate on disease diagnosis. GSCo outperforms both standalone models in tasks such as medical imaging, visual question answering, and radiology report generation, while significantly reducing development costs of medical models to adapt to new clinical tasks. This marks a meaningful step toward bringing AI medical models into everyday clinical workflows. The study, titled “Towards generalizable AI in medicine via Generalist–Specialist Collaboration,” was published in the prestigious academic journal Nature Biomedical Engineering.

The GSCo framework was developed by a research team led by Prof. CHEN Hao, Assistant Professor of the Department of Computer Science and Engineering and the Department of Chemical and Biological Engineering and Director of the Collaborative Center for Medical and Engineering Innovation, in collaboration with Harvard Medical School, Weill Cornell Medicine, The Chinese University of Hong Kong, The University of Hong Kong, Sun Yat-sen University and other partner institutions.

Medical AI models today face a long-standing dilemma. Generalist foundation models, trained on vast medical corpora, can flexibly handle diverse tasks and interpret imaging modalities, yet they often lack the precision required for diagnosing specific diseases. Specialist models, while offering precise clinical reasoning and expert diagnoses, are inherently narrow. A new model is needed for every new task or hospital, and specialist models cannot effectively engage in natural language reasoning with clinicians, as their accuracy drops sharply when faced with unfamiliar medical images. Re-training a generalist model for new tasks is computationally expensive and often practically infeasible, primarily because patient data cannot be transferred outside the hospital due to privacy and regulatory compliance requirements. Consequently, neither approach alone is well suited to real clinical deployment.

To address these challenges, this international, multi-institutional team has developed a novel cooperative AI framework named GSCo. The framework is built upon MedDr, an open-source generalist model developed by Prof. Chen Hao and his team. For this research, MedDr was trained on more than two million samples across diverse medical modalities. Concurrently, a suite of lightweight specialist models was developed, requiring only a single consumer-grade graphics processing unit (GPU) for training. During inference, the specialist models identify and compare similar historical cases and provide diagnostic predictions, while MedDr acts as the core decision-maker, integrating the evidence provided by the specialist models with its own medical knowledge to produce the final diagnosis.

To validate GSCo’s performance, the research team conducted comprehensive evaluation of MedDr and GSCo on 32 public datasets, comprising approximately 260,000 medical images. These tests encompassed tasks such as medical image diagnosis, visual question answering, and radiology report generation. The results showed that MedDr outperforms prior medical generalist models such as RadFM, LLaVA-Med, and Med-Flamingo, while GSCo secured the top overall ranking in both internal and external benchmark tests, outperforming ten state-of-the-art vision foundation models.

In stringent skin lesion tests, the team trained specialist models on the HAM10000 dataset, which comprises dermatoscopic images, using GSCo, and then deployed them on the previously unseen BCN20000 dataset for validation. GSCo achieved a diagnostic performance score of 0.8420 out of a perfect 1.0. This significantly surpassed the MedDr’s score of 0.7545 under zero-shot conditions and the specialist model’s best performance of 0.8292.

In a human evaluation focused on chest X-ray report generation, six out of seven board-certified radiologists preferred GSCo’s reports over those generated by the state-of-the-art specialist model, R2GenGPT. Stress tests further confirmed that MedDr is a robust arbitrator: even when presented with systematically biased input suggesting that all pathology images were normal, MedDr was still able to correctly identify 67.6% of tumor images. This resilience significantly enhances GSCo’s overall accuracy and robustness.

Crucially, adapting GSCo for a new clinical task can reduce development costs by up to 100-fold compared with fine-tuning a generalist foundation model. Specialist models built with GSCo can be trained locally within hospitals using institutional data and then shared anonymously, ensuring that patient records never leave the institution.

Prof. Chen Hao, the corresponding author of the paper, said, “Generalists and specialists in medical AI each possess distinct strengths—one is flexible but imprecise across many diseases, while the other is accurate but narrow, focusing on a single task. The core value of GSCo lies in facilitating their synergistic collaboration, where the generalist model acts as the decision-maker, integrating expert advice with its own broader medical knowledge. We hope this novel framework offers a more practical and sustainable pathway for integrating medical foundation models into everyday clinical workflows, particularly enabling healthcare settings with limited AI computational resources to share in the benefits of AI-powered medical innovation.”

Looking ahead, the research team plans to extend MedDr’s capabilities to encompass 3D imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI), as well as medical video. They will also explore more advanced multimodal and graph-based retrieval strategies and investigate diverse collaborations between generalists and specialists. This includes utilizing test-time computation to enable models to reason more deeply, thereby enhancing the accuracy of their results. Both open-source MedDr and GSCo frameworks have been publicly released to catalyze further research and clinical adoption. Details are available at https://github.com/sunanhe/MedDr.

About The Hong Kong University of Science and Technology
The Hong Kong University of Science and Technology (HKUST) (https://hkust.edu.hk/) is a world-class university known for its innovative education, research excellence, and impactful knowledge transfer. With a holistic and interdisciplinary pedagogical approach, HKUST was ranked 33rd and 6th in the QS World University Rankings 2027 and QS Asia University Rankings 2026, respectively, and 20th globally in the Times Higher Education Sustainability Impact Ratings 2026, ranking first among universities in Hong Kong and the Chinese Mainland for the third consecutive year. Eleven HKUST subjects were ranked among the world’s top 50 in the QS World University Rankings by Subject 2026. In addition, in the Times Higher Education World University Rankings by Subject 2026, HKUST’s Computer Science discipline which encompasses areas such as artificial intelligence and machine learning, has been ranked No. 1 in Hong Kong for ten consecutive years. Our graduates are highly competitive, consistently ranking among the world’s top 30 most sought-after employees. In terms of research and entrepreneurship, over 80% of our work was rated “internationally excellent” or “world leading” in the Research Assessment Exercise 2020 of the Hong Kong’s University Grants Committee. As of May 2026, HKUST members have founded over 1,900 active start-ups, including 11 Unicorns and 22 exits (IPO or M&A).

(This news was originally published by the HKUST Global Engagement and Communications Office here.)

Computer Vision
Medical Imaging
Computer Science and Engineering
Chemical and Biological Engineering
Research
Share
Overview of the GSCo framework. In the construction stage, the team builds MedDr, an open-source medical generalist model trained on a large-scale multi-modal medical corpus, alongside a suite of lightweight specialist models tailored to individual downstream tasks. In the collaborative inference stage, the specialists provide diagnostic predictions, referencing visually similar retrieved cases as structure context. MedDr then integrates this evidence, combining specialists’ input with its own broader medic
Overview of the GSCo framework. In the construction stage, the team builds MedDr, an open-source medical generalist model trained on a large-scale multi-modal medical corpus, alongside a suite of lightweight specialist models tailored to individual downstream tasks. In the collaborative inference stage, the specialists provide diagnostic predictions, referencing visually similar retrieved cases as structure context. MedDr then integrates this evidence, combining specialists’ input with its own broader medic
Illustration of GSCo’s two core collaboration mechanisms. In Mixture-of-Expert Diagnosis (MoED, top), multiple lightweight specialist models are independently trained on the downstream dataset. During inference, the test image is fed to all specialist models, and their predictions are incorporated into MedDr’s prompt as structured references. In Retrieval-Augmented Diagnosis (RAD, bottom), an embedding model encodes each image–text pair into a medical database. At inference, the test image’s embedding is us
Illustration of GSCo’s two core collaboration mechanisms. In Mixture-of-Expert Diagnosis (MoED, top), multiple lightweight specialist models are independently trained on the downstream dataset. During inference, the test image is fed to all specialist models, and their predictions are incorporated into MedDr’s prompt as structured references. In Retrieval-Augmented Diagnosis (RAD, bottom), an embedding model encodes each image–text pair into a medical database. At inference, the test image’s embedding is used to retrieve the most similar cases, whose metadata are subsequently provided to MedDr as additional contextual evidence.
Computer Vision
Medical Imaging
Computer Science and Engineering
Chemical and Biological Engineering
Research
Share