A Unified Multi-Modal Mixture-of-Experts Model for Integrated Representation Learning in Pharmaceutical Sciences

• Anushree Bhople

  Author

The rapid advancement of large language models (LLMs) has opened up opportunities for AI applications in pharmaceutical sciences. However, integrating diverse biological data modalities continues to remain challenging.  We propose SciMind, a multi-modal mixture-of-experts (MoE) having the capability of integrated representation learning from pharmaceutical data sources, including biomedical text, DNA sequence, protein sequence, and molecular structure. The proposed method includes method-specific tokenization strategies, sparse expert routing mechanisms, and cross-modal pre-training for improved knowledge transfer across multiple biological representations. An expert initialization approach based on limited K-means and adaptive top-k routing can use the parameters effectively while preserving domain-specific knowledge. Through experimental evaluations across four applications, including biomedical natural language processing, molecular understanding, promoter prediction and protein-related tasks, SciMind achieves competitive performance against existing domain-specific and general-purpose models.  According to the results, unified multi-modal learning can assist in the representation quality, drug reasoning capabilities, and applications in drug acceptance, molecule analysis, and personalized medicine.

• PDF

Copyright (c) 2026 International Journal of Clinical Research and Medical Sciences

This work is licensed under a Creative Commons Attribution 4.0 International License.