HMSA-Net: A hierarchical multi-scale aggregation network for multimodal biomedical image segmentation

Medical image segmentation plays a vital role in clinical workflows such as disease diagnosis, treatment planning, and outcome monitoring. However, achieving robust segmentation across different anatomical regions, imaging modalities, and resolution scales remains a significant challenge. This paper presents a novel segmentation model, Hierarchical Multi-Scale Aggregation Network (HMSA-Net), designed to enhance segmentation performance in medical imaging. HMSA-Net follows a hierarchical encoder–decoder structure, where the encoder is built upon Res2Net, leveraging bottleneck layers to effectively extract multi-scale contextual features. The decoder integrates Hierarchical Attention Refinement Blocks (HARBs), which employ convolutional layers and squeeze-and-excitation mechanisms to dynamically recalibrate channel-wise feature responses, improving the model’s ability to emphasize critical anatomical structures. Additionally, HMSA-Net incorporates a multi-scale aggregation module, enabling effective fusion of features at different resolutions, thereby refining segmentation accuracy. Experimental evaluations on the BraTS2020 dataset demonstrate the model’s effectiveness, achieving Dice scores of 0.89 for whole tumor (WT), 0.81 for tumor core (TC), and 0.73 for enhancing tumor (ET). Furthermore, HMSA-Net was assessed on three unimodal medical imaging datasets: CVC ClinicDB, the 2018 Data Science Bowl, and ISIC-2018 skin lesion segmentation, achieving Dice scores of 90.5, 87.8, and 88.2, respectively. These results validate HMSA-Net’s capability to serve as a robust segmentation framework across both 2D and 3D medical imaging modalities.