BLoss-DDNet: bending loss and dual-task decoding network for overlapping cell nucleus segmentation of cervical clinical LBC images.

Introduction: Cervical cancer has become one of the most malignant tumors that threatens women's health worldwide. Liquid-based cytology (LBC) examination has become the most common screening method for detecting cervical cancer early and preventing it. Currently, nuclear segmentation technology for cervical clinical LBC images based on convolutional neural networks has become a vital means of assisting in the diagnosis of cervical cancer. However, the existing nuclear segmentation techniques fail to segment the nuclei of severely overlapping nuclei in highly aggregated cell clusters, which will inevitably lead to the misdiagnosis of cervical cancer pathology. Methods: Therefore, a novel bending loss and dual-task decoding network (Bloss-DDNet) is proposed for overlapping cell nucleus segmentation of cervical clinical LBC images. First, the network architecture search method is introduced to search and optimize the architecture of the decoding module in the dual-task branch, determining the mask and boundary decoding modules (dual-task decoding modules) of the Bloss-DDNet. Second, two feature maps, separately generated from dual-task decoding branches composed of a shared encoder module and dual-task decoder modules, are fused to enhance the sensitivity to cell nucleus boundaries. Third, a bending loss is introduced to the loss function to focus on the curvature variation characteristics of the intersection of overlapping cell nucleus boundaries, thereby constraining the training process of the dual-task decoding branch and increasing the constraint on the cell nucleus boundary. Results: The results show that all evaluation metrics of the proposed Bloss-DDNet achieved the best performance on public datasets. Discussion: Therefore, the proposed Bloss-DDNet can effectively address the segmentation problem of overlapping cell clusters and nuclei in clinical LBC images, providing strong support for subsequent clinical auxiliary diagnosis of cervical cancer.