Implementing a hybrid deep learning technique for detecting malicious DNS over HTTPS (DoH) traffic

In cybersecurity, the detection of malicious activities within domain name system (DNS) over HTTPS (DoH) traffic is of paramount importance. However, traditional classification methods often struggle to generalize across diverse network environments and effectively handle the complexities inherent in DNS traffic data. To solve these challenges, this article proposes a novel deep learning based on DoH traffic classification. A hybrid deep learning technique is proposed for detecting malicious DNS traffic. This approach forces the strengths of graph neural networks and capsule networks for effective classification. The method aims to enhance detection accuracy and improve response times in identifying threats. This study focuses on enhancing performance through optimal hyperparameter selection. The golden jackal optimization algorithm is employed for this purpose, hybridized with capsule networks. This approach aims to improve classification accuracy and efficiency in the targeted application. For the L1-DoH-NonDoH dataset, results show strong performance across metrics with an accuracy of 99.7%, precision of 98.5%, recall of 99.7%, and F1-score of 99.3%. Similarly, for the L2-Benign DoH-Malicious DoH dataset, the model achieved an accuracy of 99.8%, precision of 98.9%, recall of 99.8%, and F1-score of 99%. These results validate the efficacy of the method in distinguishing both benign and malicious traffic across various metrics.