A Q-learning-based hierarchical routing protocol in underwater acoustic sensor networks

In order to guarantee a reliable data forwarding process, underwater acoustic sensor networks (UASNs), which are widely used in water environments like oceans and seas, need efficient routing protocols. Because sensor nodes are expensive to deploy in underwater environments and have limited energy capacities, energy optimization is a significant and practical issue, particularly for extending network lifetime. Today, various energy-efficient routing strategies have been suggested by combining opportunistic routing (OR) and reinforcement learning (RL). However, this subject deals still with different challenges. This paper introduces a Q-learning-based hierarchical routing protocol (QHRP) in UASNs. This approach builds a Q-learning-based routing tree, which contains a state set filtered by a two-step filtering process. It effectively increases the convergence speed of the Q-learning algorithm and lowers delay due to the tree construction process. In QHRP, the reward function considers network conditions and is obtained based on four metrics, namely remaining energy, strategic depth, the size of the state set, and successful transmission probability. Moreover, QHRP solves the void area problem in the routing tree by redefining the set of states and reward function. To evaluate QHRP compared to the three routing methods, namely RLOR, EE-DBR, and MURAO, various experiments are performed in terms of packet delivery rate (PDR), end-to-end delay (EED), data integrity, consumed energy, and the number of hops in the forwarding routes. These results show that QHRP improves PDR, delay, data integrity, energy consumption, and the number of hops by 9.068%, 9.03%, 9.84%, 15.61%, and 10.31%, respectively.