SafetyGPT: An autonomous agent of electrical safety risks for monitoring workers’ unsafe behaviors

Workers’ unsafe behavior is one of the major causes of accidents in electric power production. Intelligent monitoring of workers’ unsafe behaviors can effectively prevent the expansion of safety risks, thereby blocking the development process of risks to accidents. Electric power production processes are diverse in nature and require the frequent switching of operating scenarios. This makes it difficult to identify what is “unsafe” since worker behaviors within the given electrical context also exhibit variability and diversity. Existing methods have insufficient generalization and adaptability, which makes them inadequate for the case of electric power production. Therefore, this paper proposes Safety Generative Pre-trained Transformers (SafetyGPT), an autonomous agent of safety risk based on a multi-modal large language model, which incorporates a human–machine collaborative monitoring mode for unsafe behaviors of workers. SafetyGPT loads the electric power production video, and the backend supervisors set instructions for SafetyGPT based on task requirements. The model encodes visual and textual features into corresponding tokens, realizes multi-modal feature alignment and fusion through the cross-attention mechanism, and then generates targeted responses through the large language model. Next, the proposed method is applied to real production site data to confirm the effectiveness and superiority through comparison with other methods designed to identify unsafe behaviors. Experimental results show that the accuracy of the proposed method for the identification of unsafe behaviors in complex environments is 96.5%, and that it can generate reasonable recommended plan based on the identification results, assist backend supervisors in making decisions, and effectively improve the safety level of power production.