In order to reduce the negative impact of the horror image on social development and adolescent growth, a violent image annotation algorithm based on ensemble learning is proposed, assisting in screening out the horror information in the webpage. The annotation of violent image is considered as a multi-label classification problem in this method. Multiple sub-networks are trained through transfer learning, and then the ensemble learning is introduced to fuse the outputs of sub-networks. In the process of fusion, weights are allocated according to the precision of each label on different networks, thus the annotation result is obtained through a series of matrix operations. The experimental results show that the proposed method achieves a great improvement in precision and recall than traditional machine learning algorithm, and also improves the problem that the precision of model annotation on different labels varies greatly due to the label category imbalance.