Abnormal driving behavior recognition is to find a method to recognize abnormal driving behaviors correctly by analyzing the driver’s activities using image processing and pattern recognition technology. This method is composed of a structure of covariance matrices of image features, which is able to extract information from data. The proposed classification framework consists in a new multi-class boosting method, working on the manifold Sym+d of symmetric positive definite d*d (covariance) matrices. The correct recognition rate for the same target can reach 98%, and above 70% for different targets. The result shows that this method effectively improves the accuracy of abnormal driving behavior recognition.