Bag of Visual Words(BoVW) is the main solution in the current image classification field, whereas the synonymity and ambiguity of the visual words restrict the semantic expression ability of the model and reduce the accuracy of image classification. Aiming to the problem, an adaptive soft assignment method is proposed. Firstly, it analyzes the distance of the Scale Invariant Feature Transform(SIFT) features mapping to visual words, classifies these SIFT features according to certain rules, and applies adaptive allocation strategies to SIFT features with different fuzziness. Then, this paper analyzes the correlations between visual words and image categories via Chi-square model, and then removes the Visual Stop Words(VSW) and reconstructs the histograms. Finally, the images are classified by Support Vector Machine(SVM). The experimental results show that, the method can effectively reduce the impact of the visual words synonymity and ambiguity, and enhance the distinction of visual words, so as to improve the image classification accuracy.