Aiming to the problem of video target tracking in complex environment, a hybrid visual tracking method based on similarity optimization is proposed. Firstly, local cosine similarity is utilized to measure the similarity between target and candidate template, which can effectively suppress impulse noise caused by occlusion and light mutation, and improve the template matching accuracy. Secondly, discrimination weights of local targets are deduced based on the quadratic programming method of objective function, which effectively improves the discrimination ability of algorithm to target and background. Finally, in the process of system updating, discriminant updating of template is introduced, which effectively improves the model drift problem. The experimental results show that the proposed method can improve the tracking robustness and accuracy in complex and challenging environments.