An auto-adaptive tracking algorithm for fast moving target is put forward based on the improved traditional Mean Shift tracking algorithm, in order to achieve good tracking of fast moving target with variable scale. This algorithm firstly adopts the color image constituted by the pixels of target region with spatial weighting as initial frame object template, and the true position of target is obtained by the iteration of Mean Shift algorithm, therefore the spatial localization of fast moving target is realized. Then the features of adjacent frame targets are matched by Scale Invariant Feature Transform(SIFT) operator; the kernel bandwidth of next frame is updated in real time according to the scaling factor of target; the tracking window size of the algorithm is amended, which can adapt to the variable scales of the target, so the scale localization of fast moving target is achieved. Finally, the experiments demonstrate that compared with the traditional Mean Shift tracking algorithm, the tracking accuracy rate of the algorithm is above 97%, and the algorithm can accurately track the fast moving target with variable scales.