It is difficult to detect the small-signal due to the influence of strong interference signals. Based on the analysis of recursive Least Squares Algorithm(RLS) principle and its improved forms, adaptive cancellation is performed on the detected large-signal with the original aliasing signal to reduce the shielding effect by the large signal, then the small signal can be detected. The simulation shows that this method can effectively detect small FM signal with large AM signal interference in little distortion; the pros and cons of various algorithms are also compared, it is concluded that the VFF(Variable Forgetting Factor)-RLS algorithm bears not only a faster convergence rate, but also good stability after the convergence.