With rapid development of communication technology and the way of communication, the modulation type of communication signal and communication system become more and more complex. Due to the high spectrum utilization rate, Continuous-Phase Frequency Shift Keying(CPFSK) is widely used in satellite communication. Considering parameters estimation of CPFSK, an algorithm of CPFSK modulation index estimation is studied based on Lyapunov exponent. With the mathematical relationship between Lyapunov exponent and modulation index, modulation index is estimated by extracting Lyapunov exponent, and the performance is simulated by modulation index Root Mean Square Error(RMSE) vs. Signal-to-Noise Ratio(SNR) curves. With low complexity, this algorithm can be applied to CPFSK and common Frequency Shift Keying(FSK) signal.