Aiming at intelligent traffic flow detection system, a fast and light weighted background modeling algorithm based on embedded system is proposed. It filters video sequences by frame difference method, buffers the extracted motion-less frames, and adopts an improved Gaussian model to learn the buffered frames in result of a basic model. To improve the defects of Gaussian mixture model, an adaptive updating manner combining frame difference and pixel statistic have been realized, which shows a better real-time image processing performance and ideal weather adaptability on a TMS320DM648 platform.