A multi-layer Optical Camera Communication(OCC) system based on M-ary detection is proposed to adapt to the needs of multiuser. According to different priorities, the message is loaded on each layer which has diverse degrees of diversity. The higher-priority data is transmitted in higher diversity mode with higher robustness but lower data rate, whereas the lower-priority data is sent using higher multiplexing scheme with higher data rate but lower robustness. Combined with the OCC system model, the message can be detected by using an M-ary detection algorithm based on Maximum Likelihood (ML) detection. Finally, computer simulations verify the system feasibility. With reliable and low- complexity detection, the multi-layer OCC transmission system can be achieved stably.