Based on the common features of the factorized matrices, a hardware sharing architecture for multiple standards video codec is proposed. By customizing the processing element, butterfly network and transpose hardware, the proposed architecture constructs an efficient phase-pipelined hardware architecture. The proposed architecture can not only be generally used to compute 8×8 Discrete Cosine Transform(DCT) of Audio Video Coding Standard(AVS), H.264, VC-1 and High Efficiency Video Coding(HEVC) in a low cost way, but also can be used to decode Full-HD and Wide Quad extended Graphics Array(WQXGA) format video sequences in real time. The design has been synthesized in 0.13 μm technology. The synthesis results show that the proposed architecture achieves 44% reduction in gate count, 20% decrease in power consumption in comparison with other existing designs.