Multiple reference frame motion estimation to improve video coding performance is newly adopted in the H.264/AVC video coding standard. However, much complexity is burdened in these features. Therefore, we propose a fast reference frame selection algorithm based on information from the reference region used for a motion estimation process. Simulation results show that the proposed algorithm effectively reduces the complexity of multiple reference frame motion estimation with negligible degradation of video quality.