Disclosed are an inter prediction mode-based image processing method and apparatus therefor. Particularly, a method for processing an image on the basis of inter prediction may comprise the steps of: adjusting a motion vector of a current block on the basis of a ratio of a difference between a current picture's picture order count (POC) and a first reference picture's POC to a difference between the current picture's POC and a second reference picture's POC; and deriving a predictor for each pixel in the current block by applying pixel unit-based inter prediction to each pixel of the current block on the basis of the adjusted motion vector of the current block.