Systems and methods are provided for controlling video playback speed using user interaction. An user input may be detected during an aspect of video playback. The user input may correspond to, for example, adding user notes or drawings using one or more input devices (e.g., a stylus, a finger, etc.) to a video being played back. A set of features of the interaction may be determined using one or more sensors. Based on the determined features, a video playback speed may be determined and applied to the video. The video may be played at the determined video playback speed for the duration of the detected user input. When the user input is no longer detected, the video playback may automatically resume at the normal playback speed.