This disclosure describes systems and methods for vision-based navigation of an aerial vehicle. A method includes operations of acquiring a first image, and identifying features in a first image pyramid of the first image. The method includes acquiring a second image after the first image, and identifying the features in a second image pyramid of the second image. The method includes determining current navigation information of the aerial vehicle according to changes in position of the respective features between the first image pyramid and the second image pyramid. The method also includes predicting whether a sufficient number of the features will be identifiable at a third time that is in the future and after the second image was taken, and limiting flight as compared to the user flight instructions if an insufficient number of the features are predicted to be identifiable.