Determining a situational annotation of a vehicle based on an annotated environmental model is disclosed. A computing system receives an annotated environmental model for a vehicle. The annotated environmental model can include data from a plurality of modalities. The computing system uses a classifier to determine an annotation of the vehicle situation based, at least in part, on the annotated environmental model. The computing system provides the situational annotation to a driving functionality system. The driving functionality system is configured to use the situational annotation to generate signals to control operation the vehicle.