[Solution] A portable terminal apparatus of this voice recognition system is provided with: a CPU capable of switching the operation mode between normal operation mode and power-saving mode; and an MPU including an event detection unit that detects a currently occurring event from among a plurality of events estimated in advance on the basis of observation information indicating a physical quantity for detecting an event, and a voice recognition unit that executes voice recognition on the basis of voice information and selected dictionary information. One voice dictionary candidate is selected from among a plurality of voice dictionary candidates according to the detected event, and the selected voice dictionary candidate is stored, as selected dictionary information, in a storage device. Further, the MPU is designed such that power consumption when operating the MPU while the CPU is in the power-saving mode is less than power consumption when operating the CPU in the normal operation mode.