Preserving memory values during replay includes identifying trace sections that each represents events executed by an entity over a period of time. A parallel replay of trace sections is performed at a plurality of processing units. While performing the parallel replay, a persistence data structure corresponding to each trace section is maintained. This includes, for each trace section, storing, in the trace section's persistence data structure, a record of each memory address consumed by the processing unit while replaying the trace section, and a most recent memory value stored at each memory address. Returning a memory value during replay includes identifying relevant persistence data structures, and searching these data structures, in turn, based on a defined ordering. When a relevant memory address is identified during the search, the search is ceased and the value associated at the memory address, as stored in a persistence data structure, is returned.