Processing

Please wait...

Settings

Settings

Goto Application

1. WO2020092778 - MASTERLESS RAID FOR BYTE-ADDRESSABLE NON-VOLATILE MEMORY

Note: Text based on automatic Optical Character Recognition processes. Please use the PDF version for legal matters

[ EN ]

MASTERLESS RAID FOR BYTE-ADDRESSABLE NON-VOLATILE MEMORY

STATEMENT OF GOVERNMENT RIGHTS

[0001] This invention was made with Government support under Contract Number H98230-15-D-0022/0003 awarded by the Maryland Procurement Office. The Government has certain rights in this invention.

BACKGROUND

[0002] A computing system’s storage has historically had characteristics different than its main memory. Storage includes physical devices such as hard disk drives, which provide relatively large amounts of persistent capacity that was accessible at relatively slow throughput. By comparison, main memory includes physical devices such as dynamic random-access memory (DRAM), which provides relatively small amounts of non-persistent capacity that was accessible at relatively fast throughput. The design of modern computing systems has to great degree been driven by these differing characteristics.

BRIEF DESCRIPTION OF THE DRAWINGS

[0003] FIG. 1 is a diagram of an example storage-class memory (SCM) system having media controllers that cooperatively provide for masterless redundant array of independent disks (RAID) functionality for byte-addressable non-volatile memory (NVM) modules attached to the controllers.

[0004] FIG. 2 is a diagram of an example memory address space that that the media controllers of FIG. 1 provide using the NVM modules attached to the controllers.

[0005] FIG. 3 is a diagram of an example RAID group mapping table that each media controller of FIG. 1 maintains.

[0006] FIG. 4 is a diagram of an example as to how memory addresses can be assigned to the media controllers of FIG. 1 over RAID groups.

[0007] FIGs. 5A and 5B are flowcharts of an example method for performing a read request within the SCM system of FIG. 1.

[0008] FIGs. 6A, 6B, and 6C are flowcharts of an example method for performing a write request within the SCM system of FIG. 1.

[0009] FIG. 7 is a diagram of an example media controller of the SCM system of FIG. 1.

DETAILED DESCRIPTION

[0010] As noted in the background, the storage of a computing system has historically differed from the main memory of a computing system in physical characteristics, driving the modern computing system design. For instance, block-addressable file systems have been developed to permit computing systems to persistently store data on storage like hard disk drives. By

comparison, byte-addressable memory topologies have been employed to permit computing systems to quickly access data within main memory like dynamic random-access memory (DRAM).

[0011] However, the physical characteristics dividing storage from main memory have become blurred. With the advent of solid-state drives (SSDs), which employ non-volatile semiconductor memory in lieu of rotating magnetic platters, storage has become increasingly fast. Recent advances in bus design, such as the Peripheral Component Interconnect Express (PCIe) bus, have leveraged such increasing speeds, albeit while maintaining block addressability of the underlying storage.

[0012] With the advent of phase-change memory (PCM) and resistive RAM (ReRAM), main memory has become persistent. However, in other ways main memory of a computing system has remained unchanged. An on-chip or off-chip memory controller accepts memory access requests from processing cores of a processor of a computing system, and responsively accesses on or off-board main memory over a dedicated memory bus, regardless of whether the memory is DRAM, PCM, ReRAM, or another type of main memory.

[0013] A more recent advance benefiting main memory of computing systems is storage-class memory (SCM). SCM leverages the larger sizes and increasing speed of NVM within the context of main memory. That is, unlike storage and like main memory, SCM can be byte addressable. Whereas traditionally a memory controller monolithically accepts memory access requests and directly handles those requests in relation to memory modules like DRAM at the behest of requesting processing cores, the controller functionality is further disaggregated for SCM by abstracting the underlying media, such as NVM.

[0014] A memory semantic fabric can be used to interconnect processors and memory, permitting a processor to access SCM regardless of where the memory is located on the fabric. A memory semantic fabric includes an underlying high-speed input/output (I/O) network that can connect processors to the physical media implementing main memory, at potentially significantly longer distances than otherwise possible. Rather than a memory controller also having media access capability, the memory access requests that a memory controller fields from processing cores are transmitted over the fabric to media controllers particular to their attached physical media. That is, the media-specific aspects of control are moved from the memory controller to the media controller. An example standard governing such interconnection on a memory semantic fabric is Gen-Z

[0015] SCM can permit larger amounts of memory to be accessed by a processor, no differently than how it already can access locally connected main memory. That is, SCM still remains byte addressable, and is not accessed as via a file system or in another way that uses block addressability. Physical chip, board, and bus constraints, which continue to worsen, can effectively be overcome.

[0016] However, SCM can introduce vulnerabilities into computing systems, particularly when SCM is used in lieu of storage as well as to implement main memory. At a memory cell level, SCM may leverage techniques, like error-correcting code (ECC) methodologies, that have been used with more traditional DRAM to prevent data corruption. However, at broader levels, redundancy is

unavailable to SCM: redundancy techniques like redundant arrays of

independent disks (RAID) are not readily applicable to byte addressable main memory like SCM (or DRAM). This can be because the length of time for processing a RAID transaction can be too large in the context of SCM-related transactions.

[0017] Techniques described herein provide for a way to extend RAID to SCM. Specifically, the techniques apply to byte-addressable NVM modules that can serve as main memory of a computing system. The NVM modules have corresponding media controllers that are each attached to and responsible for its corresponding NVM module. The media controllers of the NVM modules cooperatively provide RAID functionality, but at a granularity at which the NVM modules are byte addressable. Furthermore, the RAID is provided in an operating system (OS)- and indeed processor-agnostic basis; the overarching processors and OSs can even be unaware that RAID is being implemented at the underlying main memory level. RAID can be provided in a way that supports shared memory applications, ensuring consistency even when multiple processors or computing systems address the same underlying NVM modules.

[0018] FIG. 1 shows an example storage-class memory (SCM) system 100 in which masterless RAID is implemented. The system 100 includes SCM media controllers 102A, 102B, 102C, . . ., 102N, which are collectively referred to as the media controllers 102. The media controllers 102A, 102B, 102C, . . ., 102N are respectively directly physically attached to byte-addressable NVM modules 104A, 104B, 104C, . . ., 104N, which are collectively referred to as the NVM modules 104. In the example of FIG. 1 , each media controller 102 has one NVM module 104 directly physically connected thereto. Each media controller 102 is responsible for the NVM modules 104 to which the media controller 102 is attached. The media controllers 102 may be physically located on their respective NVM modules 104

[0019] The system 100 includes processing nodes 106A, 106B, . . ., 106M, which are collectively referred to as the processing nodes 106. There may be fewer processing nodes 106 than media controllers 102, as in the example of FIG. 1 , or there may be fewer media controllers 102 than processing nodes 106. The media controllers 102 and the processing nodes 106 are communicatively connected to one another over a memory semantic fabric 108. Each media controller 102 and each processing node 106 is directly connected to the memory semantic fabric 108. Each media controller 102 is responsible for the NVM module(s) 104 attached to the media controller 102 in question.

[0020] As SCM media controllers 102, the media controllers 102 abstract the NVM modules 104 from the processing nodes 106. Therefore, the media controllers 102, and not the processing nodes 106, each handle the specifics as to communication with the NVM modules 104 to which the controllers 104 are attached. The processing nodes 106 nevertheless can use the NVM modules 104 as their (persistent) main memory, as opposed to as storage, and indeed may not have other, local main memory.

[0021] The processing nodes 106 may each be a computing device, and can include one or more processors that each has multiple processing cores.

The processing nodes 106 can include memory controllers that may be integrated within their processors or external to the processors. The processing cores of the processors of the processing nodes 106 may each have multiple caches, such as level-one (L1 ) and level-two (L2) caches, and share another cache, such as a level-three (L3) cache, with the other cores of the

same processor.

[0022] The fabric 108 is a memory semantic fabric, such as a Gen-Z memory fabric. The memory semantic fabric 108 can include one or more SCM switches and other components that physically interconnect the media controllers 102 with the processing nodes 106. Load and store operations that the processing cores of the processors of the processing nodes 106 issue result in the memory controllers of the nodes 106 correspondingly communicating with the media controllers 102 over the memory semantic fabric 108, which satisfy the operations in relation to their attached NVM modules 104.

[0023] The media controllers 102A, 102B, 102C, . . ., 102N respectively include RAID group mapping tables 1 10A, 1 10B, 1 10C, . . ., 1 10N, which are collectively referred to as the RAID group mapping tables 1 10. The media controllers 102 use the RAID group mapping tables 1 10 to cooperatively provide RAID functionality. The media controllers 102 provide this RAID functionality at the granularity at which the NVM modules 104 are byte-addressable, without employing a master RAID controller. For instance, this granularity may be a cache line granularity at which the processing nodes 106 cache data persistently stored within the NVM modules 104, such as sixty-four bytes. The granularity may be the granularity of individual load and store operations issued by the processing nodes 106, such as eight bytes.

[0024] FIG. 2 shows an example memory address space 200 that the media controllers 102 provide using the NVM modules 104. The memory address space 200 includes individual memory addresses 202A, 202B,

202C, . . ., 202N, which are collectively referred to as the memory addresses 202. Each memory address 202 is assigned to a media controller 102, and each media controller 102 is thus responsible for the memory addresses 202 assigned to the controller 102.

[0025] The memory addresses 202 of the memory address space 200 include memory addresses 204 that are accessible by the processing nodes 106, and memory addresses 206 that are not accessible by the processing nodes 106. The memory addresses 204 are depicted as contiguous to the memory

addresses 206 in FIG. 2 for illustrative clarity, but that may be interleaved within or otherwise discontiguous to the memory addresses 206. The media controllers 102 use the memory addresses 206 to provide redundancy for the memory addresses 204 that the processing nodes 106 can access.

[0026] The memory addresses 204 of the memory address space 200 that the processing nodes 204 thus store data accessible by the processing nodes 106 over the memory semantic fabric 108, including individually and/or on a pooled basis. (That is, some memory addresses 204 may be exclusively assigned to individual processing nodes 106, whereas other memory addresses 204 may be pooled for use by more than one processing node 106.) The

memory addresses 206 of the memory address space 200 store parity for this data. The media controllers 102 can provide RAID functionality in a manner that ensures data integrity regardless of whether such assignment is on an individual or pooled basis.

[0027] Each memory address 202 assigned to a media controller 102 is part of a RAID group that can include a memory address 202 assigned to every other media controller 102. In one RAID implementation, the memory addresses 202 of a RAID group assigned to all but one media controller 102 may be memory addresses 204 that store data accessible by the processing nodes 106. In this implementation, the memory address 202 of the RAID group assigned to the remaining media controller may be a memory address 206 that stores parity for the other memory addresses 202 of the group. More generally, the memory addresses 202 of a RAID group assigned to some of the media controllers 102 store data accessible by the processing node, whereas the memory address 202 of the RAID group assigned to another media controller 102 stores parity data for this data.

[0028] FIG. 3 shows an example of the RAID group mapping table 1 10 that each media controller 102 maintains. A media controller 102 stores a table entry 302 for every memory address 202 assigned to the controller 102 in question. The table entry 304 includes a field 304 identifying the memory address 202 corresponding to the entry 304. The table entry 304 includes a field 306 indicating whether the memory address 202 stores data or parity. That is, for the memory address 202 to which a table entry 302 corresponds, the entry 304 identifies whether the memory address 202 is a memory address 204 or a memory address 206.

[0029] Each table entry 302 of the RAID group mapping table 1 10 maintained by a media controller 102 includes a field 308 identifying the other media controllers 102 that have also been assigned memory addresses 202 for the same RAID group as the memory address 202 to which the entry 302 corresponds. For each such media controller 102, the field 308 indicates whether the controller 102 stores data or parity. The table entry 302 includes a field 310 identifying the status of the RAID group including the memory address 202 to which the entry 302 corresponds. The status of the RAID group may be normal or degraded.

[0030] FIG. 4 shows an example as to how the memory addresses 202 can be assigned to the media controllers 102 over RAID groups 402A, 402B, 402C, and 402D, collectively referred to as the RAID groups 402. For each RAID group, one media controller 102 stores parity, and the other media controllers 102 store data. For example, FIG. 4 explicitly depicts four memory addresses 402A, 402B, 402C, and 402D, collectively referred to as the memory addresses 402, within the RAID group 402A, which are assigned to respective media controllers 102A, 102B, 102C, and 102N. The controllers 102A, 102B, and 102C store data at the memory addresses 402A, 402B, and 402C, respectively, whereas the controller 102N stores parity for the data at the addresses

402A, 402 B, and 402C.

[0031] FIGs. 5A and 5B show a method 500 of example performance of a read request (i.e., a load operation) within the SCM system 100. The example of FIGs. 5A and 5B is described in relation to the media controller 102A receiving a read request for the data stored at the memory address 404A. As such, within the RAID group 402C including the memory address 404A, the media controller 102B stores parity at the memory address 404B from the data at the memory address 404A, and from the data that the media controllers 102C and 102N respectively store at the memory addresses 404C and 404D.

[0032] The parts of the method 500 are thus organized over three columns. The media controller 102A performs the parts in the left column, and the media controller 102B performs the parts in the middle column. Each of the media controllers 102C and 102N performs the parts in the right column. The media controllers 102 can each implement its corresponding parts of the method 500 by executing instructions stored on a non-transitory computer-readable data storage medium.

[0033] The media controller 102A receives a load operation (i.e., a read request) for the memory address 404A from a processing node 106 over the memory semantic fabric 108 (502). The media controller 102A retrieves the data at the memory address 404A from the NVM module 104A attached to the controller 102A (504). If the media controller 102A does not encounter any media error within the NVM module 104A while retrieving the data (506), then the controller 102A returns the successfully retrieved data back to the requesting processing node 106 over the memory semantic fabric 108 (508). Thus, when

no read errors occur, the media controller 102A can respond to read requests without involving any other media controller 102.

[0034] However, if the media controller 102A encounters a media error in retrieving the data from the NVM module 104A (506), then the controller 102A will have not have successfully retrieved the data. Therefore, the media controller 102A requests, over the memory semantic fabric 108, that the media controller 102B rebuild (i.e., reconstruct) and return the requested data (510). That is, the media controller 102A can send the load operation in question to the media controller 102B, flagging the operation as a rebuild operation (i.e., an operation to rebuild the data in question from parity). The media controller 102A can inspect its RAID group mapping table 1 10A to identify the media controller 102B as the controller 102 storing parity at the memory address 404B that has been calculated from the data stored at the memory address 404A. After requesting the rebuilt data from the controller 102B, the controller 102A can continue to process requests for other addresses stored within the NVM module 104A while waiting for the rebuilt data for the address 404A (512).

[0035] The media controller 102B, upon receiving the request from the controller 102A, locks the parity for the address 404A (and thus also for the addresses 404C and 404D) stored at the address 404B within the NVM module 104B (514). The media controller 102B also requests that the media controllers

102C and 102N respectively return the data at the memory addresses 404C and 404D from which the parity has also been calculated (516), over the memory semantic fabric 108. The media controller 102B can inspect its RAID group

mapping table 1 1 OB to identify the media controllers 102C and 102N as the controllers 102 that also store data, at the memory addresses 404C and 404D, from which the parity has been calculated. After requesting the data at the memory addresses 404C and 404D from the controllers 102C and 102N, the controller 102B can continue to process parity requests for other addresses (518).

[0036] The media controllers 102C and 102N lock the addresses 404C and 404D within the NVM modules 104C and 104N (520), so that no new data can be written to the addresses 404C and 404D until the existing data at these addresses have been retrieved. The media controllers 102C and 102N further wait until any outstanding parity updates for addresses 404C and 404D have been completed by the controller 102B (522). As will be described later in the detailed description, when the media controller 102B performs a parity update for an address, the controller 102B notifies the controller 102A that requested the parity update when the update has been completed. Each of the media

controllers 102C and 102N thus delays retrieving the data at the respective memory address 404C or 404D if any write requests to the address in question are pending (i.e. , which will result in the controllers 102C and 102N sending parity updates to the controller 102B), to maintain data consistency.

[0037] Once any outstanding parity updates have been completed, the media controllers 102C and 102N can respectively retrieve the data at the memory addresses 404C and 404D from the NVM modules 104C and 104N (524). The media controllers 102C and 102N return the retrieved data to the media controller 102B over the memory semantic fabric 108 (550). Upon

receiving the read data at the memory addresses from the media controllers 102C and 102N, the media controller 102B rebuilds the data at the memory address 404A and returns the rebuilt data to the media controller 102A (552), over the memory semantic fabric 108. The media controller 102B rebuilds (i.e., reconstructs) the data from the parity at the memory address 404B, and from the data at each of the memory addresses 404B and 404C. For example, the media controller 102B may perform an exclusive-or (XOR) operation on the parity and the data retrieved from each of the media controllers 404B and 404C to rebuild the data at the memory address 404A.

[0038] The media controller 102A returns the received rebuilt data back to the requesting processing node 106 over the memory semantic fabric 108 (558). The media controller 102A also writes the rebuilt data to the memory address 404A at the NVM module 104A attached to the controller 102A and attempts to reread the address 404A (i.e., again attempts to retrieve the data from the address 404A) (560). If the media controller 102A does not encounter an error when rereading the data from the NVM module 104A (562), then the error that was initially encountered in part 506 can be considered a corrected error (564). The media controller 102A may report that a corrected error has occurred at the address 404A, such as to a management computing device.

[0039] However, if the media controller 102A encounters a media error when rereading the data from the NVM module 104A (562), then the controller 102A may be considered as not have successfully written the rebuilt data to the memory address 402A. The error is thus a persistent error (566), having

occurred at the initial read and at the subsequent reread. The media controller may similarly report that the persistent error has occurred at the address 404A, such as to a management computing device.

[0040] FIGs. 6A, 6B, and 6C show a method 600 of example performance of a write request (i.e. , a store operation) within the SCM system 100. The method 600 described in relation to the media controller 102A receiving a write request to store new data at the memory address 404A. As such, within the RAID group 402C, including the memory address 404A, the media controller 102B stores parity at the memory address 404B from the data at the memory address 404A, and from the data that the media controllers 102C and 102N respectively store at the memory addresses 404C and 404D.

[0041] The parts of the method 600 are thus organized over three columns. The media controller 102A performs the parts in the left column, and the media controller 102B performs the parts in the middle column. Each of the media controllers 102C and 102N perform the parts in the right column. The media controllers 102 can each implement its corresponding parts of the method 600 by executing instructions stored on a non-transitory computer-readable data storage medium.

[0042] The media controller 102A receives a store operation (i.e., a write request) to store new data at the memory address 404A from a processing node 106 over the memory semantic fabric 108 (602). The controller 102A waits until any outstanding parity updates for the address 404A have been completed by the controller 102B (604). That is, if the controller 102A relatively recently performed a prior write request to the address 404A, the controller 102B may not have yet indicated that it updated the parity for the address 404A. Therefore, the controller 102A waits to process the new write request until the parity for the address 404A has been updated pursuant to the prior write request.

[0043] Once any outstanding parity updates have been completed, the media controller 102A locks the address 404A within the NVM module 104A, and attempts to retrieve existing data from the NVM module 104A (606). Locking is performed so that subsequent write requests for the address 404A remain pending until the currently received write request has been processed. If the media controller 102A does not encounter any media error within the NVM module 104A while retrieving the data (608) - which can be referred to as a write error insofar as the error occurs during the write process of the method 600 -then the controller 102A writes the new data received in the write request to the memory address 404A (610). After writing the new data, the media controller 102A performs an XOR operation on the existing data retrieved in part 606 and the new data of the write request stored in the NVM module 104A in part 610, and transmits the XOR operation result, which can be referred to as a new XOR, to the media controller 102B (612).

[0044] The media controller 102B, which is responsible for maintaining parity for the RAID group 402C, responsively retrieves the parity at the memory address 404B from the NVM module 102B attached to the controller 102B (614). If the controller 102B does not encounter an error in retrieving the existing parity (616), then the controller 102B generates and stores new parity to the memory

address 404B (618). For instance, the media controller 102B can perform an XOR operation on the XOR operation result transmitted by the media controller 102A in part 614 (i.e. , the new XOR) and on the existing parity retrieved in part 616 (i.e., the existing XOR). The result of this XOR operation performed by the media controller 102B is the new parity.

[0045] The media controller 102B sends an acknowledgment (“ACK”) indicating that the parity update process was successful to the media controller 102A (620). Upon receiving this acknowledgment, the controller 102A unlocks the address 404A so that pending and future write requests to the address can be processed (622). The controller 102A sends an acknowledgment to the processing node 106 from which the write request was received (624), indicating that the write request was successfully completed. The method 600 is then finished. It is noted that completion of the write request when no write error occurs does not involve the media controller 102A sending the new data indicated in the write request to the media controller 102B. Rather, the controller 102A may just send the XOR of this new data and the existing data stored at the memory address 404A to the controller 102B.

[0046] If the media controller 102B was unable to retrieve the existing parity for the address 404A from the NVM module 104B in part 614, however (616), the controller 102B instead sends a different acknowledgment (“NACK”) indicating that the parity update process was unsuccessful to the media controller 102A (626). The controller 102A, upon receiving this acknowledgment, sends an acknowledgment to the processing node 106 from which the write request was

received (628), indicating that the write request was not successfully completed. The controller 102B, after sending the acknowledgment in part 626, locks the parity for the address 404A (and thus also for the addresses 404C and 404D) stored at the address 404B within the NVM module 104B (630).

[0047] The media controller 102B also requests that the media controllers

102A, 102C, and 102N respectively return the data at the memory addresses 404A, 404C, and 404D from which the parity has been calculated (632). Upon receiving their requests, the media controllers 102C and 102N lock the

addresses 404C and 404D within the NVM modules 104C and 104N (638), so that no new data can be written to the addresses 404C and 404D until the existing data have been retrieved. The media controller 102A does not have to correspondingly lock the address 404A within the NVM module 104A at this time, however, because the address 404A was already locked in part 606.

[0048] The media controllers 102C and 102N further wait until any outstanding parity updates for addresses 404C and 404D have been completed by the controller 102B (642). Each of the media controllers 102C and 102N thus delays retrieving the data at the respective memory address 404C or 404D if any write requests to the address in question are pending (i.e., which will result in the controllers 102C and 102N sending parity updates to the controller 102B), to maintain data consistency. The media controller 102A does not have to wait for any parity updates for address 404A to be completed, since the controller 102A previously waited in this regard in part 604.

[0049] The media controller 102A retrieves the existing data at the memory address 404A from the NVM module 104A (644). The media controllers 102C and 102N likewise respectively retrieve the data at the memory addresses 404C and 404D from the NVM modules 104C and 104N once any outstanding parity updates have been completed (648). The media controller 102A sends its retrieved data to the media controller 102B (646), as do the media controllers 102C and 102N (650). The controller 102B generates and stores new parity to the memory address 404B and attempts to re-read the address 404B (i.e., again attempts to retrieve the parity from the address 404B) (652). The controller 102B can perform an XOR operation on the retrieved data received from the controllers 102A, 102C, and 102N, the result of which is the new parity.

[0050] If the media controller 102B does not encounter an error when rereading the address 404B (654), the prior error in part 616 that was initially encountered can be considered a corrected error (656). The controller 102B may report that a corrected error has occurred at the address 404B, such as to a management computing device. However, if the controller 102B encounters a media error when rereading the address 404B (654), the resulting error may be considered a persistent error (658), since the error occurred both at the prior reading of the address 404B in part 614 and the subsequent reading of this address in part 652. The media controller may similarly report that the persistent error has occurred at the address 404B, such as to a management computing device.

[0051] Regardless of whether the media controller 102B was able to retrieve the parity from the memory address 404B after having newly generated this parity in part 652, the controller 102B instructs the controllers 102A, 102C, and 102N to respectively release their locks from the memory addresses 404A, 404C, and 404N (660). The controller 102A thus releases its lock from the address 404A upon receiving this instruction (662), and the controllers 102C and 102N likewise release their locks from the addresses 404C and 404D, respectively (664). The controller 102B similarly releases the lock previously placed on the parity stored at address 404B (666).

[0052] It is noted that a persistent failure of the controller 102B in being able to read the parity from the address 404B does not mean that the data stored at the addresses 404A, 404C, and 404N is presently unavailable. Rather, such a persistent failure means that the data stored at the addresses 404A, 404C, and 404N has no redundancy. Should any of the controllers 102A, 102C, and 102N be subsequently unable to respectively retrieve the data at these addresses, the data cannot be regenerated as has been described in relation to the method 500 of FIGs. 5A and 5B.

[0053] If back in part 606 the media controller 102A encountered a media error when attempting to retrieving the existing data from the NVM module 104A (608), the controller 102A sends the new data of the write request to the media controller 102B and requests that the controller 102B build new parity for the address 404A using the new data (668). The media controller 102B therefore itself has to generate the new XOR that the media controller 102A generated

when there was no error. Because the new XOR is generated from existing data stored in the NVM module 104A that the media controller 102A was unable to read, the media controller 102B itself has to rebuild the existing data.

[0054] After the media controller 102A sends the new data and the new parity build request to the media controller 102B, the controller 102A writes the new data to the NVM module 104A and attempts to read the just written data (670). Because the controller 102A encountered an error when previously attempting to read the address 404A, however, the controller 102A may instead write the new data to a spare address for the address 404A within the same NVM module 104A. If the controller 102A does not encounter an error during this attempted read (672), then the prior error that was encountered in part 608 can be considered a corrected error at the address 404A (674), which may be reported to a management computing device. If the controller 102A does encounter an error during the attempted read (672), the resulting error may be considered a persistent error (676), since the error also occurred during the prior read in part 606. The persistent error may also be reported to a management computing device.

[0055] Upon receiving the new data and the new parity build request, the media controller 102B locks the parity for the address 404A within the NVM module 104B (678) (i.e. , locking the address 404B at which the parity for the address 404A is stored). The controller 102B requests that the media controllers 102C and 102N respectively return the data at the memory addresses 404C and 404D (680), and continues processing parity requests received for other

addresses (682). The media controllers 102C and 102N in turn respectively lock the addresses 404C and 404D (684) upon receiving the request for the data stored at these addresses.

[0056] The media controllers 102C and 102D respectively wait until any outstanding parity updates for addresses 404C and 404D are completed (686) before retrieving the data stored at their respective NVM modules 104C and 104D at these addresses (688). The controllers 102C and 102D send their retrieved data to the media controller 102B (690). The controller 102B generates and stores new parity to the memory address 404B of the NVM module 104B (692). To generate the new parity, the controller 102B can perform an XOR operation on the retrieved data received from the controllers 102C and 102N and the data of the write request received from the controller 102A.

[0057] The media controller 102B sends an acknowledgment (“ACK”) to the media controller 102A indicating that the new parity build has been completed (694). Upon receiving the acknowledgment, the controller 102A releases the lock from the address 404A (i.e., the controller 102A unlocks the address 404A), and sends an acknowledgment to the processing node 106 from which the write request was received (698), indicating that the write request has been completed. The controller 102B instructs the controllers 102C and 102N to respectively release their locks from the memory addresses 404C and 404N (603), which the controllers 102C and 102N do upon receiving this instruction (605). The controller 102B likewise releases its lock on the parity for the address 404A (and for the addresses 404C and 404N (607). That is, the controller 102B releases the lock from the address 104B that was placed in part 678.

[0058] The read process of FIGs. 5A and 5B and the write process of FIGs. 6A, 6B, and 6C that have been described accounts for the scenario when a media controller 102 encounters an error when performing a data read or a data write at a memory address 404 mapped to the NVM module 104 to which the controller 102 is directly attached. Specifically, the processes relate to the media controller 102A performing a data read or a data write at the memory address 404A mapped to the NVM module 104A to which the controller 102A is directly attached. The processes by which the other media controllers 102 perform data reads or data writes at memory addresses 404 mapped to the NVM modules 104 to which the controllers 102 are respectively directly attached performed in a similar manner.

[0059] The RAID that has been described in relation to FIGs. 5A and 5B and FIGs. 6A, 6B, and 6C can accommodate failure of one NVM module 104 without loss of data. If a media controller 102 is continually encountering errors when reading from or writing to a particular NVM module 104, then the module 104 in question may have to be replaced. When such an NVM module 104 is replaced with a new NVM module 104, any data and parity stored by the replaced NVM module 104 has to be recreated and stored on the new

NVM module 104.

[0060] Specifically, for each memory address 404 mapped to the new NVM module 104 that stores data, the data rebuilding process described

beginning at part 514 of the method 500 is performed. The media controller 102 maintaining parity for the RAID group 402 of which the memory address 404 in question is a part rebuilds the data, and the media controller 102 to which the new NVM module 104 is directly attached stores the rebuilt data at the new NVM module 104. For each memory address 404 mapped to the new NVM module 104 that stores parity, the parity rebuilding process involves the media controller 102 directly attached to the new NVM module 104 requesting data from which the parity is generated from the other controllers 102 storing this data at their own directly attached NVM modules 104. Once the media controller 102 directly attached to the new NVM module 104 receives this data, it can generate and store the parity on the new NVM module 104.

[0061] FIG. 7 shows an example media controller 702, which can implement any of the media controllers 102 that have been described. As such, the media controller is directly attached to a NVM module 704, which can be one of the NVM modules 104 that have been describe, and is connected to a memory semantic fabric 108. The media controller 702 can be implemented completely in hardware, and is discrete from the NVM module 704 to which it is directly attached. An example of a media controller in this respect is an SCM media controller, which may be able to be attached to multiple such homogeneous or heterogeneous NVM modules 704.

[0062] The media controller 702 can include three separate hardware components: an NVM controller 706, a fabric bridge 708, and a link layer device 710. The NVM controller 706 may be particular to the type of the NVM module

704 to which the controller 706 is physically attached. The same NVM controller 706 may be able to be attached to multiple NVM modules 704 of the same type, in which case there may be a different NVM controller 706 for each kind of NVM module 704. In another implementation, an NVM controller 706 may just be able to be physically attached to one NVM module 704, in which case the number of NVM controllers 706 present dictates the maximum number of NVM modules 704 that can be physically attached to the media controller 702.

[0063] The link layer device 710 is the hardware component that connects to the memory semantic fabric 108, such as via an intervening Gen-Z switch, for the media controller 702 to communicate with other media controllers. The fabric bridge 708 interconnects the NVM controller 706 and the link layer device 710, and may be implemented as silicon-on-chip (SoC), as another hardware component, or as a combination of hardware and software. The fabric bridge 708 performs the cooperative masterless RAID functionality that has been described as it involves the media controller 702. For instance, the fabric bridge 708 can perform parts of the methods 500 and 600 that have been described, and may maintain a corresponding RAID group mapping table 1 10, either within the NVM module 704, or within other memory, which may be internal to the bridge 708 or the media controller 702 itself.

[0064] The techniques that have been described herein thus provide for a way to extend RAID to SCM. The RAID is masterless, in that no master RAID controller has to be involved. Rather, the media controllers that may otherwise already be present within an SCM architecture can be enhanced to achieve the cooperative distributive RAID that has been described. The RAID provides for consistency even when multiple processors or system address the same underlying NVM modules; write operations are properly sequenced when data rebuild operations are currently in process. The RAID that has been described further is OS and processor agnostic; OSs and processors mapping addresses to the memory to which the media controller govern access do not have be aware that RAID is being implemented.