Search International and National Patent Collections

1. (WO2017031961) DATA PROCESSING METHOD AND APPARATUS

Pub. No.:    WO/2017/031961    International Application No.:    PCT/CN2016/074042
Publication Date: Fri Mar 03 00:59:59 CET 2017 International Filing Date: Fri Feb 19 00:59:59 CET 2016
IPC: G06F 9/50
Applicants: HUAWEI TECHNOLOGIES CO., LTD.
华为技术有限公司
Inventors: LI, Yong
李勇
ZHU, Junhua
朱俊华
Title: DATA PROCESSING METHOD AND APPARATUS
Abstract:
A data processing method and apparatus. The method comprises: if the number of completed map tasks within a pre-set time period is greater than or equal to a pre-set starting threshold, acquiring the overheads of unassigned partitions in outcomes output from the completed map tasks within the pre-set time period and before the pre-set time period (S201); according to the overhead of each of the unassigned partitions and starting states of reduce tasks, assigning corresponding reduce tasks to N partitions with the highest overheads (S202), where N is an integer greater than or equal to 1; and pulling data of the N partitions to the reduce tasks corresponding to the partitions for processing (S203), which can improve the resource utilization rate, and assigning reduce tasks according to the overheads of the partitions makes the assignment more rational and improves the overall performance of a MapReduce system.