Certains contenus de cette application ne sont pas disponibles pour le moment.
Si cette situation persiste, veuillez nous contacter àObservations et contact
1. (WO2017209788) FRAGMENTATION AUTOMATIQUE PONDÉRÉE
Note: Texte fondé sur des processus automatiques de reconnaissance optique de caractères. Seule la version PDF a une valeur juridique

CLAIMS

1. A computer implemented method, comprising:

a data processing apparatus that includes one or more processors; and

a non-transitory computer readable medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and that when executed cause the data processing apparatus to perform operations comprising: partitioning a data set for an application job into a plurality of partitions according to values of a key, wherein the key is an atomic unit of work placement;

assigning, to each worker computer in a set of worker computers, partitions of the data set to the worker computer, wherein each worker computer receives a different set of partitions of the data set;

iteratively determining workload distribution for the application job across the worker computers and determining whether a load balancing operation is required based on the workload distribution;

in response to determining a load balancing operation is required:

selecting a first worker computer having a highest load measure relative to a respective load measure of at least one other worker computer;

for one or more partitions assigned to the first worker computer, determining one or more move operations for the partition, and, for each move operation, a weight for the move operation; and

selecting the move operation with a highest weight relative to the weights of each of the other move operations and implementing the selected move operation on the partition for which the move operation with the highest weight was determined.

2. The method of claim 1, wherein the weight for each move is based on a benefit determined for the move and a cost determined for the move.

3. The method of claim 2, wherein determining one or more move operations for the partition, and, for each move operation, a weight based on benefits and costs determined for the move comprises, for each of the one or more partitions:

determining a migration move that migrates the partition from the first worker computer to a second worker computer;

if the partition is not replicated on the second worker computer, determining a replication move that replicates the partition on the second worker computer; and

if the partition is replicated on another worker computer, determining a de-replication move that eliminates the partition on the first worker computer.

4. The method of claim 3, wherein determining the migration move comprises selecting a worker computer having a lowest load measure relative to respective load measure of the other work computers.

5. The method of any of claims 2 to 4, wherein determining, for each move operation, the weight based on benefits and costs of the determined for the move operation comprises:

determining a benefit that is proportional to a reduction in load measure in the worker computer that will result from the move operation;

determining a cost that is proportional to an amount of key values that will be reassigned from the move operation; and

determining the weight based on a ratio of the benefit to the cost.

6. The method of any preceding claim, wherein determining whether the load balancing operation is require based on the workload distribution comprises:

determining that at least one worker computer has a load measure that meets a maximum load measure threshold.

7. The method of any preceding claim, the operations further comprising, for each worker computer:

determining, for each partition of the worker computer, a constituent load measure for the partition;

determining pairs of adjacent partitions, where each pair is two partitions that collectively have a contiguous range of key values; and

for each pair of adjacent partitions for which a sum of the constituent load measures of the partition does not meet load measure merger threshold, merging the adj acent partitions into a single partition.

8. The method of claim 7, wherein merging the adjacent partitions into a single partition occurs prior to an iteration to determine the workload distribution for the application job.

9. The method of claim 7 or 8, wherein:

determining pairs of adjacent partitions includes determining one or more pairs of adj acent partitions that includes a first partition on a first worker computer and a second partition on a second worker computer; and

merging the adj acent partitions into a single partition includes merging the adj acent partitions that includes the first partition on the first worker computer and the second partition on the second worker computer into the single partition onto the worker computer of the first and second worker computer that has a lowest load measure.

10. The method of any of claims 7 to 9, wherein the load measure merger threshold is an average of constituent load measures of all partitions.

1 1. The method of any of claims 7 to 10, wherein merging the adjacent partitions into a single partition comprises merging the partition into a single partition only if the merger will not result in a number of partitions on the worker computer being less than a minimum partition threshold that is greater than unity.

12. The method of any preceding claim, the operations further comprising, for each worker computer:

determining, for each partition of the worker computer, a constituent load measure for the partition;

determining, for each partition, whether the constituent load measure exceeds a load measure split threshold; and

for each partition for which the constituent load measure exceeds the load measure split threshold, splitting the partition into two separate partitions.

13. The method of claim 12, wherein splitting the partition occurs after implementing the selected move operation.

14. The method of claim 12 or 13, wherein the load measure split threshold is a double of an average of constituent load measures of all partitions.

15. The method of any of claims 12 to 14, wherein splitting the partition occurs only if the splitting will not result in a number of partitions on the worker computer being greater than a maximum partition threshold.

16. The method of any preceding claim, wherein determining one or more move operations for one or more partitions assigned to the first worker computer comprises determining the one or more move operations for only the partition with the highest constituent load measure.

17. The method of any preceding claim, wherein implementing the selected move operation on the partition for which the move operation with the highest weight was determined comprises implementing the selected move operation only is a determined benefit for the move meets a minimum benefit threshold.

18. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform the operations of any preceding claim.

19. A system comprising:

a data processing apparatus that includes one or more processors; and a non-transitory computer readable medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and that when executed cause the data processing apparatus to perform operations comprising the steps of any of claims 1 to 17.