Processing

Please wait...

Settings

Settings

Goto Application

1. WO2019083519 - NATURAL LANGUAGE PROCESSING WITH AN N-GRAM MACHINE

Note: Text based on automatic Optical Character Recognition processes. Please use the PDF version for legal matters

[ EN ]

WHAT IS CLAIMED IS:

1. A computing system, comprising:

at least one processor;

a machine-learned natural language processing model comprising:

an encoder model, wherein the encoder model is trained to receive a natural language text body and, in response to receipt of the natural language text body, output a knowledge graph; and

a programmer model, wherein the programmer model is trained to receive a natural language question, and, in response to receipt of the natural language question, output a program; and

at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:

obtaining the natural language text body;

inputting the natural language text body into the encoder model; receiving, as an output of the encoder model, the knowledge graph;

obtaining the natural language question;

inputting the natural language question into the programmer model;

receiving, as an output of the programmer model, the program; and executing the program on the knowledge graph to produce an answer to the natural language question.

2. The computing system of claim 1, wherein the program output by the programmer model comprises a function that causes a search of the knowledge graph.

3. The computing system of claim 1, wherein the knowledge graph comprises a plurality of n-gram sequences.

4. The computing system of claim 1, wherein the natural language text body comprises a plurality of sentences.

5. The computing system of claim 1, wherein at least one of the programmer model and the encoder model was trained based at least in part on a total objective function for the machine-learned natural language processing model, wherein the total objective function comprises both an autoencoder objective function and a question answer objective function, wherein the autoencoder objective function describes a reconstruction loss between the natural language text body and a reconstruction of the natural language text body that is output by a decoder model based on the knowledge graph, and wherein the question answer objective function describes a reward that is based on a comparison of the answer and an expected answer.

6. The computing system of claim 1, wherein at least one of the encoder model or the programmer model comprises a neural network.

7. The computing system of claim 6, wherein at least one of encoder model and the programmer model comprises a recurrent neural network.

8. The computing system of claim 6, wherein at least one of encoder model and the programmer model comprises a sequence-to-sequence neural network.

9. A computer-implemented method, the method comprising:

obtaining, by one or more computing devices, a natural language processing model that comprises an encoder model, a decoder model, and a programmer model, wherein the encoder model is configured to receive a natural language text body and, in response to receipt of the natural language text body, output a knowledge graph, wherein the decoder model is configured to receive the knowledge graph and, in response to receipt of the knowledge graph, output a reconstruction of the natural language text body, and wherein the programmer model is trained to receive a natural language question, and, in response to receipt of the natural language question, output a program;

inputting, by the one or more computing devices, a training data set that comprises the natural language text body and the natural language question into the natural language processing model to receive an answer to the natural language question;

evaluating, by the one or more computing devices, a total objective function that comprises an autoencoder objective function and a question answer objective function,

wherein the autoencoder objective function describes a reconstruction loss between the natural language text body and the reconstruction of the natural language text body, and wherein the question answer objective function describes a reward that is based on a comparison of the answer and an expected answer included in the training data set; and

training, by the one or more computing devices, the natural language processing model based on the total objective function.

10. The computer-implemented method of claim 9, wherein training, by the one or more computing devices, the natural language processing model based on the total objective function comprises training, by the one or more computing devices, the programmer model based at least in part on the question answer objective function.

11. The computer-implemented method of claim 9, wherein training, by the one or more computing devices, the natural language processing model based on the total objective function comprises jointly training, by the one or more computing devices, the encoder model and the decoder model based at least in part on the autoencoder objective function.

12. The computer-implemented method of claim 9, wherein the question answer objective function also describes at least one of an encoder probability distribution that describes an input of the encoder model with respect to an output of the encoder model or a programmer probability distribution that describes an input of the programmer model with respect to an output of the programmer model.

13. The computer-implemented method of claim 9, further comprising defining, by the one or more computing devices, the autoencoder objective function as a first autoencoder objective function in a first training stage and as a second autoencoder objective function in a second training stage following the first training stage, and wherein:

the first autoencoder objective function describes the reconstruction loss between the natural language text body and the reconstruction of the natural language text body that is output by the decoder model based on the knowledge graph, and

the second autoencoder objective function describes the reward that is based on the comparison of the answer and the expected answer.

14. The computer-implemented method of claim 9, further comprising applying, by the one or more computing devices, a code assist function to the programmer model, wherein applying the code assist function comprises providing the programmer model with a set of potential functions, and wherein the programmer model is configured to select a function from the set of potential functions to generate the program.

15. The computer-implemented method of claim 9, further comprising:

detecting, by the one or more computing devices, if the answer comprises an empty set; and

when it is detected that the answer comprises an empty set, applying, by the one or more computing devices, a structural tweaking function to the knowledge graph that is associated with reducing a probability of the answer comprising an empty set.

16. The computer-implemented method of claim 9, further comprising:

during a first training stage, applying, by the one or more computing devices, a code assist function to the programmer model, wherein applying the code assist function comprises providing the programmer model with a set of potential functions, and wherein the programmer model is configured to select functions from the set of potential functions to generate the program;

during a second training stage that follows the first training stage, detecting, by the one or more computing devices, if the answer comprises an empty set, and when it is detected that the answer comprises an empty set, applying, by the one or more computing devices, a structural tweaking function to the knowledge graph that is associated with reducing a probability of the answer comprising an empty set, and wherein the structural tweaking function is applied during the second training stage at a greater frequency than the code assist function is applied during the second training stage.

17. A computing system for responding to a natural language query; comprising:

an encoding system configured to receive a natural language text body and generate, using a machine-learned natural language encoder model, a knowledge graph based on the natural language text body;

a query programming system configured to receive a natural language input query, and generate, using a machine-learned natural language query programmer model, a program for querying the knowledge graph based on the natural language input query; and

a query execution system configured to execute the generated program on the generated knowledge graph and to output a query response.

18. The computing system of claim 17, wherein at least one of the machine-learned natural language encoder model and the machine-learned natural language query programmer model is a recurrent neural network or a sequence-to-sequence neural network.

19. The computing system of any preceding claim, wherein the machine-learned natural language encoder model is trained by:

generating, using a machine-learned natural language decoder model, a reconstruction of the natural language text body based on the knowledge graph;

calculating a reconstruction loss between the natural language text body and the reconstruction of the natural language text body; and

training the machine-learned natural language encoder model and the machine-learned natural language decoder model using an autoencoder objective function which describes the reconstruction loss.

20. The computing system of claim 19, wherein the machine-learned natural language decoder model is a recurrent neural network or a sequence-to-sequence neural network.

21. The computing system of claim 19 or claim 20, wherein the machine-learned natural language encoder model and the machine-learned natural language query programmer model are trained by:

receiving a natural language test query for the natural language text body;

generating a test program for querying the knowledge graph based on the natural language test query;

executing the generated test program on the knowledge graph to obtain a test query response;

comparing the test query response with an expected query response for the natural language test query; and

training the machine-learned natural language encoder model and the machine-learned natural language query programmer model using a question answer objective function which describes a reward that is based on the comparison of the test query response and the expected query response for the natural language test query.

22. The computing system of claim 21, wherein the machine-learned natural language encoder model and the machine-learned natural language query programmer model are trained based on a total object function comprising the autoencoder objective function and the question answer objective function.

23. The computing system of any preceding claim, wherein the encoding system is configured to generate the knowledge graph by converting the natural language text body into a plurality of n-gram statements.

24. The computing system of claim 23, wherein the generated program comprises one or more functions comprising an argument; and

wherein executing each function of the program causes the query execution system to discover at least one n-gram statement including the argument of the function, and to return a variable based on the function and the at least one discovered n-gram statement.

25. The computing system of claim 24, wherein the argument of at least one function in the generated program refers to a variable returned by at least one other function of the generated program.

26. The computing system of claim 24 or claim 25, wherein the encoding system is configured to generate a timestamp in association with each of the plurality of n-gram statements, the timestamps representing an order in which the n-gram statements are generated from the natural language text body.

27. The computing system of claim 26, wherein the generated program comprises at least one function which causes the query execution system to discover at least one n-gram statement including the argument of the function, and to return a variable based on the function and the timestamp of the at least one discovered n-gram statement.

28. The computing system of any one of claims 25 to 28, wherein the encoding system is configured to provide the query programming system with information on the knowledge graph, the information indicating at least one argument for a function that would return a variable when the function is executed.

29. The computing system of any one of claims 24 to 28, wherein the query programming system, in response to receiving an empty result returned by the query execution system for a function of the program, is configured to provide the encoding system with at least one instruction to modify the knowledge graph, the instruction indicating at least one

modification to the knowledge graph which would reduce a probability of an empty result for the function.