Processing

Please wait...

Settings

Settings

Goto Application

1. US20210125108 - TRAINING A RANKING MODEL

Note: Text based on automatic Optical Character Recognition processes. Please use the PDF version for legal matters

[ EN ]

Claims

1. A method comprising:
receiving training data for a ranking machine learning model that is used to rank documents in response to received search queries, the training data including a plurality of training examples, and each training example of the plurality training examples including data identifying:
a search query,
result documents from a result list for the search query, and
a result document that was selected by a user from the result list of result documents,
receiving position data for each training example of the plurality of training examples in the training data, the position data identifying a respective position of the selected result document in the result list for the search query in the training example;
determining, for each training example of the plurality of training examples in the training data, a respective selection bias value that represents a degree to which the position of the selected result document in the result list for the search query in the training example will impact the selection of the result document;
determining, for each training example of the plurality of training examples in the training data and from the respective selection bias value for the training example, a respective importance value that is inversely proportional to the respective selection bias value for the training example; and
training the ranking machine learning model on the training data, the training comprising, for each training example:
determining a loss for the training example that is based on (i) ranking scores generated by the ranking machine learning model for result documents in the result list that is identified in the training example and (ii) the result document that is identified in the training example as the result document that was selected by the user from the result list that is identified in the training example;
generating an adjusted loss for the training example by adjusting the loss for the training example using the respective importance value for the training example that is inversely proportional to the respective selection bias value for the training example, wherein the adjusted loss for weights losses for training examples having higher respective importance values more strongly than losses having relatively lower respective importance values in the training of the ranking machine learning model wherein the adjusted loss L(f) satisfies:

           L( f)= w·l( Q,f),
wherein w denotes the importance value for the training example that is inversely proportional to the respective selection bias value for the training example, and l(Q, f) denotes the loss for the training example; and
 training the ranking machine learning model on the adjusted loss.
2. The method of claim 1, further comprising:
receiving experiment data identifying a plurality of experiment search queries and, for each experiment search query, a respective position in an experiment result list of experiment result documents for the experiment search query of an experiment result document that was selected by a user, wherein the positions of experiment result documents in the experiment result lists were randomly permuted before being presented to users.
3. The method of claim 2, further comprising:
determining, for each of the plurality of positions, a respective count of selections of experiment result documents at the position by users in response to the plurality of experiment search queries in the experiment data; and
determining, for each of the plurality of positions, a respective position bias value for the position based on the respective count of selections for the position.
4. The method of claim 3, wherein determining, for each training example of the plurality of training examples in the training data, a respective selection bias value comprises:
assigning, for each training example of the plurality of training examples in the training data, the respective position bias value corresponding to the position of the selected result document in the result list of result documents for the training example to be the selection bias value for the training example.
5. The method of claim 2, wherein the experiment search queries in the plurality of experiment search queries each belong to a respective query class of a plurality of query classes, and wherein the method further comprises, for each of the plurality of query classes:
determining, for each of the plurality of positions, a respective count of selections of experiment result documents at the position by users in response to experiment search queries belonging to the query class in the experiment data, and
determining, for each of the plurality of positions, a respective class-specific position bias value for the position based on the respective count of selections for the position.
6. The method of claim 5, wherein determining, for each training example of the plurality of training examples in the training data, a respective selection bias value comprises:
obtaining data identifying a query class to which the search query for the training example belongs;
assigning the class-specific position bias value for the query class to which the search query belongs and corresponding to the position of the selected result document for the training example in the result list of result documents for the training example to be the selection bias value for the training example.
7. The method of claim 2, further comprising:
obtaining a respective feature vector for each experiment search query,
generating training data for training a classifier that receives a respective feature vector for an input search query and outputs a respective query-specific position bias value for each of a plurality of positions for the input search query, and
training the classifier on the training data.
8. The method of claim 7, wherein determining, for each training example of the plurality of training examples in the training data, a respective selection bias value comprises:
obtaining a feature vector for the search query in the training example;
processing the feature vector using the trained classifier to generate a respective query-specific position bias value for each of the plurality of positions for the search query in the training example; and
assigning the query-specific position bias value corresponding to the position of the selected result document for the training example in the result list of result documents for the search query to be the selection bias value for the training example.
9. The method of claim 7, wherein generating training data for training a classifier that receives a respective feature vector for an input search query and outputs a respective query-specific position bias value for each of a plurality of positions for the input search query comprises:
for each experiment search query:
labeling the experiment search query as a positive example for the position in the experiment result list of result documents for the experiment search query of the experiment search result that was selected by the user; and
labeling the experiment search query as a negative example for other positions of the plurality of positions.
10. (canceled)
11. (canceled)
12. The method of claim 1, wherein training the machine learning model using the adjusted losses for the plurality of training examples in the training data comprises:
training the machine learning model by minimizing a sum of the adjusted losses for the plurality of the training examples in the training data.
13. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving training data for a ranking machine learning model that is used to rank documents in response to received search queries, the training data including a plurality of training examples, and each training example of the plurality training examples including data identifying:
a search query,
result documents from a result list for the search query, and
a result document that was selected by a user from the result list of result documents,
receiving position data for each training example of the plurality of training examples in the training data, the position data identifying a respective position of the selected result document in the result list for the search query in the training example;
determining, for each training example of the plurality of training examples in the training data, a respective selection bias value that represents a degree to which the position of the selected result document in the result list for the search query in the training example will impact the selection of the result document;
determining, for each training example of the plurality of training examples in the training data and from the respective selection bias value for the training example, a respective importance value that is inversely proportional to the respective selection bias value for the training example; and
training the ranking machine learning model on the training data, the training comprising, for each training example:
determining a loss for the training example that is based on (i) ranking scores generated by the ranking machine learning model for result documents in the result list that is identified in the training example and (ii) the result document that is identified in the training example as the result document that was selected by the user from the result list that is identified in the training example;
generating an adjusted loss for the training example by adjusting the loss for the training example using the respective importance value for the training example that is inversely proportional to the respective selection bias value for the training example, wherein the adjusted loss for weights losses for training examples having higher respective importance values more strongly than losses having relatively lower respective importance values in the training of the ranking machine learning model wherein the adjusted loss L(f) satisfies:

           L( f)= w·l( Q,f),
wherein w denotes the importance value for the training example that is inversely proportional to the respective selection bias value for the training example, and l(Q, f) denotes the loss for the training example; and
 training the ranking machine learning model on the adjusted loss.
14. The system of claim 13, the operations further comprising:
receiving experiment data identifying a plurality of experiment search queries and, for each experiment search query, a respective position in an experiment result list of experiment result documents for the experiment search query of an experiment result document that was selected by a user, wherein the positions of experiment result documents in the experiment result lists were randomly permuted before being presented to users.
15. The system of claim 14, the operations further comprising:
determining, for each of the plurality of positions, a respective count of selections of experiment result documents at the position by users in response to the plurality of experiment search queries in the experiment data; and
determining, for each of the plurality of positions, a respective position bias value for the position based on the respective count of selections for the position.
16. The system of claim 15, wherein determining, for each training example of the plurality of training examples in the training data, a respective selection bias value comprises:
assigning, for each training example of the plurality of training examples in the training data, the respective position bias value corresponding to the position of the selected result document in the result list of result documents for the training example to be the selection bias value for the training example.
17. The system of claim 14, wherein the experiment search queries in the plurality of experiment search queries each belong to a respective query class of a plurality of query classes, and wherein the method further comprises, for each of the plurality of query classes:
determining, for each of the plurality of positions, a respective count of selections of experiment result documents at the position by users in response to experiment search queries belonging to the query class in the experiment data, and
determining, for each of the plurality of positions, a respective class-specific position bias value for the position based on the respective count of selections for the position.
18. The system of claim 17, wherein determining, for each training example of the plurality of training examples in the training data, a respective selection bias value comprises:
obtaining data identifying a query class to which the search query for the training example belongs;
assigning the class-specific position bias value for the query class to which the search query belongs and corresponding to the position of the selected result document for the training example in the result list of result documents for the training example to be the selection bias value for the training example.
19. The system of claim 14, the operations further comprising:
obtaining a respective feature vector for each experiment search query,
generating training data for training a classifier that receives a respective feature vector for an input search query and outputs a respective query-specific position bias value for each of a plurality of positions for the input search query, and
training the classifier on the training data.
20. The system of claim 19, wherein determining, for each training example of the plurality of training examples in the training data, a respective selection bias value comprises:
obtaining a feature vector for the search query in the training example;
processing the feature vector using the trained classifier to generate a respective query-specific position bias value for each of the plurality of positions for the search query in the training example; and
assigning the query-specific position bias value corresponding to the position of the selected result document for the training example in the result list of result documents for the search query to be the selection bias value for the training example.
21. (canceled)
22. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving training data for a ranking machine learning model that is used to rank documents in response to received search queries, the training data including a plurality of training examples, and each training example of the plurality training examples including data identifying:
a search query,
result documents from a result list for the search query, and
a result document that was selected by a user from the result list of result documents,
receiving position data for each training example of the plurality of training examples in the training data, the position data identifying a respective position of the selected result document in the result list for the search query in the training example;
determining, for each training example of the plurality of training examples in the training data, a respective selection bias value that represents a degree to which the position of the selected result document in the result list for the search query in the training example will impact the selection of the result document;
determining, for each training example of the plurality of training examples in the training data and from the respective selection bias value for the training example, a respective importance value that is inversely proportional to the respective selection bias value for the training example; and
training the ranking machine learning model on the training data, the training comprising, for each training example:
determining a loss for the training example that is based on (i) ranking scores generated by the ranking machine learning model for result documents in the result list that is identified in the training example and (ii) the result document that is identified in the training example as the result document that was selected by the user from the result list that is identified in the training example;
generating an adjusted loss for the training example by adjusting the loss for the training example using the respective importance value for the training example that is inversely proportional to the respective selection bias value for the training example, wherein the adjusted loss for weights losses for training examples having higher respective importance values more strongly than losses having relatively lower respective importance values in the training of the ranking machine learning model wherein the adjusted loss L(f) satisfies:

           L( f)= w·l( Q,f),
wherein w denotes the importance value for the training example that is inversely proportional to the respective selection bias value for the training example, and l(Q, f) denotes the loss for the training example; and
 training the ranking machine learning model on the adjusted loss.
23. The computer-readable storage media of claim 22, wherein determining, for each training example of the plurality of training examples in the training data, a respective selection bias value comprises:
obtaining a feature vector for the search query in the training example;
processing the feature vector using the trained classifier to generate a respective query-specific position bias value for each of the plurality of positions for the search query in the training example; and
assigning the query-specific position bias value corresponding to the position of the selected result document for the training example in the result list of result documents for the search query to be the selection bias value for the training example.