Processing

Please wait...

Settings

Settings

Goto Application

1. WO2021067443 - SEQUENCE EXTRACTION USING SCREENSHOT IMAGES

Note: Text based on automatic Optical Character Recognition processes. Please use the PDF version for legal matters

[ EN ]

CLAIMS

What is Claimed:

1. A method for sequence extraction using screenshot images to generate a robotic process automation workflow, the method comprising:

capturing a plurality of screenshots of steps performed by a user on an application using a processor;

storing the screenshots in memory;

determining action clusters from the captured screenshots by randomly clustering actions into an arbitrary predefined number of clusters, wherein screenshots of different variations of a same action is labeled in the clusters;

extracting a sequence from the clusters, and discarding consequent events on the screen from the clusters; and

generating an automated workflow based on the extracted sequences.

2. The method of claim 1, wherein the capturing includes templating to find a plurality of words and a corresponding location for each of the plurality of words to cluster in forming a template.

3. The method of claim 2, wherein the templating utilizes a threshold in indicating the plurality of words.

4. The method of claim 3, wherein the threshold comprises approximately 70%.

5. The method of claim 1 , wherein the capturing includes adaptive parameter tuning to iterate the template and tune the capturing for subsequent iterations.

6. The method of claim 1 , wherein the capturing includes random sampling utilizing particle swarm optimization.

7. The method of claim 1 , wherein the capturing includes clustering details incorporating a binary feature vector indicating the presence of template items..

8. The method of claim 1 , wherein the capturing includes novelty by learning sparse representation of screens and tuning cluster granularity.

9. The method of claim 1 , wherein the extracting includes forward link estimation utilizing a forward link prediction module to consider each event and link future events with each event.

10. The method of claim 1, wherein the extracting includes graphical representation with each graph node corresponding to a screen type discovered in the clustering.

11. The method of claim 10, wherein the edges of the graph represent each event and the events linked events.

12. The method of claim 1, wherein the clustering leverages optical character recognition (OCR) data to extract word and location pairs.

13. A system for sequence extraction using screenshot images to generate a robotic process automation workflow, the system comprising:

a processor configured to capture a plurality of screenshots of steps performed by a user on an application; and

a memory module operatively coupled to the processor and configured to store the screenshots;

the processor further configured to:

determine action clusters from the captured screenshots by randomly clustering actions into an arbitrary predefined number of clusters, wherein screenshots of different variations of a same action is labeled in the clusters;

extract a sequence from the clusters, and discarding consequent events on the screen from the clusters; and

generate an automated workflow based on the extracted sequences.

14. The system of claim 13, wherein the capturing includes templating to find a plurality of words and a corresponding location for each of the plurality of words to cluster in forming a template.

15. The system of claim 13, wherein the capturing includes adaptive parameter tuning to iterate the template and tune the capturing for subsequent iterations.

16 The system of claim 13, wherein the capturing includes clustering details incorporating a binary feature vector indicating the presence of template items.

17. The system of claim 13, wherein the capturing includes novelty by learning sparse representation of screens and tuning cluster granularity.

18. The system of claim 13, wherein the extracting includes forward link estimation utilizing a forward link prediction module to consider each event and link future events with each event.

19. The system of claim 13, wherein the extracting includes graphical representation with each graph node corresponding to a screen type discovered in the clustering.

20. A non-transitory computer-readable medium comprising a computer program product recorded thereon and capable of being run by a processor, including program code instructions for sequence extraction using screenshot images to generate a robotic process automation workflow by implementing the steps comprising:

capturing a plurality of screenshots of steps performed by a user on an application using a processor;

storing the screenshots in memory;

determining action clusters from the captured screenshots by randomly clustering actions into an arbitrary predefined number of clusters, wherein screenshots of different variations of a same action is labeled in the clusters;

extracting a sequence from the clusters, and discarding consequent events on the screen from the clusters; and

generating an automated workflow based on the extracted sequences.