Appium OCR Plugin

This is a Tesseract-based OCR plugin for Appium. It relies on Tesseract.js for the OCR processing.

Features

1. New OCR endpoint - call a new Appium server endpoint to perform OCR on the current screenshot, and return matching text and metadata.
2. OCR context - switch to the OCR context and the page source will be updated to respond with XML corresponding to text objects found on the screen.
3. Find elements by OCR text - When in the OCR context, using XPath will find "elements" based on the OCR XML version of the page source. These found elements can then be interacted with in minimal ways (click, getText) based purely on screen position.

Prerequisites

• Appium Server 2.0+

Installation - Server

Install the plugin using Appium's plugin CLI:

appium plugin install --source=npm appium-ocr-plugin

Installation - Client

The only feature which requires an update on the client is the new getOcrText server endpoint. There are currently not any official client plugins for this feature. Some may be developed in the future. But reference, here is how to add the command to WebdriverIO:

browser.addCommand('getOcrText', command('POST', '/session/:sessionId/appium/ocr', {
    command: 'getOcrText',
    description: 'Get all OCR text',
    ref: '',
    variables: [],
    parameters: []
}))

Also for reference, here is how you might add the command to the Appium python client:

from appium import webdriver
from appium.webdriver.webdriver import ExtensionBase


# define an extension class
class OCRCommand(ExtensionBase):
    def method_name(self):
        return 'ocr_command'

    def ocr_command(self, argument):
        return self.execute(argument)['value']

    def add_command(self):
        return ('post', '/session/$sessionId/appium/ocr')


caps = {
    # set up your actual capabilities
}

# Load the driver with the extension
driver = webdriver.Remote("https://127.0.0.1:4723", desired_capabilities=caps, extensions=[OCRCommand])

# now you can use `driver.ocr_command`
result = driver.ocr_command({})

driver.quit()

Activation

The plugin will not be active unless turned on when invoking the Appium server:

appium --use-plugins=ocr

Usage

Response value terminology

Here is the meaning for the various response values you might find while using this plugin:

• confidence - Tesseract's confidence level (on a scale of 0 to 100) for the result of the OCR process for a given piece of text
• bbox - "bounding box", an object containing values labeled x0, x1, y0, and y1. Here, x0 means the left-hand x-coordinate of the box defining the discovered text, x1 means the right-hand x-coordinate, y0 means the upper y-coordinate, and y1 the lower y-coordinate.

getOcrText endpoint

Sending a POST request to /session/:sessionid/appium/ocr will perform OCR on the current screenshot and respond with a JSON object containing three keys:

• words - Tesseract's guess at individual words
• lines - Tesseract's guess at lines of text
• blocks - Tesseract's guess at contiguous blocks of text

Each of these keys references an array of OCR objects, themselves containing 3 keys:

• text: the text discovered
• confidence: the confidence of the correctness of the resulting text
• bbox: the bounding box of the discovered text (see above)

The OCR context

With this plugin active, you will notice an extra context available in a call to getContexts: OCR. When you switch to the OCR context (via driver.setContext('OCR') or equivalent), certain commands will have new behaviours.

Get Page Source

When retrieving page source in the OCR context, the result will be an XML document with basically the same data as that returned by the getOcrText command. Here is an example:

    
        photo
        library
        Picker
    
    
        9:38
        T -
        The App
        Choose An Awesome View
        Echo Box
        Write something and save to local memory
    
    
        9:38
        T -
        The App
        Choose An Awesome View
        Echo Box
        Write something and save to local memory
    

Find Element(s)

When in the OCR context, you have access to a single locator strategy: xpath. The value of your selector will form the basis of a query against the page source as retrieved and described in the previous section. Any matching elements will be returned to your client. These elements will not be standard UI elements (i.e., XCUIElementTypeText or android.widget.TextView). Instead they are a sort of "virtual" element that only allows the following methods:

• Click Element: perform a single tap action at the center point of the bounding box for the selected element
• Is Element Displayed: always returns true, since if the element weren't displayed, it wouldn't be amenable to OCR
• Get Element Size: returns data from the bounding box in the appropriate format
• Get Element Location: returns data from the bounding box in the appropriate format
• Get Element Rect: returns data from the bounding box in the appropriate format
• Get Element Text: returns the text discovered via the OCR (same text as in the page source output)
• Get Element Attribute: only one attribute (confidence) can be retrieved, and it returns the confidence value

As an example of how this might be used, assuming we're in the OCR context and that the page source matches the example above, we could do the following (in WebdriverIO; adjust as appropriate for other client libraries):

const element = await driver.$('//lines/item[text() = "Echo Box"]')
await element.click()

Similarly, to click on an element using the OCR plugin in Appium-Python-Client, we could do the following:

## To switch the context to OCR (if already not in OCR), use the following line:
# driver.switch_to.context("OCR")

element = driver.find_element(AppiumBy.XPATH,'//lines/item[text() = "Echo Box"]')
element.click()

This clicks the center of the screen region where Tesseract has found the "Echo Box" text line to be located.

Settings

Sometimes it will be necessary to tweak the operation of the plugin in various ways. These settings are available to you to do that. You can either set them as capabilities (e.g., appium:settings[] = ) or use your client library's settings update functionality (e.g., driver.updateSettings(...)).

Development

PRs welcomed!

Setup

1. Clone repo
2. npm install
3. npx tsc

Run tests

1. Link this repo into an Appium server (e.g., appium plugin install --source=local $(pwd) from this plugin development directory)
2. Start the Appium server (e.g., appium --use-plugins=ocr)
3. export the TEST_APP_PATH env var to a path to TheApp.app.zip (https://github.com/cloudgrey-io/the-app/releases)
4. npm run test:unit
5. npm run test:e2e