extractTextWithAI
Extract structured data from the screen using AI vision capabilities.
Experimental, needs an AI CLI
This command uses an AI model, so it needs an AI CLI configured in Settings → Intelligence. See the AI assistant. The AI can answer differently between runs.
The extractTextWithAI command reads a piece of text from the current screen with an AI model and stores it in a variable. By default the variable is aiOutput.
The AI returns the text exactly as shown on the screen, or an empty value when the text is not there.
Syntax
Use a string for a simple query:
- extractTextWithAI: CAPTCHA value
- inputText: ${aiOutput}Or a map for more options:
- extractTextWithAI:
query: 'CAPTCHA value'
outputVariable: 'theCaptchaValue'
- inputText: ${theCaptchaValue}Parameters
| Parameter | Type | Description |
|---|---|---|
query | string | Required. A natural language description of the text to extract. |
outputVariable | string | Optional. The variable that receives the text. Default aiOutput. |
optional | boolean | Optional. When true, a failed step is shown in red but the flow continues. Default true. |
timeout | number | Optional. Maximum time to wait for the AI, in milliseconds. Default 60000. |
Usage examples
Read the title of the first result, then tap it. With optional: false the flow stops if the AI cannot find a title:
- extractTextWithAI:
query: Title of the first item on this page
optional: false
- tapOn: ${aiOutput}Best practices
Use this command when a selector is not practical, for example:
- Content whose text or ID you do not know beforehand, such as dynamic search results.
- Information shown as an image, such as a promotional banner or a CAPTCHA.
It is not a replacement for selectors. When you can, prefer stable IDs or text with commands like tapOn.
How Astra Scout runs it
- The step sends one AI request with the screenshot and the list of UI elements, using the provider and model chosen in Settings → Intelligence → Defaults → AI commands in flows.
- After a step types a secret (for example
inputText: ${PASSWORD}), later AI steps send no screenshot until the app is reopened. See No screenshots after typing a secret.