Uindow SDK
Founding principles
Defined by the Roman architect Vitruvius, the principles of Strength, Utility, and Delight (firmitas, utilitas, venustas) form the foundation of good architecture, and they are central to Uindow's philosophy.
Firmitas
Uindow opens the door to automations that are simply not possible with other software. It maintains a good balance between AI flexibility and JavaScript programming determinism, all while protecting your privacy.
Utilitas
Effortlessly install, run, and edit modules directly from the built-in source code editor. The editor supports all the methods and properties listed below, and comes with autocomplete, auto-formatting, and linting.
Venustas
Uindow is designed with simplicity at its core. You don't need to be a programmer to get value from this software. Just search our repository for a module that solves your problem, install it, run it, and modify it to your liking.
Getting started
Uindow is a desktop application built on top of Chromium that allows you to automate web actions.
Create an account and download the latest version to get started.
1. Agents
Agents are autonomous browser windows, each operating in its own isolated session. For security reasons, agents do not have direct access to the file system or to each other's data. You must manually specify input files as needed.
2. Modules
Agents execute small JavaScript programs called modules. Each module defines inputs, outputs, functions, and a finite-state machine. For security reasons, modules do not have direct access to the browser window. Instead, the dollar sign object ($) is used to interact with the web page, pause execution, chat with a locally running large language model, fetch user input, save results, and more.
2.1. Finite-state machine
Complex browser interactions are best modeled as a finite-state machine.
When a user starts an agent, the state machine begins execution from the entry state.
It then jumps from state to state with
return { next: "state-key" }The state machine stops if an error occurs or if no next state is specified.
2.2. Functions
Functions are used to avoid code duplication within state machine states. Unlike states, a function's return value has no special significance. Functions can be called from either a state or another function using $.fn("function-key")
2.3. Inputs and outputs
Each module can define up to 128 inputs and 128 outputs.
Supported input and output types are: integer, string, boolean,table and files
Inputs for each run are specified in the Settings tab. Fetch inputs in states and functions with $.ioInput*()
Outputs are collected in the Results tab. Store outputs in states and functions with $.ioOutput*()
3. Collaboration
You can publish your modules to the module repository, allowing others to install and use your work. You can also import and export your automations as .js.yaml files, giving you complete privacy and control over your work.
Example
Here is an example of a small Uindow module (search.js.yaml) that performs a Google search.
Please note that the module is exported as a YAML file, but each function and finite-state machine state is written in pure JavaScript and relies on the dollar sign object ($) for operations.
srcStateMachine:
- key: start
code: |
await $.fn("visit-google");
await $.fn("search");
srcFunctions:
- key: visit-google
code: |
// Navigate to Google and wait for page to load
await $.navLoad("https://google.com/");
- key: search
code: |
// Accept the terms
const acceptBtn = await $.doAwaitPresent('[role="dialog"] button:nth-of-type(2)', {
timeout: 3
});
if (acceptBtn) {
await $.doScrollTo(acceptBtn);
await $.doClick(acceptBtn);
}
// Find the input field
const inputKey = await $.doQuery("[name='q']");
if ("string" !== typeof inputKey) {
throw new Error("Could not find input field");
}
// Get search term from user settings
const searchTerm = $.ioInputString("search-term");
// Replace previous string and send enter key
await $.doType(inputKey, searchTerm, { replace: true, submit: true });
srcInputs:
- key: search-term
type: string
name: Search term
desc: A search term for Google
max: 1024
isList: false
default: uindow
srcOutputs: []
Import this and other modules by following these steps:
- Launch Uindow, select agent, source codesource control, import from YAML
- Choose file (
search.js.yaml)
The SDK
All available Uindow SDK methods and properties are described below. The SDK has a flat structure, with every method and property attached to the dollar sign object ($).$.args
{array} State or function arguments
- When used inside a state:
array passed by previous state as { next: 'state-key', args }
- When used inside a function:
array passed as the second argument to $.fn( 'function-key', args )Although you can use $.global*() methods to store and retrieve data from a global store, it is sometimes better to simply pass arguments from one state to another.
srcStateMachine:
- key: start
code: |
// Generate a random number locally (or use $.osRand())
const randomNumber = await $.fn("random", [2, 10]);
// Pass it to the next state
return { next: "math", args: [randomNumber] };
- key: math
code: |
// Passed with { next: "surprise", args: [randomNumber] };
const randomNumber = $.args[0];
$.log(`The number I was thinking of was ${randomNumber}`);
// Put the LLM to work
const prompt = `Multiply ${randomNumber} by itself.`;
const response = await $.llm(prompt);
srcFunctions:
- key: random
code: |
if (2 !== $.args.length) {
throw new Error("Expecting 2 arguments");
}
// Destructure the function arguments
const [from, to] = $.args;
// Generate a random number
return Math.floor(Math.random() * (to - from + 1)) + from;
srcInputs: []
srcOutputs: []
$.current
{string} Current state key
The key of the current Finite-State Machine (FSM) state.
Use this property in functions to customize behavior based on the FSM state that called the function.At the core of every Uindow module is a finite-state machine where each state is uniquely identified by its key.
You may need to reference the $.current or the $.previous state key inside functions.
srcStateMachine:
- key: start
code: |
await $.fn("write-haiku");
return { next: "middle" };
- key: middle
code: |
await $.fn("write-haiku");
return { next: "end" };
- key: end
code: |
await $.fn("write-haiku");
srcFunctions:
- key: write-haiku
code: |
switch ($.current) {
case "start":
$.log("An old silent pond");
break;
case "middle":
$.log("A frog jumps into the pond");
break;
case "end":
$.log("Splash! Silence again.");
await $.sleep(1000);
$.log("-- The Old Pond by Matsuo Bashō (1644-1694)", "success");
break;
}
await $.sleep(1000);
srcInputs: []
srcOutputs: []
$.previous
{string|null} Previous state key
The key of the previous Finite-State Machine (FSM) state, or null if this is the first (entry) state.
Use this property in functions to customize behavior based on the FSM state that called the function.Following the example for the $.current property, here's how one would use the previous state key.
srcStateMachine:
- key: start
code: |
await $.fn("write-haiku");
return { next: "middle" };
- key: middle
code: |
await $.fn("write-haiku");
return { next: "end" };
- key: end
code: |
await $.fn("write-haiku");
srcFunctions:
- key: write-haiku
code: |
// The Old Pond by Matsuo Bashō (1644-1694)
switch ($.previous) {
case null:
$.log("An old silent pond");
break;
case "start":
$.log("A frog jumps into the pond");
break;
case "middle":
$.log("Splash! Silence again.");
await $.sleep(1000);
$.log("-- The Old Pond by Matsuo Bashō (1644-1694)", "success");
break;
}
await $.sleep(1000);
srcInputs: []
srcOutputs: []
async $.fn( fnKey, fnArgs = [] )
Call a function asynchronously.
Useful if you're running into source code size limits or when you want better
separation of concerns in your module.
@param {string} fnKey Function key, 1 to 32 alphanumeric characters or dashes
@param {array} fnArgs (optional) Function arguments; accessed with $.args
Functions allow you to organize your module better and prevent code duplication.
Here's a simple example for counting down using functions - recursively.
srcStateMachine:
- key: start
code: |
await $.fn("countdown", [3]);
$.log("That was easy");
srcFunctions:
- key: countdown
code: |
const number = $.args[0];
if (number <= 0) {
return;
}
$.log(number, "success");
await $.sleep(1000);
// Recursion is fun!
await $.fn("countdown", [number - 1]);
srcInputs: []
srcOutputs: []
async $.llm( prompt )
Prompt the locally running large language model.
@param {string} prompt Prompt - up to 4096 characters long
@return {string} LLM response
@throws {Error} If the LLM is not ready
Uindow provides easy access to a locally running large language model for complex tasks such as text summarization and sentiment analysis. Please note that LLMs are neither accurate nor deterministic.
The example below shows when not to use a large language model: mathematical operations are much faster and more accurate in pure JavaScript.
srcStateMachine:
- key: start
code: |
const startTime = performance.now();
// Ask the magic box
const response = await $.llm("Answer with one number: 2 + 2");
// Log the execution time
$.log(`Finished in ${((performance.now() - startTime) / 1000).toFixed(2)} seconds`);
srcFunctions: []
srcInputs: []
srcOutputs: []
$.log( message, status = "info" )
Append a message to the agent logs.
@param {any} message Message
@param {"info"|"success"|"warning"|"error"} status (optional) Status; default info
A maximum of 250 logs are retained in the logs panel for each agent. Logs are stored in session and are removed when the app is closed. There are four log types, each with their own color:info,success,warning, anderror.
srcStateMachine:
- key: start
code: |
$.log("🍰 A new cake recipe was added.", "info");
$.log("🧁 The cake has baked successfully!", "success");
$.log("🔥 The oven temperature is far too high.", "warning");
$.log("💀 Cake failed to rise, check your ingredients.", "error");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.sleep( ms )
Pause the execution for a specified number of milliseconds.
@param {number} ms Sleep time in milliseconds
Sometimes you may need to slow down your script to prevent overloading a website's resources, and other times you might just want a bit of showmanship.
srcStateMachine:
- key: start
code: |
$.log("It's the final countdown:", "warning");
await $.sleep(1000);
let counter = 5;
while (counter-- > 0) {
$.log(`⌛ ${counter + 1}...`);
// Wait for it...
await $.sleep(1000);
}
$.log("🚀 Liftoff!", "success");
srcFunctions: []
srcInputs: []
srcOutputs: []
$.pause( message = "" )
Pause the execution of the current state indefinitely.
When resumed, the current state will be re-executed from the start, not from the current line!
@param {string} message (optional) Message displayed in dialog when agent is (re-)selected
Uindow modules do not access or store any personal data, such as passwords or cookies.
If a user needs to log into a website or verify they are human, simply pause the script at the current finite-state machine state and kindly request their input.
srcStateMachine:
- key: start
code: |
// Check if we already asked the user to perform action
if ($.globalRunGet("asked-user")) {
$.log("🧙 Mischief managed");
return;
}
// Mark this so we don't enter an infinite loop
$.globalRunSet("asked-user", true);
// Ooops! (reCaptcha, login wall etc.)
$.pause("Some actions simply cannot continue without a human touch.");
srcFunctions: []
srcInputs: []
srcOutputs: []
$.stop( message = "" )
Stop the execution of the current state.
When resumed, the finite-state machine will start from the first state (the Entry Point 🏁).
@param {string} message (optional) Message displayed in dialog when agent is (re-)selected
Unlike $.pause(), the current run is abandoned so all values stored with $.globalRunSet() are discarded. The next time you start the agent, it will execute normally from the entry state.
srcStateMachine:
- key: start
code: |
// Check if we already asked the user to log in
if (await $.globalEnvGet("asked-user")) {
return { next: "work" };
}
// Mark this so we don't enter an infinite loop
// Use the environment cache instead of the run store
await $.globalEnvSet("asked-user", true);
// Abandon the current run (and clear values in the run store)
$.stop("Work your magic, then restart the agent.");
- key: work
code: |
$.log("🧙 Mischief managed");
srcFunctions: []
srcInputs: []
srcOutputs: []
$.setTimeout( callback, ms )
Delays execution of a function by a specified number of milliseconds.
@param {function} callback JavaScript function
@param {number} ms The number of milliseconds to wait before executing the callback
@return {int} Timeout ID
Setting a timer is useful for a wide range of algorithms, but you will likely find it most valuable when setting up a listener for an event that has not occurred yet.
srcStateMachine:
- key: start
code: |
// Open test page
await $.navLoad("about:home/test");
// Wait for "Get source" button to be present
const buttonKey = await $.doAwaitPresent("[data-role=dl-source]");
// Trigger download of "test.js.yaml" in the future
$.setTimeout(async () => await $.doClick(buttonKey), 500);
// Grab the next download and store it in outputs
await $.ioSaveDownload("yaml");
// Mark the download
$.doTick("download");
$.log("Successfully triggered click 500ms into the future");
srcFunctions: []
srcInputs: []
srcOutputs:
- key: yaml
type: files
name: Yaml files
desc: ""
extensions:
- yaml
visible: true
$.clearTimeout( timeoutId )
Cancels a timeout previously established by $.setTimeout.
@param {function} timeoutId The identifier of the timeout to cancel, as returned by $.setTimeout
srcStateMachine:
- key: start
code: |
// Set the bomb
const timerId = $.setTimeout(async () => {
$.log("💥 Boom!", "error");
}, 500);
// Clear the bomb
$.clearTimeout(timerId);
$.log("All clear", "success");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.osRequest( url, options = {} )
OS: Make a request directly from the computer.
Useful for bypassing CORS (Cross-Origin Resource Sharing) constraints set by the browser.
If you need to make an authenticated request from the currently loaded page, use $.doRequest instead.
Note that these requests do not have access to your browser session's cookies.
Keywords: ajax, fetch, request, get, post, push.
@param {string} url Request URL
@param {Object} options (optional) Request options
@param {string} options.method (optional) Request method; default GET
@param {object} options.data (optional) Request data; default {}
@param {object} options.headers (optional) Request headers; default {}
@param {boolean} options.json (optional) JSON request; default true
@param {int} options.timeout (optional) Request timeout in seconds; default 60
@param {boolean} options.resData (optional) Parse and return the response data; default true
@return {{ ok:boolean, status:number, headers:object, data:mixed}} Response object
@throws {Error} If request failed
This method acts like a proxy, bypassing any CORS restrictions.
If you need to pass along cookies with your request, first nagivate to the target domain using $.navLoad() then issue the request with $.doRequest() or $.ioSaveRequest().
srcStateMachine:
- key: start
code: |
const url = "http://localhost:7199/manifest.json";
// JSON request (CORS is bypassed)
$.log(`Fetching ${url} from OS`, "success");
const response = await $.osRequest(url);
$.log([response?.status, response?.headers, response?.data]);
// Fetch headers only
$.log(`Fetching ${url} without data from OS`, "success");
const responseNoData = await $.osRequest(url, { resData: false });
$.log([responseNoData?.status, responseNoData?.headers, responseNoData?.data]);
srcFunctions: []
srcInputs: []
srcOutputs: []
$.osFileGetUrl( filePath )
OS: Prepare file:/// URL from file path.
Convert file path to file URL to be used in table outputs.
Use the src_output_set_table tool to define a table output.
Use the src_output_set_files tool to define a hidden files output (visible set to false).
@param {string|null} filePath File path generated with $.ioSave* methods or $.ioInputFiles
@return {string|null} URI encoded file:/// URL or null on error
Table cells automatically enrich file:// links with previews. This allows you to show file previews alongside other information in table rows while hiding redundant file outputs from the Results tab.
srcStateMachine:
- key: start
code: |
// Read input URL
const url = $.ioInputString("url");
// Load page
await $.navLoad(url);
// Get page title
const pageTitle = await $.navGetTitle();
// Save a screenshot - but the file explorer is hidden for this output (visible = false)
const { path, width, height } = await $.ioSaveScreenshot("screenshots", { extension: "png" });
// Prepare the screenshot string
const screenshot = $.osFileGetUrl(path) + ` (${width}x${height})`;
// Store complete data as a table row
await $.ioOutputRow("pages", { url, screenshot });
// Increment the temporary tick
$.doTick("screenshot");
srcFunctions: []
srcInputs:
- key: url
type: string
name: Page URL
desc: ""
max: 1024
isList: false
default: https://uindow.com/
srcOutputs:
- key: screenshots
type: files
name: Screenshots
desc: Page screenshots
max: 32
extensions:
- png
visible: false
- key: pages
type: table
name: Visited pages
desc: ""
columns:
- url
- screenshot
async $.osFileGetSize( filePath )
OS: Get file size.
@param {string|null} filePath File path generated with $.ioSave* methods or $.ioInputFiles
@return {{ int:int, string:string}|null} File size in bytes and as a human-readable string expressed in KiB, MiB, GiB, and TiB
The two fields serve different purposes: int is the raw byte count you compare against, and string is a preformatted label in KiB, MiB, GiB or TiB, ready to drop into a log line or a table cell.
Checking the size is the cheapest way to notice that a download went wrong. A saved login wall or error page is still a successful HTTP response, so $.ioSaveUrl hands back a path either way - but the file is almost always far smaller than the real one.
An unreadable or missing path returns null, so check for it before reaching into the result.
srcStateMachine:
- key: start
code: |
const filePath = await $.ioSaveUrl("downloads", "http://localhost:7199/manifest.json");
if (null === filePath) {
throw new Error("$.ioSaveUrl failed");
}
const size = await $.osFileGetSize(filePath);
if (null === size) {
throw new Error("Could not read the file size");
}
// A near-empty file usually means an error page was saved instead of the real thing
if (size.int < 128) {
$.log(`Only ${size.int} bytes - looks truncated, skipping`, "warning");
return;
}
// "size.int" is for comparisons, "size.string" is for humans
$.log(`Outputs total: ${size.string}`);
await $.ioOutputRow("files", { path: filePath, size: size.string });
// Add-up input file sizes
let totalBytes = 0;
for (const inputPath of $.ioInputFiles("attachments")) {
const size = await $.osFileGetSize(inputPath);
if (null === size) {
continue;
}
totalBytes += size.int;
}
$.log(`Inputs total: ${totalBytes} B`);
srcFunctions: []
srcInputs:
- key: attachments
type: files
name: Attachments
desc: ""
extensions:
- json
- txt
multiple: false
srcOutputs:
- key: downloads
type: files
name: Downloads
desc: ""
max: 1024
extensions:
- json
visible: false
- key: files
type: table
name: Saved files
desc: ""
columns:
- path
- size
async $.osFileShow( filePath )
OS: Show file in folder.
@param {string} filePath File path generated with $.ioSave* methods or $.ioInputFiles
@return {boolean}
srcStateMachine:
- key: start
code: |
const url = "http://localhost:7199/manifest.json";
// Save the file to disk
const filePath = await $.ioSaveUrl("json-files", url);
// Open containing folder (if the user allows it)
if ($.ioInputBoolean("show-file-after-download")) {
await $.osFileShow(filePath);
}
srcFunctions: []
srcInputs:
- key: show-file-after-download
type: boolean
name: Show files
desc: Show downloaded files when the script finishes
srcOutputs:
- key: json-files
type: files
name: JSON files
desc: ""
max: 1024
extensions:
- json
$.osRand( min, max, options = {} )
OS: Generate a random signed integer between the specified minimum and maximum values (inclusive), or a random alphanumeric string with a length between the specified minimum and maximum values.
@param {int} min Minimum signed integer value OR minimum string length
@param {int} max Maximum signed integer value OR maximum string length
@param {Object} options (optional) Random generator options
@param {boolean} options.string (optional) Return a random string instead; default false; if true, min and max define the length of the returned string
@return {int|string} A random signed integer between min and max (inclusive) OR a random string between min and max characters long, but not longer than 512 characters
Introducing randomness into the behavior of modules is so useful that we decided to dedicate a helper function to it.
You could use Math.floor(Math.random() * (max - min + 1)) + min instead, but this is cleaner.
srcStateMachine:
- key: start
code: |
const minTemp = -15;
const maxTemp = 25;
const predictedTemp = $.osRand(minTemp, maxTemp);
$.log(`🌤️ Forecast says: ${predictedTemp}°C`, "success");
switch (true) {
case predictedTemp < 0:
$.log("Brrr... this is cold! 🥶");
break;
case predictedTemp < 20:
$.log("Perfect for a walk. 😄");
break;
default:
$.log("Time for some ice cream. 😎");
break;
}
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.globalEnvGet( envKey = null )
Environment globals: Get environment variable(s). Values are JSON serializable.
These values persit between runs but are reset on module install, fork or release.
@param {string|null} envKey (optional) Environment variable key or null for all values as a key-value object; default null
@return {object|any|null}
In this example, we're using the environment cache to perform an action only once per day.
srcStateMachine:
- key: start
code: |
// Prepare date in YYYY-MM-DD format
const dateToday = new Date().toISOString().split("T")[0];
// Fetch the date stored in environment cache (persistent between runs)
const dateStored = await $.globalEnvGet("date");
if (dateToday !== dateStored) {
// Do something new!
$.log("New day, new possibilities ☀️");
// Store today in environment cache
await $.globalEnvSet("date", dateToday);
}
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.globalEnvSet( envKey, envValue )
Environment globals: Set environment variable. Values must be JSON serializable.
These values persit between runs but are reset on module install, fork or release.
The total environment cache size must not exceeded 512kB per agent.
@param {string} envKey Environment variable key
@param {any|null} envValue Environment variable value; if null, the key is removed
@return {boolean}
In this example, we're listing and removing all values from the environment cache.
srcStateMachine:
- key: start
code: |
// Set some random values to environment cache (persistent between runs)
for (let i = 1; i <= 3; i++) {
const randomInt = $.osRand(10, 99);
const randomString = $.osRand(10, 15, { string: true });
await $.globalEnvSet(`key-${randomInt}`, randomString);
}
// Get all values stored in environment cache
const envValues = await $.globalEnvGet();
// Log them as an object
$.log(envValues);
// Log them individually
for (const envKey of Object.keys(envValues)) {
$.log(`✨ ${envKey} = ${envValues[envKey]}`, "success");
}
// Clean the cache
for (const envKey of Object.keys(envValues)) {
// Setting value to null deletes it from the environment cache
await $.globalEnvSet(envKey, null);
$.log(`🗑️ ${envKey} environment value removed`, "warning");
}
srcFunctions: []
srcInputs: []
srcOutputs: []
$.globalRunGet( runKey = null )
Run globals: Get global variable(s) for this run.
These values are reset before each run.
@param {string|null} runKey (optional) Run variable key or null for all values as a key-value object; default null
@return {object|any|null}
The run store is the scratchpad for a single run: set a value in one state, read it in the next, and it's gone by the time the agent starts again. Use $.globalEnvGet when a value needs to outlive the run instead.
Being synchronous is the practical difference from the environment store, which has to be awaited. A missing key reads as null rather than throwing, so ?? is the usual way to supply a starting value.
Guarding re-entry is the pattern worth knowing. A state resumed after $.pause runs again from the top, so a flag checked at the start is what stops the work before the pause from being repeated. Calling it with no key returns the whole store, which is mostly useful for a one-off look while debugging.
srcStateMachine:
- key: start
code: |
if ($.globalRunGet("visited")) {
$.log("Done!");
return;
}
$.log("Visiting the test page...");
await $.navLoad("about:home/test/");
$.globalRunSet("visited", true);
return { next: "start" };
srcFunctions: []
srcInputs: []
srcOutputs: []
$.globalRunSet( runKey, runValue )
Run globals: Set global variable for this run.
These values are reset before each run.
@param {string} runKey Run variable key
@param {any|null} runValue Run variable value; if null, the key is removed
@return {boolean}
Values live for exactly one run and are synchronous to write, which makes this the right place for anything a run needs to carry between states - a cursor, a set of things already seen, a flag saying the user has been asked something.
The return value is more informative than it looks. It reports false when nothing actually changed, which covers both an undefined value and writing a value identical to the one already stored - so a false here doesn't necessarily mean anything went wrong.
Setting null deletes rather than storing null, so there's no separate remove call. Note that $.stop abandons the run and discards everything stored here, while $.pause keeps it - which is what makes the guard above work.
srcStateMachine:
- key: start
code: |
// Store any JSON serializable value
$.globalRunSet("cursor", 0);
$.globalRunSet("temporary", ["alpha", "beta"]);
// Writing the same value again reports false (nothing changed)
if (!$.globalRunSet("cursor", 0)) {
$.log("The cursor was already 0");
}
// Remove the temporary value
$.globalRunSet("temporary", null);
if (Object.keys($.globalRunGet()).includes("temporary")) {
throw new Error("$.globalRunSet failed to delete the key");
}
return { next: "ask-user" };
- key: ask-user
code: |
// A paused state re-runs from the top when the agent resumes,
// so a flag is what keeps this from asking twice
if (!$.globalRunGet("asked-user")) {
$.globalRunSet("asked-user", true);
$.pause("Unpause the agent to continue");
}
$.log("Carrying on where we left off...", "success");
$.doTick("success");
srcFunctions: []
srcInputs: []
srcOutputs: []
$.ioInputInt( ioKey )
IO: Get input integer(s).
This method returns an integer or an array of integers based on the selected input format.
@param {string} ioKey Integer input key
@return {int | int[]} Integer(s) supplied by user
@throws {Error} If ioKey is not a valid input integer key
min and max bound the value the user may enter, so the guessing range is enforced in the Settings tab rather than in your code. isList accepts up to 128 numbers instead of one.
An agent will not start until every input is set, so whatever arrives here is already valid - there is nothing to check and no null to guard against.
srcStateMachine:
- key: start
code: |
// How many tries does the machine get?
const tries = $.ioInputInt("tries");
// Which numbers should the machine guess?
const numbers = $.ioInputInt("lucky-numbers");
// Roll the dice
return { next: "roll-dice", args: [tries, numbers] };
- key: roll-dice
code: |
const [tries, numbers] = $.args;
// Let's roll the dice
for (let i = 1; i <= tries; i++) {
// Guess a number (same min,max restrictions as input lucky-numbers)
const guess = $.osRand(1, 100);
// Did we get it?
if (numbers.includes(guess)) {
$.log(`Guessed it - you were thinking of ${guess}!`, "success");
$.doTick("success");
return;
}
}
$.log("No luck this time. Try again!", "warning");
$.doTick("warning");
srcFunctions: []
srcInputs:
- key: tries
type: int
name: Number of tries
desc: How many times should the machine attempt to guess one of our lucky numbers?
min: 1
max: 100
isList: false
default: 10
- key: lucky-numbers
type: int
name: Lucky numbers
desc: List of lucky numbers - if the machine guesses just one of them, we win!
min: 1
max: 100
isList: true
srcOutputs: []
$.ioInputString( ioKey )
IO: Get input string(s).
This method returns a string or an array of strings based on the selected input format.
@param {string} ioKey String input key
@return {string | string[]} String(s) supplied by user
@throws {Error} If ioKey is not a valid input string key
options turns a text field into a fixed choice and isList accepts up to 128 lines instead of one. min and max bound the length of the text, where the integer equivalents bound the number itself.
An agent will not start until every input is set, so whatever arrives here is already valid - there is nothing to check and no null to guard against.
srcStateMachine:
- key: start
code: |
const mood = $.ioInputString("mood");
const wishes = $.ioInputString("wishes");
const wish = wishes[$.osRand(0, wishes.length - 1)];
const fortune = await $.llm(`Grant this wish in one ${mood} sentence: ${wish}`);
$.log(`🔮 ${fortune}`, "success");
srcFunctions: []
srcInputs:
- key: mood
type: string
name: Oracle mood
desc: How should the oracle feel today?
max: 16
isList: false
options:
- cheerful
- grumpy
- cryptic
default: cheerful
- key: wishes
type: string
name: Your wishes
desc: One wish per line - the oracle picks one at random
max: 128
isList: true
srcOutputs: []
$.ioInputBoolean( ioKey )
IO: Get input boolean.
@param {string} ioKey Boolean input key
@return {boolean} Boolean supplied by user
@throws {Error} If ioKey is not a valid input boolean key
The result is always a real boolean - a switch the user never touched reads as false rather than null or undefined, so there's nothing to default and no need for ??.
Booleans are the natural way to expose the choices that change what a module does rather than what it works on: dry run, verbose logging, whether to open a folder when the run finishes.
They also double as UI controls. Any other input can name a boolean in its depends property, as log-prefix does above, and it stays hidden from the Settings tab until that switch is on - so a whole section of the form can fold away behind one checkbox. Only non-boolean inputs can depend on a boolean, so the dependencies never nest.
Only the key is looked up here, so a typo throws rather than quietly returning false.
srcStateMachine:
- key: start
code: |
const dryRun = $.ioInputBoolean("dry-run");
const verbose = $.ioInputBoolean("verbose");
verbose && $.log("Verbose logging is on");
if (dryRun) {
verbose && $.log("Dry run", "warning");
// Not home yet
if (!(await $.navGetUrl()).match(/\/home\/?$/)) {
verbose && $.log("Navigating home...");
await $.navLoad("about:home/");
}
return;
}
verbose && $.log("Visiting test page...");
await $.navLoad("about:home/test/");
verbose && $.log("Typing some words of wisdom...");
await $.doType('[name="input-textfield"]', "Code is poetry");
await $.ioSaveScreenshot("previews");
verbose && $.log("Navigating home...");
await $.navLoad("about:home/");
srcFunctions: []
srcInputs:
- key: dry-run
type: boolean
name: Dry run
desc: Preview the result without typ anything
- key: verbose
type: boolean
name: Verbose logging
desc: ""
srcOutputs:
- key: previews
type: files
name: Previews
desc: ""
max: 128
extensions:
- png
async $.ioInputRow( ioKey, index = null )
IO: Get the next available table row and increment the row index internally.
Alternatively, get the row at the specified index.
For example, index 0 returns the first table row object with the defined columns, or null if the table is empty
Use the src_input_set_table tool to declare input table columns.
@param {string} ioKey Table input key
@param {int} index (optional) Table index; defaults to null
@return {(Object<string, string> | null)} Current row, or null if the end of the table has been reached
@throws {Error} If ioKey is not a valid input table key.
For performance reasons, tables are not loaded into memory; instead, they are accessed one row at a time.
srcStateMachine:
- key: start
code: |
// Go through the clients table one row at a time
let row = null;
while ((row = await $.ioInputRow("clients"))) {
$.log(`Client name: ${row.name}`);
$.log(`Client age: ${row.age}`);
}
srcFunctions: []
srcInputs:
- key: clients
type: table
name: Company clients
desc: List of clients
columns:
- name
- age
srcOutputs: []$.ioInputFiles( ioKey )
IO: Get input file paths.
@param {string} ioKey Files input key
@return {string[]} Input file paths
@throws {Error} If ioKey is not a valid input files key
Get absolute file paths, as configured by the user or the agent_settings_set tool.
These paths can be used for $.doChooseFiles, $.osFileGetSize, $.osFileShow or $.osFileGetUrl.
There is no need to guard against an empty array because agents cannot not start until every input is set.
srcStateMachine:
- key: start
code: |
const files = $.ioInputFiles("uploads");
const lucky = files[$.osRand(0, files.length - 1)];
$.log(`🎁 Picking 1 of your ${files.length} files at random...`, "success");
await $.osFileShow(lucky);
srcFunctions: []
srcInputs:
- key: uploads
type: files
name: Your files
desc: Pick a few - the module opens one of them at random
extensions:
- png
- jpg
- jpeg
- pdf
multiple: true
max: 25
srcOutputs: []
async $.ioOutputInt( ioKey, int )
IO: Set output integer.
Subsequent calls override previous values.
@param {string} ioKey Integer output key
@param {int} int Integer
@return {boolean} true on success, false for invalid integer
@throws {Error} If ioKey is not a valid output integer key
Within a run each call replaces the last. Across runs the Results tab keeps one value per run and draws them as a line graph - leave this on a schedule and you get a chart of a page quietly changing shape.
Values go through parseInt, so 4.7 lands as 4.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const sections = await $.doQueryAll("h2");
await $.ioOutputInt("sections", sections.length);
$.log(`📈 ${sections.length} sections on the test page`, "success");
await $.navLoad("about:home/");
srcFunctions: []
srcInputs: []
srcOutputs:
- key: sections
type: int
name: Sections found
desc: ""
min: 0
async $.ioOutputString( ioKey, string )
IO: Set output string.
Subsequent calls override previous values.
If you need to store longer strings, use $.ioSaveText instead.
@param {string} ioKey String output key
@param {string} string String
@return {boolean} true on success, false for invalid string
@throws {Error} If ioKey is not a valid output string key
The max you declare is the real ceiling - longer text is cut rather than refused, so an LLM that ignores your character limit loses its ending quietly. Use $.ioSaveText for anything long.
Only strings are accepted, and each run keeps its own value, so the Results tab builds a back catalogue of headlines rather than overwriting one.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const title = await $.navGetTitle();
await $.ioOutputString("headline", title);
await $.ioOutputString("verdict", await $.llm(`Review "${title}" in under 40 characters. Be dramatic.`));
$.log("📰 Today's edition is in the Results tab", "success");
srcFunctions: []
srcInputs: []
srcOutputs:
- key: headline
type: string
name: Headline
desc: ""
max: 120
- key: verdict
type: string
name: Dramatic verdict
desc: ""
max: 40
async $.ioOutputBoolean( ioKey, boolean )
IO: Set output boolean.
Subsequent calls override previous values.
@param {string} ioKey Boolean output key
@param {boolean} boolean Boolean value
@return {boolean} true on success, false for invalid boolean
@throws {Error} If ioKey is not a valid output boolean key
Boolean outputs are rendered as green or red dots in the Results tab.
Subsequent calls override previous values for each agent run.
srcStateMachine:
- key: start
code: |
// Get the current minute
const minute = new Date().getMinutes();
// This minute is even (divisible by 2)
const rightTime = 0 === minute % 2;
// Save/override the result as a boolean flag
await $.ioOutputBoolean("right-time", rightTime);
srcFunctions: []
srcInputs: []
srcOutputs:
- key: right-time
type: boolean
name: Right time
desc: The agent ran at an even minute
async $.ioOutputRow( ioKey, row )
IO: Append a row to output table.
Use the src_output_set_table tool to declare output table columns.
@param {string} ioKey Table output key
@param {Object<string,string>} row Row object
@return {boolean} true on success, false for invalid row object
@throws {Error} If ioKey is not a valid output table key
Output tables are append-only: each call adds one row at the end, so results are written out as you go rather than collected in memory and flushed at the end. If the agent stops halfway through, the rows written so far are still there.
That's the opposite of how the scalar outputs behave. Where an integer or string output is replaced by each call and keeps one value per run, tables and files accumulate - and the Results tab retains every run, so rows can be filtered and queried across the whole history rather than just the last pass.
The two failure modes are distinct. An ioKey that isn't a declared table output throws, because that's a mistake in the module rather than in the data. A row that isn't an object returns false instead, so a single malformed record can be logged and skipped without abandoning the remaining rows.
The keys of the row object correspond to the columns declared in the output. A table takes up to 6 columns of up to 32 characters each, and a module can define at most 6 output tables - so a wide scrape needs either a narrower shape or a file written with $.ioSaveText.
srcStateMachine:
- key: start
code: |
// Read the input table one row at a time and write a result row for each
let row = null;
while ((row = await $.ioInputRow("subjects"))) {
const subject = row.subject;
$.log(`Working on ${subject}`);
const joke = await $.llm(`Tell me a short joke about ${subject}`);
// Keys map onto the columns declared in srcOutputs.
// A row that isn't an object returns false rather than throwing.
const saved = await $.ioOutputRow("jokes", { subject, joke });
if (!saved) {
$.log(`Could not append a row for ${subject}`, "warning");
}
}
$.log("Every subject has been written to the output table", "success");
srcFunctions: []
srcInputs:
- key: subjects
type: table
name: Subjects
desc: A list of subjects to joke about
columns:
- subject
srcOutputs:
- key: jokes
type: table
name: Jokes
desc: One joke per subject
columns:
- subject
- joke
async $.ioSaveText( ioKey, text, options = {} )
IO: Save text to disk as new file.
Useful for saving arbitrary strings in custom formats like JSON, YAML, INI etc.
For strings that are shorter than or equal to 1024 characters, you can use $.ioOutputString.
@param {string} ioKey Files output key
@param {string} text Text to save
@param {Object} options (optional) Save options
@param {string} options.extension (optional) File extension; default null; must match one of the extensions declared in output; falls back to first file extension declared in output
@return {string | null} File path on success, null if download failed
@throws {Error} If ioKey is not a valid output files key
Every call writes a new file rather than appending, so build the whole string in memory and save it once. For short text that doesn't need to be a file, $.ioOutputString is simpler - as long as it fits inside the max length declared on that output.
The extension has to be one of those declared on the output, and an unrecognised one quietly falls back to the first in the list - so a file you expected as .csv can arrive as .json if the declaration doesn't mention CSV.
The min and max declared on a files output are size bounds in megabytes, enforced as the file is written. Both are real constraints - a file that comes out too small is rejected just as one that comes out too large is - and either way the throw is followed by the partial file being cleaned up, so a failed save won't leave a truncated document in the results. Because Uindow runs locally, the ceiling can be as generous as your disk allows.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Anything you can build as a string: JSON, YAML, INI, CSV
const report = JSON.stringify(
{
url: await $.navGetUrl(),
title: await $.navGetTitle(),
at: new Date().toISOString()
},
null,
2
);
const jsonPath = await $.ioSaveText("reports", report, { extension: "json" });
if (null === jsonPath) {
throw new Error("$.ioSaveText failed");
}
$.log(`Wrote ${report.length} characters to ${jsonPath}`, "success");
return { next: "csv" };
- key: csv
code: |
// Building a CSV by hand is worth it when the shape doesn't
// suit a table output, or when the file is the deliverable
const rows = [["url", "heading"]];
const pageUrl = await $.navGetUrl();
for (const headingKey of await $.doQueryAll("h2")) {
rows.push([pageUrl, await $.doGetContent(headingKey)]);
}
const escape = (value) => '"' + String(value).replace(/"/g, '""') + '"';
const csv = rows.map((cells) => cells.map(escape).join(",")).join("\n");
const csvPath = await $.ioSaveText("reports", csv, { extension: "csv" });
if (null !== csvPath) {
$.log(`Saved ${rows.length - 1} rows`, "success");
$.doTick("collect", rows.length - 1);
}
srcFunctions: []
srcInputs: []
srcOutputs:
- key: reports
type: files
name: Reports
desc: ""
max: 64
extensions:
- json
- csv
- txt
async $.ioSaveDownload( ioKey, options = {} )
IO: Capture the next downloaded file and save it to disk.
Useful for saving any file download, regardless of how it was trigerred.
Defer the event that triggers the download with $.setTimeout() before calling $.ioSaveDownload.
@param {string} ioKey Files output key
@param {Object} options (optional) Save options
@param {string} options.extension (optional) File extension; default null; must match one of the extensions declared in output; falls back to first file extension declared in output
@param {int} options.timeout (optional) Download timeout in seconds; default 600
@return {string | null} File path on success, null if download failed
@throws {Error} If ioKey is not a valid output files key
The ordering here is the whole trick. This call waits for the next download, so the thing that starts the download has to happen while it's already waiting - hence deferring the click with $.setTimeout. Clicking first and calling afterwards means the download has come and gone before anything was listening.
Use it when you can't address the file directly: downloads produced by a form submission, generated on the fly behind a button, or sitting behind a one-time link. When you already know the URL, $.ioSaveUrl is simpler, and $.ioSaveRequest covers requests you need to shape with headers or a body.
A download that never arrives returns null once the timeout expires. The default of 600 seconds is generous for a large export but a long time to wait on a click that silently did nothing, so lower it when you know the file is small.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Arm the capture first, then let the click happen while it waits
const srcButton = await $.doQuery('[data-role="dl-source"]');
$.setTimeout(async () => await $.doClick(srcButton), 500);
const savePath = await $.ioSaveDownload("yaml", { timeout: 120 });
if (null === savePath) {
throw new Error("$.ioSaveDownload failed");
}
$.log(`Saved to ${savePath}`, "success");
$.doTick("download");
return { next: "custom-extension" };
- key: custom-extension
code: |
const logoButton = await $.doQuery('[data-role="dl-logo"]');
if (null === logoButton) {
return;
}
$.setTimeout(async () => await $.doClick(logoButton), 500);
// Stored under a declared extension whatever the URL happens to say
const imgPath = await $.ioSaveDownload("images", { extension: "jpeg" });
if (null === imgPath) {
$.log("Nothing was downloaded", "warning");
return;
}
$.log(`Image saved as ${imgPath}`);
srcFunctions: []
srcInputs: []
srcOutputs:
- key: yaml
type: files
name: Yaml files
desc: ""
max: 512
extensions:
- yaml
- key: images
type: files
name: Images
desc: ""
max: 512
extensions:
- jpeg
- png
async $.ioSaveScreenshot( ioKey, options = {} )
IO: Take a screenshot of the web page and save it to disk.
@param {string} ioKey Files output key
@param {Object} options (optional) Save options
@param {string} options.extension (optional) File extension; default null; must match one of the extensions declared in output; falls back to first file extension declared in output
@param {boolean} options.full (optional) Grab a full-page screenshot; default false to grab a screenshot of the visible viewport
@param {boolean} options.dpr (optional) Use Device Pixel Ratio (DPR); default false; output video at true scale, which might be 2:1 instead of 1:1 on MacOS
@param {int} options.wait (optional) Wait time in milliseconds before grabbing screenshot; default 100; [100, 10000]
@return {{ path:(string|null), width:int, height:int, error: (string|null)}} Screenshot details
@throws {Error} If ioKey is not a valid output files key
The returned object reports the failure in error and leaves path as null rather than throwing, so check path before treating the file as saved. An invalid ioKey is the one case that does throw.{ full: true } captures the entire scrollable document instead of the viewport, which is what you usually want for archiving a page. Raise wait when the page loads images or charts lazily, and set dpr to true for retina-scale output at the cost of a much larger file.
The extension must be one of those declared in the files output, otherwise the first declared extension is used instead.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Just what the user would see, as PNG
const viewportShot = await $.ioSaveScreenshot("images", { extension: "png" });
if (null === viewportShot.path) {
throw new Error(`Screenshot failed: ${viewportShot.error}`);
}
$.log(`Saved ${viewportShot.width}x${viewportShot.height} to ${viewportShot.path}`, "success");
// The whole document as JPEG, giving lazy-loaded images a moment to settle
const fullShot = await $.ioSaveScreenshot("images", {
extension: "jpeg",
full: true,
wait: 1500
});
if (null === fullShot.path) {
throw new Error(`Full page screenshot failed: ${fullShot.error}`);
}
$.log(`Full page is ${fullShot.height} pixels tall`);
return { next: "evidence" };
- key: evidence
code: |
// Screenshots are cheap - grab one whenever something goes wrong
try {
await $.doClick(await $.doQuery("button", { contains: "submit" }));
} catch (error) {
const shot = await $.ioSaveScreenshot("images", { full: true, dpr: true });
$.log(`Click failed, evidence saved to ${shot.path}`, "error");
}
srcFunctions: []
srcInputs: []
srcOutputs:
- key: images
type: files
name: Screenshots
desc: ""
max: 1024
extensions:
- png
- jpeg
async $.ioSaveVideo( ioKey, options = {} )
IO: Record a video of the session and save it to disk.
The video is rendered in real time with no sound.
@param {string} ioKey Files output key
@param {Object} options (optional) Save options
@param {string} options.extension (optional) File extension; default null; must match one of the extensions declared in output; falls back to first file extension declared in output
@param {int} options.fps (optional) Frames per second; default 20; an integer between 1 and 30
@param {"av1"|"h264"} options.codec (optional) Video codec; default "av1"; available codecs:
- "av1": better compression
- "h264": wider support across devices
@param {boolean} options.dpr (optional) Use Device Pixel Ratio (DPR); default false; output video at true scale, which might be 2:1 instead of 1:1 on MacOS
@param {boolean} options.rwp (optional) Record While Paused; default false; continue recording video even when agent is paused
@return {function(): { path:(string|null), error: (null|string), width:int, height:int, duration:int}}
Returns an async function that stops the page recorder.
Calling this function returns an object with final video details.
@throws {Error} If ioKey is not a valid output files key, or trying to record more than one video at a time
In the following example we're recording smooth scrolling a web page at 150 pixels per second. Note that $.ioSaveVideo returns a callback function that stops the recording.
srcStateMachine:
- key: start
code: |
// Go home
await $.navLoad("about:home/");
// Start recording
const recStop = await $.ioSaveVideo("video");
// Wait, then load the test page
await $.sleep(1000);
await $.navLoad("about:home/test/");
// Smooth-scroll through the entire page
await $.doScroll(750, { speed: 150 });
// Log recorded video file path
const filePath = await recStop();
$.log(filePath);
srcFunctions: []
srcInputs: []
srcOutputs:
- key: video
type: files
name: Recordings
desc: ""
max: 512
extensions:
- mp4
async $.ioSaveUrl( ioKey, url, options = {} )
IO: Capture the file stored at this URL and save it to disk.
Useful for saving files that are not available as links on page.
@param {string} ioKey Files output key
@param {string} url URL to download
@param {Object} options (optional) Save options
@param {int} options.timeout (optional) Download timeout in seconds; default 600
@param {string} options.extension (optional) File extension; default null; must match one of the extensions declared in output; falls back to first file extension declared in output
@return {string | null} File path on success, null if download failed
@throws {Error} If ioKey is not a valid output files key, or request failed
Reach for this when you already know the address of the file you want. It's the counterpart to $.ioSaveDownload, which waits for the page to start a download on its own, and to $.ioSaveRequest, which is for saving the result of a request you had to shape yourself with headers or a body.
The URL is resolved against the page currently loaded, so relative paths scraped straight out of src or href attributes can be passed through unchanged.
A failed download returns null rather than throwing, so a single broken asset doesn't have to end the run. An ioKey that isn't a declared files output does throw.
srcStateMachine:
- key: start
code: |
// Save an asset that is never exposed as a link on the page
const imgPath = await $.ioSaveUrl("images", "http://localhost:7199/img/pages/page-404.png");
if (null === imgPath) {
throw new Error("$.ioSaveUrl failed");
}
$.log(`Saved to ${imgPath}`, "success");
// Store it under a different declared extension, and allow longer for a big file
const jpegPath = await $.ioSaveUrl("images", "http://localhost:7199/img/pages/page-401.png", {
extension: "jpeg",
timeout: 120
});
$.log(`Saved as ${jpegPath}`);
return { next: "scrape-gallery" };
- key: scrape-gallery
code: |
await $.navLoad("http://localhost:7199/");
// Relative URLs are resolved against the page that is currently loaded
await $.ioSaveUrl("images", "/img/pages/page-404.png");
// Pull every image the page references
const imageKeys = await $.doQueryAll("img[src]");
for (const imageKey of imageKeys) {
const src = await $.doGetAttribute(imageKey, "src");
if (null === src) {
continue;
}
const savedPath = await $.ioSaveUrl("images", src);
if (null === savedPath) {
$.log(`Skipped ${src}`, "warning");
continue;
}
$.doTick("download");
}
srcFunctions: []
srcInputs: []
srcOutputs:
- key: images
type: files
name: Images
desc: ""
max: 1024
extensions:
- png
- jpeg
async $.ioSaveRequest( ioKey, url, options = {} )
IO: Capture the result of this fetch request and save it to disk.
Useful for saving the result of fetch requests made from the current domain/page.
For direct access to JSON or text responses, use $.doRequest instead.
Keywords: ajax, fetch, request, get, post, push.
@param {string} ioKey Files output key
@param {string} url URL to save locally
@param {Object} options (optional) Request options
@param {string} options.method (optional) Request method; default GET
@param {object} options.data (optional) Request data; default {}
@param {object} options.headers (optional) Request headers; default {}
@param {boolean} options.json (optional) JSON request; default true
@param {int} options.timeout (optional) Request timeout in seconds; default 60
@param {string} options.extension (optional) File extension; default null; must match one of the extensions declared in output; falls back to first file extension declared in output
@return {string | null} File path on success, null if download failed
@throws {Error} If ioKey is not a valid output files key, or request failed
This example demonstrates how to save a file with a custom extension. Note that the file extension must first be declared in the output configuration. If the specified extension is not included in the declared list, the first listed extension, "json" in this case, will be used instead.
If you don't specify a file extension, the script will attempt to deduce it from the URL.
srcStateMachine:
- key: start
code: |
const url = "http://localhost:7199/manifest.json";
// Navigate to origin so the browser allows the request (CORS)
await $.navLoad(new URL(url).origin);
$.log("Saving JSON as simple text file...");
const jsonPath = await $.ioSaveRequest("manifest", url, { extension: "txt" });
if ("string" !== typeof jsonPath) {
throw new Error("$.ioSaveRequest failed");
}
srcFunctions: []
srcInputs: []
srcOutputs:
- key: manifest
type: files
name: JSON files
desc: ""
max: 1024
extensions:
- json
- txt
async $.handleAlert()
Browser: Prevent the next window.alert() from bubbling.
Unhandled alert dialogs are automatically closed,
and their message is passed as a toast notification.
Each call arms a guard for a single dialog and then steps aside, so a page that raises three alerts needs three calls. Arm it before the action that triggers the dialog, never after.
An alert nobody handled won't stall the run - it's closed automatically and its message surfaces as a toast. Handling it explicitly is about the alerts you already know are coming, so an expected dialog doesn't get reported as if something had gone wrong.
For the other two dialog types, use $.handleConfirm and $.handlePrompt. Note that these cover the browser's own window.alert family only - a modal the page draws in HTML is just markup, so dismiss it with $.doClick.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Disarm the dialog before the click that raises it
await $.handleAlert();
await $.doClick(await $.doQuery('[data-role="alert"]'));
// The guard covers exactly one dialog, so a second alert
// needs a second call
await $.handleAlert();
await $.doClick(await $.doQuery('[data-role="alert"]'));
$.log("Both alerts handled", "success");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.handleConfirm( accept = true )
Browser: Prevent the next window.confirm() from bubbling, and either accept or reject it.
Unhandled confirmation dialogs are automatically accepted,
and their message is passed as a toast notification.
@param {boolean} accept (optional) Accept or reject the next confirmation message; default true
Left alone, a confirmation dialog is accepted on your behalf. That's convenient for cookie notices and harmless enough most of the time, but it means an accidental click on something destructive goes through without resistance.
Passing false is the interesting case: it lets a module click the button and still cancel, which is exactly what a dry run wants. The value can be computed, as above, so the same code path serves both modes.
One dialog per call, armed before the click that raises it.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const confirmButton = await $.doQuery('[data-role="confirm"]');
const resultKey = await $.doQuery('[data-role="confirm-result"]');
// Accept the next confirmation
await $.handleConfirm();
await $.doClick(confirmButton);
$.log(`The page received: ${await $.doGetContent(resultKey)}`);
// Reject the one after that
await $.handleConfirm(false);
await $.doClick(confirmButton);
$.log(`The page received: ${await $.doGetContent(resultKey)}`);
return { next: "guarded-delete" };
- key: guarded-delete
code: |
// Rejecting matters more than accepting: an unhandled confirmation
// is accepted for you, which is the wrong default for a dry run
const dryRun = $.ioInputBoolean("dry-run");
const deleteKey = await $.doQuery("button", { contains: "delete" });
if (null === deleteKey) {
return;
}
await $.handleConfirm(!dryRun);
await $.doClick(deleteKey);
$.log(dryRun ? "Deletion cancelled" : "Deletion confirmed", dryRun ? "warning" : "success");
srcFunctions: []
srcInputs:
- key: dry-run
type: boolean
name: Dry run
desc: Cancel destructive confirmations instead of accepting them
srcOutputs: []
async $.handlePrompt( response = "" )
Browser: Prevent the next window.prompt() from bubbling, and answer it.
Unhandled prompts automatically return with their default value or an empty string,
and their message is passed as a toast notification.
@param {string} response (optional) Prompt response text
A prompt left unhandled returns its default value, or an empty string when it has none, and the message is reported as a toast. That's rarely what the page was waiting for, so any prompt that gates real work needs an answer queued in advance.
Queue it before the click, not after - the guard covers the next prompt only, and by the time the dialog is open it's too late to decide what to say.
Passing an empty string answers with an empty string, which is a different thing from leaving the prompt unhandled. Sites that treat blank as a valid response will accept it.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Queue the answer, then trigger the prompt
await $.handlePrompt("foo");
await $.doClick(await $.doQuery('[data-role="prompt"]'));
const answer = await $.doGetContent(await $.doQuery('[data-role="prompt-result"]'));
if ("foo" !== answer) {
throw new Error(`The page received "${answer}" instead`);
}
$.log("Prompt answered", "success");
return { next: "answer-from-input" };
- key: answer-from-input
code: |
// The answer usually comes from somewhere else - an input,
// a table row, or a value carried over from an earlier state
const code = $.ioInputString("code");
await $.handlePrompt(code);
await $.doClick(await $.doQuery('[data-role="prompt"]'));
$.log(`Answered the prompt with ${code.length} characters`);
srcFunctions: []
srcInputs:
- key: code
type: string
name: Confirmation code
desc: Sent as the answer to the page's prompt
max: 64
default: uindow
srcOutputs: []
async $.doQuery( selector, options = {} )
Document: Find the first HTML element that matches the CSS selector and return its corresponding element key.
@param {string} selector CSS selector
@param {Object} options (optional) Query options
@param {string} options.parent (optional) Parent CSS selector OR element key; default null to search the entire Document
@param {string} options.contains (optional) Text contained by element (case insensitive); default null for no restrictions
@param {boolean} options.scrollable (optional) Restrict results to elements that have active scrollbars; default false
@param {boolean} options.viewportDown (optional) Restrict results to elements placed in the viewport and below it; default false
@return {string|null} 24 characters long element key or null on error
Matching on the text an element shows survives redesigns that break a structural selector. parent narrows the hunt to one subtree, viewportDown skips what you have scrolled past.
Nothing found is null rather than an error - it is handing that null to $.doClick that throws.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const treasure = await $.doQuery(".MuiButton-root", { contains: "foo" });
await $.doHighlight(treasure);
$.log(`🎯 Found a <${await $.doGetTag(treasure)}> saying "${await $.doGetContent(treasure)}"`, "success");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doQueryAll( selector, options = {} )
Document: Find all HTML elements that match the CSS selector and return their corresponding element keys.
@param {string} selector CSS selector
@param {Object} options (optional) Query options
@param {string} options.parent (optional) Parent CSS selector OR element key; default null to search the entire Document
@param {string} options.contains (optional) Text contained by element (case insensitive); default null for no restrictions
@param {boolean} options.scrollable (optional) Restrict results to elements that have active scrollbars; default false
@param {boolean} options.viewportDown (optional) Restrict results to elements placed in the viewport and below it; default false
@return {string[]} Array of 24 characters long element keys
Unlike $.doQuery, this always hands back an array - an empty one when nothing matches - so there's no null check to forget. Test length when an empty page is worth reporting.
Pairing it with the parent option is the standard way to scrape repeated markup: collect the row containers here, then run narrow queries inside each one. That keeps the per-row selectors short and stops a query from wandering into a neighbouring row.
All the filters from $.doQuery apply, and contains is often the quickest way to keep only the rows that mention something you care about.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Every match, in document order
const headingKeys = await $.doQueryAll("h2");
$.log(`Found ${headingKeys.length} sections`);
for (const headingKey of headingKeys) {
$.log(await $.doGetContent(headingKey));
}
return { next: "scrape-rows" };
- key: scrape-rows
code: |
// The usual shape of a scrape: find the repeated container,
// then query inside each one with { parent }
const rowKeys = await $.doQueryAll("[data-role=row]");
if (!rowKeys.length) {
$.log("Nothing to scrape on this page", "warning");
return;
}
for (const rowKey of rowKeys) {
const nameKey = await $.doQuery(".name", { parent: rowKey });
const linkKey = await $.doQuery("a[href]", { parent: rowKey });
if (null === nameKey || null === linkKey) {
continue;
}
await $.ioOutputRow("items", {
name: await $.doGetContent(nameKey),
url: (await $.doGetAttribute(linkKey, "href")) ?? ""
});
$.doTick("collect");
}
srcFunctions: []
srcInputs: []
srcOutputs:
- key: items
type: table
name: Scraped items
desc: ""
columns:
- name
- url
async $.doQueryParent( element, options = {} )
Document: Find the parent of this HTML element that matches the CSS selector and return its corresponding element key.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {Object} options (optional) Query options
@param {string} options.selector (optional) CSS selector for parent element; default null to stop at first ancestor
@param {string} options.contains (optional) Text contained by parent element (case insensitive); default null for no restrictions
@param {boolean} options.scrollable (optional) Restrict results to parent elements that have active scrollbars; default false
@return {string|null} 24 characters long element key or null on error
Selectors only ever point downwards, which is awkward when the element you can reliably identify is a leaf and the data you want lives on its siblings. Climbing to a shared container and querying down from there is the way around it.
That pairs naturally with the parent option on $.doQuery: climb once to establish scope, then run narrow queries inside it. The result is far steadier than a long descendant selector that breaks the moment a wrapper is added.contains matches text anywhere inside the ancestor, which on an outer container will match almost anything - combine it with selector to keep the climb from overshooting. No match returns null.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const switchKey = await $.doQuery("input[name='s1'][value='1']");
// With no selector, you get the immediate ancestor
const firstParent = await $.doQueryParent(switchKey);
$.log(await $.doGetAttribute(firstParent, "class"));
// With one, it climbs until something matches
const groupKey = await $.doQueryParent(switchKey, { selector: "div", contains: "Switches" });
if (null === groupKey) {
throw new Error("Could not find the enclosing group");
}
$.log(`Group: ${await $.doGetAttribute(groupKey, "data-stack")}`);
return { next: "climb-to-row" };
- key: climb-to-row
code: |
// The usual shape: find the one thing you can identify,
// then climb to the container holding everything you want
const priceKey = await $.doQuery("[data-role=price]", { contains: "99" });
if (null === priceKey) {
$.log("No matching price on this page", "warning");
return;
}
const rowKey = await $.doQueryParent(priceKey, { selector: "[data-role=row]" });
if (null === rowKey) {
return;
}
// Now query downwards again, scoped to that row
const nameKey = await $.doQuery(".name", { parent: rowKey });
$.log(await $.doGetContent(nameKey));
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doQuerySiblings( element, options = {} )
Document: Find the siblings of this HTML element that match the CSS selector and return their corresponding element keys.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {Object} options (optional) Query options
@param {string} options.selector (optional) CSS selector for sibling elements; default null to match all siblings
@param {string} options.contains (optional) Text contained by sibling elements (case insensitive); default null for no restrictions
@param {boolean} options.scrollable (optional) Restrict results to sibling elements that have active scrollbars; default false
@return {string[]} 24 characters long element keys
Siblings answer the questions a downward selector can't phrase: the other options in this group, the cells beside this one, the rows around the one that matched. The starting element is excluded, so the count is "the others" rather than "all of them".
It returns an array, empty when nothing matches, so there's no null to guard against - unlike $.doQueryParent, which returns a single key or nothing.
Which element you start from decides what counts as a sibling. Climbing to the wrapper first, as above, usually gives the set you meant - starting from the raw input would return the pieces of that one control instead.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const switchKey = await $.doQuery("input[name='s1'][value='1']");
const switchRoot = await $.doQueryParent(switchKey, { selector: ".MuiSwitch-root" });
// The element you started from is never included in the result
const siblingKeys = await $.doQuerySiblings(switchRoot);
$.log(`${siblingKeys.length} sibling switches`);
for (const siblingKey of siblingKeys) {
const checkboxKey = await $.doQuery("input[type='checkbox']", { parent: siblingKey });
if (null === checkboxKey) {
continue;
}
$.log(`Sibling value: ${await $.doGetAttribute(checkboxKey, "value")}`);
}
return { next: "narrow" };
- key: narrow
code: |
const switchKey = await $.doQuery("input[name='s1'][value='1']");
const switchRoot = await $.doQueryParent(switchKey, { selector: ".MuiSwitch-root" });
// Narrow by selector, or by the text the sibling contains
const labelKeys = await $.doQuerySiblings(switchRoot, { selector: "label" });
$.log(`${labelKeys.length} labels alongside it`);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doQueryAt( left, top, options = {} )
Document: Find the first HTML element that matches the CSS selector at the specified coordinates,
and return its corresponding element key.
@param {int} left Left coordinate in pixels
@param {int} top Top coordinate in pixels
@param {Object} options (optional) Query options
@param {string} options.selector (optional) CSS selector for top element; default null to stop at first ancestor
@param {string} options.contains (optional) Text contained by element (case insensitive); default null for no restrictions
@param {boolean} options.scrollable (optional) Restrict results to elements that have active scrollbars; default false
@return {string|null} 24 characters long element key or null on error
This is the inverse of $.doGetBox: instead of asking where an element is, it asks what is at a place. That makes it the sanity check for coordinate work - confirm what a $.doClickAt would land on before firing it, rather than discovering afterwards that a sticky header was in the way.
Without selector you get whatever is on top at that point, which on a nested layout is often an inner wrapper. Passing a selector climbs to the first matching ancestor instead, so you can ask for the button rather than the span inside it.
Coordinates are viewport-relative and must be non-negative integers; anything else returns null without throwing.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// What is sitting at this point in the viewport?
const elementKey = await $.doQueryAt(50, 20);
if (null === elementKey) {
$.log("Nothing at that point", "warning");
return;
}
$.log(`Found a <${await $.doGetTag(elementKey)}>`);
// The topmost element is often a wrapper or an overlay.
// A selector climbs to the ancestor you actually meant.
const buttonKey = await $.doQueryAt(50, 20, { selector: "button" });
$.log(null === buttonKey ? "Not inside a button" : "Inside a button");
return { next: "follow-pointer" };
- key: follow-pointer
code: |
// Check what a coordinate click would hit before committing to it
await $.doHoverAt(400, 400);
const { left, top } = await $.doGetMouse();
const targetKey = await $.doQueryAt(left, top);
if (null === targetKey) {
$.log("The pointer is over empty space", "warning");
return;
}
$.log(await $.doGetContent(targetKey));
await $.doClickAt(left, top);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doRequest( url, options = {} )
Document: Make a request from the current page.
Useful for JSON and simple text responses. For large files or binary data use $.ioSaveRequest instead.
If you need to bypass CORS and send the requests directly from the computer (outside of the browser session) use $.osRequest instead.
Keywords: ajax, fetch, request, get, post, push.
@param {string} url Request URL
@param {Object} options (optional) Request options
@param {string} options.method (optional) Request method; default GET
@param {object} options.data (optional) Request data; default {}
@param {object} options.headers (optional) Request headers; default {}
@param {boolean} options.json (optional) JSON request; default true
@param {int} options.timeout (optional) Request timeout in seconds; default 60
@param {boolean} options.resData (optional) Parse and return the response data; default true
@return {{ ok:boolean, status:number, headers:object, data:mixed}} Response object
@throws {Error} If request failed
This example describes how to fetch data in the browser when CORS is an issue.
If you don't care about cookies you can use $.osRequest() to bypass CORS instead.
srcStateMachine:
- key: start
code: |
const url = "http://localhost:7199/manifest.json";
// Navigate to origin so the browser allows the request (CORS)
await $.navLoad(new URL(url).origin);
// JSON request
$.log(`Fetching ${url} from browser`, "success");
const response = await $.doRequest(url);
$.log([response?.status, response?.headers, response?.data]);
// Fetch headers only
$.log(`Fetching ${url} without data from browser`, "success");
const responseNoData = await $.doRequest(url, { resData: false });
$.log([responseNoData?.status, responseNoData?.headers, responseNoData?.data]);
srcFunctions: []
srcInputs: []
srcOutputs: []
$.doTick( name, amount = 1 )
Document: Increment a named counter in the Status bar.
Up to 5 counters can be displayed at a time.
@param {string} name Counter name. The following strings are displayed as icons:
"contact", "view", "like", "post", "repost", "comment",
"upload", "download", "screenshot", "collect",
"success", "warning"
@param {int} amount (optional) Strictly positive number; [0,1000]; default 1; 0 won't increment the counter
Logs scroll away and a run that prints one line per item quickly becomes unreadable. A counter stays put and keeps climbing, which makes it much easier to tell at a glance whether an agent is working or stuck.
The names listed above are drawn as icons; anything else appears as its own label, so a module can invent counters that suit its own work. Only five are shown at a time, so pick a handful of things worth counting rather than ticking everything.
The amount is clamped to between 0 and 1000, and 0 is a no-op rather than an error - convenient when the increment is computed from something that might turn out to be empty.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Counters are the cheapest progress signal in a long run
const rowKeys = await $.doQueryAll("[data-role=row]");
for (const rowKey of rowKeys) {
await $.ioOutputRow("items", { name: await $.doGetContent(rowKey) });
// No await - ticking is synchronous
$.doTick("collect");
}
return { next: "named-counters" };
- key: named-counters
code: |
// These names render as icons rather than text
$.doTick("screenshot");
$.doTick("download", 3);
$.doTick("success");
// Anything else is shown as a plain label
$.doTick("pages");
// An amount of 0 is accepted but changes nothing
$.doTick("collect", 0);
srcFunctions: []
srcInputs: []
srcOutputs:
- key: items
type: table
name: Collected items
desc: ""
columns:
- name
async $.doHighlight( element, options = {} )
Document: Highlight an HTML element in the viewport for 1 second.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {Object} options (optional) Highlight options
@param {boolean} options.scroll (optional) Scroll element into view before highlighting; default true
@param {boolean} options.hover (optional) Move mouse over the center of the element after scrolling into view; default true
The highlight is drawn over the agent's viewport and never touches the page, so nothing you highlight is modified and no handler on the page fires because of it.
Each call waits out the full animation, a little over a second, before returning. That's the point when you're recording with $.ioSaveVideo or watching an agent to see why a selector went wrong - and the reason to leave it out of loops that just need to get work done.
By default the element is scrolled into view and the mouse moves to its center. Pass { scroll: false, hover: false } to draw the box without disturbing either. To highlight an area that isn't an element, use $.doHighlightBox with a box from $.doGetBox.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Show what the agent is about to touch before it touches it
const inputKey = await $.doQuery("[name=input-textfield]");
await $.doHighlight(inputKey);
await $.doType(inputKey, "uindow", { replace: true });
// Already on screen, and the mouse is needed elsewhere
await $.doHighlight("h1", { scroll: false, hover: false });
return { next: "tour" };
- key: tour
code: |
// Walking a set of matches makes a recording much easier to follow
const headingKeys = await $.doQueryAll("h2");
for (const headingKey of headingKeys) {
await $.doHighlight(headingKey);
}
$.log(`Highlighted ${headingKeys.length} sections`, "success");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doHighlightBox( box )
Document: Highlight a box in the viewport for 1 second.
@param {Object} box Rectangle details; obtained with $.doGetBox
@param {int} box.left Left coordinate in pixels
@param {int} box.top Top coordinate in pixels
@param {int} box.width Width in pixels
@param {int} box.height Height in pixels
Where $.doHighlight takes an element and does the work around it - scrolling it into view, moving the pointer to it - this takes four numbers and draws exactly that rectangle. Nothing is scrolled and nothing is hovered, so the coordinates need to be on screen already.
It also returns immediately rather than waiting out the animation, which $.doHighlight does. Add your own $.sleep when you want the box to be seen before the next one is drawn - otherwise a run of highlights will overwrite each other faster than anyone can follow.
A box narrower or shorter than two pixels is skipped, so a degenerate rectangle produces no error and no output. Like $.doHighlight, the overlay is drawn over the agent's viewport and never touches the page.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const box = await $.doGetBox("[name=input-textfield]");
if (null === box) {
return;
}
// A box from $.doGetBox can be passed straight through
await $.doHighlightBox(box);
await $.sleep(1200);
// Or adjusted to point at part of an element
await $.doHighlightBox({ left: box.left, top: box.top, width: 50, height: 50 });
await $.sleep(1200);
// Boxes thinner than 2px in either direction are not drawn at all
await $.doHighlightBox({ left: box.left, top: box.top, width: 1, height: 1 });
return { next: "region" };
- key: region
code: |
// The box doesn't have to correspond to an element - here we mark
// the top third of the viewport before capturing it
const viewport = await $.doGetViewport();
await $.doHighlightBox({
left: 0,
top: 0,
width: viewport.width,
height: Math.round(viewport.height / 3)
});
await $.sleep(1200);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetMouse()
Document: Get mouse position.
@return {{ left: int, top: int}}
Coordinate methods such as $.doHoverAt, $.doClickAt and $.doTypeAt take absolute viewport coordinates. $.doGetMouse is what lets you work relative to the current pointer instead, which is handy for drag-like gestures, nudging along a slider, or resuming a movement in a later state.
Positions are relative to the viewport, not the document, so they change meaning after a scroll. Pair it with $.doQueryAt when you want to know what is actually underneath the pointer.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Park the pointer in the middle of the viewport
const viewport = await $.doGetViewport();
await $.doHoverAt(viewport.width / 2, viewport.height / 2);
// Read it back
const { left, top } = await $.doGetMouse();
$.log(`Mouse is at ${left} x ${top}`);
// Walk it to the right in small steps, relative to wherever it is now
for (let i = 0; i < 5; i++) {
const position = await $.doGetMouse();
await $.doHoverAt(position.left + 40, position.top);
await $.sleep(200);
}
const moved = await $.doGetMouse();
$.log(`Mouse travelled ${moved.left - left} pixels`, "success");
return { next: "inspect" };
- key: inspect
code: |
// Mouse position carries across states, so you can pick up where you left off
const { left, top } = await $.doGetMouse();
// What is sitting under the pointer right now?
const elementKey = await $.doQueryAt(left, top);
if (null === elementKey) {
$.log("Nothing under the pointer", "warning");
return;
}
$.log(`Hovering a <${await $.doGetTag(elementKey)}> element`, "success");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetBox( element )
Document: Get the box of an HTML element.
@typedef {Object} Box
@property {int} left Left coordinate (px)
@property {int} top Top coordinate (px)
@property {int} width Width of border-box, including padding and borders (px)
@property {int} height Height of border-box, including padding and borders (px)
@property {int} scrollLeft Distance of scrolled content from the left (px)
@property {int} scrollTop Distance of scrolled content from the top (px)
@property {int} scrollWidth Total width of content inside element, including overflow (px)
@property {int} scrollHeight Total height of content inside element, including overflow (px)
@param {string} element CSS selector OR element key obtained with $.doQuery*
@return {Box|null} Element box or null on error
The box is where an element sits in the viewport right now, which makes it the bridge to the coordinate methods - $.doClickAt, $.doTypeAt, $.doHoverAt, $.doHighlightBox. That's the way in when you need to hit a specific point inside an element rather than its center, such as a position along a slider.
Because the numbers are viewport-relative they go stale the moment the page scrolls, so read the box immediately before you use it rather than holding on to one across states.
The scroll fields describe the element's own overflow: comparing scrollHeight against height tells you whether it has more content than it's showing, which is how you find the inner panels that scroll independently of the page.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const inputKey = await $.doQuery("[name=input-text]");
const box = await $.doGetBox(inputKey);
if (null === box) {
throw new Error("$.doGetBox failed");
}
$.log(`${box.width}x${box.height} at ${box.left},${box.top}`);
// Coordinates are viewport-relative, so they feed the *At methods directly
await $.doTypeAt(box.left + box.width / 2, box.top + box.height / 2, "baz", {
replace: true
});
// Highlight a region of your own choosing
await $.doHighlightBox({ left: box.left, top: box.top, width: 50, height: 50 });
return { next: "detect-inner-scroll" };
- key: detect-inner-scroll
code: |
// scrollHeight larger than height means the element holds more
// than it can show - a panel that scrolls on its own
const listBox = await $.doGetBox("[data-role=long-list]");
if (null === listBox) {
return;
}
if (listBox.scrollHeight > listBox.height) {
$.log(`List shows ${listBox.height}px of ${listBox.scrollHeight}px`, "warning");
// Wheel events land wherever the pointer is
await $.doHover("[data-role=long-list]");
await $.doScroll(listBox.scrollHeight - listBox.height);
}
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetSelector( elKey )
Document: Generate an optimal CSS selector for an HTML element.
@param {string} elKey Element key obtained with $.doQuery*
@return {string|null} CSS selector or null on error
Note the asymmetry with the rest of the $.do* family: those accept either a CSS selector or an element key, but this one takes an element key only. It converts in one direction, from a match you've already made into a selector describing it.
That's useful in two situations. While building a module, log the selector for an element you found by text or position, and paste the result into your source as a faster, more explicit query. At runtime, hand the selector to another state, since a plain string survives being stored with $.globalRunSet or $.globalEnvSet.
The selector is generated from the page as it looks right now. Markup that changes between visits - hashed class names, generated ids, position-dependent paths - can produce a selector that won't match later.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Matching on visible text is convenient, but the text may be
// translated, reworded, or simply slow to find on a large page
const buttonKey = await $.doQuery(".MuiButton-root", { contains: "foo" });
if (null === buttonKey) {
throw new Error("Could not find the 'foo' button");
}
// Turn the match into a plain CSS selector
const selector = await $.doGetSelector(buttonKey);
$.log(`Matched: ${selector}`);
// Selectors are ordinary strings, so they can be stored
$.globalRunSet("action-button", selector);
return { next: "reuse" };
- key: reuse
code: |
const selector = $.globalRunGet("action-button");
if (null === selector) {
return;
}
// Every $.do* method takes a CSS selector wherever it takes an element key
await $.doHighlight(selector);
await $.doClick(selector);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetTag( element )
Document: Get the tag name of an HTML element.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@return {string|null} Element tag name or null on error
Tag names come back lowercase, so they compare cleanly against string literals.
Most useful for checking what a coordinate lookup actually found - $.doQueryAt returns whatever sits at a point, which may be a wrapper or an overlay. The tag alone will not tell you the kind of input, since <input> covers text, checkbox, radio and file; read the type attribute for that.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
await $.doHoverAt(50, 20);
const spot = await $.doQueryAt(50, 20);
$.log(`Under the pointer: <${await $.doGetTag(spot)}>`);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetValue( element )
Document: Get the value of an HTML element. Supported elements:
- input (includes checkbox and radio)
- textarea
- select
Returns multiple values for checboxes and <select multiple/>.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@return {string|string[]|boolean|null} Value or null on error
A string for text fields, radios and ordinary dropdowns; an array for checkbox groups and <select multiple>.
This reads the live value, unlike the value attribute, which still reports whatever the markup shipped with. Reading it back after typing catches input masks and validators that rewrite what you sent.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const field = await $.doQuery("[name=input-text]");
await $.doType(field, "uindow", { replace: true });
$.log(await $.doGetValue(field));
const boxes = await $.doQuery("input[type=checkbox][name=c1]");
await $.doCheck(boxes, ["2", "4"]);
$.log(await $.doGetValue(boxes));
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetOptions( element )
Document: Get all options for a select element.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@return {{ value:string, selected:true, text: string}[]} List of options
$.doSelect matches on an option's value, which is rarely what a user would recognise: a country dropdown may label an option "Germany" while its value is DE or 82. Reading the options first lets you match the label and pass along the value behind it.
It's also how you check that a choice exists at all. Options are frequently filtered by earlier answers in the same form, so listing them before selecting turns a silent mis-selection into something you can log and handle.
Use $.doGetValue when you only need what's currently selected; this method is for seeing the full set of possibilities.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const selectKey = await $.doQuery("[data-role=select]");
// Find out what the dropdown actually offers before touching it
const options = await $.doGetOptions(selectKey);
$.log(`${options.length} options available`);
// What is selected right now?
const current = options.find((o) => o.selected);
$.log(`Currently selected: ${current ? current.text : "nothing"}`);
return { next: "choose-by-label" };
- key: choose-by-label
code: |
const selectKey = await $.doQuery("[data-role=select]");
const options = await $.doGetOptions(selectKey);
// Users think in labels, but $.doSelect works on values.
// Match the visible text, then pass along the value behind it.
const wanted = $.ioInputString("choice").toLowerCase();
const match = options.find((o) => o.text.toLowerCase().includes(wanted));
if (!match) {
$.log(`No option matching "${wanted}"`, "warning");
$.log(options.map((o) => o.text));
return;
}
await $.doSelect(selectKey, match.value);
$.log(`Chose "${match.text}"`, "success");
return { next: "choose-many" };
- key: choose-many
code: |
// Multi-selects work the same way, with an array of values
const multiKey = await $.doQuery("[data-role=select-multi]");
const multiOptions = await $.doGetOptions(multiKey);
await $.doSelect(
multiKey,
multiOptions.slice(0, 2).map((o) => o.value)
);
srcFunctions: []
srcInputs:
- key: choice
type: string
name: Option to pick
desc: Matched against the visible label of the dropdown
max: 64
default: "8"
srcOutputs: []
async $.doGetAttribute( element, attr )
Document: Get a single HTML element attribute.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {string} attr HTML attribute (lowercase)
@return {string|null} Attribute value or null on error
Names must be lowercase. Boolean attributes are present-or-absent, so a present one reads as an empty string - compare against null rather than testing truthiness.
A null result is ambiguous: the attribute is missing, or the element is. Reading several attributes off one element is a single call to $.doGetAttributes rather than several here.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const radio = await $.doQuery("input[type=radio][name=r1]");
$.log(await $.doGetAttribute(radio, "value"));
const disabled = await $.doGetAttribute(await $.doQuery("button"), "disabled");
$.log(null === disabled ? "Button is enabled" : "Button is disabled");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetAttributes( element, attrs = [] )
Document: Get all or multiple HTML element attributes.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {string[]} attrs (optional) HTML attributes (lowercase) or empty array for all; default []
@return {Object<string,string>} Map of attribute and value
Each call crosses into the page, so pulling five attributes off one element costs five round trips with $.doGetAttribute and one here. Over a long list that difference is the whole runtime.
Calling it with no list returns everything, which is mostly a tool for exploration: log it once while writing the module to see what a site actually exposes, then narrow to the handful you need.
Attributes that aren't set don't appear in the result at all, so reach for ?? rather than expecting an empty string - particularly when the values are heading straight into an output table.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const linkKey = await $.doQuery("a[href]");
if (null === linkKey) {
return;
}
// Ask for the ones you need, in a single round trip
const attrs = await $.doGetAttributes(linkKey, ["href", "target", "rel"]);
$.log(`href=${attrs.href} target=${attrs.target ?? "(none)"}`);
// Omit the list to see everything the element carries -
// handy while working out which attributes are worth reading
$.log(await $.doGetAttributes(linkKey));
return { next: "scrape" };
- key: scrape
code: |
// Attributes that aren't set are simply missing from the map,
// so fall back rather than assuming every key is there
for (const cardKey of await $.doQueryAll("[data-role=card]")) {
const attrs = await $.doGetAttributes(cardKey, ["data-id", "data-price"]);
await $.ioOutputRow("cards", {
id: attrs["data-id"] ?? "",
price: attrs["data-price"] ?? ""
});
}
srcFunctions: []
srcInputs: []
srcOutputs:
- key: cards
type: table
name: Cards
desc: ""
columns:
- id
- price
async $.doGetContent( element, asHtml = false )
Document: Get the content of an HTML element.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {boolean} asHtml (optional) Use innerHTML instead of innerText; default false
@return {string|null} Element contents or null on error
The default reads rendered text, with whitespace collapsed the way a person sees it. Pass true for the underlying HTML when the structure carries meaning you would lose - links inside a paragraph, a table you mean to parse.null means the element was not found; an element with no content returns an empty string.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const heading = await $.doQuery("h1");
$.log(await $.doGetContent(heading));
$.log(await $.doGetContent(heading, true));
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetStyle( element, props = [] )
Document: Get the resolved values of this element's CSS properties.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {string[]} props List of CSS properties to return; default [] to return all
@return {object|null} Element CSS properties or null on error; invalid CSS properties are discarded from the result object
The values are resolved rather than whatever a stylesheet declared, so this reads the state a page presents through CSS - a row greyed out by a class, an element faded to nothing, a control the design marks as inactive without setting disabled.
For the plain question of whether an element is shown, $.doGetVisible is the simpler answer; this is for when you need the actual value, or a property that has nothing to do with visibility.
Properties that aren't recognised are dropped from the result rather than reported, so a missing key means either an unknown property name or an element that wasn't found. Requesting a short explicit list keeps the result readable and makes a typo obvious.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const buttonKey = await $.doQuery(".MuiButton-root", { contains: "foo" });
if (null === buttonKey) {
return;
}
// Ask for the handful of properties you care about
const style = await $.doGetStyle(buttonKey, ["display", "visibility", "opacity", "color"]);
if (null === style) {
throw new Error("$.doGetStyle failed");
}
$.log(style);
// Values are resolved, so state driven by a class is readable
if ("none" === style.display || "0" === style.opacity) {
$.log("The button is hidden by CSS", "warning");
return;
}
await $.doClick(buttonKey);
return { next: "explore" };
- key: explore
code: |
// Omit the list to get everything - a lot of output, but useful
// once while working out which properties carry the state you need
const all = await $.doGetStyle("h1");
if (null === all) {
return;
}
$.log(`${Object.keys(all).length} properties resolved`);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetVisible( element )
Document: Get whether HTML element is visible on page.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@return {boolean}
A one-shot check covering display, visibility and opacity. Use $.doAwaitVisible to wait for it to become true rather than test it now.
Worth running over scraped rows: filter interfaces often hide non-matching entries instead of removing them, so $.doQueryAll happily returns rows no user can see.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const ghost = await $.doQuery("#alert-visible");
$.log(`Before: ${await $.doGetVisible(ghost)}`);
await $.doClick(await $.doQuery('[data-role="toggle-visible"]'));
$.log(`After: ${await $.doGetVisible(ghost)}`);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetScrollable( element )
Document: Get whether HTML element has scroll bars.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@return {{ horizontal: boolean, vertical: boolean}}
Nested scroll areas are a common reason automation quietly does nothing: the script scrolls, the page moves, and the list you actually wanted stays exactly where it was. Checking first tells you whether you're dealing with one.$.doScroll sends wheel events at the pointer's current position, so the fix is to $.doHover the panel before scrolling. The horizontal flag pairs with { vertical: false } for carousels and wide tables.
To find these elements in the first place, $.doQuery and $.doQueryAll both take a scrollable option that filters to elements with active scrollbars. This method then tells you which axis you're working on.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Find a panel that scrolls on its own
const panelKey = await $.doQuery("div", { scrollable: true });
if (null === panelKey) {
$.log("Nothing on this page scrolls internally");
return;
}
const { horizontal, vertical } = await $.doGetScrollable(panelKey);
$.log(`Scrollbars - vertical: ${vertical}, horizontal: ${horizontal}`);
return { next: "scroll-inside" };
- key: scroll-inside
code: |
const panelKey = await $.doQuery("div", { scrollable: true });
if (null === panelKey) {
return;
}
const scrollable = await $.doGetScrollable(panelKey);
// Wheel events go wherever the pointer is, so hover the panel
// first - otherwise the page scrolls and the panel doesn't move
if (scrollable.vertical) {
await $.doHover(panelKey);
await $.doScroll(300, { speed: 400 });
}
if (scrollable.horizontal) {
await $.doHover(panelKey);
await $.doScroll(300, { vertical: false });
}
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetInViewport( element )
Document: Get whether HTML element is even partially located in the viewport.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@return {boolean}
A question about scroll position, not CSS: any overlap with the viewport counts, so one pixel inside the bottom edge reports true.
Keep it apart from $.doGetVisible, which asks whether an element is hidden. An element can be visible but off screen, or on screen but hidden.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const heading = await $.doQuery("h1");
$.log(`On screen: ${await $.doGetInViewport(heading)}`);
await $.doHoverCenter();
await $.doScroll(1000);
$.log(`After scrolling: ${await $.doGetInViewport(heading)}`);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doGetViewport()
Document: Get the viewport box.
@typedef {Object} Box
@property {int} left Left coordinate (px)
@property {int} top Top coordinate (px)
@property {int} width Width of viewport (px)
@property {int} height Height of viewport (px)
@property {int} scrollLeft Distance of scrolled content from the left (px)
@property {int} scrollTop Distance of scrolled content from the top (px)
@property {int} scrollWidth Total width of content, including overflow (px)
@property {int} scrollHeight Total height of content, including overflow (px)
@return {Box}
Every coordinate method - $.doClickAt, $.doHoverAt, $.doTypeAt, $.doQueryAt - works in viewport space, and this is where you find its bounds. Deriving positions from width and height keeps a module working across window sizes in a way that hard-coded pixels don't.
The scroll fields describe the document rather than any element: scrollHeight against height tells you how much page there is, and scrollTop tells you where you are in it. That comparison is also how a full-page screenshot works out its capture height.
Anything derived from these numbers goes stale as soon as the page scrolls or lazily loads more content, so read the viewport again rather than reusing an earlier copy.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const viewport = await $.doGetViewport();
$.log(`Viewport is ${viewport.width}x${viewport.height}`);
$.log(`The document is ${viewport.scrollHeight}px tall`);
// Coordinate methods work in this space, so the centre is easy
await $.doHoverAt(viewport.width / 2, viewport.height / 2);
// How far down the page are we?
const scrollable = viewport.scrollHeight - viewport.height;
if (scrollable > 0) {
$.log(`${Math.round((viewport.scrollTop / scrollable) * 100)}% scrolled`);
} else {
$.log("The page fits on one screen");
}
return { next: "page-by-page" };
- key: page-by-page
code: |
// Scroll exactly one screen at a time, whatever the window size
await $.doHoverCenter();
const viewport = await $.doGetViewport();
const screens = Math.ceil(viewport.scrollHeight / viewport.height);
for (let i = 1; i < screens && i < 10; i++) {
await $.doScroll(viewport.height);
await $.ioSaveScreenshot("screens", { extension: "png" });
$.doTick("screenshot");
}
srcFunctions: []
srcInputs: []
srcOutputs:
- key: screens
type: files
name: Screens
desc: ""
max: 512
extensions:
- png
async $.doClick( element, options = {} )
Document: Click or double-click on HTML element.
Automatically scroll to element before action.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {Object} options (optional) Click options
@param {int} options.left (optional) Left coordinate relative to element in pixels; default null to horizontally center on the element
@param {int} options.top (optional) Top coordinate relative to element in pixels; default null to vertically center on the element
@param {boolean} options.double (optional) Double-click; default false
@param {boolean} options.hover (optional) Hover after click; default true; use false to move mouse to the side after clicking
@throws {Error} If element not found
The element is scrolled into view before the click, so you don't need to call $.doScrollTo first. A missing element throws rather than failing quietly, which is why it's worth checking the result of $.doQuery when the element may legitimately be absent.left and top are measured from the element's own top-left corner, not the viewport - use $.doClickAt when you want absolute viewport coordinates instead. Both are ignored unless they're integers, in which case the click lands on the center of the element.
The mouse stays where it clicked by default. Pass { hover: false } when that would leave a dropdown or tooltip open over whatever you need to reach next.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Find a button by the text it displays, then click it
const fooButton = await $.doQuery(".MuiButton-root", { contains: "foo" });
if (null === fooButton) {
throw new Error("Could not find the 'foo' button");
}
await $.doClick(fooButton);
// Confirm the page actually reacted
const clicked = await $.doGetContent(await $.doQuery(".clicked-button"));
$.log(`Last clicked: ${clicked}`, "success");
// A CSS selector works anywhere an element key does
await $.doClick("button", { double: true });
// Click 10 pixels in from the element's top-left corner instead of its center,
// and park the mouse aside afterwards so it doesn't sit on top of a tooltip
await $.doClick(fooButton, { left: 10, top: 10, hover: false });
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doClickAt( left, top, options = {} )
Document: Click or double-click at coordinates in viewport.
@param {int} left Left coordinate in pixels
@param {int} top Top coordinate in pixels
@param {Object} options (optional) Click options
@param {boolean} options.double (optional) Double-click; default false
@param {boolean} options.hover (optional) Hover after click; default true; use false to move mouse to the side after clicking
@return {boolean} true on success, false on failure
Use $.doClick whenever an element can be queried - it scrolls the target into view and can't miss because the page shifted. This is for the cases where position is the input: a point along a slider, a spot on a chart, a region of a canvas.
Nothing is scrolled for you, so the point has to already be on screen. Getting there usually means $.doScrollTo first and $.doGetBox afterwards, since a box read before scrolling describes where the element used to be.
Negative or non-integer coordinates return false instead of throwing. Pairing the call with $.doQueryAt is a cheap way to confirm the point lands on what you expect rather than on an overlay.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Absolute viewport coordinates
await $.doClickAt(250, 20);
// Double-click, then move the pointer aside
await $.doClickAt(50, 20, { double: true, hover: false });
// Check what you are about to hit before hitting it
const targetKey = await $.doQueryAt(400, 400);
if (null === targetKey) {
$.log("Nothing at that point - skipping the click", "warning");
return;
}
$.log(`About to click a <${await $.doGetTag(targetKey)}>`);
await $.doClickAt(400, 400);
return { next: "slider" };
- key: slider
code: |
// Controls with no clickable child - a slider track, a chart,
// a colour picker - are driven by position
const box = await $.doGetBox("[data-role=slider]");
if (null === box) {
$.log("No slider on this page", "warning");
return;
}
// Three quarters of the way along
const clicked = await $.doClickAt(
box.left + Math.round(box.width * 0.75),
box.top + Math.round(box.height / 2)
);
if (!clicked) {
$.log("The click was refused", "warning");
}
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doHover( element, options = {} )
Document: Move mouse over an HTML element.
Automatically scroll to element before action.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {Object} options (optional) Click options
@param {int} options.left (optional) Left coordinate relative to element in pixels; default null to horizontally center on the element
@param {int} options.top (optional) Top coordinate relative to element in pixels; default null to vertically center on the element
@throws {Error} If element not found
Hovering is a real mouse move, so CSS :hover rules and mouseenter handlers fire exactly as they would for a person. That makes it the way in to menus, tooltips and toolbars that don't exist in the DOM until the pointer arrives.
The element is scrolled into view first, and a missing element throws. left and top are measured from the element's own top-left corner; when you'd rather work in viewport coordinates, use $.doHoverAt.
Since $.doScroll sends wheel events wherever the pointer currently is, hovering an element first is also how you scroll inside a panel rather than the page.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Menus that only exist while the pointer is over the trigger
await $.doHover(await $.doQuery("h1"));
if (await $.doAwaitVisible(".dropdown-menu", { timeout: 5 })) {
await $.doClick(".dropdown-menu a");
} else {
$.log("No menu appeared", "warning");
}
return { next: "offsets" };
- key: offsets
code: |
// Hover near a corner instead of the middle - useful for wide elements
// where the center lands on a gap or on a child that swallows the event
const selectKey = await $.doQuery("[data-role=select]");
await $.doHover(selectKey, { left: 5, top: 5 });
// Where did the pointer end up?
const { left, top } = await $.doGetMouse();
$.log(`Pointer at ${left} x ${top}`);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doHoverAt( left, top )
Document: Move mouse to coordinates in viewport.
@param {int} left Left coordinate in pixels
@param {int} top Top coordinate in pixels
@return {boolean} true on success, false on failure
The coordinate counterpart to $.doHover. Use the element version when you have something to query - it scrolls the target into view first - and this one when the position is what matters, or when the thing under the pointer has no selector worth writing.
Its most common job isn't hovering at all: it decides where $.doScroll sends its wheel events, and where $.doQueryAt and $.doGetMouse read from. Placing the pointer is often the setup step for something else.
Coordinates must be non-negative integers within the viewport; anything else returns false rather than throwing. Nothing is scrolled on your behalf, so a point below the fold refers to whatever happens to be there now, not to the content you were thinking of.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const viewport = await $.doGetViewport();
// Move to the centre of the viewport
await $.doHoverAt(viewport.width / 2, viewport.height / 2);
// Coordinates outside the viewport are refused rather than clamped
if (!(await $.doHoverAt(-10, 50))) {
$.log("Negative coordinates are rejected", "warning");
}
return { next: "scroll-a-panel" };
- key: scroll-a-panel
code: |
// Wheel events go wherever the pointer is, so putting it over a
// panel is how you scroll the panel instead of the page
const box = await $.doGetBox("[data-role=long-list]");
if (null === box) {
$.log("No inner list on this page", "warning");
return;
}
await $.doHoverAt(box.left + box.width / 2, box.top + box.height / 2);
await $.doScroll(300);
// The pointer stays put, so it can be picked up again later
const { left, top } = await $.doGetMouse();
$.log(`Pointer resting at ${left} x ${top}`);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doHoverCenter()
Document: Move mouse just slightly outside of viewport center.
@return {boolean} true on success, false on failure
A one-line way to get the pointer somewhere harmless without first asking how big the viewport is. It lands slightly off dead centre, which keeps it clear of whatever a page has decided to put exactly in the middle.
The usual reason to call it is scrolling. $.doScroll sends wheel events at the pointer, so leaving the mouse over a dropdown or an inner panel scrolls that instead of the page - moving to a neutral spot first avoids the whole class of problem.
It's also worth a call before a screenshot or a recording, so a stray hover state doesn't end up in the capture. For anywhere more specific, use $.doHoverAt with coordinates from $.doGetViewport.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Put the pointer somewhere neutral before scrolling, so the wheel
// events reach the page rather than a panel or an open menu
await $.doHoverCenter();
await $.doScroll(600);
return { next: "clean-capture" };
- key: clean-capture
code: |
// Leaving the pointer on a control means its hover state ends up
// in the screenshot - step away first
const buttonKey = await $.doQuery(".MuiButton-root", { contains: "foo" });
if (null !== buttonKey) {
await $.doClick(buttonKey, { hover: false });
}
await $.doHoverCenter();
await $.ioSaveScreenshot("images", { full: true, extension: "png" });
srcFunctions: []
srcInputs: []
srcOutputs:
- key: images
type: files
name: Screenshots
desc: ""
max: 256
extensions:
- png
async $.doJiggle( radius = 50 )
Document: Jiggle the mouse at the current coordinates.
@param {int} radius (optional) Jiggle radius in pixels; [10,500]; default 50
@return {boolean} true on success, false on failure
Real pointer movement around wherever the cursor already sits, which is why it follows a hover. Pages that hold content back until they have seen a mouse move are what this is for.
The radius is clamped to between 10 and 500 pixels.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const { width, height } = await $.doGetViewport();
await $.doHoverAt(width / 2, height / 2);
await $.doJiggle(20);
await $.sleep(500);
await $.doJiggle(300);
$.log("🪄 Somebody gave the cursor coffee", "success");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doScroll( amount, options = {} )
Document: Issue mouse wheel (scroll) events at the current cursor position.
@param {int} amount Scroll amount in pixels
@param {Object} options (optional) Scroll options
@param {int} options.speed (optional) Scroll speed in pixels/second; default 500; between 1 and 5000
@param {boolean} options.vertical (optional) Vertical or Horizontal scroll; default true for vertical
@return {boolean} true on success, false on failure
These are real wheel events issued wherever the pointer currently sits, which is the detail that catches people out: with the mouse over a scrollable panel the panel moves and the page doesn't. Position the pointer first with $.doHover, $.doHoverAt or $.doHoverCenterdepending on what you mean to scroll.
Because the page really receives the events, lazy loading and scroll-triggered animations behave as they would for a person - which is exactly why the loop above works, and why jumping straight to the bottom wouldn't.speed is in pixels per second and controls how long the call takes rather than how far it goes. Leave it high for throughput and drop it for a recording. Watching scrollHeight stop changing is the reliable way to know an endless list has ended - always bound the loop as well.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Positive scrolls down, negative scrolls back up
await $.doScroll(500);
await $.doScroll(-500);
// Slow it right down - worth it when recording with $.ioSaveVideo
await $.doScroll(750, { speed: 150 });
// Horizontal, for carousels and wide tables
const carouselKey = await $.doQuery("[data-role=carousel]");
if (null !== carouselKey) {
await $.doHover(carouselKey);
await $.doScroll(400, { vertical: false });
}
return { next: "infinite-list" };
- key: infinite-list
code: |
// Park the pointer over the page itself so the wheel events
// don't get swallowed by a panel or an open menu
await $.doHoverCenter();
let lastHeight = 0;
for (let i = 0; i < 20; i++) {
const viewport = await $.doGetViewport();
// The page stopped growing, so there is nothing left to load
if (viewport.scrollHeight === lastHeight) {
$.log(`Reached the end after ${i} screens`, "success");
return;
}
lastHeight = viewport.scrollHeight;
await $.doScroll(viewport.height);
await $.sleep(500);
}
$.log("Gave up after 20 screens", "warning");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doScrollTo( element, options = {} )
Document: Scroll to HTML element.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {Object} options (optional) Scroll to options
@param {int} options.top (optional) Top margin in pixels; default 0; target element distance to the top of the viewport in pixels
@param {boolean} options.hover (optional) Hover mouse over center of element after scrolling; default true
@throws {Error} If element not found
Most actions - $.doClick, $.doType, $.doCheck, $.doSelect - already scroll to the element themselves, so you rarely need this beforehand. Reach for it when the default landing position is wrong, or when you need content to enter the viewport so the page will load it.top is the gap left between the element and the top of the viewport. A fixed header that overlaps your target is the usual reason to set it: give it a little more than the header's height and the element lands below it rather than underneath.
The mouse follows to the center of the element unless you pass { hover: false }, which matters when hovering would open something over whatever you need next.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Bring something into view before working with it
const fileInput = await $.doQuery("input[type=file]");
await $.doScrollTo(fileInput);
// Leave headroom so a sticky header doesn't end up covering the target
const buttonsHeading = await $.doQuery("h2", { contains: "buttons" });
await $.doScrollTo(buttonsHeading, { top: 120 });
// Scroll without parking the mouse on the element
await $.doScrollTo("footer", { hover: false });
$.log(`Footer in viewport: ${await $.doGetInViewport("footer")}`);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doType( element, text, options = {} )
Document: Type text to specified HTML element.
Automatically scroll to element before action.
Skips typing if the element is not editable.
To save time on large texts, the first part of the text is pasted,
and only the last 250 characters are typed one character at a time.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {string} text Text to type
@param {Object} options (optional) Typing options
@param {boolean} options.replace (optional) Replace current text; default false to append text
@param {boolean} options.submit (optional) Press the Enter key when finished typing; default false
@param {int} options.speed (optional) Typing speed in characters per second; [1,250]; default 10
@param {int} options.sequence (optional) Type last N characters in sequence; use clipboard for the rest; [1,1000]; default 250
@throws {Error} If element not found
Typing is simulated at the keyboard level, so fields that only react to real key events - autocompletes, validation-as-you-type, character counters - behave the way they would for a person. That fidelity costs time, which is why long strings are pasted up to the last sequence characters.speed and sequence are the two dials worth knowing. Raise speed and lower sequence when you're filling in bulk data and nobody is watching; leave the defaults when the page is picky about how input arrives, or when you're recording a video of the run.
Typing appends unless you pass { replace: true }, and non-editable elements are skipped silently rather than throwing - only a missing element is an error.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const inputKey = await $.doQuery("[name=input-text]");
await $.doHighlight(inputKey);
// Successive calls append by default
await $.doType(inputKey, "abc");
await $.doType(inputKey, "def");
$.log(`Field now holds: ${await $.doGetValue(inputKey)}`);
// Clear the field first
await $.doType(inputKey, "foobar", { replace: true });
return { next: "search" };
- key: search
code: |
// Fill a search box and press Enter in one call
const searchKey = await $.doQuery("[name=input-textfield]");
await $.doType(searchKey, "uindow automation", { replace: true, speed: 40, submit: true });
return { next: "long-text" };
- key: long-text
code: |
// Long text is pasted up front, with only the tail typed character by character.
// Lowering "sequence" pastes more of it and finishes sooner.
const essay = Array.from({ length: 20 }, (_, i) => `${i + 1}. Lorem ipsum dolor sit amet.`).join("\n");
await $.doType(await $.doQuery("[name=textarea]"), essay, {
replace: true,
speed: 250,
sequence: 50
});
$.doTick("success");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doTypeAt( left, top, text, options = {} )
Document: Type text at coordinates in viewport.
To save time on large texts, the first part of the text is pasted,
and only the last 250 characters are typed one character at a time.
@param {int} left Left coordinate in pixels
@param {int} top Top coordinate in pixels
@param {string} text Text to type
@param {Object} options (optional) Typing options
@param {boolean} options.replace (optional) Replace current text; default false to append text
@param {boolean} options.submit (optional) Press the Enter key when finished typing; default false
@param {int} options.speed (optional) Typing speed in characters per second; [1,250]; default 10
@param {int} options.sequence (optional) Type last N characters in sequence; use clipboard for the rest; [1,1000]; default 250
@return {boolean} true on success, false on failure
Same typing behaviour as $.doType - the replace, submit, speed and sequence options all mean the same thing - addressed by position rather than by element.
Prefer $.doType whenever you have something to query: it scrolls the element into view first, and it can't be thrown off by the page moving underneath it. This is for the cases where there is no element to name, which in practice means editors that manage their own caret, canvas-based widgets, and controls buried in shadow DOM.
Nothing is scrolled for you here, so make sure the point is on screen. Invalid coordinates return false rather than throwing, which is worth checking - typing that silently went nowhere is hard to spot later.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const inputKey = await $.doQuery("[name=input-text]");
const box = await $.doGetBox(inputKey);
if (null === box) {
throw new Error("$.doGetBox failed");
}
// Type into the middle of the field
const typed = await $.doTypeAt(
box.left + box.width / 2,
box.top + box.height / 2,
"baz",
{ replace: true }
);
if (!typed) {
$.log("Those coordinates were rejected", "warning");
return;
}
$.log(`Field holds: ${await $.doGetValue(inputKey)}`);
return { next: "editor" };
- key: editor
code: |
// Rich editors and canvas widgets often have no input to query.
// Click to place the caret, then type at the same point.
const editorBox = await $.doGetBox("[data-role=editor]");
if (null === editorBox) {
$.log("No editor on this page", "warning");
return;
}
const left = editorBox.left + 20;
const top = editorBox.top + 20;
await $.doClickAt(left, top);
await $.doTypeAt(left, top, "Hello from Uindow", { speed: 30 });
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doSelect( element, values )
Document: Select zero, one or more options, replacing previous selection.
Automatically scroll to element before action.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {string|string[]} values A single value or an array of values for <select multiple/>
@throws {Error} If select element not found
Every call replaces the current selection rather than adding to it, which is why an empty array clears a multi-select and why a single value is enough for an ordinary dropdown. You never have to deselect anything first.
Matching happens on each option's value, not its visible label - a country list may show "Germany" while its value is DE. When you're working from something a user typed or a table column, read the options with $.doGetOptions first and map the label across.
This is for <select> elements. Radios and checkboxes are handled by $.doCheck, and dropdowns built out of divs and list items need ordinary $.doClick calls instead.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// A single value
const selectKey = await $.doQuery("[data-role=select]");
await $.doSelect(selectKey, "8");
$.log(`Selected: ${await $.doGetValue(selectKey)}`);
// An array for <select multiple/> - this replaces the whole selection,
// so anything not listed ends up deselected
const multiKey = await $.doQuery("[data-role=select-multi]");
await $.doSelect(multiKey, ["3", "15"]);
$.log(await $.doGetValue(multiKey));
// An empty array selects nothing at all
await $.doSelect(multiKey, []);
return { next: "by-label" };
- key: by-label
code: |
// Values are rarely what the user sees. Read the options,
// match the label, then select the value behind it.
const selectKey = await $.doQuery("[data-role=select]");
const options = await $.doGetOptions(selectKey);
const match = options.find((o) => o.text.toLowerCase().includes("8"));
if (!match) {
$.log("No matching option", "warning");
return;
}
await $.doSelect(selectKey, match.value);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doCheck( element, values, options = {} )
Document: Check radio or checkbox values.
The element's siblings must share the same name attribute.
Automatically scroll to element(s) before action.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {string|string[]} values A single value or an array of values
@param {Object} options (optional) Check options
@param {boolean} options.hover (optional) Hover after check; default true; use false to move mouse to the side after clicking
@throws {Error} If checkbox or radio input element not found
$.doCheck addresses a group rather than a single input: it resolves the siblings sharing the same name attribute, so you only ever have to query one member of the group. Pass a single string for radios and an array for checkboxes.
Values are matched against each input's value attribute. They're stringified internally, so 2 and "2" are equivalent. Custom widgets such as MUI switches and checkboxes are driven the same way as plain HTML inputs, since MUI renders a real input underneath. Read the result back with $.doGetValue, which returns a string for a radio group and an array for a checkbox group.
Use { hover: false } when the group sits under a tooltip or a hover menu that would otherwise cover the next element you want to reach.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
// Radios take a single value - any sibling sharing name="r1" can be targeted
const radioKey = await $.doQuery("input[type=radio][name=r1]");
await $.doCheck(radioKey, "2");
const radioValue = await $.doGetValue(radioKey);
$.log(`Radio r1 is now ${radioValue}`, "success");
// Checkboxes take an array of values
const checkboxKey = await $.doQuery("input[type=checkbox][name=c1]");
await $.doCheck(checkboxKey, ["2", "4"], { hover: false });
// $.doGetValue returns an array for a checkbox group
const checkboxValues = await $.doGetValue(checkboxKey);
$.log(`Checkbox c1 is now ${checkboxValues.join(", ")}`, "success");
// MUI switches are backed by a real input, so they behave identically
const switchKey = await $.doQuery("input[type=checkbox][name=s1]");
await $.doCheck(switchKey, ["1", "3"]);
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doChooseFiles( element, filePaths )
Document: Choose files for <input type="file" /> HTML element.
Automatically scroll to element before action.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {string|string[]} filePaths File path(s) generated with $.ioSave* methods or $.ioInputFiles
@throws {Error} If file input element not found, or could not choose files
Files go straight onto the input without the operating system's picker ever appearing, so nothing blocks waiting on a native dialog.
Paths have to come from somewhere Uindow knows about - a files input, or something the module wrote itself with one of the $.ioSave* methods. A single path can be passed on its own or in an array.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
await $.doChooseFiles("input[type=file]", $.ioInputFiles("uploads"));
$.log("📎 Attached - and no file picker ever opened", "success");
srcFunctions: []
srcInputs:
- key: uploads
type: files
name: Files to attach
desc: ""
extensions:
- png
- jpg
- jpeg
- pdf
multiple: true
srcOutputs: []
async $.doAwaitDomReady( options )
Document: Wait for page to load (DOM ready).
@param {Object} options (optional) Query options
@param {int} options.timeout (optional) Timeout in seconds; default 60
@return {boolean} true on success, false on timeout
$.navLoad and $.navReload already wait for DOM ready, so this is for the navigations you did not start yourself - a clicked link, a submitted form, a redirect fired by the page, or a step through history.
DOM ready means parsed, not painted, and a single page app may never fire it again after the first load. For specific content, $.doAwaitPresent is the better tool. A timeout returns false.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
await $.navGoBack();
const ready = await $.doAwaitDomReady({ timeout: 15 });
$.log(ready ? "🏠 Home again, DOM and all" : "⏳ Still loading", ready ? "success" : "warning");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doAwaitPresent( selector, options = {} )
Document: Wait for an element to be present in the DOM.
@param {string} selector CSS selector
@param {Object} options (optional) Query options
@param {string} options.parent (optional) Parent CSS selector OR element key; default null to search the entire Document
@param {string} options.contains (optional) Text contained by element (case insensitive); default null for no restrictions
@param {boolean} options.scrollable (optional) Restrict results to elements that have active scrollbars; default false
@param {int} options.timeout (optional) Timeout in seconds; default 60
@param {boolean} options.all (optional) Return all matches (string[] instead of string); default false
@return {string|string[]|false} Element key if options.all; array of 24 characters long element keys; false on timeout
Defer the thing that triggers the change, then wait for the change itself - far steadier than sleeping and hoping. A timeout returns false, so a plain truthiness check covers it.
Present means in the DOM, not on screen. Follow with $.doAwaitVisible when that difference matters.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const doorbell = await $.doQuery('[data-role="toggle-present"]');
$.setTimeout(async () => await $.doClick(doorbell), 1000);
const guest = await $.doAwaitPresent("#alert-present", { timeout: 10 });
$.log(guest ? "👋 There you are!" : "🚪 Nobody came", guest ? "success" : "warning");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doAwaitNotPresent( element, options = {} )
Document: Wait for an element to be removed from the DOM.
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {Object} options (optional) Query options
@param {int} options.timeout (optional) Timeout in seconds; default 60
@return {boolean} true on success, false on timeout
Passing the element key rather than a selector makes the check specific: a selector is satisfied the moment nothing matches, which on a list of similar rows tells you far less than watching one row disappear.
Removal is not hiding - an element merely given display: none stays in the DOM and this will time out.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const toggle = await $.doQuery('[data-role="toggle-present"]');
await $.doClick(toggle);
const alert = await $.doAwaitPresent("#alert-present", { timeout: 5 });
$.setTimeout(async () => await $.doClick(toggle), 1000);
const gone = await $.doAwaitNotPresent(alert, { timeout: 5 });
$.log(gone ? "💨 Poof - out of the DOM entirely" : "🪨 Still hanging around", gone ? "success" : "warning");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doAwaitVisible( element, options = {} )
Document: Wait for an element to become visible to the user (display, visibility, opacity).
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {Object} options (optional) Query options
@param {int} options.timeout (optional) Timeout in seconds; default 60
@return {boolean} true on success, false on timeout
Visible means genuinely shown - not hidden by display, visibility or opacity. Modals are in the DOM long before they finish appearing, and a click mid-transition often misses.
The element must already exist, so for content not yet rendered use $.doAwaitPresent first.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const ghost = await $.doQuery("#alert-visible");
$.setTimeout(async () => await $.doClick(await $.doQuery('[data-role="toggle-visible"]')), 1000);
const seen = await $.doAwaitVisible(ghost, { timeout: 5 });
$.log(seen ? "👻 It materialised" : "🕳️ Nothing appeared", seen ? "success" : "warning");
srcFunctions: []
srcInputs: []
srcOutputs: []
async $.doAwaitNotVisible( element, options = {} )
Document: Wait for an element to become invisible to the user (display, visibility, opacity).
@param {string} element CSS selector OR element key obtained with $.doQuery*
@param {Object} options (optional) Query options
@param {int} options.timeout (optional) Timeout in seconds; default 60
@return {boolean} true on success, false on timeout
Cookie banners, modal backdrops and loading masks are usually hidden rather than removed, so this is what they need - $.doAwaitNotPresent would sit there until it timed out.
Worth waiting for: an overlay mid-fade still swallows pointer events, so a click sent too early lands on the overlay instead.
srcStateMachine:
- key: start
code: |
await $.navLoad("about:home/test/");
const ghost = await $.doQuery("#alert-visible");
const toggle = await $.doQuery('[data-role="toggle-visible"]');
await $.doClick(toggle);
$.setTimeout(async () => await $.doClick(toggle), 1000);
const gone = await $.doAwaitNotVisible(ghost, { timeout: 5 });
$.log(gone ? "🫥 Faded away" : "😑 Still staring back", gone ? "success" : "warning");
srcFunctions: []
srcInputs: []
srcOutputs: []