testpilot test [path/to/plan.pilot.yaml | path/to/directory/with/pilot/plans] [flags]
Examples
# Run all tests in the current directory
testpilot test
# Run tests from a specific pilot plan file
testpilot test ./my-test.pilot.yaml
# Run tests from a specific directory
testpilot test ./tests/
# Run only tests with names containing "login"
testpilot test --filter login
# Run tests and save reports to a custom directory
testpilot test --outdir ./test-results
# Run tests in shard 1 of 3 total shards
testpilot test --shard 1/3
Options
| Option | Description |
|---|---|
-f, --filter string | only run tests with names containing the given string |
-h, --help | help for test |
--concurrency int | number of tests to run concurrently (default 10) |
--driver string | driver to use for the test (playwright or appium) (default “playwright”) |
| —launch-vars stringToString | Environment variables passed when launching the application under test. Currently only applies to iOS apps and tests (default []) |
--os string | operating system to run tests on (local, linux, macos, windows) (default “local”) |
--os-version string | operating system version to run tests on (e.g., “13”, “18.3”) |
--outdir string | Directory in which to save reports (default “testpilot-out”) |
--runtime string | runtime environment to execute the test in (chromium, firefox, webkit, native) (default “chromium”) |
--shard value | Split tests into n shards and only run tests in shard x |
--udid value | specify the UDID of the iOS device you want to test. Note: this cannot be used along with --os-version |
Advanced Options
The following options are available but not commonly used:| Option | Description |
|---|---|
--show-browser | show the browser |
--viewport ints | screen dimensions to use for your web browser (playwright driver only) |
Environment Variables
The following environment variables are required or affect the behavior of the test command:| Variable | Required | Description |
|---|---|---|
OPENAI_API_KEY | Yes | OpenAI API key for LLM interactions |
ANTHROPIC_API_KEY | No | Anthropic API key (optional) |
OPENAI_BASE_URL | No | Custom OpenAI API base URL |
SEE ALSO
- testpilot - Testpilot CLI