Testing for a team

Set up a tester's Mac and iPhone from scratch and run the team's flows without the app's source code.

This page takes a tester from an empty Mac to a finished run of the team's flows. The tester needs no source code: the team hands over a folder of flows and a QA build of the app. Do the setup once per Mac and once per iPhone.

Who	Provides
The tester	A Mac, an iPhone, a free Apple ID
The app team	The flows folder, a QA build of the app that includes the tester's iPhone, and any secret values the flows need
The owner of the Apple Developer account	Registers the tester's iPhone so QA builds can be installed on it

A simulator needs only the first two parts of the Mac setup. The steps for a real iPhone follow.

1. Set up the Mac

1. Install Astra Scout from the DMG. See Installation. You install it once: later versions arrive through the app's own update window.
2. Install Xcode from the App Store, open it once and let it install its components.
3. In Xcode → Settings → Accounts, add your Apple ID. A free Apple ID is enough. Select the team, then Manage Certificates… → + → Apple Development.
4. In Astra Scout, open Settings → iOS (section Signing Team) and choose that team from the Team list.

Astra Scout uses this team only to sign WebDriverAgent, the helper app that drives the iPhone. A free Apple ID's signature lasts 7 days; when it expires, connect the iPhone again and Astra Scout signs it again. The QA build of the app is signed by the app team, not by you. See Devices for details.

2. Set up the iPhone

1. Connect the iPhone with a cable, unlock it and tap Trust when it asks "Trust This Computer?".
2. In Xcode, open Window → Devices and Simulators, select the iPhone and wait until it is ready.
3. On the iPhone, turn on Settings → Privacy & Security → Developer Mode. It restarts; unlock it and confirm.
4. Set Settings → Display & Brightness → Auto-Lock → Never while testing.
5. Copy the Identifier (the UDID) shown for the iPhone in Devices and Simulators and send it to the owner of the Apple Developer account. A QA build installs only on iPhones listed in its provisioning profile, so the app team needs a build made after your iPhone was added.

3. Get the flows and the QA build

Ask the app team for:

• The flows folder. A folder with the .yaml flows, the subflows/ folder and the tracking-specs/ catalog. Keep the whole folder together: flows call subflows by relative path.
• The QA build for iPhone: an .ipa signed for development or ad hoc testing that includes your iPhone. A TestFlight or App Store build cannot be used for clearState.
• Secret values the flows use, such as a sandbox account password. Enter them as variables, never in the flow files. See Environments and variables.

4. Open the workspace and choose the app file

1. In Astra Scout, choose File → Open Workspace… and pick the flows folder. See Run flows.
2. Open the toolbar's app menu → App for Real iPhone, enter the app's App ID (bundle id), then Choose file… and pick the QA .ipa. Settings says whether the file can be used on the connected iPhone and why not.

Flows that start with clearState uninstall the app and install this file again, so every run starts like a fresh install. See clearState on a real iPhone.

5. Connect and run

1. Choose the iPhone in the device picker. The first time, Astra Scout builds and installs WebDriverAgent, which takes a few minutes.
2. If the iPhone shows "Untrusted Developer", go to Settings → General → VPN & Device Management, choose the Apple Development: … certificate, tap Trust, then connect again.
3. Open a flow and click Run Test (Cmd+R), or Run All Tests (Shift+Cmd+R).

6. Read the results

• The Test Runs navigator shows each run with its verdict: Passed, Failed, Incomplete, Unsure or Blocked. Blocked means a problem with the device or the setup, not a bug in the app. See Verdicts.
• Show Recording plays the video of the run; screenshots taken by the flow are kept with the run.
• If the flow checks tracking, the runner's Tracking tab shows which events arrived and which are missing or wrong. See Tracking.

When you report a problem, attach the run's report, the video and ~/Library/Logs/Astra Scout/server.log.

Troubleshooting

Problem	What to do
No events arrive from the iPhone	Check that the QA build is the one chosen in Settings, that Receive events from devices on the network is on in Settings → Amplitude, and that the Mac and the iPhone are on the same Wi-Fi. The first time the app opens, iOS asks for Local Network access: allow it.
"App file required" (APP_FILE_REQUIRED)	No usable app file for this iPhone. Choose the QA .ipa in Settings; if Settings says the iPhone is not in its profile, ask the app team for a newer build.
WebDriverAgent stops opening after a week	The free Apple ID's signature expired. Connect the iPhone again.
Taps on the mirror do nothing	Choose the iPhone again in the device picker (or Reconnect).
"iPhone screen is locked" or DEVICE_LOCKED	Unlock the iPhone and set Auto-Lock to Never. See When the iPhone locks itself.

More problems are listed in Troubleshooting.

For the app team

To let any tester run the flows without the source code:

• Read the address Astra Scout passes. On iOS, launchApp passes the launch argument scoutServerUrl. A QA build that uses it as the analytics serverUrl sends its events to the Mac that runs the flow, so one build works for every tester. See One QA build for every Mac.
• Build the QA configuration, with analytics events sent immediately and an App Transport Security exception for plain HTTP to the local network.
• Export for development or ad hoc testing, not for the App Store, so testers can install the .ipa and clearState can reinstall it.
• Register each tester's iPhone (UDID) in the Apple Developer account before building. A development or ad hoc profile holds up to 100 iPhones per year and expires after one year.
• Share the flows folder (for example as a Git repository the testers can pull) and tell testers which variables to set.