Automated end-to-end acceptance tests for the wp-calypso client and WordPress.com in general.
npm install
npm pack lib/reporter
npm install ./spec-xunit-slack-reporter-0.0.1.tgz
./run.sh -g
./node_modules/mocha/bin/mocha specs/wp-log-in-out-spec.js lib/after.js
Note: you can also change the spec temporarily the use the .only syntax so it is the only spec that runs (making sure this isn't committed)
eg.
test.describe.only( 'Logging In and Out:', function() {
The run.sh script takes the following parameters, which can be combined to execute a variety of suites
-R - Use custom Slack/Spec/XUnit reporter, otherwise just use Spec reporter
-p [jobs] - Execute [num] jobs in parallel (experimental)
-s - Screensizes in a comma-separated list (defaults to mobile,desktop,tablet)
-g - Execute general tests in the specs/ directory
-i - Execute i18n tests in the specs-i18n/ directory
-v [all/critical] - Execute the visdiff tests in specs-visdiff[/critical]. Must specify either 'all' or 'critical'.
-h - This help listing
All tests should be written to work in three modes: desktop (1440 wide), tablet (1024 wide) and mobile (375 wide).
You can run tests in different modes by setting an environment variable BROWSERSIZE to either desktop, tablet or mobile.
Eg:
env BROWSERSIZE=tablet ./node_modules/mocha/bin/mocha --compilers js:babel-register specs lib/after.js
Or you can use the -s option on the run.sh script:
./run.sh -g -s mobile
./run.sh -g -s desktop,tablet
The tests use the node config library to specify config values for the tests.
Under the config directory, there are files for each environment: default.json is the base for all environments, then development.json for local, and test.json for CI.
You can also use local config files that are not committed.
The config files should be added under the config/ tree and should follow the naming scheme: local-<env>.json
The properties in the local configuration override the default.json properties. This is useful for local testing of different configurations on local, e.g. testing on local Calypso instance, instead of live one.
If the configuration doesn't exist, the code falls back to using the environmental variables.
Note: NODE_ENV is still required, as it is used to determine what <env> config to load.
For example: export NODE_ENV='personal'
An example configuration file is provided.
The local configurations are excluded from the repository, in order to prevent accidental commit of sensitive data.
Please don't commit usernames and passwords in these files!
Note: As the NODE_CONFIG variable is growing unwieldy for use in CircleCI, some environment variables are also being split out into separate entries. These are listed below the Config Values section.
A full list of config values are:
| Name | Description | Example | Required | Store in file? |
|---|---|---|---|---|
| browser | The browser to use: either firefox or chrome |
chrome |
Yes | Yes |
| proxy | The type of proxy to use: either system to use whatever your system is configured to use, or direct to use no proxy. |
direct |
Yes | Yes |
| saveAllScreenshots | Whether screenshots should be saved for all steps, including those that pass | false |
Yes | Yes |
| neverSaveScreenshots | Overrides the screenshot function so nothing is captured. This is intended for use with the Applitools visual diff specs, since the screenshots are handled via their utilities instead. | false |
Yes | Yes |
| authURL | The page where you authenticate to use calypso - this should not include the redirect | https://wordpress.com/wp-login.php | Yes | Yes |
| calypsoBaseURL | The home page for calypso | https://wpcalypso.wordpress.com | Yes | Yes |
| explicitWaitMS | The explicit wait time in milliseconds to wait for an element to appear - for example a widget loading data via an API | 10000 | Yes | Yes |
| mochaTimeoutMS | This is the maximum total time in milliseconds a single mocha end to end test can take to finish - otherwise it will time out. | 120000 | Yes | Yes |
| startBrowserTimeoutMS | This is the maximum total time in milliseconds that the browser can take to start - this is different from test time as we want it to fail fast | 10000 | Yes | Yes |
| afterHookTimeoutMS | This is the maximum total time in milliseconds that an after test hook can take including capturing the screenshots | 20000 | Yes | Yes |
| knownABTestKeys | An array of expected, known AB testing keys used in localstorage. If a key is found on any page that isn't in here, the test will fail | [ "freeTrials_20160112", "plansPageBusinessAATest_20160108" ] | Yes | Yes |
| testUserName | This is an existing test WordPress.com account for testing purposes - this account should have a single site | testuser123 | Yes | NO |
| testPassword | This is the password for the test WordPress.com account | testpassword$$$%### | Yes | NO |
| testUserNameMultiSite | This is an existing test WordPress.com account for testing purposes that has multiple sites | testuser123 | Yes | NO |
| testPasswordMultiSite | This is the password for the test WordPress.com account that has multiple sites | testpassword$$$%### | Yes | NO |
| testUserNameJetpack | This is an existing test WordPress.com account for testing purposes that has a single jetpack site | testuser123 | Yes | NO |
| testPasswordJetpack | This is the password for the test WordPress.com account that has a single jetpack site | testpassword$$$%### | Yes | NO |
| jetpackSiteId | This is the test jetpack site id | 12344 | Yes | NO |
| testSiteForInvites | This is wordpress.com site that is used for testing invitations | e2eflowtesting.wordpress.com | Yes | NO |
| privateSiteForInvites | This is a wordpress.com private site that is used for testing invitations | e2eflowtestingprivate.wordpress.com | Yes | NO |
| mailosaurAPIKey | This is the API key from mailosaur used for testing email | hsjdhsjdh7878sd | Yes | NO |
| inviteInboxId | This is an inbox id from mailosaur used for invite email testing | sad34id44ss | Yes | NO |
| signupInboxId | This is an inbox id from mailosaur used for signup email testing | sad34id44ss | Yes | NO |
| domainsInboxId | This is an inbox id from mailosaur used for domains email testing | sad34id44ss | Yes | NO |
| publicizeTwitterAccount | This is the name of the test twitter account connected to your test username | @endtoendtesting | Yes | NO |
| passwordForNewTestSignUps | This is the password that will be set for new sign ups created for testing purposes | alongcomplexpassword%### | Yes | NO |
| storeSandboxCookieValue | This is a secret cookie value used for testing payments | No | NO | |
| slackHook | This is a Slack incoming webhook where notifications are sent for new accounts that are created (https://my.slack.com/services/new/incoming-webhook -- requires Slack login) | https://hooks.slack.com/services/XXXXXXXX/XXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXX | No | NO |
| emailPrefix | A string to stick on the beginning of the e-mail addresses used for invites and signups | username | No | NO |
| testAccounts | A JSON object with username/password pairs assigned to keynames for easy retrieval. The necessary keys are shown in the example to the right. | {"defaultUser": ["username1","password1"], "multiSiteUser": ["username2","password2"], "jetpackUser": ["username3","password3"], "visualUser": ["username4", "password4"]}, "privateSiteUser": ["username5", "password5"] | No | NO |
| Name | Description | Example | Required | Store in file? |
|---|---|---|---|---|
| slackToken | This is a token for uploading files/screenshots to Slack (https://api.slack.com/tokens -- requires Slack login) | xxxx-##########-##########-###########-XXXXXXXXXX | No | NO |