CircleCI E2E Testing: a Case for Autify Integration

What is CircleCI?

CircleCI is a CI/CD platform designed to assist in DevOps. By monitoring repositories on supported version control systems like GitHub and Bitbucket, it can launch builds each time new code is committed. Automated tests are triggered in a container –like Docker– or a virtual machine, each job running in separate containers or VMs. Via Slack or IRC notifications, it can then alert the team on pass or fail results on test completion. Builds can be optimized for faster machines, resulting in faster job runs, even in complex pipelines, thanks to advanced caching, docker layer caching, and resource classes.

It supports deployment to multiple environments like AWS, MS Azure and Heroku.

Go, Java, Ruby, Python, JavaScript, PHP, and any other language that runs on Linux or macOS is supported.

SSH support for locally running jobs, parallelism, pre-built Docker images, self-hosted runners, test insights –and more, are some of the other powerful features CircleCI presents.

Integrating E2E Tests to the CI/CD Pipeline with CircleCI

CircleCI provides a configuration file called config.yml. It’s basically divided into two parts: workflows and jobs.

Workflows help establish rules for defining the jobs to be run and their order with their respective prerequisite combination, which may generally look something like this:

• Check out code: by pulling it on the branch.
• Installation –get values from integrated API if required.
• Run tests –provided all eventual prerequisites.
• Deploy –only on test success, when code is checked in on the master branch.

Jobs, the main blocks of configuration, are sets of steps, which run commands/scripts as required. Each job declares an executor which can be docker, machine, windows, or macos.

As well, test jobs have a specific, basic set of steps, which usually are:

• Restore cache: restores a previously saved cache –modifications performed in previous jobs– based on a key.
• Run steps –which may imply initialization of certain services (server, webdriver, etc) as well as coordination between steps and any background process (pauses) so as not to interfere with the correct run order.
• Notify –via Slack or IRC.

Working locally with Circle CI Runners [1]

CircleCI runner enables you to use your own infrastructure for running jobs. This means you will be able to build and test on a wider variety of architectures, as well as have additional control over the environment.[2] It is a process installed on your infrastructure that communicates with CircleCI. Runners function as their own resource class. Your config.yml still defines the pipeline, jobs, and workflows, but instead of using CircleCI’s cloud-based resources like containers and virtual machines, you execute it on your own.

First, install the runner agent locally. This is a daemon process that runs in the background, checking for prompts from CircleCI. When CircleCI signals that a job must be triggered on the runner, the agent executes the workload. It also has access to the same credentials as the cloud version of CircleCI, so it runs seamlessly and can fetch the code from your repositories.

Once the job is complete, the results and any artifacts are sent back to the cloud service. Only the actual job execution is done on your infrastructure.

Implementation

Implementation consists of these processes:

• Setting up the runner agent locally
• Invoking the runner from your CircleCI config.yml

Setting up the runner agent

Installing the runner agent depends on which platform you are using. Refer to the setup instructions in the CircleCI docs.

For my sample application, I installed the runner agent on a Mac computer.

Note: These steps are just a brief overview and not a complete guide.

The main steps to set up the runner are:

1. Register a new runner in your organization
2. Install the runner agent, following the instructions for your platform
3. Start the runner daemon, and make sure it runs on system startup

Use the CircleCI CLI tool to register a new runner in your organization. If it is your first runner, you may need to create a namespace in that organization. You will be referring to that specific runner from the CircleCI config using that fully qualified name. In my case, that was zan_demos_namespace/mac_runner; mac_runner is the name of the runner, and zan_demos_namespace is the namespace I used.

Here is a sample command:

$ circleci runner resource-class create zmarkan-demos-namespace/local-mac-runner "Local Mac OS Runner" --generate-token

This command will also set up a new API token for this runner to use with CircleCI API in the organization. That is made possible because of the –generate-token flag. Be sure to store that token for later.

After registering the runner with CircleCI, install the runner agent following the instructions in the docs. The setup requires software like curl, sha256sum, tar, and gzip preinstalled. There may be other software required.

Set up some daemon scripts, and pass in these values:

• The API token created in the previous step
• The runner namespace
• The name you specified

The particulars of these items vary based on the platform you are using. For example, on Mac you need to create a launchd.plist and start it with launchctl. On Linux it will be a new service that starts with systemctl.

Invoking the runner from a pipeline

Once the runner agent is in motion, you can invoke it from a job in the .circleci/config.yml file. Make sure to specify the machine executor type and the resource class with your runner’s namespace and runner name. Here is an example:

runner-test:machine: trueresource_class: zmarkan-demos-namespace/local-mac-runnerenvironment:JAVA_HOME: /Users/zmarkan/libs/jdk-11.0.2.jdk/Contents/HomeANDROID_SDK_ROOT: /Users/zmarkan/Library/Android/sdksteps:- checkout- run:name: Run connected check on local runnercommand: |./gradlew connectedDebugAndroidTest

‍

In my example, I had some Android devices connected to my Mac, so the runner could access that local hardware to run tests on them.

I also needed to pass in some specific values that come out of the box when using CircleCI’s own infrastructure, like JAVA_HOME and ANDROID_SDK_ROOT. This is because they were not readily accessible in the specific shell that CircleCI runner uses. On most modern Macs the shell used is Zsh; CircleCI runs Bash by default.

The rest of the steps are straightforward and will depend on your application and what you want to execute. In my case that was checking out the code from the repo and running the tests. You could also use store_artifacts and store_test_results to process the tests and save outputs. The outputs would then be available in your CircleCI cloud dashboard.

Autify integration with CircleCI

To integrate Autify E2E tests on our CircleCI pipeline we must do some modifications to the config.yml file and make use of orbs. Orbs are reusable packages of YAML configurations with repeated pieces of configuration condensed into a single line of code. They help automate repeated processes, speed up project setup, and make it easy to integrate with third-party tools.

The modifications required for the config.yml should look like this:

1. Use CircleCI version 2.1 at the top of your .circleci/config.yml file.

version: 2.1

If you do not already have Pipelines enabled, you’ll need to go to Project Settings -> Advanced Settings and turn it on.

1. Add the orbs stanza below your version, invoking the orb which runs an Autify E2E test plan:

orbs:
mobile: autify/mobile@1.0.0

1. Use mobile elements in your existing workflows and jobs.
2. Opt-in to use of third-party orbs on your organization’s Security settings page

Usage examples:

version: '2.1'orbs:autify-for-mobile: autify/mobile@1.2.3workflows:use-my-orb:jobs:- autify-for-mobile/run_test_plan:build_id: 4d5E6f7gtest_plan_id: 1a2B3c

‍

Commands

run_test_plan

‍

Runs E2E test plan on Autify For Mobile –below, a description of parameters.

PARAMETERDESCRIPTIONREQUIREDDEFAULTTYPEbuild_idBuild ID that you want to run test againstYes–stringresponse_file_pathFile path that keeps the response from the APINoresponse-mobile-run-test-plan.txtstringtest_plan_api_base_url–No‘https://mobile-app.autify.com/api/v1/test_plans/’stringtest_plan_idTest Plan ID that you want to runYes–string

‍

How to Publish

• Create and push a branch with your new features.
• When ready to publish a new production version, create a Pull Request from feature branch to master.
• The title of the pull request must contain a special semver tag: [semver:<segment>] where <segment> is replaced by one of the following values.

IncrementDescriptionmajorIssue a 1.0.0 incremented releaseminorIssue a x.1.0 incremented releasepatchIssue a x.x.1 incremented releaseskipDo not issue a release

‍

Example: [semver:major]

• Squash and merge. Ensure the semver tag is preserved and entered as a part of the commit message.
• On merge, after manual approval, the orb will automatically be published to the Orb Registry. *

We only know as far as our goals and needs are defined. Many tools in the market offer a wide range of utilities and some of them are in fact really good, but in an ever evolving world in which technology and demand for it grows at quantum leap pace, certain features become a must-have: no-code or low-code technology, being AI powered, a SaaS delivery model, to just name a few.

However, pricing is not always as honest as we would want. You just need to check a few lists online to ascertain that. Avoiding companies that do not have any pricing available and going with companies that are not simply tools, but have real humans and outstanding customer success behind them to support your QA team is what we consider the way to go.

AI-powered test automation software like Autify learns of software user interface changes and notes them for QA testers. Instead of the entire test failing for a small UI change, it can point out the change to testers for faster testing cycles. Testers can also record a test case and re-run without failures even if the UI changes.

Moreover, Autify’s tech support makes for an absolutely ideal overall customer experience.

You can see our client’s success stories here: https://autify.com/why-autify

• Autify is positioned to become the leader in Automation Testing Tools.
• We got 10M in Series A in Oct 2021, and are growing super fast.

We have different pricing options available in our plans: https://autify.com/pricing

• Small (Free Trial). Offers 400~ test runs per month, 30 days of monthly test runs on 1 workspace.
• Advance. Offers 1000~ test runs per month, 90~ days of monthly test runs on 1~ workspace.
• Enterprise. Offers Custom test runs per month, custom days of monthly test runs and 2~ workspaces.

All plans invariably offer unlimited testing of apps and number of users.

Give our Free Trial a try!