Jenkins Test Automation: its CI/CD Cycle and Why Integration with Auify is a Smart Choice

What is Jenkins?

Jenkins is an open-source server written in Java, which is designed for the automation of certain aspects of the software development process like CI/CD. Thanks to the great variety of plugins available, it is notably extensible. As it continuously automates integration and testing of software builds, it alerts developers about any eventual errors at early stages, and it can also print test results as a post action. It supports multiple build systems, testing frameworks, script types, version control tools and lets the user create multiple test scenarios to widen test coverage. Being a Java application, it is portable, hence easy to install on almost any operating system. It is easy to configure and to distribute machine/platform-wise. Most importantly, it provides a standardized environment for developers to build and test code changes.

Jenkins uses a Master-Agent architecture. What does this mean? A Master-Agent logic, much like the master-slave logic which is used in parallel testing, is a form of distributed architecture that consists in nodes, which Jenkins uses to manage builds. Both Master and Agent communicate with each other through TCP/IP.

Master –or Master node, as the role denotes, is the main Jenkins server, who’s in charge scheduling build jobs, selecting the appropriate agent in the master-agent ecosystem for dispatching the builds, monitoring agents and putting them online/offline as required, showing build results (and reports) to the development team.

The Jenkins Agent is a remote machine connected to the Master. Option for ‘n’ number of agent nodes can be added to the project, depending on its requirements. These agents can be run on different OSs and the Master will choose the appropriate one to build and test based on the type of build request.

The Jenkins CI/CD Cycle

1. Developers check-in code changes in the remote repository.
2. Only Master is connected to the repository, and periodically checks for changes. Agents are connected to the Master.
3. Master sends the request (for build and test) to the chosen Agent. This lets you perform builds and execute tests in different environments across the entire architecture.
4. Agent executes the tests, generates test reports, and sends them to the Master for monitoring.

As developers continue to push code, Agents can run different builds/versions for different platforms. Master controls the operation of each build.

Testing with Jenkings

Jenkins features:

• Plugins which can integrate with tests developed in many popular automated testing platforms.
• Test suites can be scheduled in a job and executed automatically.
• Test results can be obtained in HTML format thanks to Jenkins plugins. Besides, the results can be represented in the form of graphs/tables, highlighting the failures, duration of tests, total number of tests, and so on.
• Projection of result trends based on past executed tests.
• Smooth CI processing, resulting in time efficiency.

Jenkins is bundled with a test harness, which, being based on the JUnit framework, simplifies test development. Some of the benefits of the test harness:

• Automated setup and teardown of a Jenkins installation, allowing each test method to run in a clean, isolated environment.
• Helper classes and methods to simplify the creation of jobs, agents, and more.
• Annotations for specifying test methods.
• Direct access to the Jenkins object model, allowing tests to assert directly against the internal state of Jenkins.
• HtmlUnit support, simplifying test interaction with the web UI and other HTTP calls.

Tests can be run via command line scripts prompt or from any Java IDE.

Jenkins provides a way of creating pipelines via its Pipeline functionality. Jenkins Pipeline is a suite of plugins that supports implementing and integrating continuous delivery pipelines into Jenkins. Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines “as code” via the Pipeline domain-specific language. The definition of a Jenkins Pipeline is written into a text file –a Jenkinsfile– which can be committed to a project’s source control repository, making the CD pipeline as part of the application to be versioned and reviewed like ordinary code.

Integrating Jenkins with Autify

Autify provides an API.
 By using the API, you can integrate with CI/CD tools such as CircleCI and Jenkins, and it can be included in the development flow, such as executing a specific test plan automatically when the source code is merged.

One of the best ways to use Autify is to run tests after adding a feature or before a stable release, to ensure there are no problems with existing features. You can automate this process by running tests with Autify in the Jenkins pipeline. This will require a set of previous conditions, like a workflow and a test plan specifying the test scenario to execute.

Prerequisites [1]

In this article, we will assume that the following two points already exist:

• A CI/CD workflow created using tools such as CircleCI, Jenkins, and GitHub Actions.
• We also assume that URLs of generated verification environments can be identified in the workflow.
• A [Test Plan] that specifies which [Test Scenario] you want to execute automatically at a specific timing.

As of October 2021, Autify has the following four APIs. In this article, we will use the APIs in bold.

• schedules‍
  • Executes a [Test Plan]

• scenarios
  • ‍Gets a [Test Scenario]

• results
  • Gets test results
• url_replacements
  • Actions to replace a URL

Add a job that calls an Autify Web API to an existing workflow

Add a job for the following process:

3-1. Add a [URL Replacement] for the target [Test Plan].

3-2. Execute the target [Test Plan]

3-3. Delete the [URL Replacement] you added in 3-1.

Let’s take a closer look.

Add a [URL Replacement] for the target [Test Plan]

Add a [URL Replacement] to the target [Test Plan] using a [url_replacements API].

For example, suppose the [Test Plan] runs a [Test Scenario] in which user interaction with https://example.com is recorded. Also assume that, when the workflow is executed, a verification environment with the URL https://1234567.example.com is generated. In this case, you can add a [URL Replacement] for the [Test Plan] by issuing the following API request:

curl -X POST -H "Authorization: Bearer PERSONAL_ACCESS_TOKEN" https://app.autify.com/api/v1/test_plans/{test_plan_id}/url_replacements -H "Content-Type: application/json" -d '{"url_replacement": { "pattern_url":"https://example.com", "replacement_url": "https://1234567.example.com"}}'

Replace PERSONAL_ACCESS_TOKEN with the token you generated in 1. and

{test_plan_id}

with the ID of the target [Test Plan] (for example, when you open the [Test Plan] on a web console and the URL is https://app.autify.com/projects/1/test_plans/3, the Test Plan ID is 3).

Issuing an API request is the same as adding the following configuration in the web console:

For more information about this feature, please see this document.

By taking these steps, even with a [Test Scenario] in which user interaction with https://example.com is recorded, the test will be executed after the URL is automatically replaced to https://1234567.example.com.

If the above API request is successful, you will get the following response:

{"id":1111,"test_plan_id":12345,"pattern_url":"https://example.com","replacement_url":"https://1234567.example.com","created_at":"2021-09-15T01:23:45.678Z","updated_at":"2021-09-15T01:23:45.678Z"}

Note that 1111 is a [URL Replacement] identifier. We will use this ID later.

Execute the target [Test Plan]

Use [schedules API] to run the test. For example, you can run the target [Test Plan] by issuing the following API request:

curl -X POST -H "Authorization: Bearer PERSONAL_ACCESS_TOKEN" https://app.autify.com/api/v1/schedules/{test_plan_id}

Replace PERSONAL_ACCESS_TOKEN and

{test_plan_id}

in the same way as 2-1.

If the request is sent to the [schedules API] properly, it will return the following response:

{"data":{"id":"999999","type":"test_plan_result","attributes":{"id":999999}}}

id is a test result identifier (in this example, the ID is 999999). To proceed to the next step of the workflow, we need to get the result of the initiated [Test Plan]. Use the ID and send a request to the [results API]. For example,

curl -X GET -H "Authorization:Bearer PERSONAL_ACCESS_TOKEN" https://app.autify.com/api/v1/projects/{project_id}/results/{result_id}

Replace PERSONAL_ACCESS_TOKEN… You know what to do!

Replace

{project_id}

with the Autify Project ID (for example, if the URL when you log in to Autify is https://app.autify.com/projects/1, the Project ID is 1. For

{result_id}

, specify the response content you got from sending a request to [schedules API] (in the above example, specify 999999).

It will take some time for Autify to finish running tests. How long it will take depends on the test, so in practice, you will need to regularly run the [results API] and check whether the test is complete.

Delete the [URL Replacement] you added in 3-1.

After the test is finished, make sure to delete the [URL Replacement], which was dynamically added by the job flow (if you don’t, the test will be rerun on https://1234567.example.com the next time you run this [Test Plan]).

You can delete the [URL Replacement] added in 2-1 by sending the following API request:

curl -X DELETE -H "Authorization:Bearer PERSONAL_ACCESS_TOKEN" "https://app.autify.com/api/v1/test_plans/{test_plan_id}/url_replacements/{url_replacement_id}"

I’m sure you are already familiar with how to replace the character strings in the above command, but

{url_replacement_id}

is the [URL Replacement] identifier you got in 2-1. In this example, it is replaced with 1111.

If this API request is sent successfully, the [URL Replacement] has successfully been deleted. Now everything is back to how it was before we started.

There you have it! Now you know how to automate testing on verification environments that are dynamically generated. I hope this helped make the most of Autify!

More on examples on how to integrate Autify tests with CI/CD tools like Jenkins, here.

A brief search online will show us a lot of options in this new world of no code platforms. But, when it comes to pricing, most of them aren’t exactly what we would deem transparent. Besides, real, comprehensive tech customer support is a value that differentiates a good tool/service from an average one.

We invite you to check these amazing customer stories:

• A reduction of 1-2 working days per month. Accelerate development cycle while maintaining quality with test automation.

You can see other of our client’s success stories here: https://nocode.autify.com/why-autify