Deep Dive into to Cross Browser Testing with Selenium

Briefly defined

Like we stated in earlier posts, cross-browser testing is a type of non-functional testing which consists in making sure a web app or page works as expected at a design, responsiveness, aesthetic and accessibility level, independently of the web browser utilized. Also this definition today extends to the operating system in which the web browser runs.

Ideal tools?

Cross-browser testing, traditionally, was always performed in a manual fashion, without the assistance of any special tool or set of tools. Today that has changed. We can now find a quite fair range of both desktop and web tools oriented, specifically or not to cross-browser testing.

Normally, a reasonably good cross browser testing tool should feature certain performance abilities such as code validation (HTML, CSS, JS), assessing app performance, responsive design testing, checking for UI inconsistencies, etc.

An ideal tool for cross-browser testing should assist us in code validation (HTML, CSS, JS), assessing app performance, responsive design testing, checking for UI inconsistencies, give us the ability to perform multi-device parallel testing among other things.

A long time friend called Selenium

Selenium, as most of us know, is probably one of the most popular frameworks for automated testing. We also know that Selenium software packs a set of very useful tools within it, such as:

• Selenium WebDriver, which is a set of browser-operating bindings for various languages. It offers an interface where many methods can be declared and applied to end-to-end testing.
• Selenium IDE, which is an add-on that makes it easy to record and replay script interactions with browsers.
• Selenium Grid, which is a software program suite designed to run many test cases against a wide range of browsers, operating systems, and hardware simultaneously. This facilitates testing and scales up test suites as they expand.

But I’d like to focus on Grid especially.

What about Selenium Grid then?

As we’ve already said, Grid is the Selenium component which offers the capability to run multiple tests across different browsers –and browser configurations, operating systems, and machines (real or virtual) in parallel, through routing the commands from remote instantiations of browsers, with a server functioning as a hub. A user just has to configure the remote server to be able to execute tests. WebDriver uses W3C protocformulaol since Selenium framework version 4. This results in our cross-browser tests being more stable and less flaky.

The two main components of Selenium Grid are:

• Hub: a server that takes access requests from the WebDriver client, routing the test commands (in JSON format) to the remote drives or nodes. It receives instructions from the client and remotely executes them in parallel on the existing nodes.
• Node: a remote device which implies a native OS and a remote WebDriver. It receives requests from the hub (the JSON formatted test commands) and executes them via WebDriver. A node can carry several kinds of platforms, that is, different OSs and browsers, and it doesn’t require to use the same platform as the hub.

Selenium Grid also has a version of its own. The latest one is, like the Selenium suite indeed, version 4, which implements some important new features in it.

Since Selenium 4 has smoothed out the whole process, with the hub and node becoming part of the same jar file, this facilitates the server (once started) to act both as Hub and Node. Other improvements from Selenium 2 have been (since the Hub consisted of three processes – Router, Session Map, and Distributor) to incorporate a new architecture which supports separate processes (Router, Session Map, Distributor, and Node).

• Router: Listens to the new session request.
• Distributor: Selects the appropriate node where the test should be executed.
• Session Map: Maps the session ID to the node.
• Node: Test machine where the test execution takes place.

Hence, the three Grid modes supported in Selenium 4 are:

• Standalone Mode
• Classical Grid (Hub and Node like earlier versions)
• Fully Distributed (Router, Distributor, Session, and Node)

Grid can run parallel test suites in against multiple machines, resulting in dramatic time-saving. Grid can also run parallel tests against multiple different browsers, and multiple instances of the same browser. As an example, let’s imagine a Grid with six Nodes. The first machine has Firefox’s latest version, the second has Firefox “latest minus one”, the third gets the latest Chrome, and the remaining three machines are Mac Minis, which allows for three tests to run in parallel on the latest version of Safari.

Execution time can be expressed as a simple formula*:

Number of Tests * Average Test Time / Number of Nodes = Total Execution Time
15      *      45s        /        1        =      11m 15s   // Without Grid
15      *      45s        /        5        =      2m 15s    // Grid with 5 Nodes
15      *      45s        /        15       =      45s       // Grid with 15 Nodes
100     *      120s       /        15       =      13m 20s   // Would take over 3 hours without Grid

But how does it work?

Well, it depends on which mode you choose to run.

If we want, for example, to run the Classical mode, we will need to follow these steps:

After we’ve created a hub (java -jar selenium-server-4.0.0-alpha-2.jar hub) –counting we already have Selenium Server installed and running, we can then connect or register nodes into it (java -jar selenium-server-4.0.0-alpha-2.jar node --detect-drivers) if hub and node reside in the same machine, and (java -jar selenium-server-4.0.0-alpha-2.jar node --detect-drivers --publish-events tcp://hub:4442 --subscribe-events tcp://hub:4443) if they respectively reside in different machines.). Since the hub is the master who controls where your tests will run (nodes/slaves), it is also the one responsible for assuring our tests are performed on the right node (e.g., the details concerning machine, operating system and browser/version we specified in the test script).

When the hub is started, XPUB and XSUB sockets open which bind respectively to tcp://*:4442 and tcp://*.4443. Likewise with the node, via the command cited above, it connects to the same address. The –detect-drivers option will automatically detect the Selenium WebDrivers installed in the system.

As soon as the instance of Chrome WebDriver is instantiated, it is registered on the Grid:

DesiredCapabilities caps = new DesiredCapabilities();caps.setBrowserName("chrome");/* 
The execution happens on the Selenium Grid with the address mentioned earlier 
*/driver = new RemoteWebDriver(new URL(Node), caps);

A session ID will be created as specified in the JSON format output which will something like this:

"sessions": 
	[{"sessionId": "[alphanumeric string]", (The created session ID) 
"stereotype": {"browserName": "[browser name]"},
"currentCapabilities": 
	{"acceptInsecureCerts": [boolean value],
    "browserName": "[browser name]",
    "browserVersion": "[browser version]",
    	"chrome": {"chromedriverVersion": "[driver version]",
        "userDataDir": "[user data directory]"},"goog:chromeOptions": {"debuggerAddress": "localhost:[port]"},"networkConnectionEnabled": [boolean value],"pageLoadStrategy": "normal","platformName": "[OS name]","proxy": {},"setWindowRect": [boolean value],"strictFileInteractability": [boolean value],"timeouts": {"implicit": 0,"pageLoad": [milisecs],"script": [milisecs]},"unhandledPromptBehavior": ["accept"|"dismiss"|"accept and notify"|"dismiss and notify"],"webauthn:virtualAuthenticators": [boolean value]}}]

The Fully Distributed mode will require a certain set of steps for each component to run its own process.

1. Start Sessions Map

This will map the session IDs to the node

(java -jar selenium-server-4.0.0-alpha-2.jar sessions)

2. Start Distributor component

Will be listening to requests on port 5556

(java -jar selenium-server-4.0.0-alpha-2.jar distributor --sessions http://localhost:5556)

3. Start Router component

java -jar selenium-server-4.0.0-alpha-2.jar router --sessions http://localhost:5556 --distributor http://localhost:5553

Will get the node URI and forward the request to it. Any new request will go to the Distributor, while it’s listening on `localhost:4444`.
Create a node

4. Create a node

java -jar selenium-server-4.0.0-alpha-2.jar node --detect-drivers

Node details (and its URI) will get updated in the sessions map and once WebDriver is instantiated it will be registered on Grid.

With the Standalone mode, the server will be running the processes as a whole. Grid will automatically detect WebDrivers for both Chrome or Firefox browsers:

java -jar selenium-server-4.0.0-alpha-2.jar standalone

When the test code is executed, the WebDriver is registered in the Grid.

By executing curl http://localhost:4444/status we’ll be able to confirm the creation of a session ID with a JSON format output just like the one shown above from the Classical Grid mode.

The real stuff

Now, everything concerning the latest Selenium Grid version and its features is real cool, but we want to know crucial thing: how does it actually run tests?

We will check an example which shows how to execute tests in parallel**.

Two classes with multiple @Test methods will need to be created, plus a Browser class which will invoke the remote WebDriver with the browser parameters passed in a testng.xml file.

Browser.java classpackage com.test;

import java.net.MalformedURLException;
import java.net.URL;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;

public class Browser {public static RemoteWebDriver getDriver(String browser) throws MalformedURLException {return new RemoteWebDriver(new URL("http://10.0.0.6:4444/wd/hub"), getBrowserCapabilities(browser));}private static DesiredCapabilities getBrowserCapabilities(String browserType) {switch (browserType) {case "firefox":System.out.println("Opening firefox driver");return DesiredCapabilities.firefox();case "chrome":System.out.println("Opening chrome driver");

return DesiredCapabilities.chrome();case "IE":System.out.println("Opening IE driver");

return DesiredCapabilities.internetExplorer();default:System.out.println("browser : " + browserType + " is invalid, Launching Firefox as browser of choice..");return DesiredCapabilities.firefox();}}}

Then via the getDriver method we will call getBrowserCapabilities according to parameters specified in the following classes. If ‘chrome’ parameter is passed it means chromedriver is invoked.

We will create a class called ParallelTestA.java

package com.test;

import java.net.

MalformedURLException;

import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class ParallelTestA {public static RemoteWebDriver driver;
public static String appURL = "http://www.google.com";@BeforeClass@Parameters({ "browser" })public void setUp(String browser) throws MalformedURLException {System.out.println("*******************");driver = Browser.getDriver(browser);driver.manage().window().maximize();}@Testpublic void testGooglePageTitleInFirefox() {driver.navigate().to(appURL);String strPageTitle = driver.getTitle();Assert.assertTrue(strPageTitle.equalsIgnoreCase("Google"), "Page title doesn't match");}@AfterClasspublic void tearDown() {if(driver!=null) {System.out.println("Closing browser");driver.quit();}}}

Then we’ll also create a class ParallelTestB.java

package com.test;

import java.net.

MalformedURLException;

import org.openqa.selenium.By;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class ParallelTestB {public static RemoteWebDriver driver;public static String appURL = "http://www.google.com";@BeforeClass@Parameters({ "browser" })public void setUp(String browser) throws MalformedURLException {System.out.println("*******************");

driver = Browser.getDriver(browser);driver.manage().window().maximize();}@Testpublic void testGooglePageTitleInChrome() {driver.navigate().to("http://www.google.com");

String strPageTitle = driver.getTitle();Assert.assertTrue(strPageTitle.equalsIgnoreCase("Google"), "Page title doesn't match");}@Testpublic void testSearchGoogle() {System.out.println("Opening Google..");driver.navigate().to(appURL);driver.findElement(By.name("q")).sendKeys("Selenium Easy Grid Tutorials");driver.findElement(By.name("btnG")).click();}@AfterClasspublic void tearDown() {if(driver!=null) {System.out.println("Closing browser");driver.quit();}}}

The last thing, in order to run the tests, is to create a testng.xml file setting the parallel=”tests” attribute and the browser parameter for each test.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Main Test Suite" parallel="tests" verbose="1"><test name="Grid firefox Test">
<parameter name="browser" value="firefox"/>
<classes><class name="com.test.ParallelTestA"/></classes>
</test><test name="Grid chrome Test"><parameter name="browser" value="chrome"/><classes><class name="com.test.ParallelTestB"/></classes></test>
</suite>

When the code is executed, the console will show us information from the hub such as how many available nodes there are, or on which node it started executing the tests, etc. As long as the browser parameter is used, any default browser will be ignored and only what’s specified in the parameter will be taken into account:

-browser browserName=[browser name],version=[browser version],maxInstances=[number of browser instances],platform=[OS name]

If we execute the code passing this parameter, the Grid console (localhost:4444/grid/console) will then show us the browser name, browser version, max instances (number of instances of the same browser version able to run over the Remote System) and the platform used to register the node.

Let’s suppose the remote machine has multiple versions of Firefox. We can then map the location of each binary to a particular version on the same machine to execute multiple versions:

-browser browserName=firefox,version=[browser version],firefox_binary=[browser executable path],maxInstances=[x],platform=[OS name] -browser browserName=firefox,version=[browser version],firefox_binary=[browser executable path],maxInstances=[x],platform=[OS name]

Conclusion

It is no wonder that both Selenium WebDriver as well as Grid are invaluable tools when it comes to performing tests which require multiple instances of a same software and running tests in parallel, such as cross-browser tests.

However, the market today with its everchanging rules requires a smarter approach.

What if we could harness the same features without writing a single line of code?

Normally, a reasonably good cross browser testing tool should feature certain performance abilities such as code validation (HTML, CSS, JS), assessing app performance, responsive design testing, checking for UI inconsistencies, etc.

Autify, besides all that, supports desktop and mobile browsers, eliminating the tediousness of having to maintain and manage real devices and device farms. It features multi-device parallel testing as well, which makes Autify an ideal match when it comes to cross browser testing, since it saves hours of valuable time.

All that, packaged in one platform which also offers:

• A no code testing platform, no programming knowledge required.
• A GUI to record test scenarios and play them back.
• Test script maintenance through AI.
• Artificial intelligence algorithms which “learn” of user interface changes, adapting to them and alerting the QA team.

Try Autify for your Cross Browser Tests

You can see our client’s success stories here: https://noode.autify.com/why-autify

• Autify is positioned to become the leader in Automation Testing Tools.
• We got 10M in Series A in Oct 2021, and are growing super fast.

We have different pricing options available in our plans: https://nocode.autify.com/pricing

• Small (Free Trial). Offers 400~ test runs per month, 30 days of monthly test runs on 1 workspace.
• Advance. Offers 1000~ test runs per month, 90~ days of monthly test runs on 1~ workspace.
• Enterprise. Offers Custom test runs per month, custom days of monthly test runs and 2~ workspaces.

All plans invariably offer unlimited testing of apps and number of users.

We sincerely encourage you to request for our Free Trial and our Demo for both Web and Mobile products.

* Extracted from https://www.selenium.dev/documentation/grid/applicability/

** Extracted from https://www.seleniumeasy.com/selenium-tutorials/parallel-execution-in-selenium-grid