Mobile Testing Can Be Done In-Browser with No Code: Planning for The Future with Test Automation

As people’s demands diversify, we must create an environment where people can access the right information when they need it. mediba, Inc. develops services in a wide range of fields, including in-house services and services for the au brand, including KDDI’s au Smart Pass.

To keep up with the ever-evolving mobile industry, the Quality Control UNIT is responsible for quality assurance. The team ensures that mediba can compete in a fast-paced environment where new user experiences are continually emerging. We interviewed Ms. Izumi Narita, Ms. Rei Kazami, and Mr. Tomoki Sawada from the Quality Control UNIT of the Technology Center at mediba about the challenges they faced in developing mobile services during the coronavirus pandemic and the benefits of test automation.

Testing and development go hand in hand. Automation is a must in agile development.

– Please tell us about your company’s business and your responsibilities.

Izumi: Our company develops a wide range of services, from au’s services such as KDDI’s content platform, au Smart Pass, to our own in-house services. Some services have over 10 million users.

With the growth of the company, we have entered a phase where we prioritize quality control. After starting as a project manager, I created Quality Control UNIT about five years ago, and now I am responsible for quality control.
We develop by customizing a base structure, so it’s not entirely original, so the service is more like an app builder platform.

Rei: I’ve worked with testing for about 10 years since my last job. I joined mediba in 2017 and currently work as a part of the Quality Control UNIT. I started implementing Autify in 2019.

Tomoki: I got involved with quality control through au Web Portal.

– You mentioned there are many different services; how many team members does the Quality Control UNIT have?

Izumi: If you include subcontractors, there are usually over 30 members at any given time. There were only two of us when Quality Control UNIT was launched, but the number of media outlets and team members has grown since then.

– Is that when you started considering test automation?

Izumi: Yes. As more and more partner companies got involved with development, there was an increase in the number of technologies used. Keeping quality control consistent became increasingly challenging. Rather than testing after development, testing and development were happening simultaneously for an increasing number of products and services. To speed up the agile development cycle, we decided that test automation was inevitable.

Also, since our services are mainly used on smartphones, we needed to test on real mobile devices. Relying on engineers to write test code would be inefficient. To solve these issues, we started to consider various test automation tools. Ms. Kazami found several potential tools, and one of them happened to be Autify.

– You were looking for a tool that could run tests regardless of the development environment?

Izumi: Yes, that’s right. Quality Control UNIT generally handles testing across the board. Rather than having tests for each product, we needed to make rules that anyone could easily follow, even when we have new people or if the company culture changes. We needed a comprehensive test automation tool that could handle all of our products and services.

Autify runs itself and is easy to maintain

– Did you consider various test automation tools?

Rei: Yes, I tried using Appium to see if it could be a potential solution, but there were several issues. Writing test code may be easy at first, but it gradually becomes difficult, and it’s not realistic in the long term. For example, we needed it to assign Unique IDs to developers, but it was challenging to make sure it happened every single time. Also, no one else within the company had written tests with Appium before, so we became discouraged. I wanted to try integrating it with AWS Device Farm, but since Autify can handle cross-browser testing, I thought we might as well go with Autify.

– I remember introducing you to Autify back in 2019, and you had decided to implement it. What was the deciding factor?

Rei: The key was that tests automated with Autify aren’t dependent on humans. We knew from experience that we will be automating tests over the medium to long term. Autify makes it easy to create Test Scenarios with no code, and it’s easy to maintain. It’s even compatible with smartphone browsers. The deciding factor was that Autify could handle anything related to E2E testing seamlessly, and it would run itself without depending on human intervention.

– One of Autify’s strengths is that it can handle smartphones, so that’s great to hear. How exactly did you go about implementing Autify?

Rei: The first step was to use it myself. When I was responsible for a testing project, I made an automated test and ran both manual and automated tests. This is so that I could determine how reliable the automated tests were. I’ve heard that automated tests aren’t reliable, but reliability was not a problem with Autify.

Next, I talked to other engineers about Autify and had them try it out. The more I talked about Autify, I got a surprisingly large amount of interest. We held study sessions internally to increase awareness of Autify.

– Getting people involved is key! You picked one project, made Test Scenarios, and became familiar with it. Then, you gradually expanded it to other projects.

Start small and automate what you want to

– Do you have any tips and tricks for automating tests with Autify?

Tomoki: I think the key is to not be overzealous. That’s something I could have done better; I tried to organize areas to automate by making a whole list of things to automate. But I was spending way too much time on organizing instead of automating!

It would have been a lot more productive to trust my gut and automate areas where I wanted to automate.

– Were you able to use Autify intuitively?

Tomoki: I think people learn how to use tools at different speeds, but in our case, we were able to master it early on. We wanted to make sure the Test Scenarios’ quality was up to par, so we discussed what kind of testing and recording we should do for each feature.

– Did you encounter any challenges or roadblocks as you continued your automation efforts?

Tomoki: Finding workarounds for areas where recording doesn’t work can be challenging. Here’s an example; there was a test where you click a link, go to a new page, and then go back to the original page. However, the test didn’t work correctly, and scrolling didn’t always behave correctly. When we encountered those issues, we would discuss and share knowledge.

– What was helpful when you started using Autify, Mr. Sawada?

Tomoki: Ms. Kazami gave me a walk-through of Autify’s functions, so the process was smooth for me. I got in touch with Autify’s support team via the contact form if there was anything I wasn’t sure about. They would refer me to FAQ articles or advise me on the correct code, so I could solve each issue as they arose.

– How are you expanding the use of Autify within the company?

Izumi: If we were to get the whole company using Autify, we would need the company’s approval because of the cost. We are waiting for the right timing when another product team asks for an automation tool. Then, we would say, “Actually, we’ve already started automating with Autify. It’s a no-code tool, so you don’t need extensive programming skills to start using it.”

We would also propose it speed up delivery time since it’s one of the challenges we face. Thanks to Ms. Kazami and Mr. Sawada making steady progress, engineers and directors are already showing interest.

Technical backing from engineers has boosted implementation

– Are there anything other hacks for using Autify?

Rei: I was the only one to use it right after implementing Autify, but seeing how effective it is, I wanted more people to use it. It’s usually difficult to write program code to automate tests, so I talked about how difficult test automation can be.

Engineers especially tend to think that test automation is just a matter of writing code. However, it’s not always that easy, if you consider operation and testing. You need help from the right people, such as setting up the test environment.

– Engineers who’ve never automated before tend to overlook those challenges. It sounds like the study group was effective. What did you think, Mr. Sawada?

Tomoki: We use agile development and scrum to develop au Web Portal. We bring up issues during the morning meeting such as struggles with writing a specific JavaScript code, and we would showcase things that we made during review meetings. Engineers would sometimes give advice on how to write a specific code or point out issues. It was great that all members involved in the product cooperated.

Izumi: Not only engineers but also business managers are aware of automation, so we are sometimes asked about how much cost it could reduce. For engineers, automation makes delivery quicker and makes time management more effective. They may be able to allocate more time for development. For business managers, the benefit of automation is that although it would reduce workload in the short term, it improves effectiveness in the long term.

– I’ve heard that Autify is being used by an increasing number of people at your company, so it seems communication and sharing results is going quite well!

Izumi: As mentioned earlier, Ms. Kazami started by using Autify herself and gradually involved others. Also, Mr. Sawada came up with a way of using it with au Web Portal., which has bee another way of getting results.

Five-hour regression tests now takes only three hours. Test automation frees up time to do other tasks.

– What kind of results have you seen within your company?

Izumi: By creating and running Test Scenarios with Autify, regression tests that used to take 5 hours now only take 3 hours. The time savings may not be substantial, but the freed-up time can be spent on other tasks. I believe we can also accumulate data to quantitatively show how test automation has helped us in terms of quality.

Tomoki: We’d been manually running regression tests on five terminals, but now, we have automated three of them. Now, team members step in only when needed. We can focus on manually testing the remaining two. By only having two terminals that need our attention, we can find even minor issues that improve quality assurance.

– How about the issues you faced before using Autify? It seems Ms. Kazami has been the key person for implementing Autify.

Izumi: Merely creating Test Scenarios isn’t enough. We need a system to maintain it. Test engineers are responsible for that; they have to consider the balance between what to automate and what to test manually.

In some cases, areas that engineers and directors are responsible for could come under the test engineer’s scope of work, which would mean more tests could be automated. More services and products could be the target for test automation.

– Will engineers or each product manager be maintaining Autify from now on?

Izumi: Yes. I think each project needs someone that leads maintenance tasks. The test engineer can see things from the testing perspective, while engineers understand development design. Rather than training a leader with both sides, it’s better to create a system where communication can take place. That’s the next challenge.

Because of the pandemic, it’s been difficult to test on real devices. We need to think carefully about how we can minimize the need for real device testing. In that sense, I think Autify is a tool that can take us into the future.

Get engineers involved

– Do you have any advice for those who are planning on automating tests?

Izumi: My role has been to convince the company to give us funding so that we can implement Autify. The key is to start small and show that Autify was effective on a small scale at the right timing. In any organization, bringing in something new is always a struggle. The main thing is to go with the flow.

Rei: From my experience, automation would go smoothly if you get colleagues and engineers involved, rather than trying to do it all by yourself. Then, you can gradually get more and more people interested and accumulate small successes.

Tomoki: As Ms. Kazami said, involving engineers is a critical part of the process. In my case, I’m most friendly with the scrum team. It’s important to contribute to the team on a daily basis, such as help solve any product issues, so that you already have a relationship with teams. It makes things a lot easier.

– Finally, do you have anything you’d like to announce?

Izumi: The Quality Control UNIT is currently hiring QC members. We are looking for those who can think about how to solve quality control issues and those willing to develop their skills to achieve it. It would be great if we could create a new company culture that prioritizes quality at mediba.

(Interviewer: Autify CEO & Co-Founder, Ryo Chikazawa)