The Importance of Test Automation in Agile Development: Steady Operation is the Goal, Not Implementation

Speeding up the development cycle is critical for surviving in a rapidly changing industry. One such industry is online services, which includes games, media, and SaaS. Contract-based development companies must understand both the client and the end user’s demand and quickly develop a product that meets their demands. At the same time, they cannot compromise quality. ITI, Inc. is one such company. They develop a wide variety of in-house services such as media and game applications and develop services for client companies. Contract-based development is one of ITI’s main business pillars. Mr. Yuki Harada, the Project Planning & Development Department manager, and Mr. Makoto Yonekura, the QA/QC leader dealing with quality assurance, are two key figures who work in that domain. They talked to us about some of the issues that the team faced and the process that led them to test automation.

Left: Mr. Yuki Harada, the Project Planning & Development Department Manager Right: Mr. Makoto Yonekura, QA/QC leader
(Photo courtesy of ITI, Inc.)

Releasing every two weeks: Meeting client demand through agile development.

– Please tell me about your business.

Yuki: Originally, ITI was an advertising company. We now have a Media Department, an Application Department that develops iOS/Android game apps, a Project Planning & Development Department that handles contracted development, and an Administration Department.

We belong to the Planning & Development Department which develop products for our clients. One of our projects has been going on for eight years now. It’s a long-term contract that deals with all aspects of the product, including operation, management, and maintenance. We plan the development style based on the agile scrum approach and release every two weeks.

In the future, our vision is to take on projects of various sizes and make this service a new revenue stream for ITI.

The Planning & Development Department develops apps for stores such as restaurants, beauty salons, and physical therapy clinics. In the last three years, we have developed 5,000 apps. If you combine iOS apps and Android apps, we’ve developed 10,000 apps in total.

We develop by customizing a base structure, so it’s not entirely original, so the service is more like an app builder platform.

– What are your responsibilities? Can you also tell us about your teams?

Yuki: I started as an engineer and joined ITI about six years ago. I’m mainly in charge of the server-side programming but also some front-end and infrastructure programming. Recently, I became responsible for development as well as managing the Planning & Development Department.

The Planning & Development Department has a total of 25 members. The department is subdivided into several teams such as Service Desk, which is the contact point for clients, Development, PO (Product Owner), Infrastructure, and QA/QC (Quality Control).

Makoto: The company I used to work for merged with ITI eight years ago, and here I am now. I joined the Planning & Development Department four years ago, and I am currently in charge of QA (quality assurance). The QA team consists of three members.

If we don’t automate, the system will eventually stop: The challenges in the operational structure

– What challenges were you faced with during agile development before implementing Autify?

Yuki: The new service we’re currently developing for a client didn’t have a dedicated QA/QC tester. This meant that the development team had to debug and test in addition to their usual development work. So, we created a QA/QC team consisting of two members, including Mr. Yonekura.

The app has a CRM system that allows members to make reservations and distributes coupons. The app has been used for around ten years and uses some legacy code. The source code is outdated, and we were adding new features and making modifications like patchwork. We started working on unit tests three years ago to prevent issues, but building tests was problematic.

– It must have been hard to build unit tests for legacy code.

Makoto: Yes. I knew that Mr. Harada wanted to automate and that we needed to automate to keep going, so I’d been interested in automation for a while. However, I thought I would have to learn Selenium from scratch if we were to automate. Honestly, I didn’t want to do that.

However, we could tell that we would eventually have to automate if we continued using this system, or it would stop working. Mr. Harada told me that he’d be happy to implement Autify, so we started considering the specifics.

Yuki: We did talk about test automation before Autify. We implemented tests using Selenium, but it was a struggle. Getting to the point of implementation was an uphill battle.

Our development cycle releases every two weeks, and we need to test each time. Doing it manually was time-consuming for our engineers. Even though we wanted to automate, engineers need to focus on development. We had to fix the tests every time the specifications changed. This meant that we were too busy on maintenance that we couldn’t work on development, which is the central part of our jobs.

The whole point of automation was to make things more efficient, but it had the opposite effect. Selenium was too impractical. We began searching for a better way, and that’s when we came across Autify.

– Selenium was too tricky.

Yuki: It wasn’t easy at all! We used to test everything manually, which was quite hard as well. We couldn’t check everything in detail, and we didn’t have time to do regression testing. There were times when we could only check areas that were modified and hope for the best.

Since we implemented Autify, we’ve been able to test thoroughly. For example, even when only a small part is modified, we can test the entire system. We still manually test some parts of our apps, but our goal is to automate as much as possible. I don’t want my team members doing repetitive tasks.

– Did you used to find issues after release?

Yuki: Yes, we did. Before Autify, we would find issues quite frequently after release.

Work from home was an opportunity. Building team momentum through knowledge sharing

– How was the process of implementing Autify? Did you face any struggles?

Makoto: There wasn’t anything in particular that we struggled with. I like how intuitive it is. We didn’t think much about the process of implementation. We just went with it, solving issues as they came up. Because of this, our system and Autify didn’t work very well at first. We contacted Autify’s customer support via chat, and I was extremely grateful that they responded quickly.

I was the only one who used Autify for the first two years. But when a new member joined the QA team, I encouraged other members to use it more. This freed some of my time, so I created resources and ran workshops to share my knowledge. I think the whole team is capable of using it now.

– That’s great. When people hear the word automation, they immediately think it’s too difficult and avoid using it. How did you overcome that?

Makoto: I think having to work from home because of the coronavirus pandemic was a blessing in disguise. I was able to demonstrate how easy it is through video calls. If I hadn’t done that, there’s a good chance I’d still be doing it all by myself!

– How did you proceed with automation after you implemented Autify?

Makoto: I wanted to start small and focus on understanding how Autify works. I started by creating Test Scenarios by interacting with the screen and checked how it was being recorded.

– Did you come up with any tips and tricks for using Autify?

Makoto: I made it so that test completion notifications from Autify are automatically exported to a spreadsheet. I set up a Webhook using GAS (Google Apps Script) for that. It’s helpful when debriefing my team in morning meetings. Now that we’ve exported several thousand pieces of data, I’m hoping to analyze test results to discover any trends and apply the findings in the future.

Also, I create a rough blueprint before creating Test Scenarios. This makes it easier to modify the test later.

– Do you mean you wrote test cases on a spreadsheet and used that as a blueprint?

Makoto: The point of the blueprint is so that everyone can study it. Now, I write each step in a spreadsheet for each Test Scenario that I plan to make. I started this so that other members can refer to it when they make Test Scenarios. I ask them to think about what the test should focus on first. That gets checked, and if it’s OK, they start working on it. Eventually, I’d like to consolidate it as a test case.

I see. It seems like a long way round but sharing knowledge with members is an essential step for using it independently.

TIt can be used to prove the stability of the service.

What changes did you see after implementing Autify?

Makoto: It’s helped us test all areas comprehensively, including parts we didn’t have time to check before. We can run tests automatically every day between midnight to early morning when server load is low. We don’t cause trouble to our clients anymore. It’s fantastic that we’re able to run tests and assure quality while we sleep. Also, every test is recorded once it’s run, so we can efficiently run the same test as needed.

Yuki: It gives me a sense of security that the service is stable. By regularly running tests with Autify, we’ve been able to find critical bugs. We’re able to fix it immediately, which is excellent. We must be able to find issues before it gets to the client. In that sense, it’s one of the things that we can list in the SLA (service-level agreement).

Then there is cost reduction. The entire team, including engineers and the service desk, used to do all the functional testing after release. Now that we’ve automated with Autify, we don’t need their help to run functional testing. Engineers and the service desk can focus on their jobs, and that’s a big step in the right direction.

As we increase the number of contracted projects, applying the know-how we’ve accumulated so far to new projects will be a significant benefit.

– How much of the tests are currently automated with Autify?

Makoto: Roughly 60% of all tests are automated. As far as the admin panel is concerned, it is 90% automated. We’re planning to automate even more; the target is 80% automated overall.

– What are your plans for the future?

Makoto: I’d like to analyze the data that we’ve accumulated so far and discover trends in what issues we deal with. Also, I want to work on automating tests for native apps. That’s something we haven’t started yet. Some parts of the browser operation screens cannot be automated, so we still have to test those manually. I’d like to fix that.

– Please let us know the details on where you’re unable to automate. We’ll investigate it so that you can automate it as well. We’ll also help you with the native app.

Makoto: That’ll be great. I appreciate it.

After giving up on automation so many times, Autify has finally made it happen.

– Finally, do you have any advice for those who are planning to automate tests?

Makoto: If you try to automate tests by writing your own code, it’s going to be very time-consuming. Using Autify will make it a lot easier. I tried and failed many times, but with Autify, we finally made it happen. If you’re about to try and automate testing, it’s worth giving Autify a try. Also, I’d like to thank Mr. Harada for taking the initiative to implement Autify!

Yuki: Likewise, I’m grateful that you agreed to it! Like anything, it takes a lot of work and patience to implement something new. There are quite a few cases where people are satisfied with just implementing it, even if they aren’t making full use. We’ve made Autify a part of our routine; we run it every day and report the results in the morning meeting. I think it’s been adopted by the entire team now. I want to say to those considering automation that once you implement and utilize Autify, the rest will be easy.

– Thank you very much. By leaving automation to automation specialists, you’re able to focus on the important work. That’s what Autify is all about.

(Interviewer: Ryo Chikazawa, CEO & Co-Founder of Autify, Inc.)