Things that were impossible became possible. A benefit could be seen which cannot be measured by cost alone

DeNA Co., Ltd. (DeNA) is one of Japan’s most well-known leading venture companies that is constantly working on providing new value. The vision is to “delight the world as a permanent venture by utilizing the Internet and AI.” DeNA’s QA team continues to provide many products and services and has grown to over 100 staff in areas other than games. Autify has been adopted as a tool to automate the enormous number of software testing that they deal with. We interviewed Naoki Kashiwakura from Quality Control, and Kenji Serizawa from System Headquarters about the road to test automation and its benefits.

– First, please introduce yourselves.

Naoki: I’m the leader of the QA team that verifies services and products in areas other than games, specifically those related to entertainment. As for other activities, I hold a study session called “DeNA QA Night“.

Kenji: I am also leader of the QA team in areas other than games, mainly focusing on entertainment content like mobile game platforms, cartoon apps, novel apps. Like Naoki, I have also been involved with other activities such as taking part in “Software Quality Symposium” in 2019.

– Thank you. It’s been about six months since DeNA introduced Autify (at the time of the interview). I’d like to ask you again about the sequence of events that led you to introducing it.

Escape from the Excel screenshot. Issues prior to introducing Autify

– First, please tell us about how testing was before introducing Autify.

Kenji: Before introducing Autify, the quality control department mainly conducted manual tests. With the diversification of DeNA services, the test area and scale were gradually increasing, and there was no good timing to consider introducing the tool. We were overwhelmed with daily manual testing.

– DeNA has a specialized team for Software Engineer in Test (SWET). How were the SWET team and manual testing separated?

Naoki: The SWET team is better at the development side of test automation, such as CI/CD. When it comes to E2E testing that Autify supports, it’s mostly manual. Automation wasn’t advanced. Some of our team members were able to handle Selenium. People who could use it were using it willingly, but when it came to the whole department, it was too difficult so we couldn’t go ahead with it.

– Then you learned about Autify. What was the deciding factor for its introduction?

Naoki: Autify is amazingly simple and extremely easy to get into. For example, it’s easy to use even without any coding skills, and that point alone made it viable to introduce it to the whole team.

Kenji: Some team members commented that the screenshots of the test result screen and other evidence, log tracking, and confirmation were very easy. I guess you could say that the traceability is good. That was something I had never seen before.

– How was traceability resolved when you were performing manual testing?

Kenji: We were so-called “Excel screenshot craftspeople.” We used to manually take screenshots and save them and manage where and what the images were. The main focus was taking those screenshots, rather than conducting tests. In the case of Autify, the screens are lined up according to the procedure, so you can follow the flow just by sliding. Because of this, I immediately felt this is very easy to use when I first say the screen. We were freed from “craftsmanship” thanks to Autify.

Naoki: ‘Excel screenshots’ are just so tedious! I’d always wanted to free myself and team members from those chores.

– It’s not something people should do. I am very happy that we were able to help you in that respect.

In-house information sharing and starting small were key for test automation

– Please tell us how you proceeded when introducing Autify. Where did you start from?

Kenji: First of all, we tried it. We started off by operating it and getting used to it. As for test items, we started by simple items rather than complex ones. For example, we started creating scenarios for short test cases such as checking the display and page transitions.

– Did you have any problems at that point?

Kenji: There were operation manuals and lecture videos, and documentation was very comprehensive so introduction was very easy. It was very easy to use. I think the initial introduction went quickly. Sometimes Autify gave us tutorials directly, which was also very helpful. We may have been able to expand its use quicker if there were video tutorials, operation steps and manuals for when we wanted to do something more advanced, such as assertion. This is something that Autify could improve on!

– We will work on that immediately! sIt seems that it’s being used in a variety of products (services), departments, etc. Has there been anything you’ve done differently when using it in a wide range of settings? Is there anything that is unique to a large QA team like DeNA?

ー What did you do as a small start?

Kenji: When we introduced Autify, each team took some time for test automation. We made plans on what will be automated in what order and we set goals such as KPIs. As we hadn’t taken the time to think about those things, I think it was very beneficial for future operations to have had the opportunity to do so.

– I see. What kind of actions did you take as a result?

Kenji: Taking one regression item as an example, we didn’t think about automating everything all at the same time. Instead, we automated things one by one, and did it properly. We thought about the scenario before assembling. That’s where we started from. Especially repeated tests, which is where automation can be particularly beneficial. We’ve raised the priority of repetitive tests, gradually turned it into scenarios, and run it on a regular basis. That’s how we gradually increased the number of tasks delegated to Autify.

– You started small. How did you decide the priority?

Kenji: For each team, we proceeded with systematic reviews while also having a mutual understanding of where we were going to start automating. That may have been the key for getting results quickly.

Naoki: We have various products, but each team introduced Autify at the same time. At DeNA, we hold regular meetings where team leaders gather to discuss each issue, and one of the items on the agenda was Autify and test automation. We were able to share ideas during the in-house information sharing meeting. As Kenji mentioned, the following points were made during those meetings:

Select items before creating scenarios

– Start small
– Think about how to effectively utilize Autify’s functions
– For example, assemble a scenario so that the data driven test function can be used.

– I can see that on a large scale like in DeNA, having a place to share each team’s know-how was key to accelerate the use of Autify after starting small.

Things that were impossible became possible. A benefit could be seen which cannot be measured by cost alone

– Next, I’d like to ask you about the outcome. What kind of effects and benefits were there specifically?

Kenji: Just starting off by automating simple items, we were able to spend time on other tasks. Efficiency of test execution has improved. Also, I think the biggest benefit has been that we can do tests that we didn’t get to do previously. By that, I mean things like repeated tests. At our scale, doing it manually is impractical. I can’t imagine how many people we would need. We also sometimes use Autify to perform regular migration checks to detect abnormalities in the production environment. Again, this is something we couldn’t do until Autify was brought in. It’s been a major benefit that other tasks can be carried out in parallel by constructing and executing a scenario. The specific number will depend on the team, but I feel that overall, we’ve been able to perform tests efficiently. For example, for a product (service) that I manage, workload has already reduced by 10-15%. A recent example is that it was effective when we had to work remotely. While the operations department quickly switched to remote work, staff in quality control had restrictions on taking out actual devices used for testing. They had to take turns taking home frequently used terminals, and to do that, rental management was necessary. It took a long time to fully switch to remote work. Only the quality control department was left behind. The tests that used Autify were totally fine in this regard. The fact that it is a cloud tool has been a great help for us even under difficult circumstances.

– Thank you. I am very happy that Autify has been useful in the recent situation. How about you, Naoki?

Naoki: Our team is focused more on performing tests that we previously could not do, rather than reducing workload. For example, we are continuing to increase the number of regression test items, and we are performing these tests regularly. If this were done manually, it would be very expensive. We think we are able to pursue higher quality without the cost. For example, if our development team modifies the code, we can find it through our verification mechanism. However, if the OS or browser version goes up, we couldn’t perform all items in the test every time. Autify allows us to run a wider range of tests than ever before. Of course, we are also trying to reduce workload. For manual testing, we had to write test items. But now, we’ve eliminated this step; instead, we’ve moved forward with implementation. Evidence can be obtained automatically, so creating items on Excel is an unnecessary process. We think this will lead to a significant reduction of workload. Furthermore, the more we do this, the number of regression test cases will increase at an accelerating rate. This is still experimental, but when I tried it for about a month with a product with a quick release cycle, I was able to reduce workload by about 10%. I think it is likely that we will be able to reduce workload from the start.

– We will also aim to support you in those areas in terms of functionality.

Change in thinking due to test automation

Kenji: There’s been another big benefit. Because we used to mainly do manual testing, there’s been an increased number of team members who are interested in automation itself since the introduction of automated testing. People started to think ‘I want to accept and utilize test automation.’

– Does this mean a departure from repetitive work?

Kenji: We’ve begun thinking about what should be automated, what should be done manually, and how we can reduce visual confirmation and manual work that depends on people. There’s also more time to think. Anyone can automate with Autify, so we start off by talking about whether it can be automated. This means that we can execute tests that are closer to the user’s perspective by being more aware of the scenarios.

– Thank you very much. Finally, please tell us about your future outlook.

Naoki: As a vision, I would like the entire department to minimize manual script testing. I would like to gradually expand the area of exploratory testing, focusing mainly on exploratory and automated testing, and concentrate on sensory testing. I believe that utilizing automated tests will allow us to have more time for it.

Kenji: This is similar to what Naoki said, but regression tests are very tedious. It’s something that we want to automate. The most important thing I want to do is to broaden the scope of automation and thoroughly pursue automation, so that we can devote time to places where human intervention is needed.

– In other words, you would like to concentrate on essential parts of QA which cannot be automated. I’m very proud that you are continuing to share your experiences as a large QA team like DeNA. Thank you very much.