How MoT Is Expanding Test Coverage for Better Reliability in Quality and Getting the Most Out of Test Automation

I’m sure many have used cab-hailing services where you can order a taxi on your smartphone.

Mobility Technologies, Co. Ltd (MoT) is the company behind the most downloaded taxi app in Japan, GO. Aside from the taxi-hailing service, the company launched DRIVE CHART, a service that helps reduce traffic accidents by utilizing AI, in 2019. Primarily adopted by logistics and taxi companies, the demand for the service is growing.

MoT aims to solve various social issues surrounding mobility; congestion, traffic accidents, and the lack of public transportation in depopulating areas. In this interview, we spoke to Mr. Sawada Yuichi and Mr. Niko Ryo, who work on DRIVE CHART’s quality control.

DRIVE CHART uses AI to detect dangerous driving and reduces the risk of traffic accidents

– Who are you, and what do you do?

Yuichi: I’m Yuichi Sawada. I belong to the Quality Management Department at MoT, and I oversee organizational management in general.

Ryo: I’m Ryo Niko, the QA leader of our AI-based service, DRIVE CHART, which aims to reduce traffic accidents. I’ve been using Autify to automate testing since last year.

– What does your business do?

Yuichi: As our company name suggests, Mobility Technologies is a company that provides a variety of mobility-related services. Our mission is to “make people happy through mobility.”

Along with our taxi app GO, we also develop DRIVE CHART. This service aims to reduce traffic accidents using a dashcam with a next-generation AI feature. Another service is GO Dine, a food delivery app that allows users to order food from Michelin restaurants and other renowned restaurants and have it delivered by taxi. With the desire to revolutionize mobility, we aim to offer solutions for various social issues related to mobility.

– I understand that you use Autify mainly for DRIVE CHART. Can you tell us a bit more about the service?

Yuichi: DRIVE CHART is a service that helps reduce traffic accidents. We achieve this by collecting various data from our dashcams placed in taxis, sales vehicles, and long-distance trucks. The data is used to automatically detect dangerous driving likely to lead to traffic accidents (distracted driving, speeding, ignoring stop signs, etc.), analyze and report driving habits. We are working hard to develop more and better features so that our service can reduce traffic accidents.

Service expansion and the accumulation of technical debt caused testing to be highly time-consuming

– What kind of challenges did you face before implementing Autify?

Ryo: Although there weren’t any major issues in terms of quality, things like code refactoring were causing a lot of minor regressions, which affected other parts of the service. We needed to figure out a way to run regression tests for the entire service, but we couldn’t allocate enough staff. That’s why we started considering whether automation would be an option.

– Have quality expectations changed as well?

Ryo: Quality expectations have remained the same, but the number of things we need to check keeps increasing. In that sense, I think the level of QA has gone up.

Yuichi: We have to expand the functionality and develop faster, but we’ve also accumulated a lot of technical debt, such as internal logic. In a situation like this, the time required for testing becomes the bottleneck.

We wanted to create a secure environment for development engineers so that they could go ahead with aggressive programming. That’s why we decided to automate testing. We thought increasing test coverage through test automation would improve product quality in all areas.

No-code and excellent customer support made Autify more accessible

– What were the deciding factors for implementing Autify?

Ryo: It was my first time automating testing, so there was a significant psychological barrier. Along with its ease of use, great support was huge when we took the first step.

The support staff helped us implement Autify. They also quickly responded to any questions we had via chat, which helped clear doubts one by one.

– What kind of psychological barriers did you have?

Ryo: We had no one dedicated to the project and didn’t know where to start. I’d heard that automation is prone to failure, and I had a vague sense of anxiety, so I first searched the Internet for anti-patterns to learn what we needed to do.

Yuichi: When people hear about test automation, they tend to think that you have to write a lot of scripts first. However, not many test engineers can write scripts, so I guess we were worried we didn’t have the technical capability.

– With Autify, you can create and run automated tests without writing code. Did that ease your worries?

Ryo: Yes. Autify provides a lot of snippets, so it was easier than expected. The more I edited snippets and created different test patterns, the more familiar I became with scripts. I gradually learned how to use them in different ways.

Yuichi: During the free trial period, I checked how Autify behaves on each of DRIVE CHART’s screens and features, which was the app we wanted to automate testing for. I contacted chat support with various questions and got answers immediately. Doing an extensive feasibility study helped everyone be confident that they could automate tests, too. I think that led us to overcome the psychological barrier.

Visualizing test coverage potential by conducting a feasibility study

– Once you implemented Autify, how exactly did you go about automating tests?

Ryo: We started by reevaluating quality requirements. We then analyzed the testing requirements and ran a feasibility study to examine whether they could be met with Autify. We concluded that Autify fits the bill for the most part, so we designed tests and moved on to implementation.

The original regression test items were there, but they were supposed to be done manually, so we made some adjustments for automation. I’ll use a car as an analogy… With manual testing, the car runs on a manual engine. If you replace it with an automatic engine called Autify, it won’t run well. We figured we might as well build a new car compatible with the Autify engine and began designing tests from scratch.

Yuichi: In the feasibility study, we began sifting through features and screens, determined whether Autify could test each of them, and created our own test case. Once test cases were complete, it became clear that Autify could cover about 50%.

Another critical factor when implementing test automation is making sure you have the budget; people would often ask roughly how many tests can be automated and how effective it’s likely to be. Ryo gave me a rough estimate and visualized the cost-effectiveness. As someone responsible for the budget, this was extremely helpful in convincing me. I thought, “I think we can do this. Let’s go all in!”

Setting up an operational structure with a focus on avoiding specialization

– Did you have any unique ideas when you implemented Autify?

Ryo: First, I focused on avoiding specialization. When you automate testing, it’s not just about running tests on Autify. You need to create an entire system, including operations. Right from the planning stage, I thought about how I could avoid key-person dependency.

For example, when we created scenarios, we also prepared an itemized document listing what to check at which step. This would allow anyone to maintain the scenario.

Yuichi: Our test automation plan incorporated an educational element. Along with milestones of creating scenarios, we worked on training our staff and accumulating knowledge.

The main objective isn’t to automate testing, so we often discussed what kind of results and effects we need to achieve through test automation and carefully prepared systems to measure results.

Ryo: When we implemented Autify, we communicated with development engineers about the automation system and how they could utilize it. I think that’s why we were able to start using it smoothly, and people were keen to try it.

Yuichi: Involving the development engineers was hugely beneficial. Once we ensured we were on the same page about expectations and requirements, we created and ran the scenarios. As a result, they were able to get the expected results and were satisfied.

– Absolutely. It’s vital to ensure that each team is on the same page in terms of expectations before implementing the platform. How did you go about measuring effectiveness?

Ryo: The effect must be measured quantitatively. During the planning stage, we created a system that calculates the work-hours required to manually run each scenario and estimated how much work-hours automation would save. Numbers like how many items have been automated and coverage ratio are irrelevant to non-QA staff, so we make sure to avoid them.

Yuichi: The scope of implementation was areas that manual testing wasn’t covering already – we didn’t completely replace tests we’d been doing manually. We saw this as an effort to cover areas that even manual testing couldn’t handle. The idea wasn’t necessarily to save work-hours but to do more with the same number of people.

As test coverage improves, the more reliable the quality will be. We would also get benefits like fewer bugs and better performance from our engineers.

Preparing for the future by building a system that accumulates and analyzes test result data

– Do you use Autify in any creative ways?

Yuichi: Utilizing the API provided by Autify, the QA team built two tools: a tool that automatically aggregates a list of test results after execution and a tool that uses the list to automatically create a detailed test result report. To reduce operational costs as much as possible, we wanted to automate as much as possible; test execution as well as related tasks.

As for Slack notifications of test results, we filter them using keywords and automatically add only the necessary data to a spreadsheet. Monitoring specific parts of test results and accumulating data will make it easier to analyze what kind of bugs occur in what kind of conditions. We created those tools in anticipation of those needs.

– Are there any other features that you frequently use?

Ryo:  We use Step Groups quite a lot. I also like that there are a lot of snippets. We also use JS Steps and the URL Replacement feature.

Testing quickly and having peace of mind: the benefits of test automation

– You mentioned earlier about measuring effectiveness. What kind of results and changes have you seen since implementing Autify?

Yuichi: I think 70% of regression tests have been automated so far. The most significant change is that we can instantly run tests whenever we choose. There are many situations when you’d want to run a test, such as when the development side wants to refactor or upgrade the programming language. Previously, we often couldn’t test for various reasons, including a lack of QA resources and limited time we could allocate for testing before release. Since implementing Autify, we’ve been able to respond instantly when development engineers request us to check all features.

For example, we found several issues after refactoring, so there have been tangible results. The fact that we can find problems early is exactly what we had hoped for, and when that happens, I really appreciate the benefit of automation. As someone responsible for quality control, it’s very reassuring that we can check the automated areas at various times of the development phase and ensure there aren’t any issues (things are working as expected).

Development engineers have commented that being able to quickly and comprehensively test all screens gives them peace of mind and is extremely helpful. The impact on engineers and the product as a whole has been greater than we’d expected.

Ryo: Even if you think making a minute change in the code won’t have any adverse effects, refactoring can cause unexpected issues. It’s great that we can fix those bugs before release by looking at results from automated tests.

70% of test items have been automated. Cost-effectiveness exceeded expectations.

– How has the cost-effectiveness of test automation been compared to what you’d expected?

Yuichi: It has exceeded expectations. However, the cost-benefit can be less than the operating cost (*1) if you don’t run tests frequently enough. For us, value-for-money is exceptionally high if we run all test scenarios about four times per month. However, if development is slow and tests aren’t run as frequently for a month, the cost-effectiveness may be low if you only look at that month. However, if you look at it quantitatively over the long term, we think Autify is cost-effective.

• 1 Autify license fee, creating/editing scenarios, compiling/analyzing/reporting test results, and other costs (knowledge, training, formulating rules, creating various tools, etc.) – It pays for itself if you run automated tests more than four times. Do you know how many work-hours automation saves?

Yuichi: If we were to manually run all test scenarios, it would take about 15-16 work-days. We have a large number of test items, around 1400-1500. Using Autify, we can run it all with a click of a button, meaning Autify saves 16 work-days worth of testing. This frees up time for the QA engineers to focus on other tasks.

– How many times per month do you run all scenarios for regression tests?

Yuichi: On average, about three to four times per month.

Ryo: In the past few months, we’ve identified the results and shared how effective Autify has been with the development team. We’ve gotten some requests for automating some other things as well. As we respond to those requests moving forward, tests will naturally be executed more frequently, and we’ll get even more out of it.

Delegate as much as possible to machines so that people can create value

– What would you like to work on next? What’s your vision?

Ryo: Our goal is to lower operating costs and further improve cost-effectiveness. In cooperation with the development team, we hope to improve test coverage by automating complicated test cases. We would also like to incorporate Autify’s E2E testing into the CI/CD pipeline we are developing. It would be ideal if we could also use Autify to test other products.

Yuichi: First, I want to build a system that regularly monitors quality and visualizes it. Then, I would like to make it so that anything that can be left to machines is done by machines so that people can focus on creating value in ways that only we can. We hope to achieve this with Autify’s help.

– That’s exactly our mission. It would be our pleasure to help. Do you have any advice for those planning on automating testing?

Ryo: The hardest thing is to take the first step, but if you identify all the issues surrounding automation, the rest isn’t that hard. Autify supports you at each step, from implementation to utilization, so you simply need to take a small step towards automation.

Yuichi: First off, defining your goals and objectives is critical. Why do you want to automate testing? What issues do you have, and what do you want to solve through automation? If you skip this step, you automate for the sake of automating. Once you’ve done that, it’s a matter of planning what you need to do to reach the goal without getting distracted by immediate results and effects and getting them done one by one. That’s a piece of advice that applies to anything, really.

– Finally, do you have any announcements?

Yuichi: We are hiring for various positions. In particular, the Quality Management Department, responsible for quality control, has many open positions. We are looking for people willing to grow and take on challenges and difficult issues together. If you are at all interested, please feel free to contact us for a casual interview.

– Thank you very much!

(Interviewer: Ryo Chikazawa, CEO, Autify Inc.)