Automating Test Maintenance: Autify’s AI Detects Differences and Auto-Corrects Test Scenarios

How has Coincheck improved its quality assurance process?

Bitcoin has been traded since 2009 and has recently gained attention as a speculative investment due to its volatility. In June 2021, El Salvador passed a bill that made Bitcoin legal tender, attracting even more attention.

Coincheck, Inc., a member of the Monex Group, provides the crypto trading service Coincheck. Bitcoin is the main cryptocurrency traded on the platform.

Until August 2012, the company’s leading service was STORYS.JP, a service for posting life stories (under the former company name ResuPress, Inc.). In August 2014, they entered the cryptocurrency trading industry with the launch of Coincheck. The app has become the No. 1 downloaded app in Japan※, with over 2.84 million users.

Mr. Yuji Eto is the CRE Group Leader whose mission is to develop focusing on customer relations and improve internal operations. We interviewed him about the process of E2E test automation and the effects of adopting Autify.

※Among domestic crypto asset trading apps. Research period: January 2021 – June 2021. Data supplied by App Tweak, As of June 30, 2021

Changing the system where engineers write code and members test in their spare time

– First, can you tell us about Coincheck and what you do?

Yuji: Coincheck, Inc. is the company behind Coincheck, the cryptocurrency trading service. Coincheck has gained its popularity mainly by young people due to its simple and easy-to-use design. From 2020, we have started providing new services such as Coincheck NFT(beta version) and Sharaly, a virtual shareholder meeting support service.

I am a CRE (Customer Reliability Engineer). The team’s mission is to support customers and ensure our in-house tools are used to their fullest potential.

– What challenges did you have with E2E testing?

Yuzi: Since we don’t have a dedicated QA department, our engineers had to write Test Scenarios and work with customer support and the legal department to execute the tests. This would happen every time we develop something.

When I joined the company in December 2018, I was initially responsible for E2E testing. I was later assigned as the manager of the anti-money laundering project. In that project, we create a feature we call KYC (know your customer, customer management measures), which captures the information needed for verifying the customer’s identity. In addition to collecting the customer’s information and improving our services’ security, we have also been working on upgrading systems that monitor suspicious transactions and systems related to the revised Act on Prevention of Transfer of Criminal Proceeds.

At that time, Test Scenarios had to be created from scratch. We took the Test Scenarios created for the AML (Anti-money laundering) project and applied them to test systems related to the Act on Prevention of Transfer of Criminal Proceeds. Customer support, legal, and engineering departments all coming together to create Test Scenarios for each user characteristic, such as occupation, age, nationality, etc., and then manually tested them on the web and mobile.

We could delegate to a certain extent if we had a QA department, but we don’t. Engineers were creating and executing tests in addition to their regular work. Overtime kept increasing, and there was not enough time to run regression tests thoroughly. That’s when I came across Autify and decided to adopt it so that we could run tests automatically.

– Regression testing is a lot of work. How often do you release?

Yuji: For larger projects, we release once every three to six months. There are also minor releases once a day and sometimes even several times a day. We only test the difference in those cases, but we want to avoid regression from adding new features.

Autify detects differences and auto-edits the test scenario, significantly reducing maintenance costs.

– What was the deciding factor for implementing Autify?

Yuji: Mr. Kakizaki, an ex-colleague at my previous job who now works at GA technologies, tweeted about their experience using Autify. I want to automate E2E testing but writing the code myself with Selenium was too challenging. As Mr. Kakizaki pointed out, the key factor for choosing Autify was how easy it is to create tests.

– Did you also consider Selenium?

Yuji: I had used Selenium at my previous job, but it was a struggle! We had to rewrite the code whenever the UI or stylesheet changed, so maintenance was time-consuming. Autify is great because it detects any changes in the UI, displays the difference, and automatically modifies the Test Scenario. It makes life so much easier.

– What challenges did you face after adopting Autify, and how did you go about using it?

Yuji: Our service requires a high level of security, but sometimes, the two-step verification during the login process doesn’t work. The user must use either a TOTP (Time-based One-Time Password) or receive a security code via SMS, so I wanted to create a test to verify the authentication flow worked properly.

The authentication step didn’t work in the test environment, so we skipped it as a workaround. From there on, it was just a matter of adding more scenarios. It went smoothly.

– We’ve recently created an endpoint that generates one-time passwords. Your test could work if you used that.

Yuji: That would be ideal. All we need is to be able to enter the secret key by scraping the TOTP. I already knew that you could add a JS Step in Autify, but I wondered whether it could handle TOTPs.

– End-to-end testing tends to have common issues like TOTPs and two-step verification. My hope is that Autify will eventually be able to deal with all of them. We’ve recently added a feature for testing email authentication.

Yuji: Our service requires email authentication quite a lot, so that’ll be very helpful. I personally test to make sure the email authentication works but haven’t managed to add it to Test Scenarios yet. I’m hoping to do that soon.

– Did you act as a leader when implementing Autify?

Yuji: Yes. There were three team members at the time, so we worked together to create Test Scenarios for testing the identity verification system. Then, we ran the test once a week. It’s inconvenient to create a new test environment and run tests yourself. By changing the URL, we can have Autify test it for us. I like that Autify can run connectivity tests, too.

– How often do you have to create a test environment?

Yuji: Sometimes it’s once a week, sometimes it’s once a month. We can test locally if it’s a test that can be handled by the development department alone. However, if we want the legal department or customer support to test it too, we build a test environment for them.

Sometimes, customer support’s test would come back okay, but the legal department’s test fails, or vice versa. Depending on the project, we ask the legal department to check for any legal issues. That’s something that may be unique to our industry.

Automating tests to have peace of mind

– Are there any tips and tricks for implementation and operation?

Yuji: We create Step Groups and share what we can. Periodic execution and regression testing for various environments aren’t tips and tricks, but they are very useful.

– What kind of obstacles did you face when using Autify?

Yuji: I’d like to find more time to create more tests. If we added tags for each item, we’d be able to manage and maintain the test environment better, but in reality, we haven’t done that yet. Currently, the CRE team uses Autify to run tests, but I’d like the legal department to use it as well.

With new projects starting all the time and other things to prioritize, our main challenge is that we haven’t yet created all the tests we want to. We are busy with development, so I’d like to set aside time to work on testing every week. I think future me would appreciate it if I completed setting up all the tests. Some tests immediately come in handy, so I think working on them is essential.

I realize how much Autify helps me when I run regression tests on new projects. I’d like more people to use it.

Spending less time staring at test specifications

– What changes or benefits have you seen by using Autify?

Yuji: Sometimes, a different page inserts itself between certain pages, and the test for that is beneficial. Even before identity verification, the page changes several times. By testing with Autify, you get a message that there’s something wrong so that we can change the scenario accordingly.

Also, we’ve saved a lot in maintenance costs. Modifying tests is very easy because you can insert new steps between existing steps. The interface is visually easy to understand, and I love how you can change a test by clicking a few buttons.

Also, I don’t have to rewrite the entire test every time I make a minor change, which has saved me from wasting time looking at the test specifications. It gives me peace of mind. As long as the test is on Autify, we humans can forget about it, making our lives so much easier.

When you need to create a new test derived from an existing test, we can duplicate the existing test on Autify and create a new scenario. That’s another area where Autify shines.

We still have to work on making tests for the admin pages, but once those are done, there will hardly be any need for manual testing. Of course, I’m not saying we will never have to check manually, but all the essential parts would be automated. I’d like to rethink how we test from the ground up while also changing how we create test data.

– Preparing the test data well can help cover more ground with automated tests. How much of your testing do you want to automate with Autify?

Yuji: I think we can cover a lot of ground if we start by creating data. For the identity verification system, we automate close to 100%. After the initial input of various user data, we can use the API to create scenarios and run tests. Once the input is done, we can run tests on admin pages. I’m very optimistic that we can eventually automate most tests.

Automating tests makes a huge difference mentally. It allows me to wake up without being too stressed out. I’d like to make it as less stressful as possible.

We’ve also been able to streamline other tasks related to testing. When a new member joins, we can tell them about the type of scenario we use for the test instead of telling them about the type of test we have to run. Members can run tests on their browsers as long as they know the URL and Autify’s account details. Conveniently, Autify takes a screenshot for each step, making it easy to explain things to other people. Sometimes there’s no need to explain because people can understand just by looking at the screenshots. Even if a bug occurs, just give the URL of the result page to the relevant person, and it can be taken care of.

– What is your future outlook like?

Yuji: Autify allows us to create tests by simply interacting with the browser. I’d like to allocate time for making more tests. Even if it’s a simple test, I think the payoff would be substantial, not just for other team members and myself but also for those who join the company in the future.

Crypto assets are highly volatile, and the market is rapidly gaining popularity. NFTs (Non-fungible tokens) are also a hot topic right now. I’d like to improve the UI and use Autify to run automated tests to ensure quality so that new customers can smoothly complete the identity verification process.

A message for those who are planning to work with test automation**

– Finally, do you have any advice for those planning on automating tests?

Yuji: It’s a great feeling to create a simple scenario with Autify and see the test working. The more positive experience you have with automated tests, the more encouraged you feel to automate more. I recommend it, and I plan to keep working on it, too.

With regression testing, humans will always miss something. Also, looking at test specifications when you’re already feeling under the weather will make you feel even worse! I think Autify helps with that.

Coincheck is hiring

– Do you have any announcements you’d like to make?

Yuji: Many of our departments are currently hiring. Crypto assets are already an exciting industry, but with the addition of the NFT, I think it has become an industry that cannot be ignored.

If you’re not sure which department you’d like to work in, we’d like to invite you to a casual interview. We want to share the joys and the struggles that we face every day, and if it interests you and you are qualified, we’d like to work with you. I look forward to working with you.

(Interviewer: Ryo Chikazawa, CEO & Co-Founder of Autify, Inc.)