RecoChoku’s Vision for Development Without a Test Phase

RecoChoku’s efforts to raise quality awareness through Autify

In the 20 years since its inception, RecoChoku Co., Ltd. has been a pioneer in the Japanese music streaming industry, providing services and platforms for music downloads and streaming. More recently, the company has begun a new music experience that utilizes blockchain technology such as NFTs. As the company develops and releases a wide range of services, it implemented Autify, our test automation platform, in 2020.

“There are things that only humans can test.”

Being in a creative industry, RecoChoku prioritizes allocating time for creative work precisely because the content they provide, music, is creative. In this interview, we spoke to Mr. Tomoyuki Ito and Mr. Yasufumi Kiyosaki, both from RecoChoku’s IT Infrastructure Department, about their views on the future of quality assurance.

Provide music as an experience

– Please tell us about your business.

Mr. Tomoyuki Ito of RecoChoku: RecoChoku Co., Ltd. was founded in 2001 as a joint venture by major Japanese record companies, and this year marks its 20th anniversary. In 2002, we launched a service called Chaku-Uta®. We currently offer music download/streaming services for a wide range of devices, including PC, smartphones, and game consoles.

We also offer services in collaboration with companies such as NTT Docomo, Inc.; USEN Corporation; and Tower Records, Ltd. In October 2021, we started providing a D2F (direct-to-fan) platform that allows artists to connect directly with their fans. In 2022, we began offering new music experiences, including a unique platform that supports NFTs.

– What are your responsibilities?

Tomoyuki: I’m the head of IT Infrastructure. The IT Infrastructure Department is responsible for information systems, security, and QA (quality assurance). Within the department, quality assurance is handled by the QA team, which has seven members.

The QA team works with the development team to test our in-house services and our to-B services. We began Scrum development in 2021 and currently use Autify to run regression tests as a quality improvement initiative.

Mr. Yasufumi Kiyosaki of RecoChoku: I’ve been working at RecoChoku for about 15 years, and it’s been almost 5 years since I was assigned to the QA team. Currently, I’m the QA team lead. I used to be responsible for supporting and negotiating with record companies, initial development stages, and development project management. I’ve always reflected on how we can keep delivering stable services. I analyze past issues to continuously improve our QA processes.

The main issue was maintenance and lack of resources during busy periods

– It’s been two years since you implemented Autify. What challenges did you deal with before?

Yasufumi: As we offer several services, development and testing happen continuously. A full-time QA manager would be assigned to large-scale projects if it’s a large-scale project. However, projects in the maintenance/operation phase are only tested as needed when the QA team asks us to.

The scale of testing depends on what we’re developing, so we can run out of resources during busy periods. We had to rely on external help when this happened, which increased our expenses. We needed to figure out how to work efficiently on QA while maximizing test coverage.

– It’s pretty rare to take on QA while working on CS. Did you become the QA manager when you realized that QA is something that needs to be dealt with?

Masahiro: We simply didn’t have enough people and couldn’t choose who does what. All of us did what needed to be done. I used to work in the blockchain division, but when bakuraku Invoice was released, I moved to the _Invoice _team. I did sales and CS, a little bit of QA as well. All of us scrambled to get things done as a team. I quite like sorting things out in a chaotic environment; I got things under control and started improving the QA process.

– What made you interested in implementing Autify?

Yasufumi: I’ve been interested in test automation since around 2019. I attended a QA seminar where Mr. Suemura from Autify gave a presentation titled The dark side of cross-browser testing. He talked about how labor-intensive it is to run cross-browser tests on Selenium. Mr. Suemura was basically telling us to give up on automating tests with Selenium! That’s where I found out about a useful platform: Autify.

– His presentation became a topic of conversation, didn’t it?

Yasufumi: Yes, it was absolutely fascinating. I’d been questioning if it’s really worth it to keep testing manually, which was why I attended the seminar. Another reason was that my predecessor had considered test automation, but it didn’t materialize. Not testing isn’t an option but continuing to rely on manual testing has its risk, too. It was my mission to solve this issue.

As a business expands, test targets will increase, the development cycle will get faster, the number of features multiplies, and regression testing will need to cover more ground. We needed to deal with all of these issues.

– I heard that you gave up on test automation before. Can you tell us more about that?

Yasufumi: Yes, we tried to automate app-testing with Appium. However, it was not practical to implement it, considering how expensive it was to maintain.

The most expensive part was maintaining and fixing test environments. You have to edit the test code as the number of Test Scenarios increases and the target UI changes. However, we simply couldn’t create a maintenance system with Appium. We were spending more time maintaining the code than testing.

Tomoyuki: One of the most significant issues was running out of resources when several projects were happening at the same time. We can’t turn down projects when things get busy, but we can’t just add more staff because we have quiet periods too. We decided to give Autify a try after realizing that things would be more efficient if we focused on designing, thinking, and taking advantage of our know-how. Once we come up with ideas, it’s better to let the automation tool execute them for us.

Evaluate maintenance costs, execution speed, and test results

– What made you choose Autify?

Yasufumi: After learning about Autify at the workshop, I went to TechCrunch Tokyo 2019, and Autify had a booth there. I was apprehensive, so I asked in detail about how you maintain test environments and modify test scenarios on Autify. It seemed easy enough to implement, so I consulted Tomoyuki, my boss, as soon as I returned to the office and requested a demo the next day. It didn’t take long before we decided to implement it.

– I’m glad we had a booth at TechCrunch! How did you go about implementing the platform?

Yasufumi: After sending the demo request, we immediately began examining how useful it could be. We chose to investigate the PoC (Proof of concept). I picked one service to test on and asked a developer to use Autify in the testing environment as much as possible.

We evaluated three main points to test the PoC. The first point is how easy it is to create test scenarios and how costly it is to maintain them. The second point is the execution speed. The third point is how much time we need to check the test results. If a test fails, can Autify provide the evidence? We evaluated these three points objectively and decided whether we would officially implement the tool.

– What was the result of your evaluation?

Yasufumi: With Autify, you can record steps and screenshots by interacting with the screen as you usually would. Using it was straightforward, and it allowed us to run tests faster. We don’t have to search for locators or elements one by one when creating a scenario because Autify automatically finds it for you. It’s very helpful.

You can also insert or replace steps when you make a mistake, which means that any team member can create and maintain test code and contribute to quality control even if they can’t write test code. It quickly became apparent that Autify could come in handy in a variety of situations.

– How was the execution speed?

Yasufumi: A test that used to take a whole day was completed in a quarter of the time, which proved how useful Autify is. Also, Autify allows you to run 10 tests in parallel, which makes testing even more efficient. The initial preparation is inevitable: we have to create test scenarios manually. However, it’s a massive advantage that Autify can quickly run multiple tests simultaneously.

– What about checking test results?

Yasufumi: In terms of PoC, there were no issues. Even though automated tests would occasionally fail, we could quickly identify the causes, such as an issue in the test environment. The screenshots show the error messages, which helped us not get stuck.

Improve quality awareness within the development team

– Did you come up with creative ways to use Autify?

Yasufumi: First, we shared our evaluation results with the head of development at Manager Meetings to demonstrate how useful Autify is. Next, we began automating existing regression test scenarios, which we used to run manually.

During this process, we realized that humans sometimes check things that aren’t in the scenario. There’s an art to identifying them and creating automated tests that include them.

– How did the development team respond to Autify?

Yasufumi: It was very well received. Some said they’d love to use Autify. However, it’s not realistic to automate all tests at the same time. So, we identified which items are developed frequently and automated those tests first.

One thing I wanted to avoid was implementing the tool but not making use of it. We need the developers to be more involved in QA to improve quality. The point isn’t to get everyone using Autify but to instill the idea that we’re all responsible for improving quality.

– Changing people’s attitudes is to challenge the entire culture, isn’t it?

Masahiro: I wanted to create a process where the development team would run a certain set of tests, and the QA team would do a final check. The development team leader agreed with this idea, which helped implement Autify faster. The next task was to figure out how developers would execute tests. We held workshops to determine which tests were necessary, both manual and automated. Also, we explained how to design test scenarios and how to use Autify. I created a guideline outlining the most basic rules, gradually raising quality awareness.

We held meetings regularly so that the QA team and the development team could exchange information about how Autify was being used and updates on our products. It was also a place for problem management: team members can bring up any issues and discuss how we could solve them by the next meeting, as well as the next course of action. These efforts are vital if we want to get the entire team on board with the tool.

– What have been some of the struggles when trying to raise quality awareness?

Yasufumi: The quality of a test scenario depends on the test viewpoint. So, I referred to a guide called “Viewpoint List” that the QA team created when explaining. We want to ensure that the viewpoint remains consistent no matter who makes the test scenario. We are building a system where Autify is used effectively by articulating the viewpoints.

If developers can run tests properly, they could cover areas that the QA team couldn’t in the past, which would improve test coverage significantly. This frees up resources for creative work. If we can improve the quality at the development stage, the product quality will naturally improve as well, and we can rest easy.

All regression tests for primary user flows are tested on Autify

– Which Autify features are particularly useful?

Yasufumi: Lately, we’ve been using the Email Test feature a lot. We use it to ensure that email authentication works correctly when a new member registers. Tests that include steps for new user registration take longer to create. But Autify’s Email Testing feature generates random email addresses, which is very helpful.

– What benefits have you seen since implementing Autify?

Yasufumi: We’ve been able to check quality without compromising test coverage by running a combination of manual and automated tests at the final stages of development. We have a clearer view of projects even during busy periods, and we no longer need to outsource additional resources. We’ve managed to automate all regression tests for primary user flows on Autify.

For new features, we test manually first. If it passes, we replace the manual test with an automated test. Unless you test manually, you overlook some things, and we flag anything that we find odd. It’s only turned into an automated test once issues are fixed. The QA team has gotten used to using Autify, and more and more scenarios are being created by the development team, so test automation is going quite quickly. Most of the concerns I had before implementing Autify have been resolved.

Tomoyuki: Before Autify, there were times when we had so many projects and would realize that we need more time just before release. It would cause trouble for the development team, and QA team members would be worried too. After implementing Autify, we’ve been able to run automated tests efficiently and smoothly.

Right from the get-go, Yasufumi and I had agreed that implementing Autify should be the goal. If we were to simply implement the tool and tell members to use it, we would lose sight of the objective: to raise quality awareness. Initially, this objective was not sufficiently understood by the development team. However, we patiently discussed and came up with solutions, and as a result, the development team became more aware of Autify. Lately, more people are keen to use Autify and learn how to use it. As the saying goes, you reap what you sow – the seeds we planted last year have bloomed this year.

– Do you have any advice for those planning on automating tests?

Yasufumi: Before automating tests, it’s essential to clarify what you want to achieve. For example, the goal may be to run more end-to-end tests, improve test coverage, or speed up test execution. The benefits are more evident if you clarify the objective before automating tests. It also allows you to align expectations within your team and reduce the risk of it becoming a mere formality that has no benefits.

I’m a member of a Slack community where Autify users can ask questions and exchange information. I recommend joining the community. It makes test automation a more enjoyable experience.

Tomoyuki: There are two important things. First, don’t lose sight of why you’re automating tests. Second, get the entire company involved in the automation process, not just the QA team.

Allocate resources to QA tasks that only humans can do

– What’s next?

Yasufumi: Ultimately, I’d like to eliminate the test phase from the development process. It won’t be easy, but I think it’s possible if we use Autify to create an environment where we can run tests at any time. Even if we get rid of the test phase, it doesn’t mean the QA team won’t be needed. The QA team will be able to use the know-how they’ve accumulated so far and perform exploratory testing with a combination of timing and conditions. We also want to build a structure where the QA team takes the lead so that we can release at any time.

Because our company provides music and video streaming services, there are some tests that only humans can perform. To provide an excellent user experience where people can enjoy music, we plan to focus on understanding music fans’ needs by gaining user insight. That’s what ‘QA that only humans can do’ is. Automating tests will allow us to spend more time on creative tasks.

Tomoyuki: Our aim of automating tests has always been to make it so that we as humans can focus on only what we can do. I hope that the team will evolve into one that works on improving quality and adding value by providing better services.

– Do you have any announcements you’d like to make?

Tomoyuki: In 2021, We launched an e-commerce solution for the music/entertainment industry called Murket that allows users to set up an online store. We are hiring development team members for this project. If you’re interested, please contact us on our website. We’re looking forward to hearing from you!

(Interviewer: Ryo Chikazawa, CEO, Autify, Inc.)