Development team and customer success team work together to meet customer needs.Behind the scenes of Billing Management Robot test automation

Many companies have been working on accounting automation. Financial technology (fintech) has been gaining momentum globally, and remote work is becoming increasingly common due to the coronavirus pandemic.

Nineteen years before these trends started, ROBOT PAYMENT Inc. has been working on automating payment, fund transfer, billing, and fee collection.

ROBOT PAYMENT released Billing Management Robot, a tool that creates invoices automatically, in 2014. The demand for this service has been steadily increasing and 500 companies have introduced it so far. While paper invoicing and the hanko stamp is still common in Japan, they have started the “More freedom in Japanese accounting” project. This project questions why only the accounting department is forced to go to the office.

We interviewed Mr Ryo Tamoto, Mr Junpei Yamashita, and Mr Yonosuke Imokawa, who work on “Billing Management Robot” at ROBOT PAYMENT. They talked about what goes on in the background of quality control and development, and their experience with introducing the test automation tool Autify.

– Please tell us about your jobs.

RYO: I am the project manager of Billing Management Robot project at ROBOT PAYMENT. I had started as an intern at our company in the second year of university. I have been making requirement definitions, acceptance tests, and scenario tests since then, so I’d already experienced how difficult testing is. I officially joined the company as a new graduate as an engineer, and have been a project manager for the past two years.

Junpei: I’m in charge of customer success for Billing Management Robot. My mission is to improve corporate onboarding and their satisfaction.

Yonosuke: I work as a QA (Quality Assurance) engineer. I’m in charge of Billing Management Robot as well as applications provided on Salesforce and other payment services.

In the current climate, there’s no one to receive paper invoices at the office even if you manage to send it. Things are becoming more and more digitized so there’s a lot of demand for your services right now.

RYO: With the revision of the Electronic Bookkeeping Law on October 1, 2020, going paperless is becoming increasingly popular. Companies are working to implement a system in their back office.

Fixing issues to improve quality became increasingly time consuming

– What kind of issues did you have before introducing Autify?

RYO: We had many quality issues in the past. Fixing an issue would sometimes result in another issue, and there was a limit to how much testing we could do manually.

So three years ago, we decided to introduce Selenium IDE. At the time, we would manually check whether there aren’t any issues in the production environment immediately after writing and releasing all the scenarios. I’m sure people who have used Selenium will understand this but fixing takes a long time because errors such as Ajax (Asynchronous JavaScript and XML) errors would occur frequently.

Since it was difficult with IDE, we decided to write Selenium in code. However, it didn’t work depending on the environment, and we faced errors with unknown cause, too. The test code had to be modified each time according to the specification change. The man-hours kept increasing. That was the challenge for us.

– How many man-hours did it take to write the code?

RYO: With IDE, it took a three-person team a month or two to write code that covered all functions. We re-execute the part where the error occurred, but sometimes the error didn’t occur. If we can’t reproduce the error, we can’t determine the cause, which makes things very difficult. We would try adding weights and debug it or try to reproduce the error almost forcibly. Sometimes we weren’t sure if the underlying issue was solved even after spending a lot of time on it. We would just say, “I guess it seems fine now” and move on.

Customer success team who understands customer needs leads scenario creation

– You said that you considered Autify to solve the problem. How was the process up to its introduction?

RYO: I started off by telling the relevant departments about the issues we had. I explained how much maintenance cost was from using Selenium for quality control. Autify can make corrections automatically with AI analysis, so I explained that one engineer wouldn’t have to only work on maintenance. They could be assigned to development instead.

After consideration, we decided to go for it. We then started discussing which department would be in charge. Only engineers who could write code could be responsible until then, but with Autify, it can be done with a GUI. It doesn’t have to be done by someone who can write code.

We didn’t have a team that was solely responsible for QA at the time. As we were rearranging the quality control workflow, I suggested that the person in charge of writing the scenario should be the customer success (CS) team, who works closest to the user and understands the specifications.

Still, the responsibility of deliverables lies with the development team. We have decided to establish a system in which both the CS team and the development team work together.

Is this why Mr Yamashita was chosen? Because he has experience with writing code and is most knowledgeable about service specifications?

RYO: That’s right. He had a deep understanding of the user’s business, so I think he was most suitable. He knew that users would use the service in a certain way, so he could make suggestions.

**- It’s natural that the customer support team understands what customers have trouble with and how they use the tool. Were there any other candidates other than the CS team as to which department would be in charge?　**

RYO: In terms of ensuring service quality, other departments suggested that the development team should be in charge. On the other hand, the development team thought someone closer to the user should think about scenarios. They are both busy, so I’ll be honest, there was some disagreement here (laughs).

We didn’t have a QA unit that turned KPI into quality improvement, so after discussing that it would be good to have one, Mr Yamashita and myself ended up being in charge together.

Junpei: I wanted to learn about systems in general, such as coding and programming. I wanted to understand what kind of difficulty the development team faces. I was also interested in how to get the development team to implement the user’s request faster. So, when I was asked if I wanted to do it, I accepted it.

– Did you have any problems with resources, considering you had existing work?

Junpei: Having Autify didn’t mean that we would have less work. In fact, scenario creation and its operation were added to our to-do list. I delegated new tests to another member so that I could focus on those. It was certainly difficult to make those adjustments.

However, I think the reason why everyone cooperated was because there was an understanding among CS members that it would lead to quality improvement and eventually customer satisfaction. Another reason was that by having fewer issues, the CS team’s workload could be reduced.

Yonosuke: I wasn’t there at the time of this discussion, as I joined ROBOT PAYMENT after the fact. But when I look at it as an outsider, I think it was the CS team that had the most trouble. It’s the CS team who has to respond to inquiries when things go wrong.

Junpei: The CS team is the one that gets stuck in the middle! I really wanted to have as few issues as possible.

Peace of mind that the development team and the CS team could always check

– Did you have any creative ideas for the operation after you introduced Autify?

RYO: When creating the scenario, we first identified and categorized functions so that they could be covered comprehensively. After prioritizing each function and designing holistically, we started fixing the details as appropriate.

We scheduled the test to run automatically the day after release. We check for errors and correct the scenario if necessary.

– How do you cooperate with the development team?

Yonosuke: If there’s an error, the status and procedure will be displayed on Autify, so I share it on Slack and the development team handles it. There was one time when the batch system went down. Autify detected it and gave us feedback.

It’s linked with Slack and it’s set so that if it works without problems after release, I’ll receive an “OK” notification. I can see that it’s working properly at a glance, which I find reassuring. It gives me peace of mind that the developers and the CS team are always able to confirm that things are working properly.

One engineer can now be assigned to function development

– It’s been a while since you introduced Autify. Have you noticed any changes regarding the challenges you faced before?

RYO: Before we introduced Autify, we were manually checking if things were working properly. Given that there’s a new release once a week, it’s incredibly difficult to manually check every single case. With Autify, we’ve been able to reduce man-hours, and we can operate safely.

When we were using IDE, it was so difficult that we stopped halfway through. I think there are quite a few cases where maintenance is so difficult that we would just give up.

Since introducing Autify, we’ve managed to create a system where tests are run automatically, errors are detected immediately, and check happens simultaneously. This has been a great achievement. I think being able to develop with this reassurance is helping with faster development.

– How much maintenance man-hours have you been able to reduce?

RYO: Proper maintenance used to take one man-month, but now it’s been reduced to five man-days. That’s less than one engineer. There is no need for maintenance, and once it is made, every release is secure even if it is left alone. I feel that I don’t have to constantly worry anymore. Besides, it’s cheap.

– How about scenario creation?

Junpei: It’s very easy because you can create a scenario on the screen. I can use it smoothly without taking up man-hours and focus on CS work. Also, support has been helpful when I get stuck. I really like the new function that allows you to test and check on the spot once you have created a scenario.

– We are working on adding new features quickly. Do you have any technical matters that you are planning to work on?

RYO: I’d like to create a system that makes it easy to re-execute problems that occur during development. I also want to incorporate it into the CI environment. I would like to use Autify for checking before deploying, so that we can have a system in which we can release with more peace of mind.

Start small and eliminate the black box

– Finally, do you have any advice for those who are planning on test automation?

RYO: Trying to do test automation on your own tends to result in a black box that only a select few engineers understand. Engineers who are familiar with test automation are few and far between in the first place, so I think it’s better to replace it with SaaS and operate it. Hand-over issues and high cost can be solved by creating an environment where everyone regardless of department can check, write scenarios, operate, and understand.

Only a few companies can prove their service is cost-effective. Autify has a Micro plan so I suggest starting small. If you don’t see results, you can always stop using it. If you do see results, gradually automate more and more. I think that’s the key if you’re planning on automation.

– It was interesting to hear that you’re working to improve quality by engineers and the CS team working together. Thank you.