Build a team where all members can ensure quality. The future that Money Forward Kessai, a BtoB Payment Service, envisions.

With business-to-business transactions, managing billing and payment is very important. However, the truth is that these tasks become increasingly labor-intensive as the business grows. In particular, having have to remind clients of unpaid invoices can take a huge mental toll.

Money Forward Kessai is a settlement agency that deals with inter-business payments and transactions.

Money Forward Kessai Inc. was launched in 2017 as a new business born from Money Forward Inc., a cloud accounting service provider. We interviewed Mr. Yuki Shinohara, the CTO, about the issues they faced with quality assurance and how they solved them through test automation.

Reducing transaction costs to build a solid foundation.

– Please tell me about your business.

Yuki: Money Forward Kessai is a new company established by Money Forward in 2017. Until then, Money Forward’s corporate businesses focused on cloud services for the back office. Our company was launched so that we could concentrate on corporate payments.

We offer a settlement agent service, Money Forward Kessai, which facilitates various tasks related to business transactions, such as credit screening, billing, invoicing, and shipping. If the payment has not been made, we remind the debtor.

Credit screenings are available for corporations and sole traders, and most of them are automated. We have built a system that can return results in as little as a few seconds.

– Your company provides software that systematically solves the business operation with automation?

Yuki: That’s right. The client interacts with Money Forward Kessai when entering transactions and billing information or uploading them as a CSV file via our web service. They can also enter it through the API.

Companies that have introduced the API also wanted system integration, so screening results can now be notified with webhooks.

Our clients have different circumstances; they may or may not have engineers who can handle development. Our development system considers those demands and provides value to our users.

– What is Money Forward Kessai’s payment model like?

Yuki: We are primarily fee-based. The fee depends on the amount of money entrusted to us.

For example, for a claim of 1 million yen, a fee of 0.5%-3% is paid on the due date.

However, there are times when a client wants the payment quickly, such as when the company’s circumstances have changed. For such requests, we offer an early deposit service for an additional fee.

Our idea is that Money Forward Kessai helps remove the anxiety and administrative costs that come with inter-business transactions and create a solid foundation that allows them to focus on providing their products and services.

From Infrastructure to Quality Assurance

– Did you become the CTO around the same time as when this service was created?

Kyo: I joined Money Forward in December 2016 and worked as an SRE (Site Reliability Engineer).

When I graduated from university, I started working as a web app developer at Yahoo Japan Corporation. I later changed jobs and became a full-stack developer at a venture company. That’s when I became very interested in infrastructure, productivity improvement systems, and DevOps; things that are now called DX (digital transformation).

AWS was not being used when I joined Money Forward, so I thought if we were to increase development productivity, the old development and operation system would have to change.

I didn’t have a core mission when I first joined the company, but I became close to the founding members of Money Forward Kessai (then MF KESSAI) and started advising them on the service they were launching. I ended up becoming the executive for Money Forward Kessai.

– Working with infrastructure and DevOps led you to quality assurance.

Yuki: Exactly. On the subject of productivity, it’s not easy to incorporate the front-end into an iterative development model. Some companies manage E2E tests with code using tools like Cypress, but the code can break and hinder the iterative process.

That’s why we couldn’t work on E2E tests. The least we could do was to check if it worked after deploying.

Of course, we checked whether new features and changes worked correctly, but one of the most common problems we faced was that we couldn’t check for regression or whether a gRPC update didn’t cause any issues on the front-end. We knew that not having a comprehensive system would overwhelm us down the line.

We wanted to create a system where repetitive tasks were automated

Yuki: Needless to say, not being able to refer to something or seeing the status would be a problem. However, for Money Forward Kessai, the most important part is the transaction registration feature. We can’t have any issues there.

We used to check the system’s integrity and behavior from logs, metrics, and database monitoring. In particular, log and metrics alerts won’t happen unless there is a reaction or action between users.

We need to take action before the user does, but we’re all human; we sometimes forget. This can lead to productivity loss, so we wanted to create an environment where repetitive tasks can be done mechanically.

Another issue was that because we use Kubernetes, infrastructure rebuild events occur regularly. When this happens, the rebuilt one would be a newer version. The SRE can’t guarantee that everything is working correctly.

In those situations, we would ask our engineers to check all applications. From the application side, checking the tasks that the SRE worked on is an interruption. Tasks would take longer to complete because we couldn’t check ourselves.

– How often do you release?

Yuki: There’s a microservice element to it, so some releases affect users and some don’t. As a cycle, we release dozens of times per day. If you combine all the teams, there are about 20 releases for each Kubernetes.

Given that we release dozens of times every day, we can’t always run regression tests. That’s why automation is inevitable.

The importance of testing increased as the business grew.

– Did you try any other services before implementing Autify?

Yuki: We tried Cypress first. But a select few became specialized with Cypress, and other people couldn’t maintain it.

There are simulators and emulators, so they could make it work to an extent. However, it seemed like a lot of work to manage code and build an execution environment each time we created a new service. It didn’t stick.

In our case, minor changes and UI changes occur frequently. The web service keeps changing while the test case broke. The test case couldn’t keep up with how often things changed.

There were times when we’d think, “maybe it’s too soon to automate,” or “shall we keep doing it manually for a while longer?”

-In other words, maintenance was the issue?

Yuki: That’s right. We weren’t at the stage where we could hire a QA engineer, so we just did it all by ourselves. There was also a situation where I didn’t have the time to write test code for testing the UI.

With Cypress, it wasn’t as if all the development engineers agreed to try it. Instead, there was one person who would try it out and give feedback. And the feedback was that Cypress is just too labor-intensive.

If it was 2 or 3 years ago, we were still a small team so we could check the behavior thoroughly. At the time, we thought we could keep checking manually. Also, the whole team was developing the same product back then, so we could check each other’s work.

Regression didn’t happen very often at the time. But in reality, we would sometimes find errors on the back-end from user feedback.

Now that Money Forward Kessai has grown as a business and provides multiple services to users, one change will affect other teams’ products. So, it became critical that we perform thorough tests, which is why we began using Autify and other tools for testing.

I want to create a world where anyone can write a test.

– Having considered other test automation services, what made you choose Autify?

Yuki: The most significant factor was how powerful the auto-healing function is. It’s beneficial because there are often small changes in the UI with our services. For example, if we receive user feedback that something is hard to see, we would change the display. Autify’s auto-healing was so powerful that we found that it could handle various changes that we were trialing. So we decided to go ahead and implement it.

It’s part of our branding, but ultimately we want to create a world where non-engineers can write tests too.

If everyone could write tests, issues don’t necessarily have to be solved by an engineer. Whoever has the time can take on QA tasks. I think it’s better to make it so that the whole team can work on QA.

If E2E tests are managed with code, the work becomes something only engineers can do. On the other hand, if you were to test manually, I think anyone would be able to do it. If we made a manual or if the whole team had a common understanding, I could check the behavior as a PdM. Or the designer could take it on. There will be more cooperation.

With a service with a powerful emulator like Autify, I think we can make it so that everyone in the team can handle testing. Autify told us that their vision is to create a world where everyone can write tests. We wanted to help make that a reality, and we wanted to see it through. That’s why we ended up implementing Autify.

Product teams and SREs use Autify for testing.

Yuki: Small mistakes by the product team and the SRE can lead to bugs, which cause trouble for users. For example, an error when changing the network layer and security layer can cause a significant issue.

Even in those cases, we may not notice it until a user contacts us, or if it’s from a release that happened in the middle of the night when users aren’t using the service, we may not notice it until morning. It’s difficult never to miss errors, even if we perform operation checks ourselves.

After replacing the Kubernetes cluster, I did a full test with Autify and was able to verify that there were no issues. I think there’s a sense of security from implementing Autify, not because someone within our team has made the change but because other team members can confidently make changes.

Set expectations for E2E testing and create a course of action

– Now that you’ve been using Autify for a while, have you come up with any tips and tricks?

Yuki: No matter how user-friendly the service is, people will be reluctant to learn it if you tell them to use it without guidance. So, we started by having the SREs use it for problem-solving. After building the framework and the foundation, we gradually taught each team how to use it and maintain it.

We tried using it regularly with Money Forward Kessai’s product to have at least a basic understanding of how to use Autify. Then, we set expectations for what we expect from E2E testing and what we don’t. Then, we gradually increase the number of members who have hands-on experience with Autify.

We needed to understand what it can and cannot do. Knowing what it can’t do was especially important because it would be a waste of time if everyone got stuck on the same issue.

Someone said on Slack today that it’s straightforward than they expected. We share our opinions about it to understand it better and motivate each other.

– How do you differentiate what you verify with E2E testing and manual testing?

Yuki: In our case, we decided at the very beginning that we will not use Autify for every single test.

We regularly run tests for areas where there’s an integration between Money Forward’s services. For example, an event is fired when a new client issues the API key. We test this thoroughly because it’s a vital part of the service that must not fail. Events that only happen once in a while are tested regularly.

We also test the essential parts of the transaction registration function every day. We prioritized creating tests for critical areas where any issues would cause trouble for users. This is due to the nature of our service, but we can’t make it so that an invoice is sent on the same day as the transaction is registered. If a flow takes more than a day, we will test manually.

The status changes based on the flow; from registering a transaction, sending an invoice, to receiving payment. It would take a whole month if E2E testing handled this. We’ve come to terms with the fact that this is something we have to test manually.

We decided that E2E testing is only for things that can be done on the web. Having a rule like that saved us a lot of headaches. By reducing the workload for testing to 70%, you get to save 30% in costs every day, and productivity will increase.

Once everyone starts using it, I’m sure there will be areas where we cannot automate. I plan to keep automating to keep reducing the workload by considering it on a case-by-case basis. We’ve only just begun.

– Each microservice uses Autify when it’s merged with commit hook in the repository, right?

Yuki: Merging after the tests pass in the in-house environment would be ideal. However, due to issues with the in-house environment, we started with a quick check after deployment and fix issues as soon as possible.

It would be nice to run auto-rollback too, but we haven’t had the issues that have needed that so far, so it’s not a priority. How we will proceed in the future is an issue that remains to be seen.

Regular testing resulted in detecting issues early.

– What changes have you seen since implementing Autify?

Yuki: There are a lot of new companies these days. A while ago, it was Firebase Authentication. Now, it’s Google Cloud Identity Platform. With AWS, it’s Cognito and Auth0. Or even Okta.

With all these ID authentication services popping up, there’s an increasing number of companies that use third parties for login authentication. Those companies don’t manage IDs and passwords themselves. With microservices, it’s common to only do an operation check after deploying, but there will be events where that’s not enough.

When you implement a SaaS, especially SaaS that handles login, there will be issues where users cannot even log in. If login is the trigger, there will be no alert unless the user takes action, so it takes some time to realize that they couldn’t log in. The user would have to contact customer support, and that doesn’t give a good impression.

Thanks to Autify that allowed us to build a system that runs tests regularly, we can see that there was an issue with Firebase, for example. Incident alerts come up in reports, but you can also see that errors are happening internally at levels that reports don’t show. You can find issues like Firebase returned an error 500, but it didn’t show up as an incident.

If we can detect those issues early, customer support can let clients know about them. Autify plays an essential role as an early detection system.

– As more people take on responsibilities, it becomes increasingly difficult to know where the issue is coming from. It’s great to be able to cover that without E2E testing.

Yuki: The things I just talked about are still relatively basic, so we could have used Datadog for monitoring and checking. But maintenance would become costly if we were to write separate test cases that are similar in content. So, for us, Autify is helpful because it’s a one-stop shop for monitoring everything, including login and user actions.

The development team is focused on development. An upward spiral that creates new value.

– Finally, what are your plans for the future?

Yuki: I’d like to explore how we can automate, streamline, and speed up the DevOps cycle. We are also planning to automate tests for areas relating to security.

That includes detecting vulnerabilities in the code and ensuring that the containers generated by Docker are not vulnerable or that there is no unintended code caused by someone changing a file. I want to test everything we’ve produced before deploying so that everything’s looking good.

I think the anxiety and fear will dissipate if we can test before deploying. We can be confident because the machine tests everything, and we can focus on development. I think it will lead to an upward cycle where developing becomes fun, creating more value. It’s a worthwhile investment.

– You can create an environment where engineers can focus on development by automating tests.

Yuki: I think the performance of the application is a part of that. Even if you improve and deploy, it would be pointless to connect it to the production database if the performance worsens. I want to make it so that you notice any performance issues early on.

Even with SQL, the key is how you measure performance. It’s not just about the current performance data. The load will be different half a year from now. The question is, how do you deal with a situation like that? If we can solve those issues, we can guarantee the quality of our output in the long term. It would be nice to see the development cycle go faster without creating liabilities.

– Do you have any advice for those who are planning on working with test automation?

Yuki: This is a cliche, but E2E testing is no silver bullet. It’s not going to work well if you don’t have a company-wide guideline. Automation shouldn’t be the goal. Consider what issue you’re trying to solve and go from there. If you approach it that way, you’ll be glad you implemented a test automation tool.

Think about what would make you happy if you could automate. What would be the best way to design the automated test for your circumstance? That would make the process go smoother.

You may look at case studies and think, “if they’re doing it, we should too.” However, it’s important to realize that even you run a similar business, the product’s structure or the approach might be completely different. So, naturally, the way you go about making tests would be different. If you fail to recognize this, the implementation may be difficult, and in some cases, productivity may even decrease. Considering your product is key.

– Finally, do you have any announcements you’d like to make?

Yuki: We are hiring back-end engineers who will develop the core area of our payment service. It’s a position that allows you to work across roles as you see fit.

We’re also hiring SREs. In our company, we value making the most of cloud services, but at the same time, we don’t hesitate to purge things that aren’t important. The job is to improve software engineering based on operation cost and productivity.

Inter-business payments have increased its market value in recent years. Many companies cannot step into the market because they will not have the cash to buy bonds. It’s a tricky business to start without credit and capital. While it means there are not many competing startups, it also means that issues don’t get solved.

Money Forward Kessai will continue to help companies grow their business in the core area of business transactions. We are looking forward to hearing from those who are keen to create value with us.

(Interviewer: Ryo Chikazawa, CEO & Co-Founder of Autify, Inc.)