How Autify Helps LayerX Build a Robust QA Process

LayerX, Inc. used to be in the blockchain industry but has since shifted direction to the BtoB SaaS industry. They now provide products that enable users to digitize business processes.

With the mission to digitize all business activities, the company has been rapidly expanding by providing bakuraku Invoice (Former: LayerX Invoice), an invoice automation tool, and bakuraku WorkFlow (Former: LayerX Workflow), which streamlines the approval process.

Both are finance products, making it all the more important to focus on QA to avoid system failures. In this interview, we talked to Mr. Masahiro Kajiwara and Mr. Masataka Suzuki, who are striving to keep the development cycle going while meeting customer needs.

Creating exceptionally easy-to-use BtoB products

Masahiro: Hi, I’m Masahiro Kajiwara. I currently* work as the QA and Customer Success Lead for two of LayerX’s SaaS products: bakuraku Invoice (Former: LayerX Invoice) and bakuraku WorkFlow (Former: LayerX Workflow). (*as of October 2021)

Masataka: I joined LayerX in August as an application development engineer working on the server-side. Testing was beginning to put a strain on the team at the time, so I’m currently semi-working as a SET (Software Engineer in Test) and creating tests on the side.

Masahiro: LayerX has three businesses, and the SaaS division has the largest number of employees. The SaaS division provides two products that promote DX (digital transformation) in the back office, such as accounting. Our vision is to deliver products that are exceptionally easy to use and make work exciting.

bakuraku Invoice, which we released in January 2021, is a SaaS that allows users to receive invoices with zero manual input. Many products already deal with issuing invoices, but there aren’t many that automatically process received invoices. When I used to handle accounting tasks, receiving invoices had to be done manually. bakuraku Invoice was developed after interviewing about 100 companies and defining the needs we will meet.

We also released bakuraku Workflow in April 2021. This is a platform that makes the internal approval process quicker. We aim to quickly respond to user requests by developing these products. However, a multitude of QA issues began popping up along the way, and we’ve been working on them.

QA takes up 20% of development resources

– So, you started focusing on QA after releasing bakuraku WorkFlow (Former: LayerX Workflow)?

Masahiro: Yes, that’s right. Finance products are linked to invoice data and accounting software, and it’s an area where quality is critical. Unfortunately, we didn’t have a dedicated QA manager, so engineers, business managers, and product managers had to make test lists and run them manually.

We release once every two weeks. We first release to the staging environment, and the production release happens two days later. We run tests and fix bugs in the two days leading up to the production release.

– It’s pretty rare to take on QA while working on CS. Did you become the QA manager when you realized that QA is something that needs to be dealt with?

Masahiro: We simply didn’t have enough people and couldn’t choose who does what. All of us did what needed to be done. I used to work in the blockchain division, but when bakuraku Invoice was released, I moved to the _Invoice _team. I did sales and CS, a little bit of QA as well. All of us scrambled to get things done as a team. I quite like sorting things out in a chaotic environment; I got things under control and started improving the QA process.

– Masataka, you started out as a backend engineer, but then you were assigned to QA tasks as well?

Masataka: Yes. Testing is essential for preventing development from slowing down. It also helps us to understand specifications. Initially, my role in QA was supposed to be temporary, but now the plan is to keep going.

Masahiro: The reason we’ve allocated valuable development resources to have a SET (Software Engineer in Test) was that, back in March, we had three bugs in one production release. Fortunately, none of them were critical bugs, and we were able to catch them before customers used the platform, but the team felt a sense of urgency. Three bugs in one release was simply unacceptable. Obviously, speed is important, but it became clear that we really needed to work on quality. We decided to allocate 20% of our development resources to QA for a few months. We also started looking into using Autify.

Improving quality while keeping up the release cycle

– You were already aware that quality was an issue. It sounds like the bugs in March were the turning point.

Masahiro: The number of items we needed to check had ballooned to 500-600. I buckled down and tested as much as I could. Depending on the accounting software, one page can have different views, which means the linked parameters are different too. For example, the same feature could have six QA items depending on the accounting software.

We develop quickly, so it was becoming almost impossible to continue testing everything in two days. Speed is our strength in our business, and we didn’t want to compromise it. So, I had to find a QA tool that allowed us to do QA checks in the same two-day time span. If we were to run E2E tests as well, we would soon have over 1000 items to check, given our development speed. I decided to take action to make things more efficient.

– At first, you considered automating the testing process in the same timeframe, which was the two days you had in the staging environment before the production release.

Masahiro: That’s right. We also considered which areas we should create unit tests and API tests for. We will be adding important features, and any bugs in those features would cause serious adverse effects to the client’s workflow. Regardless, we were already aware that our quality assurance process needed to cover more ground.

– Up to that point, your main priority was to be agile. It sounds as though you also started working on improving quality as well.

Masahiro: That’s right. I’m sure it’s the same for other companies, but when you’re only starting up and adding a lot of new features, you don’t really have the luxury to allocate resources to create tests. Any available engineers were assigned to feature development. So, given the lack of resources, I started looking for ways to automate testing and make E2E testing more efficient while keeping an eye on where the business was going.

Why LayerX chose Autify after trialing seven to eight test automation tools

– How did you end up choosing Autify?

Masahiro: I consulted an engineer who’s knowledgeable in automated testing and had a quick look at seven to eight tools. Besides E2E test automation tools like Autify, we also considered Selenium, which you can’t use without developing the tests yourself.

Our requirement was that anyone, not just engineers, could use the automation tool. We narrowed it down to four choices and tried them. We spent two full weeks testing out two platforms in particular. We considered whether it’s easy for non-engineers to use and how many test items for bakuraku Invoice the platform can automate.

Initially, I didn’t expect to automate everything because some things are rather tricky. For example, can the tool test a feature, which is a checkbox when you look at the HTML screen, but the actual design is a switch? Can you upload CSV files? Can you test user registration emails? Can we check that something has been downloaded? We listed all these items and tried automating about 20.

– You mentioned that you tried automating a list of several test cases during the trial.

Masahiro: I remember being extremely excited about choosing a tool! Since we are a startup, I’m used to things changing all the time. Plus, changing and improving all sorts of things is how we work. However, when it comes to testing platforms, it’s hard to take it back once you’ve made a choice. If you create Test Scenarios with one tool, you can’t just export them and import them to a different platform. I put a lot of thought into my choice, thinking about whether we can stick with it long-term.

– What convinced you that Autify was the long-term solution you were searching for?

Masahiro: First of all, we were able to verify that Autify can do what we wanted it to do. Of the 700-800 test items we had at the time, we had hoped to automatically test about a half. The other half were things that simply had to be checked by humans.

The reason I said about a half is that there were items that could be automated depending on the tool, but it would probably be difficult. Also, we can easily make sure that basic core features work, so automating them wasn’t strictly necessary. It would have been sufficient if the tool could automate about 30% or 250-300 items. We found out that Autify could automate about 400 items if we tried, which is just over half. Autify’s functionality was satisfactory.

I also contacted CS several times a day during the trial. I was able to get a response right away every single time, which helped me resolve issues one by one. It was reassuring to find that the CS team was responsive. It was one of the biggest pros. Another key point that, as a non-engineer, I personally found it easiest to use. I like that it was quick and easy to learn.

– Why was it important that non-engineers could use the tool? Was it because engineer resources are limited, so everyone would have to use the tool?

Masahiro: Yes, that’s right. I would prefer engineers to develop new features and improve the product. In the meantime, I wanted the QA manager to cover weak points. It would be a wiser way of using resources. Autify has been very helpful.

– Were there anything you weren’t sure about?

Masahiro: I wasn’t sure how much Autify could handle in terms of API testing.

I’d seen that non-engineers could use Autify to quickly automate E2E tests, so I wondered how far it could go if engineers used it. For example, if we wanted to embed a Scenario into an E2E test to check the response after sending a request, we would need to come up with a creative solution.

Improving test cases so that anyone can work on QA

– Once you made your decision to use Autify, how did you go about implementing it?

Masahiro: From April to June, we buckled down and improved the efficiency of our QA processes so that we wouldn’t get overwhelmed. Autify was one of the main ways we were going to achieve that. It was absolutely vital to check that the scenarios were correct and complete before automating them on Autify. So, we examined existing QA items very closely. We tidied up the tests by adding items that we had omitted or combining items if they were too detailed. For example, combining three items into one.

It’s also important to paste a screenshot of the expected result into the QA table and prepare test files for uploading. We spent a long time on this stage, but it almost halved the time spent on QA. It was much better than we had anticipated.

If you create the QA process that only works for people who know the specification, new members or anyone helping out will have to work on QA while feeling unsure if they’re doing it right. I don’t know if our way was the right way, but things got a lot faster once we organized test items and simplified the QA process.

– It’s impressive that the QA became twice as fast.

Masahiro: I was astonished. Our organization has grown rapidly, and new members are joining us all the time. I don’t think we would have coped if we had stuck with the old QA table. When I ask everyone to work on QA, new members and interns join in, which allows them to understand the specifications.

– That’s great!

Masahiro: We prioritized anything that we had already decided to automate, as well as parts that could be automated or were important. Then, we started automating them with Autify. We also focused on whether the scenarios created on Autify are appropriate. In other words, we evaluated the QA items before creating the scenario. We also allocated time for two people to review scenarios we’d made.

– How did you juggle reviewing test cases and creating scenarios on Autify?

Masahiro: We did them both at the same time. We didn’t have the time to create a detailed timeline. We just dove right in. We aimed to automate 300 items within three months of introducing Autify. Halfway through, I handed this project over to Masataka, who came in like a savior!

Masataka: When I joined, tests for bakuraku Workflow _hadn’t been automated at all. _Workflow has complicated approval paths, and I began automating those first. There were too many items to check, and we’d been using several scenarios in rotation at every release. By using Autify, we can now test all scenarios every time. We’ve gotten used to making scenarios on Autify now, and so far, we’ve automated at least 272 items.

Masahiro: When we say ‘path,’ it’s a feature with almost infinite numbers of patterns if you try to create a test for each one.

It was getting beyond the scope of what humans could handle. Something like this is difficult for humans to do, and it’s also mentally draining. I started working on it because I thought Autify would make our lives a lot easier.

The priority was to start from features that needed to be tested and ones that I expected Autify could handle. However, after a while, I began picking features that were exhausting to check manually. Checking 700-800 items in two days is a lot. Plus, the number of items keeps increasing at every release. It’s overwhelming. I managed to automate a lot of it, which made us feel a lot better.

Using JS Steps to increase test coverage and improve accuracy

– What’s the secret to automating successfully? Which Autify features did you find most useful?

Masataka: When I started creating tests, we also decided to use the email test feature. When testing paths, an approval email gets sent automatically, and we had to step in just to check that part. Knowing that you have to check one part of an automated test is irritating. Ideally, we wanted a tool that could do the whole thing.

Autify develops quickly, so when I chose to implement it, I’d expected an email test feature to be released soon and make our lives even easier. Indeed, Autify did come out with the feature, and we no longer have to visually check emails. It’s made testing painless.

– What about API tests?

Masataka: I’m originally an engineer, so I use JS Steps. It allows you to do a lot more if you can use that feature. Currently, we use Autify to run small-scale API tests. Assertions are important for E2E tests, but there is a limit. For example, a record you just created and registered may not always be at the top of the list. If you check whether the record exists in the table and a record from a previous test is left, you might end up with inaccurate results.

One time, we found a bug where the same record was registered twice, creating a duplicate. This sort of bug is hard to find with an assertion. Searches are also an issue. You can check that query results show what you searched for. But with E2E tests, it’s difficult to check if nothing else is displayed. Even if it’s easy for humans to check something, creating an automated test that can do the same thing accurately is difficult.

Meanwhile, API tests aren’t good at checking uploads and asynchronous tasks. My solution is to incorporate API tests with E2E test scenarios using JS Steps as appropriate. We’ve been able to create scenarios that give us accurate results while covering the weaknesses that E2E tests and API tests have.

– When you started, you didn’t plan to use JS Steps very much. What made you change your mind?

Hiroshi: Some said it might be better to edit and update scenarios even if it takes up the engineers’ work hours. Once we realized that we needed APIs, we considered whether to implement a new tool or not. Instead of using an API test tool, it was easy to run API tests with Autify.

Masahiro: I like that we can embed API tests in scenarios. At first, I didn’t think Autify could handle API tests, so I considered Postman and Newman. Then, Masataka said, ‘if we can write JS Steps, maybe we can write API tests with Autify’s JS Steps too.’ We tried it and realized that Autify can run API tests!

Fix bugs early and release with confidence

– Before you implemented Autify, you wanted to keep the length of your release cycle and get the tests done in two days. You also mentioned that you wanted QA to be easy for non-engineers. What kind of changes did you see after implementation?

Masahiro: The most significant achievement is that we’ve been able to maintain the release cycle. It’s a big deal that we can still test everything in two days. That’s more important to us than how much we’ve saved in labor costs. So far, we’ve automated over 1,000 items on bakuraku Invoice and 600 items on bakuraku Workflow.

We don’t test every item every time. We select which items are tied to the new feature or have been affected by it, usually about 400-500 items on Invoice and about 200-300 items on Workflow. Out of 600-700 test items, close to 300 are automated. All we have to do is click the button and run it. Of course, there are times when we have to check the results and update the scenario, but knowing if there are any bugs early on is a huge advantage.

In fact, we wouldn’t be able to complete testing in two days if it wasn’t automated. We have to release in two days. There won’t be enough time to fix bugs before release if we can’t finish testing most items and identify bugs in a day and a half. That’s why it’s extremely helpful that we can run tests automatically and know if there are any bugs ahead of time.

Masataka: In fact, we need to find bugs early so that we have more time to fix them. It allows us to release our product with confidence.

– Absolutely. The sooner you test, the sooner you find bugs.

Masahiro: Once you get used to Autify, maintenance is straightforward. If you wrote code with Selenium, you would have to check every single code. It leaves you wondering if there’s any point in automating if it takes so long to maintain.

Masataka: Maintenance is easy because Autify automatically tracks changes. When we run a scenario, Autify tracks UI changes and tells us if anything changed by displaying the Review Needed icon. When I see that icon, it reassures me that changes are reflected correctly.

To further improve the development cycle, run pre-tests before testing on the staging environment

– What’s next for your team?

Masataka: We will continue to release new products in the future. We call it ‘extremely speedy development’ at our company. If we want to develop quickly, it needs to be of high quality. Otherwise, maintenance and bug fixes would take up so much time, ultimately slowing us down. We will continue implementing automated tests early in development so that we can keep creating products quickly and ensure that our users can use them with confidence.

Masahiro: Currently, we only use Autify in the staging environment in the two days we’ve allocated for QA. I hope to run tests regularly in the production environment in the future. And if branches are merged in the development environment, I want to make relevant scenarios run automatically. We can develop more confidently if tests are run regularly in each environment, and it’ll help accelerate the release cycle.

– If you can start testing earlier, the last two days could be like a backup.

Masahiro: Ideally, I’d like to use the staging environment for final checks. I’ve also recently started testing on Edge in addition to Chrome. My plan is to expand test coverage.

Automating E2E tests isn’t the only thing we are doing for quality assurance. There are non-functional requirements, not just functional ones. We need to hold ourselves accountable for anything related to quality. Sometimes we do this in collaboration with other infrastructure teams.

For example, I want the QA team to lead vulnerability assessment and penetration testing. There are so many things I want to do, and I think automating E2E testing with Autify will play an important role.

Any product should automate ASAP

– What advice would you give to those planning on automating tests?

Masahiro: Autify has become a tool that supports the growth of our product, and it’s the best tool we’ve implemented in 2021. It’s been extremely helpful. I’m sure I can give insights from a user’s perspective, so if you’re considering Autify, feel free to contact me on Twitter.

Masataka: Any product will benefit from automated testing because it will result in faster development, and engineers will have peace of mind. I recommend that you automate testing as soon as possible.

– Finally, do you have any announcements?

Masahiro: Our B2B SaaS product has gotten off to a great start, which I’m grateful for. As the organization grows, we hope to continue providing many companies with a solution that makes work exciting.

I believe that the QA’s role is to protect the solution. If the QA initiatives we discussed today interests you, we want to hear from you. We are currently hiring!

(Interviewer: Ryo Chikazawa, CEO and co-founder, Autify, Inc.)

‍