Products can’t exist without test automation: How Graffer achieves frequent release with their digitalization services for government organizations

Requesting paperwork, such as residence certificates and seal registration certificates, has become more convenient when the My Number Card was introduced in Japan. More and more local governments are going paperless, and various administrative tasks are being digitalized.

Still, there are many digitalization issues that government services face.

Graffer Inc. (Graffer) is a start-up that aims to solve these challenges by developing SaaS for governments. The company offers Graffer Smart Application, a system that allows users to apply for IDs and copies of their residence certificate by holding up their My Number card to a smartphone. Fees can also be paid electronically. Many local governments all over Japan have implemented their service.

Mr. Kyo Ago, Front Engineer, Graffer, Inc.
(Photo courtesy of Graffer, Inc.)

We interviewed Mr. Kyo Ago, a front engineer working on product development at Graffer, about development and test automation challenges.

There is a growing demand from governments and business due to the pandemic

– Please tell me about your business.

KYO: Graffer is a tech start-up. We provide SaaS mainly for local governments. Our services aim to improve the user experience for administrative procedures. It’s been adopted by local governments.

For private companies, we also offer services that are useful when applying for corporate certifications, for example.

Also, if a staff had to work on testing, it would mean there’s one less staff for application development. It was a vicious cycle.

– We use your service in our back office and we’ve been very impressed by how useful it is.

KYO: Thank you. We’ve received great feedback from many of our clients. With the pandemic, every citizen who visits the city office has a chance of spreading the virus. An increasing number of local governments are going paperless to minimize this risk. We have seen a rise in the number of inquiries about how things can be improved using our services.

– What does your work involve?

KYO: I’m a front-end specialist in charge of all technical aspects. I’m mainly responsible for E2E test management, library updates, CI acceleration and maintenance in the front-end. I also build libraries for the whole team. My latest interests revolve around security, accessibility, and quality control.

Maintenance was costly before Autify

– What challenges did you face before adopting Autify?

KYO: We would write program code to test each of our products, but maintaining them was very costly. It’s time-consuming to run each test, and it was labor some to set up and maintaining the test environment.

Developers wrote the code, set up the environment, and checked results, but that doesn’t cover the things in its entirety. When an error occurs in an E2E test, the only ones who understood the problem were engineers. Even if non-engineers found issues, they couldn’t write tests to solve or verify them.

At our company, the responsibility of implementing our services at local governments is handled mainly by the Biz Dev (Business Development) team. Our product is very versatile, so the Biz Dev team needs to have a certain level of technical skill. They also have to understand all the complicated settings and suggest them to users.

– Was testing done manually at the time?

Kyo: Engineers dealt with automated testing too. If the problem was relatively simple, they would quickly fix it. Then, the Biz Dev team would manually check to see if the fix was correctly applied based on the pre-release checksheet.

User rights aren’t defined

– What was the process like when you implemented Autify?

Kyo: We already had a foundation for E2E testing that our engineers created at a TestCafe. We’ve implemented Autify around it without forcing anything. I explained how to use Autify to non-engineers – Biz Dev members – by demonstrating how to run a test on the computer screen.

Autify’s UI is easy to understand intuitively, so we didn’t get stuck. They understood the tool quickly, so the implementation process went smoothly.

Once implementation was complete, some engineers said Autify was more straightforward, and we started using it in other areas. Half of our testing was written by engineers, and the other half was made with Autify.

– Autify aims to make things easier for our users, so it’s great to hear your positive feedback. What feature has been most useful for you?

Koji: I like the fact that you can create tests by interacting with the browser. It’s visually easy to understand. Another thing is that testing is stable. Until now, E2E tests that were manually created would sometimes fail even though there was no problem. We had to do occasional maintenance to prevent failures.

Autify doesn’t have that problem. You can just set it and forget it. If it fails, we know something is wrong and we can deal with it right away. The reliability of the system is what made it so popular with our engineers.

– You said that the implementation went smoothly. Did you have any creative ideas to successfully implement and operate it?

Kyo: We didn’t set strict user rights so that anyone could be added. It made Autify accessible, and that was one of the factors for gaining traction within the company.

I didn’t use Autify at first, so didn’t have a deep understanding of it. I intentionally didn’t make rules because the priority was for everyone to try it out.

It’s surprisingly rare for services to work as they should

– Did you consider other services before deciding on Autify?

Kyo: Yes. The initial assumption was that the tool would be used by non-engineers. So, we picked out and compared several services that are simple enough for non-engineers to use.

– What was the deciding factor for choosing Autify?

Seishi: The most significant factor was that it worked as it should when we tried it out on our service. It behaved normally without having to troubleshoot.

An example is when you bring up the slide-down menu by hovering on it. Other services didn’t support hover, so you have to right-click and press the “trigger hover event” button to make it work. Autify didn’t have that problem and worked as expected.

Also, other services were hard to understand. There were quite a few services that even engineers couldn’t figure out. I thought there aren’t many viable options other than Autify.

– We’ve put a lot of effort into engineering a product that works as intended, so I’m really happy to hear that.

Laying the foundation for E2E testing from the early days to release frequently

– How often do you deploy?

Seishi: Our company offers around 20 services and the most frequent one releases five to six times a day. If you add up all services, we deploy about 10 times a day.

– It’s amazing that government service can be deployed so frequently.

Kyo: I work as a developer, so it feels like developing any SaaS; it doesn’t feel like I’m creating a service specifically for local governments. It’s not like we can only release at night because it’s for governments.

At first, we only had a few services and engineers, so we didn’t release as frequently as we do now. As the number of services and engineers increased, the release frequency naturally increased with it. I think that’s been possible because we built the foundation for E2E testing from the get-go.

– What changes did you see after implementing Autify?

Kyo: I think it’s been great that we could add Autify to the range of tools that engineers and other members can use. Autify is a lot easier to check visually, so we can set it and forget it with Autify. It’s saved us a lot of money.

– How has maintenance been like after implementing Autify?

Kyo: Autify’s Maintenance AI feature has been invaluable because it automatically detects differences caused by updates and automatically repairs test scenarios. Maintenance is almost completely automated. We occasionally get a notification that the UI has changed, but all we have to do is take a look and click OK. We pretty much don’t have to do any maintenance.

– How are the tests made on TestCafe?

Kyo: A certain number of errors is inevitable with TestCafe. Of course, we could have greater stability by adding queuing time, but that doesn’t cut down on costs. For example, say you wait 100ms for each added test. That’s 10 seconds of waiting time if there were 100 tests in the queue.

Plus, you can’t cut down on time unless you determine whether adding 100ms is really necessary. So you can’t reduce the number of errors even if you improve the test environment. We used to adjust the queue time very carefully, and that made the maintenance cost quite expensive.

Testing is a must regardless of cost-effectiveness

– There are still very few start-ups that have automated E2E testing. Why did you invest in test automation from the beginning?

Kyo: I considered it a necessary part of the process. We have many experienced developers, so we have a common understanding that E2E testing and unit testing are a must and that it’s better to automate. No one disagreed about that.

I think the reason we have that company culture is because of how we hire. During the job interview, we ask questions such as “Do you have experience with testing?”, “What kind of testing have you done in the past?” and “What kind of problems did you have, and how did you handle them?” I think many people in our company have faced challenges and overcome them. Existing members helped establish that company culture.

Of course, each member’s experience and opinion about architecture design, supporting multilingual environments, etc., are also important.

– What do you think about the cost-effectiveness of testing?

Kyo: We knew that testing was a given. The product can’t exist without it, so we’ve honestly never considered how cost-effective it really is.

For example, when you ask people about using a cloud environment, they probably wouldn’t compare the cost-effectiveness of using a cloud environment and not using it. It’s the same with testing; not testing isn’t an option.

– It’s great to have a corporate culture that thinks that way. How do you think organizations can create a culture that values testing as part of the business?

Kyo: In my company, it was already like that when I joined the team, so I don’t know how to build it from scratch. If testing isn’t valued and you want to change that, you could try not testing and see what happens. You’d be struggling big time!

Ben Franklin once said, “the only thing more expensive than education is ignorance.” Ignorance is the best way to realize how cheap education is. If you think testing is too expensive, try going without it. The results would be blindingly obvious.

With testing, the key is to automate it. In our case, we use Autify. Testing is inefficient if you don’t automate, and you’d struggle without automation. Automation is a necessity for us.

If the code is testable, testing techniques are redundant

– Do you have a vision for how you want things to evolve?

Kyo: I’m still unsure how to delegate tasks between Autify and TestCafe, but I think TestCafe needs to be more stable. If I were to take all the tests we run on TestCafe and rebuild them on Autify, replacing them would be rather costly. It’s a tough decision.

Currently, there is a flow of obtaining an electronic signature with native apps on smartphones. We haven’t built a test for that yet, so I’m planning on adding it to Autify. It would be great if we could do that with Chrome extensions on Autify. It’ll allow us to test in a way that’s similar to our existing test flow.

Building a new system is very convenient on Autify, but migrating existing tests isn’t as straightforward. Autify is very reliable and always works as expected. That alone makes it valuable.

– Do you have any advice for those who are planning to automate testing?

Kyo: If you are going to automate tests, the best course of action is to make existing code testable. It’s a dull process, but start with code splitting and consolidate functions.

Many of our products are quite simple. For example, there aren’t many animations, and objects don’t fade in and out. I think that made the implementation process go smoothly.

Of course, you can create E2E tests in environments where testing is difficult. But that would result in too much E2E testing. It’s inefficient to try to check every single minor issue with E2E testing. Complicated code would require advanced testing techniques, but that’s not the case if the code is easily testable. I think the key is to make the code testable.

– Finally, do you have any announcements you’d like to make?

Kyo: Graffer is looking for individuals who are passionate about improving government services that we use on a daily basis. We are looking for specialists who are up for the challenge. You’ll determine the current quality, how it should be improved, and how to build tests to achieve it. Please take a look at our careers page if you are interested.

(Interviewer: Autify, Inc., CEO & Co-Founder Ryo Chikazawa)