Managing test specifications with Autify: how Nulab tackles QA through test automation

Nulab, Inc. develops several SaaS products that improve business productivity, including the project management tool Backlog, which has over 1 million users.

Nulab’s services have helped many companies move their projects forward, visualize the team’s ideas, and collaborate.

As a company that develops and operates multiple services, each with many clients, what issues do they face in terms of quality assurance, and how do they address them? We interviewed Mr. Makoto Hirayama and Mr. Teruo Kunihiro of the Service Development Team at Nulab.

Providing team collaboration services, globally

Can you tell us a bit about your business?

Makoto:  At Nulab, we develop and provide products such as Backlog, a project management tool; Cacoo, an online diagram tool; Typetalk, a chat app for teams; and Nulab Pass, an optional service that improves the organization’s information security and governance.

We offer services geared toward teams with the goal to “make team members across the world glad that they’re a part of their team.”

What are your roles?

Makoto: I am a QA engineer at Nulab. I oversee our services, of which we have several.

Teruo: Hi, I’m Teruo Kunihiro. I’m responsible for the development of Backlog, from the infrastructure to the front end, and I use Autify for E2E testing.

How is the team structured in terms of quality assurance?

Makoto: I’m solely responsible for quality assurance (QA), introducing and promoting the use of QA tools like Autify. Each development team does the actual testing. Over ten people have an Autify account, and we use it for many of our products, not just the one Teruo oversees.

Many staff work on developing Nulab’s services, and E2E testing is essential. But manual testing is labor intensive.

What was it like before implementing Autify, and what obstacles did you face in terms of QA?

Makoto: For some projects, we used frameworks such as nightwatch.js and Selenium WebDriver to run E2E tests.

However, they require knowledge and it’s quite labor-intensive to prepare the tests. It’s all fine and dandy until you finish introducing the tools but roles and responsibilities gradually become unclear; who’s going to maintain the test environment? Eventually, things just get abandoned. To put it simply, the frameworks were too difficult for us to utilize effectively.

At the same time, I also think E2E testing is necessary because it allows us to verify from the user’s perspective that our services are working correctly. We wanted to work on E2E testing, but we were stuck figuring out how to maintain and manage it.

Teruo: Several engineers are involved with developing Backlog. As each team develops new features, a feature that one team made would sometimes affect another. We wanted to run E2E tests to ensure this wasn’t happening. As soon as I joined the team, I created and managed test specifications, but continuing to do so manually would be too much to take on.

Did each team have to test their own feature when another team made changes?

Teruo: Initially, I would check the Git repository once a day to see if there were any changes in the code base and target files I was in charge of. I decided whether to perform an E2E test depending on the scale of the modification. I usually ended up testing manually twice a day.

In the long term, it would be hard to keep doing everything manually.

Teruo: Exactly. It was around that time that we started talking about implementing Autify company-wide. I tried Autify and found that I could create test cases the same way as manually running E2E tests. We began implementing Autify as if we were going to test everything with Autify.

The deciding factor was being able to create tests without coding and the ability to test login authentication.

What led you to implement Autify?

Makoto: It all started when Gyori from your marketing team asked someone in our company whether we’d be interested in trialing the alpha version of Autify for Mobile. It had already been decided that I would be in charge of QA the following year. Intrigued by their proposal, I decided to learn more about how Autify can automate testing.

Mobile is not the main focus of our product line yet, so we decided not to implement Autify for Mobile, but after learning about Autify for Web, we thought it would be invaluable. So we decided to try it out.

We were given a month to trial it, so we put it through the wringer. Just because I like it doesn’t mean the whole company would adopt it, so I created a demo video for my colleagues and held a briefing session. Using the tips I got from Autify’s study session, I’m happy to say that the session was a success. My colleagues liked Autify, and we decided to implement it.

What were the deciding factors for choosing Autify?

Makoto: It says on your website, but one of Autify’s biggest strengths is that you can create test scenarios without coding. Another reason was that Autify can automatically test our two-step authentication system. We are improving the login security of our services, so whether Autify can test it was a key factor. During the trial, we found that we could automate tests from login to the behavior of our features. This gave us confidence that Autify was the right choice for us.

Autify can also test email authentication and authenticator apps, so please try it out.

Easy-to-use UI helped Nulab automate 70% of testing.

After you decided to implement Autify, did test automation go smoothly?

Makoto: After the internal briefing session, we asked which teams would like to use it. The Git team, which Teruo is a part of, and several others raised their hands. Since then, more and more members have begun using Autify.

Teruo: I scheduled some time for making test scenarios. I had no idea how much Autify could do, so I played it by ear at first. The process of automating manual tests with Autify went smoothly, and I gradually learned what Autify can and can’t do.

The UI is intuitive and easy to understand, so it was pretty straightforward.

Makoto: Now that we have several teams creating scenarios for us, we hope to get feedback and information from them and expand its use to other teams. Autify’s Step Group feature makes it easy to create scenarios that other teams can use, so I plan to encourage my colleagues to take advantage of those scenarios.

Do you use Autify in any creative ways?

Teruo: The goal is not to automate E2E testing per se but to manage and maintain test specifications and execute them on a regular basis. Whenever we automate using Autify, we always try to link the scenario with the corresponding test specification. Specifically, we paste the URL of each Autify scenario into the test specification. We still have some test cases that don’t have a URL and have to be run manually.

How much of that specification have you automated with Autify?

Teruo: Around 70%. There are a few things that Autify can’t test, such as webhooks, but I think we’ve done pretty well.

Basic tests are easy on Autify, and customization comes in handy.

When and how often do you run regression tests?

Teruo: We run them weekly, every Monday at 9:00 a.m. I get nervous when I go and check the test results on Monday mornings. If there’s a failed test, my schedule for the day will be ruined!

I’ve spent a lot of time building tests but not so much on managing them. I want to make tweaks as appropriate so that flaky tests become more stable, tests finish faster, and we can test more frequently.

Are there any features you’ve found especially helpful on Autify?

Teruo: We use Step Groups and JS Snippets for inserting JavaScript (Autify JavaScript Snippets) quite often.

Autify doesn’t depend on id and class, and you can build tests with simple mouse clicks. One advantage is that you can write JavaScript to solve any issues. For example, we’ve used JavaScript for our “Updates” feature; to ensure that the user can check only the latest updates or updates with a specific condition. I love that you can make minute customizations like that.

Creating a simple scenario is extremely easy. That’s the best part. At the same time, Autify allows you to go further by writing JavaScript. I love that Autify is an all-rounder; the basics are easy, and you can do even more if you want to.

I really appreciate that. That’s exactly what we are focusing on at Autify. Please keep the feedback coming. We will use it to improve Autify’s features.

Makoto: As the QA engineer, I want to be able to advise my colleagues if they ask for help. However, I’ll direct them to Autify’s chat support if it’s out of my depth.

Teruo: The chat (Intercom) is updated frequently. I’ve never seen a service that gives us new announcements quite as much as Autify, so I was impressed!

Test automation enabled Nulab to address fundamental issues in QA.

Given all the challenges you used to face, how have things changed since you implemented Autify?

Makoto: You can’t maintain automated tests if you had to write code all the time; it would just put more burden on developers. It’s been about a year since implementing Autify, and I think we will soon be able to eliminate writing code.

Teruo: Thanks to Autify, we’ve been able to address the essence of operations and QA.

Building tests was an easy issue to solve… It’s great that we can now work on the difficult parts of E2E testing and the timing.

Makoto: Some engineers in our company aren’t utilizing Autify yet. I’m sure those using it see the advantages I mentioned earlier, but quite a few people aren’t fully utilizing it. That’s one of our issues.

I see. Teruo, do you also think that Autify being a no-code platform is an advantage?

Teruo: Absolutely. Using Autify is a breeze because you don’t have to think too deeply about the DOM (Document Object Model) structure, like id and class. Also, Autify shows screenshots when you make scenarios, which quickly reminds you of the page displayed at each step. I found that very convenient.

I would love it if I could modularize Step Groups and use them in various situations.

We will officially release an update of our Step Group feature.

Teruo: That’ll be great.

Makoto: That’s a feature I’ve been hoping for. I’m sure we can get even more out of Autify, so I’ll keep experimenting so that we can utilize it to the fullest.

Creating better tests to improve quality.

So, what’s next?

Makoto: As a QA engineer, I want to get my colleagues interested in testing and have them start talking about the benefits and areas for improvement. We’d like them to work on E2E testing and improve quality.

Teruo: I’m currently in charge of day-to-day operations and management, but my immediate goal is to make it so that the whole team can manage it. It would be good if we had a team discussion about how we can improve testing. This is because I tend to think, “I can just re-do it,” or “There are some parts that I don’t understand, but the test passed, so it’s fine.” I want to improve the quality of scenarios by being mindful that they will be used by others, too.

Finally, do you have any announcements?

Makoto: Nulab is growing fast, and we are hiring for various positions, including engineers. Nulab’s headquarters is in Fukuoka, and we have several offices overseas, but you can work fully remotely if you apply for a position in Japan, and there are no location requirements. You can find the details on our Careers page, so please take a look. If you’re even a little bit interested in Nulab, please don’t hesitate to contact us.

Thank you very much!

(Interviewer: Ryo Chikazawa, CEO, Autify Inc.)

‍