One year with Autify: The story of a QA team who saved over 6 million yen per year in costs.

GLOBIS CORPORATION is a company with a vision of “creating an ecosystem of people, money, and knowledge related to business management, and creating and transforming society” by education. Globis Graduate School of Management, which offers an MBA degree program, is well known.

Twenty-five years have passed since its founding, and now they are going beyond the business school framework to provide corporate training and content that deepens business knowledge.

In-house application development was started in 2016, and a cross-sectional QA organization was launched in 2019. We talked to the team leader, Mr. Ikebe, who is the leader of these efforts.

– Please introduce yourself.

Keisuke: I’m the QA team leader in the development department at Globis Inc. I have been a QA engineer for three years. I can’t say I know everything about QA, but I work in QA and lead the team.

As a business school, Globis educates business leaders and we recently released an online learning app.

Even during the pandemic, an increasing number of users are signing up for digital services, and sales continue to increase with it. In these uncertain times, we have been working to raise awareness about the importance of continued learning for the future. The success has been encouraging.

– Thank you. Let’s get right into it.
I heard that Globis has recently shifted to agile development.

Keisuke: Yes, an independent business division was created in 2016, and in-house development started at the end of the same year. We had been outsourcing development until then, but we changed our strategy to speed up development. We set up an in-house development group, which has now grown into an organization with over 100 people. There are more than 70 members involved in development, including engineers and designers.

While the organization has grown, our products are increasing too. We don’t have time to spare!

– I imagine developing in-house is very challenging. How did it go?

Keisuke: Yes, it was. I think it went well because we started entirely from scratch instead of taking over development of the existing product from the development vendor.

The first product is “GLOBIS Manabi-Hodai (All-you-can-learn)”

– How many products do you have now?

Keisuke: We are currently offering four major products. GLOBIS All-you-can-learn (a video learning service that condenses systematic knowledge necessary for business) and its English version (GLOBIS Unlimited); an investment program called “GLOBIS Alumni Growth Investment” which is for venture companies started by current students or alumni of Globis; LMS which is used in Globis training; and the platform that manages certifications and applications.

– I heard that your team is responsible for QA for all of those products.

Keisuke: Yes, that’s right.

We created a QA team in 2019 with the premise that we will manage QA for all products. This is because the original issue we had was that there were many mistakes where products connected. Our team’s organizational structure can cover those connecting areas.

– Each product’s quality control and QA were separate before that?

Keisuke: Yes, up until 2018. The underlying assumption was that releases would be managed within the Scrum team, so engineers were responsible for QA. We were able to meet quality standards for each product, but we started encountering problems when it came to the parts where products connected. Aside from technical aspects, the line began to blur with regards to who is responsible for testing. Issues increased, and our team became disjointed.

– There was an increasing need for E2E testing.

Keisuke: We didn’t do much E2E testing until then. I don’t hate it, but honestly, it’s not easy to automate. Testing manually was impractical because we lacked human resources. That’s the biggest issue that we wanted to solve.

I don’t want to create an organization that pushes its staff to their limits. Implementing Autify to work on quality assurance (QA) as a team

– Other than wanting to deal with E2E, what other issues did you face before implementing Autify?

Keisuke: I tried some automation with Selenium and Headless Chrome but it was not stable at all.

I also thought continuing this maintenance would be torture.

While it was unrealistic to do everything myself, it was difficult to assign someone else or delegate either. There are several alternatives, such as sharing the Selenium environment or asking a tester, but staff costs would be too expensive. I didn’t want to be an organization that pushes its staff to their limits.

– What do you mean by that specifically?

Keisuke: Performing E2E tests manually is time-consuming and costly. To run tests quickly with the scrum cycle, I thought we needed a flow that runs automatically along with deployment.

Automation was essential for creating a team, too. Throughout my career, I had always thought that repetitive tests aren’t something that people should be doing. I did not want to make my team members do it.

Also, too many testers would be too expensive to manage. I originally wanted to create a QA organization that could also handle engineering. But considering all those factors, I knew it wouldn’t be possible. That’s why investing in automation was a given.

– Globis has been using Autify before its official launch. I’m sure it was far from complete at that time. What did you find appealing even at that stage?

Keisuke: One is that I wanted to create a QA organization that was not entirely reliant on testers, as I mentioned earlier. The other is that we were short-staffed in the first place. I think automation engineers are a very niche occupation so hiring the right person was very difficult.

That’s why I wanted a maintenance tool that even non-engineers could use. I was skeptical that such a tool even existed!

I came across Autify around the time when I started researching it. The timing was perfect.

Non-engineers could use it, and maintenance became easier.

– So you were looking for a tool that could be used by staff who aren’t engineers, and one that would make maintenance easier?

Keisuke:Each Scrum team at Globis has a PO (Product Owner), so I thought it would be great if POs could test and do maintenance tasks as well. At that time, it was only conceptually that I could see things done the way I wanted them to be, but I was excited.

Implementing Autify started with writing test scenarios

– Can you tell me about how you implemented Autify?

Keisuke: We started by each team writing their own E2E test scenario for each product and then creating test scenarios on Autify. I don’t think it was a particularly complicated implementation process.

Aside from that, we decided to test in a staging environment.

When creating scenarios for existing test cases, some cases were not suitable for automation. Some couldn’t be done as an E2E test in the first place, and some required post-processing of data. In that sense, there was some trial and error involved with automating tests.

– You looked at the test scenario first and then considered where to start automating.

Keisuke: That’s right. The QA team did not compile the test cases shared with Autify at that time. The format was not standardized, and we didn’t have a testing perspective. We’d handed it over without any clear documentation, which I think made it difficult to interpret!

Autify assisted in creating test scenarios at the time.

– I think it’s safe to say that that project was the reason Autify exists today!
I remember this happened just after Globis’ QA team was created.
Can you tell us what you like about Autify?

Keisuke: The Customer Success team and the comprehensiveness of support are exceptional.

I also find it excellent that Autify listens to user feedback and adds features based on that. It’s a very user-centric service.

I like the transparency that the roadmap is published as well. Of course, some features are still missing. But because the roadmap outlines all the features under development, I know what to expect. I find that very helpful.

For example, I used to have to use a JavaScript step for a mouseover action, but that feature has been released since. I know that Autify is always working to solve issues, which gives me confidence in the service. I have plenty of trust in the roadmap, too.

One year since implementation: Autify has become an integral part of the QA process

– It’s been a year since you implemented Autify. Can you tell us how test automation is going right now?

Keisuke: First, we stopped getting test cases from products.

We start the process by assuming that we’ll be using Autify. We try to create scenarios in order from the most important for the user, prioritizing ones that are least expensive to maintain.

Then, we determine the importance based on the domain knowledge of each product. The next step is to figure out whether we can use Autify to automate it. Instead of trying to make all the scenarios work on Autify, we do it the other way round; we identify priority scenarios and then confirm that it can be automated with Autify.

In other words, we started assuming that we can use Autify. It’s a shift in our thinking and the process itself.

Fortunately, someone with experience in test automation joined our team, and their expertise helps us determine whether a task is suitable for automation or not.

– It’s fascinating that the flow itself has changed. You started assuming that Autify can be used instead of creating a scenario and then seeing if it works with Autify.

Keisuke: Exactly. We specify the test perspective, such as which function or user action we want to test. This is because it’s faster to record with Autify than to write the test step on Excel, which would be the subsequent process.

Writing the steps on Excel and recording with Autify are essentially the same thing, so consolidating them has made things more efficient.

– How many members do you have on your QA team now?

Keisuke: Including myself, we have six main staff. There are two part-time staff as well. The part-timers aren’t engineers and cannot write code, but they proactively create scenarios with Autify.

It’s still not enough but it’s been a considerable improvement from when I used to call myself a team!

Now I can do other things.

Successful operation from a simple smoke test

– Were there any creative ideas when you operated as a team?

Keisuke: This might be one of Autify’s uses, but we use Autify for smoke testing.

It’s similar to ping, but we also run basic tests on each product, such as whether SSO works and whether the user can log in.

I had planned to use Autify on smoke testing from the initial stage, but it was only recently that we’ve been able to achieve it.

– Do you expect it to be effective?

Keisuke: The scenario for smoke testing is short, and it’s very stable. Combining that with Autify’s stability, I think it will have a positive effect in terms of service stability.

Autify’s cost-effectiveness: Over six million yen in cost savings.

– Since implementing Autify, what kind of specific effects did you see?

Keisuke: We have not yet been able to quantify the effect, which is an internal issue, but there is a peace of mind that smoke tests and regression tests for each product are being carried out.

– What about qualitative effects?

Keisuke: Stability has improved from having a larger number of scenarios.

Workflows must be defined in organizations with multiple medium-sized products, all of which have to be managed holistically. In this regard, Autify has helped us define the workflow.

The more experience you have with test automation, the more you’re reluctant to work on complicated tasks and difficult maintenance tasks.

There’s always a part of us that screams, “Automation isn’t always that easy!”

Autify has been invaluable in that respect. It’s easy to use, and the more you use it, you realize how low the maintenance cost is. There’s a sense of relief that you can lean on it.

– We are delighted that those who’ve worked on automation in the past trust our service.
Has it been cost-effective?

Keisuke: There’s been an enormous saving in labor costs. For example, if we had to hire another highly skilled test engineer or an automation engineer, there’s no question that Autify’s usage fee is a lot cheaper.

The engineer would cost us over 6 million yen. By having Autify, we’re saving on hiring costs and labor costs.

Besides, there aren’t many test automation engineers in the first place! I’ve personally tried building E2E using Selenium myself, but I wouldn’t consider myself an engineer specializing in automation. If there were language requirements in addition to that, finding the right person would be extremely difficult. If you also think about whether the person fits our company culture, it narrows potential candidates to near zero. Realistically, I don’t think it can be compared with labor costs.

However, we’ve conducted a review of the tool in March 2020.

This was partly because of the coronavirus pandemic and the company regulations, but we decided to evaluate and compare with other products properly.

Based on that review, we concluded that there is no better product than Autify at this time, considering the fact that there is a roadmap that outlines the fixes for issues. That’s why we decided to keep using it.

– I am pleased that you continue to use our service based on the evaluation. We will always keep evolving to meet and exceed your expectations.

The future outlook for Globis: Rethink the approach to QA.

– What is Globis’ future outlook like?

Keisuke: As I mentioned before, Globis has several medium-sized products.

Several scrums are tied to each product. By incorporating frameworks like “Scrum@Scale,” we are working on effectively running numerous small scrums.

The QA team is also exploring a more flexible approach to QA.

The QA domain hasn’t figured out the correct approach yet. We are working towards defining that for ourselves.

We want to have an effective QA even with a small Scrum team, resulting in higher product quality and faster release.

The QA team will not try to achieve this by blindly adding more staff.

I want to work together as a team, thinking about what Globis’ QA should be like and the best approach to improve quality.

– Thank you. Finally, do you have any advice for those who are planning on test automation?

Keisuke: I still have a lot that I have to work on before giving advice to people!

I heard the story about Autify’s Sam-san (co-founder Yamashita) before.

The test automation tool market itself is expanding. I think QA itself will start focusing more on design tests, and I think that’s how it should be.

Automating tests alone will not improve quality. Thinking of ways to improve quality is the job for humans. I hope that we can leave automation to Autify to focus more on tasks that only people can do.

I hope many people feel the same way.

Autify’s cost-effectiveness: Over six million yen in cost savings.

Keisuke: When we first implemented Autify, we were planning on having each PO (product owner) use it during the pre-release test phase of the agile cycle.

We actually changed that, so I’d like to talk about it.

At Globis, we have switched to using Autify for Shift Right testing. In other words, we changed it so that it’s continually running regression tests on products after release.

If testing is getting in the way of releases, it goes against the concept of an agile cycle. We needed both Shift Left testing and Shift Right testing.

When we first implemented Autify, we thought that the only benefit would be that E2E testing would become more manageable. However, the more we used it, we realized we could delegate Shift Right testing to Autify as well.

At Globis, Shift Left testing is now done manually, while Shift Right testing is done on Autify. As a result of this, Scrum and QA can work asynchronously.

– It means that you don’t have to pause development because of testing.

Keisuke: Globis’ development rule ensures that the staging environment and the production environment are the same. That’s the conditions we have.

Autify is always running, and it’s continually monitoring for us. When it finds a problem, we take it from the production environment to the staging environment.

– How do you handle Shift Left?

Keisuke: Shift Left is called “PO Support” in our company. When we considered how we could speed up Scrum releases, the biggest bottleneck was the PO’s release decisions. This is an important point.

Until now, the time of release has depended on checking the release against the specifications based on the user story. The problem is that it’s checked only from the perspective of the PO.

This is still under verification, but the QA team is assisting this. We’ve added QA items to the PO’s checkpoints. For example, we communicated by saying, “Here’s a perspective that you should check based on previous issues.”

By doing so, we can have an objective and multifaceted perspective. Further, I’m hoping that by clarifying these perspectives, we will make the PO release decision itself a test.

– It’s a good combination of user story and quality assurance.

Keisuke: I would like to execute it before creating a user story in the future.

There are some differences from the so-called “shift left/shift right,” but I’m confident that this will result in faster releases.