Escaping the downward spiral of regression with Autify: from 40 hours of testing to zero using codeless test automation

In recent years, an increasing number of companies have begun to use online advertising to increase sales and raise awareness. The online marketing market is expected to grow even further as the Internet and smartphones continue to develop.

With this, Ad Tech, or tech that maximizes both the media and advertisers’ profits, is attracting attention. SMN Corporation of the Sony Group provides Logicad, an Ad Tech platform that provides services such as advertisement distribution management, analysis, and optimization.

We interviewed Mr. Takahiro Yasuda, an Executive Officer, Mr. Naoki Shigetomi, a tech development leader at Logicad, and Mr. Takahiro Yoshii, a QA (Quality Assurance) manager, about the issues they faced in development and how they solved them with our codeless test automation platform, Autify.

– What are your responsibilities at SMN?

Takahiro: I am in charge of developing Logicad, an advertising distribution platform developed in-house by SMN Corp. I’ve been involved with _Logicad _since its inception, and now I’m in a position to oversee the general tech as an executive of the tech development department.

Naoki: I belong to the team that develops Logicad’s UI. The team consists of 11 members, of which 6 are developers, and 5 are testers. I’m the team leader.

Takahiro Yoshii: I am responsible for QA at Logicad’s UI development team led by Mr. Shigetomi.

Selenium was too time-consuming and costly

– What challenges did you face with test automation before introducing Autify?

Takahiro Yoshii: As a team, we thought that Selenium would make testing easier. We considered how extensively we need to run regression tests and how much it would cost.

However, the staff cost for using Selenium was high, and we couldn’t assign staff who could perform maintenance tasks. We came across Autify around that time when we were struggling to find a solution. Using Autify doesn’t require any technical skills, and it’s relatively cheap. It seemed like the best way forward.

Takahiro: We tried to implement Selenium about five years ago. We got to the point where we could start using it, but keeping up with subsequent development and continual maintenance became too labor-intensive.

We were releasing once or twice a month at the time. The time it took to maintain Selenium and the cost of allocating skilled staff didn’t make financial sense.

Of course, I’m not saying Selenium isn’t a good tool for test automation. It’s that we had a negative experience with it, and we were anxious that we’d make the same mistakes again.

Escaping the downward spiral of regression

– You tried and failed test automation five years ago. What made you want to try again?

Takahiro: There are times when adding new features causes problems for existing functions and causes regression. Of course, the more features you have, the more likely regression will be. The test itself will become complicated too.

It’s not uncommon to find a bug and have to check for regression. You start forgetting how much of the regression testing needs to be done manually. We are reaching that point, and it was becoming increasingly difficult to handle it manually.

– You started considering Autify at the end of last year. Was there anything else you wanted to achieve with test automation other than managing regression risk?

Naoki: I wanted to re-allocate the resources for regression testing to test new functions to accelerate the development cycle.

If the cost of regression is ten today, it will be 11 tomorrow and 12 the day after; the cost keeps growing! Plus, the regression tests are for existing functions, not new ones. That means the more functions we release, the more workload we would have for testing. I wanted to get out of this downward spiral ASAP.

Since I started using Autify, I haven’t even thought about the cost of regression testing. The man-hours for testing have clearly reduced compared to before.

Prevent regression by fixing critical bugs

– With agile development, the number of functions will continue to increase. What was the process like leading up to implementing Autify?

Takahiro: Mr. Chikazawa (CEO & Co-Founder of Autify) gave us a demo for the test and development team. When the whole team all saw how it works, I think they were convinced that it would work. If you just say, “here, use it,” they wouldn’t know what to do! That demo made it seem more doable.

Takahiro Yoshii: I agree. I think it was an excellent opportunity for each person in charge to implement Autify and get things set up.

We started by prioritizing the critical elements of our business. We made the Test Scenarios for them first. As the system grows, the number of test branches increases, so we increased the number of Scenarios by building on top of the initial Test Scenarios.

– When you say “critical elements,” do you mean if there’s a bug, it will cause a lot of damage to the business?

Takahiro Yoshii: Yes. Registration and updates are essential to our business, so we want to test those areas thoroughly and make sure any bugs are fixed. On Logicad, the screen changes depending on user rights. We would sometimes test one screen but forget about the other and there was one time when there was a bug in the screen that we didn’t test. That was a bitter experience.

– Was it difficult to run tests even with Test Scenarios?

Takahiro Yoshii Running tests isn’t difficult because the Test Scenarios were already prepared. The problem is that it’s not always easy to determine when and how much we need to run them. It was impractical to test everything, given that we don’t have infinite time and resources.

Takahiro If there is a bug in the new function and development time is longer than expected, sometimes the regression test gets compromised. If we use our intuition to determine how much we test, regression is usually hiding somewhere we didn’t anticipate.

Naoki When our prediction is wrong about how much a new feature will affect the system, Autify will cover for us. It tests the system as a whole and finds the regression that we didn’t anticipate. That’s why we’ve seen less regression after implementing Autify.

Visual Test Cases drastically reduced training costs

– We’ll be releasing a new feature that covers user rights on Autify. I think it will be useful for you. Did you have any creative ideas about how to use Autify?

Takahiro Yoshii: When you initially create a Test Scenario, you know exactly what it’s testing. But as the number of Steps increases, it can get unclear. When we first implemented Autify, there were times when it was a struggle to make sure everyone could understand what each Scenario’s purpose was.

Autify has since introduced a new feature that allows adding comments and Step names. Instead of just words, you can see it move in the browser, making the Test Case more visual. It’s easier to understand that way.

If you leave a comment, it will always be clear what the Scenario is for. Also, a new employee can read the comment and create new Scenarios themselves. Not having to teach them every time has reduced training costs. That’s been an unexpected, yet pleasant, surprise.

– What’s the biggest factor about Autify that helps cut down maintenance time?

Naoki: I think it’s the simplicity of Autify’s UI. The ease of use is at a level that _Logicad _can’t emulate. Just by looking at Test Cases that the test team has made, I can intuitively understand how to make a particular Test Case.

I also appreciate that I can chat with customer support, and they respond quickly. I think it’s not easy for live chat support agents, but it gives me an immense sense of confidence as a user.

Takahiro Yoshii: Customer support allows each staff to work independently, and that’s been very beneficial.

– Do you feel that implementing Autify has produced results?

Takahiro: Yes, preventing regression has had a significant effect.

The service we offer, Logicad, receives advertising banner images and delivers them as an advertisement. There’s also a reporting feature that shows the ad’s performance results. With an increasing number of advertising creatives, there are many ways to produce reports. For example, we could report how many impressions and clicks the ad campaign got and how it performed daily. If a user submits 100 patterns of creatives, Logicad needs to produce a result report for each of them. Dozens of people go through the results and base their optimization decisions on them every day.

We often improve existing features, and regressions are likely to occur because of the nature of the service. Testing is time-consuming but it’s not something we can avoid. Having automated time-consuming tests, I can confidently say that it’s had an enormous impact.

– How many man-hours have been reduced?

Takahiro Yoshii: It depends on the function, but in most cases, something that used to take 40 hours now takes virtually zero hours. A task that one staff used to spend a whole week doing is now entirely automated.

As I mentioned earlier, the amount of time that we spend testing increases as time goes on, so the time saving will only go up from there. The 40-hour saving happens once or twice a month. Plus, the time we save can be spent on other tasks.

Naoki: Critical regressions only happen several times a year, and we haven’t had one where Autify prevented. Even so, as a developer, the overwhelming sense of security alone has been worth it.

Afternote: “Immediately after the interview, we were able to detect two regression bugs with Autify before release.”

Autify is an adaptive tool that can handle complex systems

– What are your plans for test automation and development?

Takahiro: Right now, I mainly use Autify for regression testing, but I hope to expand it to testing new functions. If we create a workflow where a developer writes a Test Scenario and the QA team runs tests, I think we can accelerate development speed. In terms of the development process, it would be ideal to use tools like Autify and keep streamlining it.

– Is there anything else that you would like to work on with Logicad’s service?

Naoki: The team’s challenge is to prevent issues in the production environment. Another thing is to keep improving the service.

There aren’t many types of user rights at the moment, but we plan to increase them next year to improve Logicad’s usefulness. I anticipate that the test load will increase even more, but it’s partly because of Autify that we can go forward with the plan. I don’t think we could do it without Autify!

– I’m glad to hear Autify has been useful. Do you have any advice for those who are planning on automating tests?

Takahiro: Autify is an adaptive test platform for maintaining functionality on complex systems. I think there are many people who maintain complex systems by doing small maintenance tasks every day.

Some people may think that the more complicated it is, the more difficult it will be to automate. Autify is a tool easy enough to use in those situations.

There aren’t many types of user rights at the moment, but we plan to increase them next year to improve Logicad’s usefulness. I anticipate that the test load will increase even more, but it’s partly because of Autify that we can go forward with the plan. I don’t think we could do it without Autify!

Naoki: Our QA team used to run regression tests every time it was necessary, and it was too time-consuming. But after implementing Autify, the number of things we don’t have to do keeps increasing, which has freed up time to do other tasks. I think it’s lead to the QA team being more motivated.

Takahiro Yoshii: Autify is a service that the QA team primarily uses, but I think the development team can also get involved with the process. Autify is a promising tool that has endless possibilities depending on how you use it.

– Thank you for talking to us about your experiences. We are continuing to improve Autify. If you have any feedback or requests, we would like to hear from you.