“As someone who can’t code, Autify gave me hope.” Paving the way to test automation: a story of a QA engineer

Contract-based development isn’t everything. In this day and age, a successful business must collaborate with clients, partners, and other third parties to develop services and solve problems.

tambourine.inc is one such company that practices this business model. They develop services that utilize cloud platforms such as Salesforce.

Of course, it’s essential to understand the client company’s demands and quickly develop services that meet them. What’s equally crucial, though, is ensuring quality in those services. Ms. Yuri Hayakawa joined Tambourine in 2020 after working for a third-party testing company. Currently, she’s responsible for QA (quality assurance). How has she paved the way to create a test automation and quality assurance system?

We also spoke with Mr. Tatsuya Nakao, CEO of Tambourine, Inc., about the importance of test automation and the unique challenges as a contract-based development company.

The limits of manual testing: comprehensive testing on multiple terminals and browsers

– What challenges did you face before implementing Autify?

Tatsuya: We were working on a project to develop an e-commerce service. Upon reviewing the deadline, work progress, and remaining tasks, we realized that we couldn’t complete them on time if we were to proceed with manual testing. We had to automate the testing early in the project to streamline development, so we considered some test automation options.

We could have used open-source tools like Selenium, but in the end, we chose Autify because we needed a tool that anyone can use to create Test Scenarios and plan regression testing. It had to be usable for non-engineers, too.

– What’s the biggest challenge with testing at your company?

Tatsuya: Testing on multiple devices and browsers is time-consuming, and if we tested manually, it would be endless. With e-commerce services, critical issues such as not making purchases would result in a considerable loss. That’s why it’s essential to test on multiple devices and browsers and pass with flying colors before launch. We wanted to improve efficiency while ensuring quality, which was the reason for considering Autify.

– With e-commerce websites, there are many devices that users may use. Screen sizes would be different, and if you include cross-browser testing to the mix, I can imagine how comprehensive testing would be a laborious task.

Tatsuya: We’d purchased test devices, but we started working from home when the pandemic hit, and using them became impractical. We had also considered BrowserStack, another testing tool, but preparing test automation with it would have taken a long time. Autify was attractive in that it could handle real device testing for smartphones.

– We’ve been hearing from many people that working from home made it difficult to manage test devices. Some even have to go to work just to use test devices or send the devices between staff by mail.

Yuri: Testing isn’t our primary focus, so we definitely lack when it comes to device coverage. Rather than preparing every possible device, it’s quicker to use a service that can handle it all. Autify is very attractive in that sense.

Automate maintenance testing and devote resources to new development

Yuri: Another issue was additional development in maintenance and operations. If we were to manually run regression testing every time a new feature is added, it would take a considerable amount of resources. Operation and maintenance is a long-term effort, so even small costs add up. Another advantage of test automation is that machines don’t make mistakes. Any human will make a mistake at some point, so testing manually will lead to bugs. Reducing the man-hour for testing allows us to allocate more time for new development. Those are some of the reasons that reducing costs while maintaining quality is essential.

**- You needed to automate testing to streamline operation and maintenance for existing clients so that you can move forward with new projects. **

Tatsuya: Exactly. Since its inception, Tambourine has created services for B2C of major companies. Most projects need continuous development, even after project completion. Previously, staff who developed the service would remain on the development team and work on additional development and maintenance.

In recent years, there’s been an increase in the number of projects for developing services that support the client’s business. As the scope of our responsibilities such as bug fixes and continuous development expands, we need to optimize how we allocate staff on previous projects. That was another challenge we needed to solve.

Codeless Test Scenario creation and screen capture function were the deciding factors

– What was the deciding factor for choosing Autify?

Tatsuya: Engineers are often busy with development. One of the deciding factors was that with Autify, PMs and directors who work closely with clients could create Test Scenarios easily. If the Test Scenario is designed right, you can reuse it in other similar projects. Alternatively, we could build on an existing Test Plan. That was what I thought when I looked at Autify’s management screen.

Yuri: Another critical factor was that Autify takes screenshots automatically. We are a contract-based company, so clients sometimes request screenshots at the time of completion. There aren’t many automated tools that have screenshot capabilities, so the ability to download screenshots in bulk on Autify was fantastic.

– After implementing Autify, you’ve been providing tutorials for each project, Ms. Hayakawa?

Yuri: Yes, this is because I think we need to make sure that all of us know how to design Test Scenarios and have a good understanding of testing. I still lack knowledge and experience, so I appreciate Autify’s customer success team to count on. I participated in the Test Automation conference the other day, which was very helpful.

Tatsuya: Ms. Hayakawa goes into the project and transfers skills on how to think about testing. She teaches the team how to achieve a goal in the shortest amount of time and how to streamline the process. This is what an ideal QA looks like. She’s raised awareness even among engineers, and communication with customers has become smoother.

Share knowledge and pave the way to self-reliance

– That’s great to hear! After introducing Autify, did Ms. Hayakawa create Test Scenarios first?

Yuri: No. I interviewed project team members about what kind of testing they wanted to automate, what tests have been done previously. From that, I started by identifying what was lacking in our quality assurance process. Then, I educated the team on how to think about testing and how to create them. Then, I asked them to create a test. It took a while, but it was worth it. Making Test Scenarios myself would be pointless; test automation would rely on me, which isn’t what I wanted.

Tambourine has 70 employees now, and I manage QA by myself. If I did everything when we first introduced Autify, I would have to create Test Scenarios for all future projects. There’s only so much one person can do, so I focused on how I can share my knowledge. If I didn’t create a system that can run itself, I would end up suffering. Not to mention it would have a negative impact on our clients! It was my job to create a system, not manually handle QA tasks. I think it’s more valuable that way.

– You focused on coaching others instead of doing it all by yourself. Was everyone able to use Autify smoothly?

Yuri: Some understood it immediately, some struggled. I have experience in education and I know that we all have different strengths and weaknesses. You have to think about how to approach it in a way that works for each person. I try to think about how I can explain it so that people find it easy to use Autify. At the same time, I try to avoid confusing explanations.

Tatsuya: Ms. Hayakawa also runs a Zoom seminar on why we should install Autify within our company. Members of the PM team were surprised about some of the things Autify can do.

Showcasing the value of test automation to clients

– Mr. Nakao, as the CEO of the company, what do you think of test automation?

Tatsuya: Test automation is one way to optimize human resource allocation. There are several steps we’ll need to take to completely solve that issue, but the first is to try Autify out and explore the effects within our company. Once we see results, we can showcase the value of test automation to our clients. Then, we can proceed with continuous development and operation by thinking about Test Scenarios with the client, for example. By following this process, I think we can propose implementing the test automation tool when we estimate the cost of the project.

Clients are divided into two types; those who are experiencing challenges with testing and those who are not. That’s why it’s essential to show them the value of test automation first. For example, if the client is spending a lot of time on acceptance testing, we could show them that by reducing the amount of time spent on that, they could focus on promotion, marketing, and creatives. I think Autify’s value is not just the ability to streamline testing. I think the value can be seen in the deliverables as well.

– What changes did you see when you implemented Autify?

Yuri: As I mentioned earlier, testing on multiple terminals/environments and the ability to run automated tests without programming knowledge have had an enormous impact. I also think that the ability to automate pre-release and post-release checks is a significant advantage. We can reuse the same Test Scenario, so there is no need to create a new test every time. Just a single click and you’re done. We’ve been able to reduce about one man-day per test.

Now, we can allocate that time to other tasks, such as running manual tests on additional development or creating resources about testing.

Tatsuya: Thanks to Ms. Hayakawa’s efforts, the number of people who can use Autify within our company is gradually increasing. I also feel that it has raised awareness of how important quality is within the development team. The next step is to expand it to other projects.

The stability of the system creates an environment where staff can express their ideas freely

– What is your future outlook like?

Yuri: We’re at a stage where I need to provide guidance when creating Test Scenarios, but I’d like to increase the number of people who can manage it themselves. PMs tend to have a huge workload, so I’d like to simplify testing so that it can be delegated to the Operation & Maintenance team. We are planning on launching a QA team, and I’d like to educate each team member to have a QA perspective so that each person can choose the appropriate means for QA. We still have many issues to resolve, but I would like the company to work on quality assurance as a whole. I think my role is to pave the way to create a system so that each team can automate testing themselves.

There are still many companies in the IT industry where QA isn’t prioritized. I’ve seen those companies myself, so I’m interested in creating opportunities for change. I want to create a world where QA staff can work more efficiently and be more motivated. Our company motto is ‘Let’s PARTY!’ Based on that motto, I’d like to create a strong QA team and discuss this at external conferences. The goal is to give a talk at JaSST (Japan Symposium on Software Testing).

It may be a challenge for the whole company to implement it, but it’s easier if you start small and gradually get other departments to try it out.

– I’m sure you’ll achieve that goal! Mr. Nakao, what is your future outlook as the CEO?

Mr. Nakao: Mr. Nakao: There’s an increasing number of companies who do business digitally, and it’s our mission to provide services that are fundamental to their business practices. Meeting the complex demands of our clients in marketing, design, and UX gives joy to all of us, including PMs, directors, and engineers.

To achieve this, just enjoying the creative process isn’t enough. The system needs to be stable. We believe that it is imperative to create an environment where you can try out new ideas safely and securely. The vision of ‘Let’s PARTY!’ is all about working with clients. However, to work with clients successfully, we need to put in the hard work and have standards.

Ms. Hayakawa supports the quality, which is a fundamental part of this. We share the importance of testing with our clients and work on the project together. Finding value is exactly what I want to do at Tambourine.

Autify gave us hope. Don’t give up on automation, even if you can’t code.

– Finally, do you have any advice for those who are planning to automate testing?

Yuri: I only have experience as a QA engineer, so I can’t code. At my previous job, I was told that automation is impossible if you can’t code. There was a part of me that gave up. But Autify came along and gave me hope. It’s was great to know that even I can automate if I set the direction for quality assurance and follow through.

If you get stuck with testing or don’t know where to start, I recommend giving Autify a try. I want people to not give up on automation, even if they can’t code. I’d be thrilled if more people and companies could feel the same hope as I did, and if there was a more active discussion on testing and quality.

(Interviewer: Ryo Chikazawa, CEO & Co-Founder of Autify, Inc.)

– I’m sure you’ll achieve that goal! Mr. Nakao, what is your future outlook as the CEO?

Mr. Nakao: Mr. Nakao: There’s an increasing number of companies who do business digitally, and it’s our mission to provide services that are fundamental to their business practices. Meeting the complex demands of our clients in marketing, design, and UX gives joy to all of us, including PMs, directors, and engineers.

To achieve this, just enjoying the creative process isn’t enough. The system needs to be stable. We believe that it is imperative to create an environment where you can try out new ideas safely and securely. The vision of ‘Let’s PARTY!’ is all about working with clients. However, to work with clients successfully, we need to put in the hard work and have standards.

Ms. Hayakawa supports the quality, which is a fundamental part of this. We share the importance of testing with our clients and work on the project together. Finding value is exactly what I want to do at Tambourine.