Online E-commerce startup saved a lot of time and money by automating E2E testing with Autify.

Autify recently interviewed AJ Almaguer, a Frontend Tech Lead and Staff Engineer at Fernish. Fernish is a furniture rental and purchasing company based in Los Angeles.

Their software testing team faced major challenges with end-to-end regression testing in their workflow. Furthermore, their small startup team of engineers wore many hats.

With the introduction of Autify for end-to-end automated testing, they were able to conquer some of their core challenges; including checking all testing environments, and alleviating testing responsibilities from their Product Manager.

AJ identified how Autify was priced right for their organization, and how the software testing tool saved their team time (which saves money) and gave them peace of mind. Discover more in our interview with him.

– What does Fernish do? And what do you do at the company?

AJ: My name is AJ Almaguer, Frontend Tech Lead and Staff Engineer at Fernish.

Fernish is a furniture rental company based in Los Angeles. Our first two markets are selling in California and Seattle. We expanded to Austin, Dallas, and Fort Worth in Texas. We recently opened markets in New York and DC. Fernish is about 4 years old, and I have been working here for 3 years now.

– How is your team structured?

AJ: We have about 8 to 10 engineers. Actually, we don’t have a QA. That’s why we needed to find a product like Autify.

– What were the challenges before introducing Autify? What kind of problems were you facing before?

AJ: We are actually effective at performing unit testing and running regression tests in our code. However, end-to-end tests presented a slight challenge to write. And, being a startup, we try to balance doing the right thing with speed. Therefore, our intended solution was lacking.

To elaborate, our Product Manager actually ended up being part of the release process. The Product Manager would smoke test the site to make sure everything works, and then sign off on the release. Thereafter, we could release it. That was the biggest challenge we faced.

The Product Manager (just like other members of the team) was been getting busier. So our goal shifted to automating that part of the release process- giving us more security about the state of our code and our system.

– So, for end-to-end tests, the Product Manager has been testing?

AJ: Yes, previously the PM was testing. In our search, we were shopping for a tool that offered end-to-end testing as well as automated end-to-end testing.

Our PM tried to help with the login flow but also the check-out flow. The biggest hurdle we faced attempting to do our own end-to-end testing was the final step of the check-out flow, which required a signature plus it interacted with iFrames. All of that was such a headache. Switching to your system (Autify,) it became so much easier.

– What kind of applications do you have right now?

AJ: We have our e-commerce website (that’s the site we use to test with Autify.) We mostly test our customer’s end-user experience. We also have our backend software for the operations team and customer service team. Unfortunately, like most startups, it’s highly neglected. We really don’t need to test that.

We have a third application, it’s a React native app. We call it our “Asset Management” application. It scans all of our inventory and helps our drivers with deliveries and pickups, plus assists with work in the warehouse.

We are talking to you (Autify) about that. However, it’s built on Android. At the moment, I don’t think you support Android end-to-end testing. If I recall, you did a year ago. I think we might have to request a sign-up for your Beta.

– Yes, we just launched the early beta version for Android. So, Sam is going to reach out to you.

– What is your release cycle? How many times do you deploy on a weekly basis?

AJ: On average we’re deploying once a day or once every other day. We have a continuous integration setup. We also have a good pipeline but it only auto-deploys to our development environment. Then we implemented a few extra checks (to be safe) when we promote from our development environment to our staging, and then from staging to production. That last part is a little manual. So, whoever is on call for the week is responsible.

As part of the release process, we promote the code to staging, then we trigger the Autify test, when Autify pings us on Slack, we can confirm that all of our tests have passed.

Lastly, we have a Slack workflow to ping everyone, reminding them of what’s going on. Then, either our Engineer Director or Product Director would sign off on the release, allowing me to complete the release.

“Autify has the right amount of price points plus features and ease of use”

– So, you are running the Autify test against the staging environment right now?

AJ: Yes. We run Autify tests against the developing environment. That runs every morning. We are so impressed by the recording feature! I think that’s one of the main things that we really liked. I personally like how it shows the results. Meaning, it shows the previous screenshot, plus the next one, and the whole review process. We can easily determine a “pass” or “fail” result. This is very beneficial!

I think you (Autify) have the right amount of price points plus features and ease of use for us. This is what motivated us to proceed.

– You mentioned that previously it was tested by the Product Manager. Is the Product Manager also using Autify?

AJ: A little, but not as much because our PM is very busy. I’m the main person at Fernish that manages Autify and whoever is on call for the week. We are the ones that run tests and monitor them. If a test fails, we determine whether it was a “flaky” test or do we need to perform a regression, etc. I think Autify has already caught a couple of regressions in the past four months and we are very grateful for that.

– After you have decided to go with Autify, how would you introduce Autify into your operation?

AJ: For the release process, instead of asking our Product Manager or someone else to do the smoke testing, we can now trigger it ourselves. It’s really nice to be able to trigger that part of the process.

Currently, I’m responsible for setting up and scheduling the tests to run every morning. In the future, we will train our team members to set up their own tests.

– Where did you start? As you mentioned, you started creating a login flow, checkout, etc.

AJ: Our very first test was to evaluate a critical path for our customers. The test involved going to the site, browsing the catalog, adding a couple of things to the cart, etc. By default, you can rent furniture. However, you can purchase furniture outright.

So, technically you can add an item for subscription and an item for purchase. When you click on the cart, we force you to log in through the login workflow, then the checkout workflow, and sign the contract (making sure that the receipt shows everything.) The last step is going to the Dashboard. Everything is on the Dashboard.

It’s paramount that we test that our critical workflow works well because it’s how we monetize. Hence, why it is the one of first tests that we created with Autify.

After that, I added other important tests that are hard to run with unit tests. Such as making sure the promotional modal popups work. Other tests included checking that forms can be submitted successfully.

We have a wish list feature, which we created a test for. A wish list is also a great way to capture customer email addresses because we require them to sign up.

Beyond those tests, I created several new tests on our item detail and product detail pages. This tests many important features of the site. It’s a critical part of our e-commerce site.

– You mentioned that the critical path for your clients is the most important one. I think that is one of the biggest values of Autify for most of the E-commerce stores. I’m wondering if there was a little bit of a push from the marketing or business side to say, listen, we really need to test this path flow of conversions daily to make sure we don’t lose money and maybe set up a cadence that is every minute or something like that. Did you hear anything like that?

AJ: We haven’t had requests from our business side to do something like that. I don’t think they know that’s a possibility. Don’t give them ideas, OK? (Laughs)

Currently, we do not conduct tests in production. And the reason why is that we have many tools for data analysis like Google Analytics for tracking. We built our own dashboards with Looker to keep track of the flow- such as conversions, etc. In theory, we could try testing production with Autify by logging in with a test user account and then flagging it so it does not skew our e-commerce conversion rates.

– Do you have any Autify features you are actively using right now, or that you think are useful?

AJ: Yes, the scenarios and test plans are the main features we are using. The next feature is triggering the test via an API.

“Autify saves a lot of time for us, which helps us sleep better at night”

– You should check out our CLI, this is available on our help page. How would you describe the changes after introducing Autify? As you mentioned previously, the Product Manager has been doing the end-to-end testing manually, and that person could be a bottleneck on the release. Now, how was that changed?

AJ: I think that’s the main thing. Autify saves a lot of time for us, which helps us sleep better at night. We are reassured, and we haven’t had major things come up.

– Now you can sleep better than before. Did Autify find any serious bugs?

AJ: Autify found a couple of regression bugs. I cannot fully recall, but it was related to bugs that were difficult to test in the entire pipeline. Currently, we have a NextJS application that doesn’t communicate with other applications. It’s separate from the Rails applications. The disparate application makes it difficult for unit and regression testing. So, Autify is the main tool we use to run those tests. It’s our only defense for actual end-to-end tests.

– How did you find Autify?

AJ: We formerly used Walrus.ai, but it has shut down. Which caused us to seek a replacement tool. Then, I received an email from the Autify team which caught my attention.

– May I ask you the reason why you decided to decouple the front end from Rails? I mean, it’s obvious but I’m just asking.

AJ: Previously, we had a few pages built as Rails controllers. Our MVC controller operates the server-side Rails page. Then we inject a single React app on that page. It’s really nice to have all the front end with one unified code.

Personally, the greatest thing I value is developer experience. It makes my fellow engineers and my life a lot easier by having full control over the front end. A couple of years ago, I could recall all of our front-end and back-end systems were all in one system. Making changes became much more difficult and slowed development life cycles.

Prior to coming to Fernish, I was acclimated to working with a backend engineer. After determining the API contract, we would work independently yet still unite the application, and it worked. Though we worked a lot slower because we worked in parallel, working sequentially, and using one monolith. Now, it’s a lot more common practice to have one engineer build the API while another engineer builds the UI. It’s much easier to build the code that way.

– I would like to ask you about your future plans. Not only testing but also as an Engineering leader yourself. How do you envision your Engineering team?

AJ: One of the coolest things we were able to do by decoupling NextJS was building a framework in NextJS. Therefore, NextJS has 2 backends that it talks to.

It has a Rails application for almost everything. Additionally, we implemented a service called ButterCMS for some of our marketing content, and it’s easy to use.

Our Marketing department asked us to implement a blog on our site. Instead of custom building it, we relied on ButterCMS which has a nice blog and CMS features. Now, NextJS is talking to ButterCMS which we don’t control. So, Autify will be very important here to test that connection between ButterCMS and us.

Soon, we will be able to power an entire page; such as the home page and several landing pages with ButterCMS. This gives Marketing a lot more flexibility, and they don’t have to submit a ticket to Engineering to make any changes on the home page or landing pages. They are empowered to do it on their own. It’s really exciting to offer that functionality!

– So, the front end is talking with the ButterCMS API to join the content on the front end. Is it mainly the blog posts or are you managing another kind of content?

AJ: Right now, the main content that ButterCMS manages is our blog posts. We are also planning to power our entire home page with ButterCMS plus enable Marketing to create new landing pages. Basically, different versions of the home page for whatever campaigns they are launching. And all of that can be powered by the same CMS.

– For the React one, do you have two repositories? Is there one for E-commerce and the other for the website?

AJ: No, we still have a monolith repository. But in the repository, we have various different applications in it. The NextJS application is one folder, the Ruby application is another, then we have our Asset Management application, and our main application is the Android app. We still have one monolith, but we are able to have all applications living nicely with each other.

– You would be a great use case for that, testing your website as well, because we are using Autify for testing our website too! It could be interesting that you involve your marketing folks to test out those kinds of landing pages because they are going to build a lot of marketing landing pages.

AJ: Yes, Marketing is asking for more flexibility to manage our sites and store. Having a CMS is an easier way for us to achieve this. Since we don’t have to rebuild the CMS in our site, we can display whatever components or content that Marketing desires to create.

– Does that requirement come from Marketing as well?

AJ: The critical things like product catalog will still probably talk to our Rails backend because it manages all of our item inventory and customer orders. We don’t have any plans to transition it to a different backend due to the need to control it. The front end communicates with ButterCMS and the Rails backend.

– Cool! Thanks a lot. I think I asked you everything I wanted to ask you.

AJ: You’re very welcome. As a matter of fact, you just gave me the idea to test our blog’s production environment.

– Yes. Let’s do it! It should be very helpful. Do you have any questions, feedback, or any request for us?

AJ: Not right now, I think so far so good. Though I have a random test question about one of our scenarios, but I will ask Support about that. Your support has been really helpful and responsive.

– Great to hear that! Sam is great. Would you recommend Autify to others?

AJ: Of course! Yes! I wouldn’t have said yes to this interview if I wouldn’t.

– Cool! It’s really great to hear that we can support your development flow. I would really want to contribute a lot more to your development process. So, feel free to reach out to me and give us any feedback, requests, etc. We are building our functionality, so your feedback is precious. Thank you, AJ!

(Interviewer: Ryo Chikazawa, CEO, Autify Inc.)