Laws in Custom Software Development: 80/20 Rule in E2E Test Automation

I’ve always been intrigued by the 80/20 principle, or the Pareto Principle as it’s often called, which asserts that 80% of results come from 20% of efforts. It’s one of those ideas that, once you start looking, you begin to see it everywhere. In sales, marketing, and even personal productivity, the rule holds true. But it wasn’t until I applied it to software testing and custom software development that I truly appreciated its power.

At CodeSuite, our aim to deliver the best custom software solutions has always driven us to explore more efficient and effective ways to handle testing. 

As an SQA engineer, I’ve spent years refining our approach to testing, especially when it comes to end-to-end (E2E) test automation. And it was during this process that I discovered just how valuable the 80/20 principle could be.

The Challenge: Balancing E2E Test Coverage with Resources

Software testing is often a challenging process. The complexity of modern applications demands thorough testing to ensure everything works as expected, but there’s always a trade-off between coverage and resources. Early in my career, I encountered two extreme viewpoints within testing teams:

1. 100% Test Coverage: This is the ideal, often seen as the gold standard in testing. However, achieving 100% coverage in E2E test automation is not only costly but often impractical. The ongoing maintenance and skill required to achieve and sustain this level of coverage can be challenging, especially for smaller teams.
2. Minimal or No Test Automation: On the other end of the spectrum, some teams opt for minimal or no E2E test automation, relying heavily on manual testing. While this might seem like a cost-effective solution initially, it quickly becomes unsustainable, particularly as the application grows in complexity.

Both approaches have significant drawbacks, and neither is suitable for most software projects. As we at  CodeSuite explored further into our projects, we found that the answer lay somewhere in the middle, guided by the 80/20 principle.

The Discovery: 20% of Automated E2E Tests Cover 80% of the Code Base

In one of our early custom software projects, we were faced with the challenge of testing a large and complex e-business site. It was clear that achieving full test coverage was not feasible given our time and resource constraints. 

That’s when I began to wonder: Could the 80/20 principle apply here?

The idea was simple yet profound: by focusing on automating 20% of the E2E tests, we could cover approximately 80% of the code base. 

This wasn’t about cutting corners but about being strategic. We identified the core business scenarios—those that were most critical to the application’s functionality and user experience—and prioritized them for automation.

For example, the site had various payment scenarios, such as an order with a single item, multiple items, or one item in multiple quantities. These scenarios were all driven by a few underlying code components—perhaps a single Java class and a few web pages with JavaScript. By targeting these key areas, we could achieve broad coverage with a relatively small number of tests.

This approach not only saved us time but also allowed us to focus our limited resources on maintaining these critical tests, ensuring their reliability over time.

Implementing the 80/20 Rule in Test Automation

With the 80/20 rule in mind, we set out to automate around 20% of the user-story-level E2E tests. Based on my experience, I estimated that this would equate to about 50 test cases out of a 250. 

It’s important to note that this isn’t a hard and fast rule—some projects might require more, others less—but it provided a practical target for our team.

To our surprise, this smaller set of tests covered the majority of the codebase effectively. We were able to identify and fix most critical issues without the need for exhaustive testing of every possible scenario. The key was focusing on the most impactful areas of the application.

One of the most significant benefits of this approach was the ability to maintain the tests more easily. With fewer tests to manage, our team could devote more time to ensuring that the automated tests we did have were robust and reliable. This was crucial, as we soon discovered that test maintenance often accounts for a significant portion of the overall test automation effort.

The Importance of Test Maintenance: 80% of Effort Goes into Test Stabilization

One of the lessons we learned early on at  CodeSuite was that test creation is only a small part of the test automation process. While creating tests can be exciting—especially with modern tools like Selenium—it’s the maintenance that truly demands attention.

In fact, I’d estimate that about 80% of our effort in test automation goes into stabilizing and maintaining the tests. A single code change, whether it’s in Java, JavaScript, or even CSS, can lead to numerous test failures. Given the frequency of changes in an Agile environment, this maintenance effort is constant.

This realization was a major turning point for our team. Rather than being disheartened by the amount of work required to maintain our tests, we considered it an essential part of the process. We invested in tools and practices that made maintenance easier, such as using more flexible locators in Selenium and implementing a robust CI/CD pipeline.

A Balanced Approach: 80% Automated Regression Testing, 20% Manual Exploratory Testing

Another area where we applied the 80/20 principle was in our approach to regression testing. Regression testing is essential for ensuring that new code changes don’t break existing functionality. However, relying solely on manual testing for regression is both costly and time-consuming.

By automating 80% of our regression tests, we could ensure that the most critical areas of the application were consistently tested with every code change. This left the remaining 20% for manual testers, who could focus on exploratory testing—an area where human intuition and creativity are invaluable.

This balance allowed us to maximize the effectiveness of our testing efforts while keeping costs under control. The automated tests provided a safety net, catching the most common regressions, while the manual testers could dive deeper into less predictable areas of the application.

The Results: High Test Coverage, High Reliability

Applying the 80/20 principle to our software testing and development efforts yielded impressive results. With just 20% of the possible E2E tests, we were able to achieve about 80% code coverage, significantly reducing the risk of critical issues slipping through the cracks.

Moreover, our focus on maintenance meant that these tests remained reliable over time. Unlike some teams that struggle to keep their automated tests passing consistently, we were able to maintain a pass rate of over 95%—even with daily production releases.

This high level of test coverage and reliability gave us the confidence to move quickly, iterating on the software with frequent releases while still ensuring a high-quality user experience.

The Takeaway: Apply the 80/20 Principle in Your Projects

The 80/20 principle is more than just a rule of thumb; it’s a powerful tool for making strategic decisions in software testing and development. 

At CodeSuite, we’ve found that by focusing on the most impactful areas—whether it’s the 20% of tests that cover 80% of the codebase or the 80% of effort that goes into maintaining those tests—we can achieve better results with fewer resources.

Of course, the exact percentages may vary depending on your project. But the underlying idea remains the same: by being selective and strategic in your approach, you can maximize the impact of your testing efforts.

If you’re struggling with the challenge of balancing test coverage and resources, I encourage you to give the 80/20 principle a try. Identify the core areas of your application, focus your testing efforts there, and don’t be afraid to invest in maintenance. The results may surprise you.

At CodeSuite, we’re committed to helping our clients overcome these challenges, leveraging our experience and expertise to deliver high-quality software that meets their needs through custom software development services. 

If you’re looking for a partner to help you implement effective software testing services, don’t hesitate to reach out. 

Together, we can make your software development process more efficient, effective, and successful.