When launching a new app, everyone faces the crucial phase when you put together the final touches before unveiling “the next big thing” to the market. Only after you've double- and triple-checked everything do you deem the product ready to take the first train to the app store.
But wait! Are you sure that all of the bugs have been squashed? According to research conducted by Ubertesters, 3 out of every 10 apps placed in the app store arrive with critical bugs that nobody noticed. Think about it: That means 30% of apps in an app store are playing Russian roulette with users, and the result could be the complete destruction of what you thought would be the perfect startup. That's why beta testing is even more critical than it was back when Java apps ruled the landscape.
But how many testers should you use? That's one of the main barriers that companies face when they start looking for beta specialists. There are many facts to consider before initiating your beta test, but determining the number of beta testers you need can be boiled down to the following four criteria.
1. Project size
The first factor affecting the size of a beta-testing team is the product target audience and market share. How many users you are expecting (project workload), and why is your target audience interested in using your app?
Generally, you only need about 5-10 testers for small, independent projects, 25-40 testers for casual use apps, and 200 or more testers for gaming apps. You can split project sizes into these five categories.
| Project size | Team size | Minimum number of beta testers needed |
| Tiny | 1-2 | 3-5 |
| Small | up to 20 | 10-15 |
| Medium | 250-400 | 30-50 |
| Large | 500+ | 200+ |
| Enterprise | 1,000+ | 500+ |
2. Project geography
Creating multi-market apps is an enormously serious challenge that can become a tombstone for your dreams. To avoid that fate, be sure that your app is:
- Is fully compatible with different carriers and local data plans (2G, 3G, 4G or LTE)
- Correctly supports local languages and dialects
- Is user-friendly for people with different cultural patterns
- Works well on local markets’ best-seller devices
Here are a few typical “minor” issues that might just kill your project:
- Currency formats that are locale dependent. A lot of dev teams do not pay attention to this point.
- Locale-based keyboards, which can cause issues. You can get unpredictable results when the language and the keyboard’s files are not configured for a specific locale.
- Database issues. Make sure that database collation doesn't do anything horrible to the non-ASCII characters that the language uses. Also, there's a specific task you must check if the Unicode format is specified.
- Image render issues. Renders for ASCII characters are generally smaller than those outside the ASCII character set, which can cause errors.
Such beta testing is not a task for one person or even a small, in-house QA team. To cover multiple demographics and geographic locations, your beta testing team should include around 100-300 testers, and you must write good test cases and execute those on actual devices.
Imagine that you need to test one app, on just one platform. With this, you want to look at how the size of the beta team will increase in response to an increase in the number of target countries.
| Number of regions (countries) | Minimum beta testers required |
| 1 | 10-25 |
| 2-4 | 75-100 |
| 5-10 | 120-250 |
| 10+ | 250+ |
3. Project complexity
If you want to test the functionality of your app, then you'll need up to 15 independent beta testers, as described above. But if you plan to have a “heavily-packed” app, with integrations with geographic information system, payments systems and POS terminals, and so on, then you'll need a much bigger testing group. Performance testing, load testing, and error testing must all be undertaken with realistic usage patterns, and under extreme loads.
Think about two different types of mobile apps: A local food delivery app and an online banking app. While both provide an easy to use app with easy navigation, these apps are extremely different in terms of complexity. Online banking needs to navigate and be compatible with many complex systems, such as ERP, Web services, APIs and more. such apps must also deal with many regulatory and security issues.
The point is, complex apps require more testers, and more testing hours. You should have each feature on your list checked at least three or four times by different people.
Typical bugs in complex projects like this include:
- Incorrect SDK/library integration
- GUI alterations that result in overlaps or misalignment of GUI elements and controls
- Missing or extra controls that lead to missing or broken functionality
4. Types of testing
While there are many different types of testing that you can use, here's a list of some of the most popular types:
Project managers calculate the time it takes for an app to be tested as 15-25% of the development time, without splitting it into different tests types. This means that even if you get the beta team of your dreams, some aspects of your app might not be well tested. That's why you should spend at least 20-25 hours for each testing type, depending on your project.
So, how many beta testers is right for your project?
How many beta testers do you need? These questions should help you develop the right answer.
- Who is my target audience?
- Do I have enough resources to handle the incoming feedback? How many people will help you correct and improve those issues?
- How will I collect the results from the test? Expect to have many people involved, and lots of reported issues. Choosing an efficient and convenient method of delivery is vital for your sake—and theirs. So select an appropriate management tool to allow testers to easily report issues from within the app itself, and for the project manager to easily get the reported issues, and manage and control the beta testing process.
- Am I conducting a technical beta or marketing beta test? A technical beta test focuses primarily on finding bugs and technical issues. A marketing beta test is all about eye candy and how users can easily interact with the interfaces. If you require a technical beta, then you need to search for a service that provides professional, experienced testers, with background and QA certificates, not just beta users.
- When will I be testing? Consider the timing of your test. Will it overlap with a holiday such as Christmas? Did your test just happen fall be during the peak season of summer, when everyone is on vacation? Think about your beta testers and how holidays may get in the way or slow down your testing.
How do you decide on the number of beta testers you need? I look forward to hearing your tips and suggestions.
Main image credit: Flickr
Infographics source: Ubertesters
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
