A Complete Mobile QA/Application Performance Management (APM) Strategy

Image Courtesy: Flickr

You want to create and maintain a valuable, optimized mobile service to your audience. There are different solutions and approaches for effective performance assurance of your mobile service over time. It is important to understand and optimize the performance of your mobile service in different scenarios: pre production, post production, under load etc. A more modern aspect of mobile data is the rapid evolution of rich media content (not only in the context of entertainment, but also as brand promotion, product description and reviews etc.).

This post describes different mobile APM and QA solutions and what constitutes and complete solution. To start, I'd like to describe several macro-trends and challenges that affect the space.

First, Mobile is unique ;).

Retailers/marketers ability to offer targeted services to their customers on mobile is better than desktop. That highly targetable audience had high expectations: in 2011, Compuware found that 71% of all users expect mobile services to match or exceed the availability and performance of desktop experience. Since the competition is around the corner, failure is expensive. More expensive on mobile.

Compared to services delivered to desktop users, mobile devices are highly fragmented: screen sizes, OS variants, OEMs and device features, behavior under different 'environmental' conditions such as the wireless (cellular/wifi) signal and protocol, number and composition of applications running on the device, driving resource consumption (memory, battery, throughput) etc. Further, mobile phones and tablets are not production-testing designed devices: under ongoing testing, they will suffer inherent stability issues. These certainly aren't enterprise-quality servers. To add, phones and tablets are expensive! Scaling a QA/APM solution is challenging.

Another uniqueness of mobile, specifically in B2C scenarios, are resident apps. Most desktop experiences are delivered via "standard", limited set of (almost) browsers. Apps are more difficult to QA/APM. A former carrier employee once told me: "The trick is not to test the phone or app. It's to test it in random combination with other apps".

Another interesting trend is recent closer integration between development and operations. Deployment cycles are getting shorter and even 'heavy-enterprise' applications are more open for experimentation in return for user feedback as genuine contributor to the optimal product composition. As a result, functional testing and application performance management are becoming closer. Last, true rendering of mobile experiences are an integral part of the QA cycle.

The market wants a unified solution: QA+APM. The breadth of the deployed solution depends on a growing maturity of the Dev/IT/Ops organizations in leveraging value from the solution. At all times, RoI from functional QA/APM solutions needs to be clearly demonstrated to the LoB (prevention of outages, revenue/customer loss and brand damage).
There will likely be more than one technology buyer. A QA/Dev representitive will usually make the initial purchase. Initial APM buyer will be somewhere between QA and IT/Ops. Eventually, every mobile APM strategy will become part of the overall APM strategy and owned by IT/Ops. Assuming functional QA starts with physical phones, the interest will be to proceed with real-device based APM.

In my view, the ultimate mobile APM/QA solution consists of the following toolsets, integrated into a central reporting solution:

• Functional Mobile QA & Automated Testing: Pre Production testing of mobile services, on real handsets. These can be offered onsite or remote (dedicated or shared). The main point is to run tests on real phones in as many scenarios as possible, representing use cases, environmental parameters (location, carrier etc.). I call it "Clean room QA and regression". While testing on real devices, ensure you get screenshots or even videos of the transaction. Companies who offer this service include Perfecto Mobile, Jamo, Device Anywhere (Keynote) etc.
• Crowd Sourced QA: Unlike the structured QA, crowd source QA can offer insight to issues caused by human users in unplanned scenarios, which may contain use cases, combinations of applications running on the phone, different locations, carriers etc. Companies who offer this service include uTest. Unsure who else ;)
• Proactive APM: Ongoing monitoring of mobile services, from the end user perspective. Both pre and post production. These are robots located where the user is, running recurring scripts. There are several options: Real phones/tablets vs. simulators. Onsite vs. SaaS. Ethernet vs. over the cellular carrier. My preference would be toward simulator-based remote monitoring in a SaaS environment, using both the wireless carrier and a wired connection. Very close to the end user experience. Simulators can be instrumented to provide signal strength, CPU consumption, application metrics and much more. It's cost-effective with lower management overhead. It's a clean room environment for ongoing monitoring. Who provides this?? unsure.
• Real-User based APM - SaaS: This approach lets your users work for you.  Passive tags are instrumented by the service owner into the app/website, and as your users access those, data is generated, describing the end user experience (therefore, it's all post production). This approach is scalable, insightful and real. Moreover, W3C is defining timing APIs for browsers as well as resource timing APIs. This is really exciting: imagine mobile users could submit complete HAR files describing the real user experience. There are a lot of startups in this space because all you need is simple (relatively) collection tags, and the heavy smarts is in the data analysis. It's been a pleasure hearing a retailer saying: "For as long as I was proactively monitoring my site, I had no idea of the countries, devices and experiences my users were going through. As soon as I turned on real user monitoring, a whole new reality dawned on me. My users are mobile, they travel and consume my service from everywhere, specifically from places my proactive monitoring does not exist". The SaaS approach here really means that there are no on-premise installed components. So you have limited insight on what happens on the backend. Companies offering this service: Compuware, Yottaa (I think), others  
• Real User-based APM - On-Prem: Like the above, except there are on-prem installed agents or appliances. The big difference is the ability to correlate between the transactions users generate and the response time of each server in the backend. It could be extremely insightful and help identify and resolve issues quickly. (It's worth though keeping in mind that a lot of mobile services leverage 3rd party components. So if your 3rd party vendor component does not perform, these on-prem agents you have installed on your backend, really don't offer much insight.). Companies offering this service: Compuware, others.

In the spirit of optimizing an app/mobile website, you can't forget analytics. So important. What's really exciting (to me) is advances in analytics done by W3C (timing APIs) and really cool innovation like what ClickTale are doing (seen it in action, it's awesome!)


OK, this came out a bit long, but I hope it gave some insight to whoever was looking for APM+QA solutions for mobile. Nobody has the complete solution, I'm sure VCs are creating complete portfolios around similar concepts.

As always, very interested in your feedback!