The Keys To Mobile Test Automation That Works!

In the world of mobile app testing, a developer’s claim “Well it works on my device” is not acceptable. Mobile is by far more fragmented than Desktop/Web platforms due to the variety of device types, OS versions, and the environment on which it operates (network impact, roaming, battery stretch etc.).

It is not sufficient to test on one or even a few devices, but instead test apps on several devices that adequately cover your end-users’ device pool. How can you keep up with testing apps on so many devices, you ask? Easy. Build a working mobile test automation strategy that runs the same script across all of your devices, making your Dev/QA teams confident that they are delivering the best app to end-users.

From the experience Perfecto Mobile gathers everyday, we came up with the must-haves for building a continuous, unattended mobile test automation strategy that provides immediate feedback to its Dev/QA teams.

Relevant Device Selection

It is one of the toughest dilemmas in today’s market to identify the right device X OS mix on which teams should develop and test their mobile app against. Base your selection on the right criterias such as:

    1. Target market and competition/End-Users –> On which devices and OS versions your end-users actually use the most (based on organizational analytic data)
    2. Generic market statistics –> Top sold devices (smartphones and Tablets) in the relevant regions. Such data can be retrieved from analyst research such as IDC, Flurry, ComScore and others.
    3. Change rate and market dynamics –> As we all see, the mobile market is in constant change and your teams need to “listen” to the trends and changes- adopt them and change the strategy accordingly

Unattended mobile app test automation which supports continuous integration and fast feedback:

    1. Mandates mobile OS system level control – If the script cannot initiate events outside of the AUT (App under test) it cannot really test the app as the end-user will use it
    2. Incoming alerts and error handling are also mandating system level control out side of the AUT
    3. Hybrid objects support – automation framework should support both OS level objects (DOM for web and hybrid apps) and Visual objects for full use case support
    4. Device logs, rich media reports (Video and Visual) and vitals (Memory status, Battery etc.) are critical evidence to Dev/DevOps teams as feedback to act upon

Device Elasticity

As your mobile app project grows/shrinks based on your SDLC phase, your automation solution needs to be flexible enough to accommodate necessary changes and provide the needed support to your team (even more critical in cases of distributed teams).

Integration is Key for Cross Team Collaboration

Whether your Agile/Dev/QA teams are using ALM tools (HP, Microsoft, etc.) or OSS (Selenium, Robotium, JUnit) tools, your automation framework solution should be able to connect to these tools and bridge any gaps in the daily continuous work flow. Use what your team knows – this is the best way to release mobile apps faster without compromising quality.

Real Devices

It is imperative to understand that testing on something other than real devices in your critical automation scripts and as your mobile app evolves is a huge risk. Only real devices running formal OEM OS flavors against real networks reveal an app’s true functionality in the same way your end-user will experience it.

To conclude, see below (Fig 1), a check list of mobile testing pillars which need to be part of your Mobile App Testing Plan.

Fig 1: The pillars of mobile app testing

Good Luck!

Leveraging Real Network Conditions As Part Of The Continuous Quality Processes

As the digital experience becomes key for organizations to offer end-users and a high user experience based on mobile moments, it is important to understand the environment in which mobile apps are “living” in.


We have taken to calling a mobile app’s environment as a wind tunnel – many different conditions and “states” occur, causing mobile apps (Web and Native) running on various screens (Smartphones, Tablets, Wearables) to behave differently with functionality and performance POV.


In order to extend a continuous quality process and enrich its coverage, mobile app quality ought to include real network conditions as part of the test matrix as early as possible in the mobile app development lifecycle.


To achieve coverage of real network conditions developers and network engineers should add the following scenarios into their test plan:

  • Network virtualization – What happens to your mobile app in various network conditions, through the use of various profiles which can simulate poor, average, good 2G/3G/4G/Wifi network connections in various geographical locations (see e.g. Fig1 below). Understanding the behavior of your mobile apps in degraded network conditions early is a key to drive your mobile app release velocity as well as increase your app quality.

Screen Shot 2015-01-22 at 20.21.07

  • Degraded application performance due to other processes, like apps running in the background, can be achieved through mobile device vitals collection, logs analysis and other examinations (e.g. many apps today leverage 3rd party apps like social media apps etc. which can impact the app under test performance)
  • Degraded server performance due to load (light load, heavy load) – how do your mobile app performs from response time and availability in peak times, in various network conditions?
  • Interoperability and events impact on mobile app functionality – how is your mobile app going to behave and respond when an incoming call, pop-ups, and other events which comes in while your mobile app is running?
    • To measure this impact mobile app vendors need to have full access to the mobile system (e.g. iPhone SpringBoard) so they can trigger the events, control them and debug them properly.

Screen Shot 2015-01-27 at 14.02.24

Figure 2: Perfecto Mobile Wind Tunnel various capabilities 

Some examples for more mobile app testing interruptions

  • Incoming Call
  • Text message
  • Other app notifications
  • Storage/RAM low
  • Battery low
  • Battery dead
  • No storage
  • Airplane mode
  • Intermittent connectivity
  • Home screen jump
  • Sleep mode



To read more about the continuous quality approach and the continuous quality lab which can enable such capabilities, visit the Perfecto Mobile web site:

Does Your Enterprise Have a Central Strategic Mobile Resource?

Mobile technologies are much more than a trend, and are no longer something that organizations and enterprises can overlook or pretend to ignore.

Mobile is already a significant market platform that generates substantial revenues, driven by the ever-growing use of smartphones and mobile applications. However, alongside these exciting business opportunities, mobile technologies and applications also introduce new types of risks that were not an issue in the desktop application world.

As mobile grows, the accompanying pains and risks require (even dictate) the adoption of a robust mobile strategy built upon a centralized organizational solution. Such a strategic solution should be based on an appropriate set of tools that can be used by management, development and testing teams wherever they may be located around the globe.

Mobile applications require enterprises to take into account several mobile-specific components that are not relevant to the desktop space:

  • Governance and availability of devices (smartphones and tablets). The extremely dynamic and fragmented mobile market requires continuous testing of mobile products. To ensure timely delivery to market, you must have access to an up-to-date set of mobile devices at all times.
  • Global solution which enables offshoring, testing and collaboration among diverse and distributed teams
  • Robust manual testing solution to handle user interface and other “look & feel” issues
  • Cross-device test automation solution in order to accelerate TTM and to enable QA to efficiently perform continuous regression testing while moving from one version of the product to the next
  • The ability to plug a dedicated, mobile-specific testing platform into your existing ALM tools, processes and organizational skill sets (i.e., seamlessly integrate with existing ALM quality suites)
  • Mobile Application Performance & Monitoring (APM) must be taken into account. Availability and response time for your mobile application are critical for business success. Studies show that mobile users are less tolerant than desktop users to performance and availability shortcomings.
Figure 1: World Quality Report 2012-2013, CapGemini

When formulating your mobile strategy, do not take the risk of overlooking these important aspects of mobile testing. In addition, the latest World Quality Report (as illustrated above) stresses the importance of using a cloud-based, centralized organizational resource to enhance agility, competitiveness and cost-effectiveness.

Using this blueprint, Perfecto Mobile has deployed mobile testing solutions for leading enterprises and mobile carriers worldwide.  Our experience shows that addressing these key pillars assures ongoing mobile quality while cost-effectively streamlining the entire mobile lifecycle for the various mobility groups (Development, Testing, Performance and Operations).

So here’s the takeaway:

Mobile is a serious business platform which is getting more complex and fragmented every day. New devices will continue to pop up and new mobile OS versions will be released with increasing frequency, making mobile testing that much more difficult.

To thrive in such an environment, enterprises ought to focus their time, energy and resources on developing high quality mobile products that serve their customers’ needs. Integrating the right mobile solution with your existing tools and processes will allow your team to focus on developing real mobile testing scenarios, automation, etc., rather than dealing with organizational issues. By supporting the pillars listed above, your mobile testing solution can serve as a central strategic resource across the enterprise IT environment.

Perfecto Mobile Introduces Full-Featured, Non-rooted iOS Mobile Testing in the Cloud


I wanted to share with you a fresh announcement from Perfecto Mobile.

Today Perfecto Mobile announced its new technology solution aimed for iOS mobile testing.

This solution enables full-featured remote iOS iPhone and iPad testing in the cloud that is 100% transparent to the device under test with no device rooting (jailbreaking).

With this enhanced and complete solution Perfecto is now able to support any iOS device or version in the MobileCloud(TM) without having to jailbreak the device.

The MobileCloud users can remotely work on any iPhone or iPad, running any iOS version, operating in various carrier networks around the world.

The solution uses a complete non-jailbreak approach with the following benefits:

  • Full device control over native application, web based services and pre-installed iOS applications as well as full springboard control via the cloud with digital video support
  • Remote manual and automated testing, as well as real-device monitoring
  • Fast time-to-market for iOS apps and services with maximum quality and device coverage


Figure 1: Perfecto’s new iOS testing solution for native applications using OCR

For the full press release, please refer to this URL (LINK)



Best practices for iOS mobile application testing


iOS changed the mobility game, no doubt about it. It paved the way for the ‘mobile era’ by offering amazing functionality with a simple user experience.  However when it comes to testing and monitoring, working with the iPhone/iPad mobile application can be anything but simple…

As the iOS app market continues to produce record growth, challenges and complexities surrounding iOS application testing also continue to interfere with development. A key challenge of iOS testing is that, unlike the open-source Android OS, Apple iOS is a closed operating system. Added complexity during the development and testing stages arises with a closed system, since users can’t extract necessary data from low level objects, which are essential for test automation. So, what’s the best approach for getting the necessary level of access to the iOS device – rooting (jailbreaking) or compile-time source instrumentation? Should you base your testing on native objects or OCR-based screen analysis?

Let’s take a deeper look into some of these challenges and why a cloud-based hybrid approach is important to offer developers and testers the necessary coverage, capabilities and flexibility to deliver better iOS apps and deploy them with confidence.

Rooting (jailbreaking) vs. Source Instrumentation (compile-time)

There are two common methods used today in the mobile testing industry to address this challenge (i.e. access to the low level objects): rooting (jailbreaking) and source instrumentation (i.e. compile-time solution).

Jailbreaking refers to the process of removing the limitations placed by Apple on the iOS device in order to get low level (root) access to the operating system. This allows the tester to be able to recognize the objects within the application being tested.

Source Instrumentation is performed by compiling the application being tested with an additional piece of code that provides access (“back door”) to the low level OS for object recognition. This code enables the tester to execute the low level calls and get the Object ID’s from the operating systems (without the need to root/jailbreak the device).

The decision as what approach to adopt strongly depends on several considerations (below are just few):

1)    The used SDLC process

2)    Corporate policies

3)    Application under test

4)    Frequency of testing

Perfecto Mobile provides its end users with the freedom to choose what fits them best, while taking into consideration the advantages and disadvantages of each approach. When customers need to quickly test either a new iOS version or a new iOS device, the jailbreaking approach is less suitable. In such a case, the compile-time method is preferable – even though it complicates the SDLC by introducing additional code to the application being tested.

On the other hand, using a jailbroken device lets you test the application with the exact code by which it will be released (compile-time mandates that before store submission, you will remove the “back-door” or be exposed to serious security issues). This eliminates the need for compilation and intrusive operations which could potentially pose a risk to quality. Companies using a compile-time approach should also consider possible regulations (such as HIPAA) which enforce testing on the final binary (and not on debug version, test friendly version, etc.)

The combined (hybrid) approach lets you choose which type of tests to implement on which iOS device according to the nature of your application, project needs, and policy. When the test devices are deployed and securely managed in a “private cloud” (such as that offered by Perfecto Mobile), such a configuration guarantees that the jailbreak method does not introduce any risks or abuse of the platform for non-testing purposes. The jailbroken device is used only for testing purposes in a closed and secure testing environment. This is analogous to the use the way iOS devices used for development have a “developer signature,” as well as the way Android devices used for development have more levels of access than those required during the normal ALM cycle.

The Need for a Hybrid Approach to Object Recognition

Testing a mobile application requires strong object recognition capabilities. The use of visual analysis might not be sufficient, for example, the OCR technology can detect UI issues and glitches on the test devices, but cannot ensure 100% accuracy due to its heuristic nature. On the other hand, low level objects might “miss” the obvious qualification that a visual analysis can easily detect. That’s why a hybrid approach incorporating both visual and Native object analysis is imperative for covering all mobile business cases. Such an approach is supported by Perfecto Mobile.

Object level analysis vs. Visual analysis

This screenshot above shows the differences of using an object level analysis as opposed to visual analysis (object level analysis would not have detected the overlapping of the button on the text).

The Perfecto Mobile Approach: Go Cloud, Go Hybrid

Perfecto Mobile’s experience as a market leader has taught us that the best approach is to present each customer with all possible alternatives making them available inside the cloud.

Real devices + emulators (in the cloud),  OCR screen analysis + OS level native objects (in the cloud), rooted/jailbroken device + non-rooted/jailbroken devices (in the cloud)

With hundreds of thousands of automation hours running every month on our platform, we are well-positioned to suggest and guide, but not to “judge” what’s best for everyone…

Perfecto Mobile hybrid object support on a rooted android and a non-jailbroken iPhone


Eran Kinsbruner

IDC 2012 report around mobile and additional post on mobile testing


Redirecting to 2 new posts i’ve made on PerfectoMobile blog: