Tuesday, February 5, 2013

Using Symbiote to Inspect the DOM on Mobile iOS Applications

If you have been looking for a firebug like application for inspecting an iOS mobile application DOM, let me introduce you to Symbiote from Pete Hodgson. We discovered this tool as we were adding support for the Frank Mobile Automation Testing into our Awetest Framework. Symbiote is an embedded web server that is added to the iOS app when you "Frankify" the app with libraries from Frank. Symbiote enables you to inspect the current page UI hierarchy to see the accessible elements which can be used for test case development and script level validations. As you move from screen to screen in the app, the Symbiote UI elements and accessible elements change; This gives feed back for you to complete your test case. To get started:

Download Frank
Install Frank
Open your Xcode project
Frankify your native app (see the install video link in step 2 above)
Compile and open the Simulator with your app
Open the symbiote viewer app by using the URL http://localhost:37265 in a browser
Install cucumber via command line
Create a test case in cucumber and add the file to the feature folder. Use the Symbiote viewer here!
Run the test case against the app from the command line by typing 'cucumber'
Your test case will run. Use the Symbiote viewer to view the run if needed.

Using Symbiote dramatically improves the speed of your test case development in Frank. As a technology the Awetest framework fully supports the creation of test cases in Frank and also improves the test case development cycle by adding specific value-added features. For example if you ran your test cases through Awetest you could do the following:

Schedule your test case to be run in the future
See detailed reports with pass/fail/validations and screenshots
Run against remote machines
Target the test case to be run against iPad/iPhone and specify the iOS SDK
Run multiple scripts at the same time
Split test cases across multiple simulators
Export reports to PDF, Excel
Use and manage a group of simulators for test case runs
View Symbiote during the test case run on remote machines

Links
http://moredip.github.com/frank_at_selenium_slides.html#slide13
https://github.com/TestingWithFrank/symbiote

Beta Testing by Distributing Your iOS (simulator) App

One of our Mobile App customers had mentioned what a challenge it is to distribute iOS apps for Beta testing and as an going commitment to supporting iOS app testing we have come up with a simple solution to this problem. To understand this challen
ge you have to understand that today if an iOS app developer wants to have his app used before publishing to the Apple store he has only 2 known options:
1) Distribute the app using Apple's Ad-Hoc mechanism: Though the registration process is straight-forward, it still involves a compile and distribution for the initial set of devices and this whole process is repeated for every new device that is used in the testing process. So if you want to test on the new iPad 3, guess what... Yes you have to register, recompile, redistribute to get that app on your new iPad. Also, using the Ad-Hoc distribution means that someone in the Dev/QA department has to pony up the funds to buy each tester an iPad which is great for the tester but might trigger some budgetary alarms. Hey, don't forget that you need to test your app on devices running different versions of the iOS. See the problem!
2) Share the source code for a recompile: This limits the flexibility of the developer. I don’t believe beta testers should be compiling code because it is error prone and could involve directory path issues and finally you may not want to share your source code.
Here's where emulator based testing becomes very useful (assuming the testers have access to Macs with Xcode). Developers can compile their application and share it with their test team for testing without requiring any additional registration or encountering a cap on the number of devices that are registered. Best of all the testers can choose the device and the iOS SDK emulate for testing. Here is the 10 minute process for doing this:

Step 1: Get your app ready to distribute

1) Create the APP file

Open the Xcode project
Under build settings/deployment change the Skip Install to "no" for both release and debug

Create the iOS simulator app by changing skip install setting

2) Under the Product Menu, build the target app for Running

create the app file by build for running in product menu

3) View the scheme for this target

Edit Scheme to control build configuration

4) Look at the scheme entries

Remember what build configuration exists in the “Run” entry in the tab on the left. This helps determine the location of the APP file.

5) Find the build dir for the project

The base directory for the build file location is set in the project settings for the project. Open the settings and look at the derived data location. Your app file will exist in that dir.

• If the build setting is set to Release the APP file will exist in [base dir]/[Release-iphonesimulator]
• If the build setting is set to Debug the APP file will exist in [base dir]/[Debug-iphonesimulator]
• NOTE: New app builds in Xcode have an additional build directory so the directory would be [base dir]/[GUID]/build/[Release-iphonesimulator] or [base dir]/[GUID]/build/[Debug-iphonesimulator]
http://stackoverflow.com/questions/5331270/why-doesnt-xcode-4-create-any-products

6) Distribute the APP files

Step 2: Running Your App On Another Computer/Emulator

1) Copy the app to a location on your Mac

# example
cp ~/Downloads/iWebDriver.* ~/[app dir]

2) Install Sim_Launcher via Terminal

# Code example
$ gem install sim_launcher

3) Launch sim_launcher via Terminal

$ sim_launcher

4) Open your app by inputting this URL into browser

http://[HOST]:[PORT]/[launch_[DEVICE]_app?[SDK]&app_path=[APP_PATH]
HOST= localhost if running on single computer
PORT= 8881
DEVICE = ’ipad’ or ‘iphone’
SDK = iOS SDK to run under
APP_PATH = Full path to app
Example: http://localhost:8881/launch_iphone_app?sdk=5.0&app_path=/Users/irschick/app/TheElements.app
Example: http://localhost:8881/launch_ipad_app?sdk=5.0&app_path=/Users/irschick/app/TheElements.app
Example: http://localhost:8881/launch_ipad_app?sdk=5.1&app_path=/Users/irschick/app/AngryBirds.app

Requirements to run your distributed app

Xcode 4.3.x
Rubygems
Ruby >=1.8.7
Have an SDK equal to source Xcode

Links:

http://stackoverflow.com/questions/3104127/ios-question-can-i-distribute-the-xcode-simulator-versions-of-my-app
https://github.com/landonf/simlaunch/
https://rubygems.org/gems/sim_launcher
http://stackoverflow.com/questions/1187611/how-to-install-iphone-application-in-iphone-simulator/4894178#4894178

About the Author

Michael Irschick, Product Manager, brings over 20 years of technical product development experience to 3Qi Labs including Systems Architect, Development and Technology Manager for Wells Fargo as well as Founder/Director for both Penguin Systems and Primitive Logic. As a product manager at 3Qilabs, Michael has molded integral portions of 3Qi Lab's approach to application development providing an expert understanding of how to leverage automation, testing best practices and specialized services to improve the quality of our web-based regression, mobile web, and native device app testing products.
More posts by Michael Irschick
Visit Website

Use a website to distribute your iOS app for Beta Testing

Steps to Distribute an iOS App using a Website

One of the challenges that exist in an iOS dev shop, is how to distribute your app to devices for Beta testing. The Ad-hoc process that Apple supplies for creating an app for distribution is well documented. But how do we actually get the app on to our devices for beta testing. Wouldn't it be great if we could just compile our app and put the app out on a website, then someone using an iOS device could just go to the website and install the app with a tap on a link. Best of all when ever you have a new version of the app you can just upload the app to the website and the users can just reinstall. Here I'm going to show you how to do it. It should take 10 minutes from start to finish.

Overall Process

Get the UDID from new device using iTunes
Add the device to portal
Create a distribution profile
Add the device to the distribution profile
Synch Xcode to bring in the new profile
Code sign your app with the new profile
Build the app for archive, create the app for distribution
Move the app, manifest file, profile to a website (code is included below)
Create a link on a website pointing to app
Give the URL out to users
User goes to website with a device and installs the app by tapping on link

Identify the UDID of the device

Open up iTunes, connect device, click on summary, click on serial number and the UDID will show up. Use the mouse to copy this. It is a long string.

Add the new device to the portal

Go to the developer.apple.com website and go to the provisioning profile. Click on Devices. Click on Add Device.

Add the UDID and Name

Paste the UDID that you copied from the device and type in a name. Do this for each device that you want to register.

Create a new Distribution profile

You can edit an existing profile and then add the devices to the existing profile, but for this process I'm going to create a new profile. Note that I'm using the distribution tab of the provisioning profile setup.
I'm assuming that your development environment has a valid certificate. If your development profile in the organizer is showing "valid signing identity not found" then go to the developer.apple.com website and follow the instructions for creating a valid certificate. You must create a certificate from the computer that you want to run or download from an existing on that has been setup. Note: Make sure that you only have one certificate in the keychain access.

Identify the devices for the profile

In the new profile, enter a name, the app ID, and then devices. These are the devices that you want to have access to your app. You only have 100 total devices per year per developer profile. Submit and the go to Xcode and from the organizer rsynch and the profile will brought in.

Give name to archive and code sign

Click on the Scheme then click on edit the scheme, assign a name, push ok

Go to the build settings, filter by code sign

Then Rebuild for Archive under the product menu. This will create an .ipa file.
Go to the Organizer/Archives click on the archive you created, click button DISTRIBUTE

Click on Ad-Hoc Distribution

Choose on the distribution profile to include

Save File

Create a simple website

Here is a simple website, with all contents that will enable a user to click a link

Website Code

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Sample iOS OTA install</title>

</head>

<body>
<a href="itms-services://?action=download-manifest&url=http://192.168.1.123/nativeApp/manifest.plist"><font size="+4">Install the App</font></a>

</body>
</html>

Plist File (use text wrangler to edit)

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<!-- array of downloads. -->
<key>items</key>
<array>
<dict>
<!-- an array of assets to download -->
<key>assets</key>
<array>
<!-- software-package: the ipa to install. -->
<dict>
<!-- required.  the asset kind. -->
<key>kind</key>
<string>software-package</string>
<!-- required.  the URL of the file to download. -->
<key>url</key>
<string>http://192.168.1.123/nativeApp/TheElements.ipa</string>
</dict>
</array><key>metadata</key>
<dict>
<!-- required -->
<key>bundle-identifier</key>
<string>com.yourcompany.TheElements</string>
<!-- required.  the download kind. -->
<key>kind</key>
<string>software</string>
<!-- required.  the title to display during the download. -->
<key>title</key>
<string>iOS Test App</string>
</dict>
</dict>
</array>
</dict>
</plist>

Send out an email with url

http://192.168.1.123/nativeApp/index.html
If the user clicks on the link the app will be installed

Links

http://support.apple.com/kb/HT4061
http://mobiledevelopertips.com/xcode/distribute-ad-hoc-applications-over-the-air-ota.html

About the Author

Michael Irschick, Product Manager, brings over 20 years of technical product development experience to 3Qi Labs including Systems Architect, Technology Manager for Wells Fargo, Founder of Penguin Systems and a Director at Primitive Logic. As a product manager at 3Qilabs, Michael leads the design and development of our first-in-class test automation software product Awetest. He actively works with customers and partners, both individually and in webinars and user conferences to understand the challenges that exist in enterprise software development and communicates how software test automation can provide solutions to these problems.
More posts by Michael Irschick
Visit Website

We need more manual testers... no really...

There will always be a need for manual testing. Certain types of interactions within an application are always too unique, too complex and too costly to develop automated test economically. You will especially need more manual testers because there is a huge wave of upcoming enterprise applications that are being developed for mobile devices and tablets.

Why is this? Well historically the enterprise has always lagged behind the consumer market in adopting and using devices, but no longer. The enterprise are now enabling their employees and their customers to use mobile devices and tablets to access applications that were typically only available on desktop browsers. QA now has to deal with this and the fact that mobile platforms have a fast release cycle and apps are released even faster. So more devices running more OS versions running more apps. Recipe for QA disaster. Even web applications that were designed for specific desktop browsers configurations (usually IE running on windows) now have to be tested on an array browsers and operating systems.

So if you kept the overall percent allocation of resources the same, the actual resources needed to handle the manual tests will still increase. But keeping the status quo is not an effective strategy in this scenario. You need to leverage technology and develop a strategy to deal with this growth. Specifically you need to evaluate the return on investment on developing automated tests for your apps and applications. Under a scenario with limited target platforms, some tests are more economical to just manually test, however under this new paradigm these same tests can now be economically automated. New situation means new decisions.

First automated tests are extremely effective in testing permutations of devices, OS, and browsers. A manual tester would have to cycle through all the combinations one click at a time to test the app or application. But an automated test which is expensive to develop initially, could usually be run against each combination with only minor alterations to meet the discrepancies that arise. Even apps that were developed for specific platforms like iOS or Android would be effectively tested with this approach.

Most importantly the return on investment would continue to accrue going forward. Manual test costs occur at every release, while automated test costs occur during their development, which mainly occurs upfront when they are initially developed. If you have the belief that your application or app has staying power for near future, then it is worth the investment to develop a long term strategy to leverage automated testing vs. staying with the status quo.

About the Author

The Enterprise can learn a lot about testing from developers

Alt Title: Why we chose to integrate Cucumber support for Enterprise automation

Cucumber is often used by a test driven developer to insure proper code functionality. He can write test cases and then code while using the test cases to insure the code works as designed. This also insures a couple of things for the developer. First it insures that he understands the business functionality; Second his code can be systematically tested to insure it still meets the business functionality over time. Because of the simplicity of cucumber, the developer can quickly develop these test cases. But the enterprise could also realize a lot of benefits if they adopt cucumber as well.

Here are the reasons to adopt cucumber:

Test cases can be understood by all key stakeholders: A key challenge with writing automation scripts in the enterprise is not that they can’t be written but can’t be understood by anyone except the person who authored it. In a world of complex business functionality with even more complex implementation, the QA developed test cases really add another layer of complexity to the SDLC. More often than not, Automation Tests have a shelf life that lasts the duration of a single release – a massive drain on the organization when you consider building automation takes 6-10x of the time of a comparable manual test. Furthermore, a lot gets lost in translation through the various stages of the SDLC from MRD> BRD >PRD > Test Cases (and Steps) >Automation Build out> Execution> Reporting> etc. etc. etc.

Test cases also take specialized skills to develop and they involve specific technology tools and knowledge like, Watir, Selenium, VBScript (for the HP/Mercury QTP crowd). With Cucumber effectively allows all 3 parties to at least understand the test cases.

There are a lot of more subtle side benefits that can accrue when you adopt cucumber.

Test cases are understood over time: The easy readability makes it easy for a new developer or QA member to alter and adjust the test case as needed. This is a big bonus for an organization.

Easy to adopt agile: It is easier to adopt agile methodologies within the organization. Because cucumber tests can actually be developed during the sprint not like the test cases developed in Watir, VBScript, Selenium. If the enterprise wants to adopt agile methodologies they need to eliminate road blocks, speed is a key road block. We have found that when we work with our clients they are constantly dealing with incumbent processes that are hard to change.

Test case building blocks: Cucumber makes it easy to develop a library of test case steps because the structure of the language has a natural separation of test case steps and the underlying implementation. Because of that there is a tendency to develop reusable steps while in watir test cases are often written in long script files without clear functional steps identified.

Can be used across all target platforms:

Mobile browser testing is complex compared to desktop based functional testing. In the pc browser, elements can be identified easily with the large range of automated testing tools that are available, but in the nascent mobile market there are few tools and for now and in the foreseeable future. Cucumber eliminates the dependency on the browser DOM by focusing and the user interactions from a visual and functional perspective.

Overall cucumber is easy to adopt within the enterprise and the potential return on investment is large enough to easily warrant its inclusion within the SDLC.