QA

Who Said Multiple Personalities Were Bad?

When asked about QA or more specifically Software Quality Assurance I always try to find creative ways to explain to people what exactly it is that we do. Often times I would compare what we do to what a building inspector does. I'd say, We basically check other people’s work to make sure they are up to code. Where the requirements would be our building code.

But in reality we are so much more than that.

If you truly wish to test software you would need to have multiple personalities. Let me explain. There are distinctly five different personalities you need to adapt when testing. Essentially you need to be the dork, the space cadet, the cheerleader, the sneak and the rebel (much like the recipe to the old 80's teen movies). These personalities make up much of the Internet's population and sadly are also the cause of many undue headaches. Allowing one personality to dominate over another really depends on your targeted audience, your required level of security and how much time you have to test your projects. The personalities are described as follows:

The dork or what I call “By the book Bob” is someone who follows all directions. He is told exactly how to use the system and he reads every instruction and never deviates from the coder’s happy path.

The space cadet or “Lazy Lucy” is someone who doesn’t bother reading the instructions at all. She would use the system the way she wants to and the way she assumes it should work.

The cheerleader or “Enthusiastic Ellen” is someone who doesn’t read the instructions and loves to try out every little function the application has. She’s extremely excited about the application and can’t wait to try literally everything the company releases out.

The sneak or “Hedging Henry” is someone who is trying to exploit the application to his advantage. He would see if maybe he wouldn’t have to pay for a service. Or maybe he would “borrow” content. Better yet why not exploit this site so that he can make some money off of it.

And finally the rebel or “Dangerous Dan” is someone who just wants to crash your system. Why? It's fun to do so. If he can make it so that your company can lose a day of revenue, all the better.

People always wonder why it takes so long to test something. Well. There are about 7 billion people in the world. All of which can be categorized into these five personalities. If you really want your system tested properly, your system would at least have to go through testing five different passes. Add to that the time you'd need to regression test all the features when something is touched. But that's a whole different post right there.

Just something to think about … Happy Testing!


Experience with Appium So far

I got a chance to go to Selenium Conference in Boston this year. It is always interesting to know how big shops like Facebook, Google, etc. manage their QA architecture, what new frameworks / technologies they are working on, and what can we learn from others. One automation testing framework for mobile which was talked about a lot was Appium.  Appium uses Webdriver Json wire  protocol to drive iOS and Android apps. Since our own iOS app at Rent The Runway is under development, I thought this would be the best time to try my hands on Appium and have some cool automation test scripts ready for our app.  And now at this time when I have some pretty scripts ready for our app, I thought why not write a blog to share my experiences with Appium so far. When I initially started setting up Appium, I had a very little idea how to read the page source for the app to interact with it. After a little bit of searching I found something that they call an Inspector. The way Inspector works is it makes Page source calls to you app and displays the element hierarchy for you, making it possible to find the name, value etc of the objects on the app.

I use the page objects framework to write automation tests for the mobile app.The way I have set my framework is I have rake tasks in the Rakefile to run regression/ smoke iOS tests. Since eventually we will have a Jenkins job to run these tests I have not hard coded  Appium serverURL and app path parameters in the code. Instead, I pass them in using the command prompt.

So if I have to run my regression suite for iOS, it is as simple as

rake ios:default  IOS_TARGET="appium server url  "" APP_PATH='location of the app path’

where ios:default is a task defined in the Rakefile.

We use watir webdriver to write scripts and by passing the correct capabilities object in the script, you are good to go.  I would say I am pretty satisfied by how Appium has come up so far and I am very happy with the progress I have made with it but there are some challenges which I am facing.

Current Challenges and things to Expect:

I have often found that once my tests fail, the Appium server goes to state saying “clearing out any previous sessions” and it never comes out after that. So if I have to run my tests again I have to restart the server in order for the tests to succeed. In my local testing that's ok but when I add this job to Jenkins this causes major problems upon test failure.

I would also love to have these mobile tests run in parallel. Unfortunately, there is not much documentation out there on how to register a node to the Grid and run parallel tests using Appium. So hopefully my next blog will cover how I managed to solve my current challenges and run Appium tests in parallel. :)

Till then Keep Using Appium!


Parallelizing Test Execution: Speed Run!

Browser tests (sometimes) take a long time to finish. If your automated tests do things such as registering for a new user, adding a dress to a shopping cart, and then renting that dress, while checking along the way that each step is good, it might take upwards of 2-3 minutes. Combine that with the fact that each test file has multiple test cases, and a file might take upwards of 15 or more minutes to finish execution. Now, with multiple test files, your entire test suite might take upwards of an hour to finish execution. This isn't conducive to agile programming. The solution: parallelize your test execution. With Ruby and RSpec, this is handled easily with the addition of two gems: parallel_tests and parallel_split_test. You can parallelize test execution of the test files and then the test cases within the files themselves.

With parallel_split_test v0.3.0, you can use regular rspec test options. This means that you can pass custom formatters and write to specific output files. This comes in handy when you want to create JUnit formatted xml files for Jenkins or other continuous integration systems.

Nat Ritmeyer has written a custom RSpec JUnit Formatter that I've been using: http://www.natontesting.com/2012/05/25/rspec-junit-formatter-for-jenkins/

After installing the parallel_split_test gem, execute the test using this command: parallel_split_test my_spec.rb -r ./junit.rb -f JUnit -o results.xml

NOTE: The one caveat is that parallel_split_test will generate an xml file that contains multiple root nodes. I've attached a function below that will normalize all .xml files in the specified folder.

[gist id=5398225 bump=1]

Using parallel_tests and parallel_split_test, we've managed to cut our execution time from 30+ minutes down to 3 minutes. Theoretically, your tests should only take as long as your slowest individual test case. Your only limitation now is how many nodes are available to your Selenium Grid. You are using Selenium Grid, right?

If you have any questions or comments about parallelization in test execution, you can leave a response in the form below or find me on Twitter at @brokenthumbs. Enjoy!