Job opportunity in the ASP.NET QA Team

There is an opening for a tester in my team. If you are interested in help building a test team that is strongly based on the context driven testing principles, then shoot your resume to Mark Berryman, we would love to talk to you.

You can see the external job post here.

- Federico

DevConnections slides and demos

Last week I presented at the DevConnections conference in Las Vegas. The topic was the Lightweight Test Automation Framework (LTAF) and the links to the powerpoint presentation and the demos are at the bottom of this post.

The conference is mostly for developers, so I tried to mold the talk towards practical applications of a test framework to enable product teams to catch regressions sooner. I asked at the beginning how many persons in the audience considered themselves as “testers” and only 4 persons raised their hands. Based on the comments that I got after the talk, there is still the stigma that testers are second class citizens below developers, which was not too surprising. However, I did find high interest from attendees to have more test oriented talks in future conferences which was a surprise and something to look forward to.

I am trying to improve my skills as a speaker, and this opportunity definitely helped (translating spanish to english on stage is harder than I thought). To my knowledge, this was the first time that a Microsoft technical conference had a session dedicated to QA, and seems like people want more. I’ll start preparing for next time. ^_^

• PowerPoint Deck
• Demos

- Federico

What I learned from James Bach?

This Friday I got to meet and interview James Bach (episode should be available soon-ish at codingqa). I really want to thank James for meeting with Matthew and I on a Friday afternoon, and taken the time to teach us some ropes of the trade. Besides learning some very useful testing exercises, there were some points that I got while talking with him about being a professional tester.

1. Create your own definitions. I thought I had taken some time to internally define what “testing” and “quality” mean, but James took it to another level. He challenged me to define and describe all my behavior so that I could explain with determinism what I was doing. I find this extremely important: to develop the capacity to synthesize behavior into its essential nature and to be able to classify it against other behaviors. In other words, to develop my own vocabulary. Why? Because once I have this framework I will be able to evaluate and standardize test work as a profession.
2. Train your intellect. Testing is a highly intellectual activity that needs to be developed. I already knew this, or more accurately, I knew it by intuition. But to be more serious and even systematic about training my mind to be a test ninja is work that I need to start doing.
3. Learn by experiment. The reason why I respect the people that I respect is because they lead by example. This is the type of the lead that I want to become in my team. One cannot preach without having tried it over and over, and of the best ways to do that is to experiment a lot, observe, learn and try it again.
4. Build your reputation. A professional works hard to build and guard his/her reputation, it is this standing among peers and others that grants you executing power and it translates to money at the end of the day. My first step is to work on point #1 and #2 to increase my confidence, that is the key ingredient.
5. Leverage the community. I care deeply about testing and the product that we are building (ASP.NET), I go to great lengths to envision new ways to test, try them out and roll them out to the team. However one thing that I haven’t done so much is reach to the community of other testers for more ideas and to help me learn. There are a lot of thought leaders out there, and I need to put myself with them to share what I got and take in what they got.

Lastly, something that was more of an experience rather than a learning, was the motivation that I got by talking with James. There is so much more out there that is exciting, just thinking of how me and my team could be better testers makes me want to get in the car and floor the gas and just go. Recklessly.

- Federico

Working with numbers and common sense

The Asp.Net QA team is going through some changes to better focus our activities and avoid randomization, I’ll talk about it in some later post. For now, one of the roles that I am assuming is to oversee the “test pass”. This is basically a set of product wide runs that all QA teams in the division must do to ascertain the state of the product against team expectations.

The moral of the story comes as we were ready to start a test run and I asked when it would be ready for testers to analyze it: “two days”. Wait, what? Two days? It takes 48 HOURS to execute our tests? I was told that’s what it always takes. I took a calculator and went to my office.

Turns out the type of run that we wanted to run generated 9,000 results. Each test takes an average of 3 minutes to run, don’t ask me why right now, just work with me. That is 27,000 minutes. That’s 450 hours, or 18 days. So, if I wanted my results to be done in one night, or 12 hours of execution, I need roughly 35 machines. And how many machines do we use in our runs? Eight. I asked why? It’s not as if we are in a super shortage of machines in our lab. There didn’t seem to be a good reason, eight just seemed to be a “good” number and we had “learned” to live with it.

We are now running with ~30 machines and I expect to see the results tomorrow.

- Federico

Has anybody used Concordion?

Leonid pointed me to this framework/tool back in August but I hadn’t had time to check it out. It is called Concordion, and it looks like a very interesting way of writing acceptance test that read pretty much like a spec. You basically generate a set of linked HTML pages that read as a  functional specification, but by putting special syntax in them you can call into your test fixture code that can interact with the system under test.

At first glance it sounds like a ton of work, but then again I don’t have a lot of experience writing end user applications in a team with agile methods (writing and testing a framework like ASP.NET is not quite the same as writing a testing an Amazon).

If you have experience using this kind of spec-to-code approach I would be very interested in hearing your experience. Drop me a line!

- Federico

Presenting at DevConnections

Hey everyone, I got a talk lined up for November in DevConnections about Testing ASP.NET. Here is the abstract:

AMS09: Testing in ASP.NET
Federico Silva Armas
As more people in the community and organizations recognize the value of testing, developers are sometimes left with the challenge of testing ASP.NET. With so many tools and approaches, it is difficult to decide what to invest in. In this introductory sessions for developers and testers alike, we'll explore different tools and strategies for testing an ASP.NET Web application. We'll focus on how to write functional tests using a variety of tools and how you can leverage them in your next project. We will also discuss some of the lessons that the ASP.NET QA team has learned while testing ASP.NET.

I am still unsure about the audience (I have never been to DevConnections before), so I hope I don’t shoot wildly off the mark. I will be talking mostly about tools and strategies to write functional automated tests. I plan to show off LTAF, Selenium and WatiN. Hopefully get the audience excited and interested in test automation.

- Federico

Using a unit testing harness to execute functional tests

As I am preparing for my talk at DevConnections, one of the things that I was advised to demonstrate is how to incorporate LTAF tests into MSTest  or NUnit. The idea is not new as certainly that’s one of the preferred ways to author Selenium or WatiN tests (if I am totally wrong, please correct me).

Personally, the idea of using a tool that was designed for unit testing to write and run functional tests is a little awkward. My reasons for this are:

1. Functional tests need a better logging mechanism than simple asserts and exceptions. Our harness produces super cool logs that contains the HTML at each step of a test, with visualizers to see what rules failed, debug them and fix them. It is definitely leaps forward than the exception reporting of a unit test harness.
2. Functional tests need a better test manager. What I have seen of unit test tools is some support for creating list of tests, but a formal test manager system will include additional test metadata like requirements, description, abstract, list of scenarios, contexts, several ways to organize them and dependencies.
3. Functional tests need more advanced analysis support. A unit test tool basically "runs and forgets” tests, it shows you the results of the current run and some support storing data of past runs. But a good functional testing harness will provide tools like comparing results of many runs, can track what was the reason for failure of each result, it can group failures that seem to be related to make it easier to analyze, it can check the bug database to see if there are similar problems to the one you are analyzing, it can correlate how much of the product code each test touches, it can organize and display the results when there are thousands of tests into views that are useful for a test team.

These are all things that are possible with the tools that we use internally in DevDiv. Now, you can argue that you can accomplish all of these with MSTest or NUnit or whatever, and that may be true. What I am saying is that functional tests have a specific set of requirements that having a specialized harness for functional tests makes more sense to me than trying to coerce a set of tools that were designed for dev unit tests.

What do you guys think? You like using a tool like MSTest or NUnit to write and execute functional tests?

- Federico