Joy and Pain with Fitnesse

(This blog is entered using Dvorak input, which I am going to write a blog about after I either master it or give it up)

Last Wednesday (May 18th.) I gave a presentation to the BayXP (http://groups.yahoo.com/group/bayxp/) titled "Joy and Pain with Fitness". I am putting the presentation material here, and try to explain the examples.

Joy and Pain with Fitnesse

Presentation of what we have learned during our usage with Fitnesse, an acceptance test framework, and how we have benefited from it. This presentation is intended on purpose to avoid raising philosophical issues, software engineering methodology, or what the correct usage of the Fitnesse should be. Rather, it concentrates on the retrospective of our experience with Fitnesse. It is more important to understane the reason behind

Overview
Fitness' own description can be found at its website "OneMinuteDescripiton", with the following in particular in our project:

1. Fitnesse is an acceptance testing framework. Its targeting audience is not only the developer but also the user who is going to write and run acceptance tests, like domain experts, QA, manager, etc. (And it has done a great job to achieve this goal).
2. Fitnesse is written in Java, which makes it easy for developer to assist the acceptance test writing.

What We Have Achieved
Just like an application product, we slowly developed a set of Fitnesse fixtures that allow everyone on the team, including the QAs and BAs, to set up the system state, simulate the events processing, and assert the end state of the system by writing XML document assertions and database assertions.

In order to document how to use the fixtures, we also created a "Fixture Tutorial" page, which contains all the fixtures that are available, with detailed descriptions. Each page is a test by itself, which means that people can change them and run them. This is very similar to the ones on Fitnesse page, e.g. JdbcSample, except ours are related directly to our domain.

How Fitnesse Works
The following is the outline of what we have learned in our project. It is quite possible that there are useful features that we have not discovered so you should make sure to check out Fitnesse website.

1. Fixtures are loaded through reflection by very simple naming convention. For example, if the header of the table is "Insert Person Data", then the class "InsertPersonData" will be loaded and instantiated. The packages in which Fitnesse should look for the classes can be specified through "Import" fixtures.
2. A page called "SuiteSetup" will run at the beginning of the suite and will not run for a single test. A page called "SetUp" will run at the beginning of each test all the time. The same rule applies to "SuiteTearDown" and "TearDown".
3. The following are the four most useful features to us in this project:
1. Column Fixture: The column name without the question mark is a public field in the fixture that Fitnesse will set value to. The column name with the question mark is a public method in the fixture that Fitnesse will call and compare to the value in the table.
2. Row Fixture: Great for doing assertion on a set of data. For example, if you want to assert that the person has three specific phone numbers (555-2222, 555-4444, 555-9999), you just need write the Fitnesse page with these numbers and implement a row fixture that simply returns all the phone numbers this person has. Fitnesse will do a fantastic job of comparing these two sets and show you the difference.
3. Action Fixture: The first column of each row is mapped to a public method. This is especially useful to run a sequence of actions and check the result between the actions. This fixture is being utilized by jWebFit.
4. If you use standard fixtures, all data type conversions are handled seamlessly. As explained in "DataTypesInFixtures".

Why We Love Fitnesse
We believe that tests are good and that we as XP and Agile developers should do everything we can to encourage developers to write unittests and QA/BA to write acceptance tests. In this project, we would never have achieved this level of tests without Fitnesse. To quote from our QA:

It is great to have a tool that allows me writing tests for our system. I have been quite uncomfortable with matching logic. With these many tests and assertions (over 23,000) I am feeling a lot better now.

Because it is easy and fun to write test, many things just follow naturally. Stories are written with test in mind, making them easy to verify. Acceptance tests are also written early, sometimes even ahead of the iteration. Consider this is the team that didn't know what a story card is and didn't have any automated tests, we are really satisfied with the result.

For BA and QA, or any non-technical member of the team, Fitnesse is great because it has very clean UI with neat user interaction design. It is much less itimidating than a programming language. You just feel that you are writing a Wiki page with tables that are runnable, instead of a program with limited way of writing comments. The pages are much more readable than the JUnit tests.

For developers, there are already fixtures helping them writing unit tests. With Fitnesse written in Java and loading fixtures through simple reflections, it is quite easy to hook up the Fitnesse with the existing testing fixtures, thus greatly reduces the code duplications athe cost of the Fitnesse fixtures.

Why We Hate Fitnesse
I believe it is as important to explain the struggle we had with Fitnesse as well as the success, because this way the reader can learn to use this tool effectively. It is also possible that in the limited amount of time and with project development to bring business value as our ultimate goal, we might not have learned the way that Fitnesse was originally designed for. And I'd very much like to be directed to the right direction.

No matter how many times that I say it is very easy to write Fitnesse, you still have to spend time writing it. Also because it is not the developers who write and run the Fitnesse, you get the same issues that you have for any project. What this means is that developers are taking requirement request. It is only natural that you want to help, and the better a developer you are, the more likely you are going to put down your story and write fixture instead. Before you know it, your velocity is down and you are only overwhelmed with more and more fixture requests. There is another member of BayXP who encountered the similar problem.

The target audience of Fitnesse are the none-technical member of the team so it is doing its best job to achieve that. This achievement, however, comes with the cost of the usability for developers. A developer who has been infected by TDD is a developer who has been spoiled by JUnit. With Fitnesse we have to again and again wait patiently for the suite to finish. While waiting, we have no idea how many have been run, how many there are in total, how long it is going to take. When there is a failure, we have to wait for all the tests to finish before we can see the report. (You can copy the URL and open it in another browser window but then you have to worry about running two acceptance test processes at the same time on the same machine, which can give you some trouble sometimes.) Even with the report, you still cannot open that line of code with one click. And it is a lot harder to put breakpoint at the place that you want in debug a problem.

In the spirit of Continuous Integration, we run some of our Fitnesse tests through CruiseControl. The best way we were able to figure out is to launch the server, then use the Fitnesse runner connecting to it. With the way Fitnesse is implemented, we ended up with a server thread serving the HTML, a client thread reading it, and another thread monitor the communication between these two. Somewhere it is not managed correctly and it is causing the process to hang. We spent a lot of time trying to figure out the reason but this problem only occurs on one of the Solaris machine (which happened to be our release machine), occurs 10% of the chance on our Linux machine (which is our build machine) and never on the windows machine. The closest thing that we found online is this post about how an earlier similar bug was fixed. We moved to an Linux machine for the release and bounce build machine every now and then, and moved on.

Fitnesse also tries too hard, in my humble opinion, to be everything. Rather than a simple webapp, it is a stand alone web server. This means that you have another server to manage, another account to manage and another port to open. It also has its own Wiki engine so it is another set of rule to remember (after played with 5 different rules among Wiki, Twiki, Confluence, Rubywiki, one really gets tired.) Fitnesse dose not extend JUnit framework. The whole suite is actually run as one test, which means memory will increase as the test is running because of the HTML page.

Patterns
As we work with Fitnesse, we have developed the following patterns that helped us to use it successfully.

Just like any product feature request, every Fitnesse fixture request needs to be considered and discussed. We include them in our iteration planning meeting, we estimate them, and we pair up to implement them.

As we develop the fixtures, we write the tutorial pages first, which serves as our tests for the fixtures themselves.

Fitnesse pages are saved in their original format in TXT files. We launch the server with the correct arguments so that it does not manage the history of the pages. Then we check the files into our version control system.

For each project, we use one singleton class to manage the data that we want the fixtures to pass to each other, and make sure that all fixtures are stateless. In this way, the fixtures can stay simple and we were able to optimize the tests to make all the test run under 5 minutes.

For each project, we designated one suite as our acceptance tests area. Once a tests pass, we move it in there so that it never fails. In this way, the QA can feel free to write any tests they want and still can check in.

Always remember, it is nice and cool to do everything, but it is worth the effort only when the benefit outweighs the cost. We didn't have Fitnesse tests for a project that involve Peoplesoft because it uses views that are oracle specific. We resort to jWebUnit tests for another webapp because it was not as easy to use jWebFit (However, with the helper classes that we developed later on for web tests, we are taking another look at it to see how much it would cost now. The QAs is getting a hang of automated testing and starts feeling bad about not able to write them for Web apps).

References

• QA: Quality Assurance
• BA: Business Analyst role in a development team. (http://www.xprogramming.com/xpmag/BizAnalysis.htm)
• Fit: The framework for integration test that Fitnesse is based on (http://fit.c2.com/)
• Other nice tools for web testing
• Watir: http://wtr.rubyforge.org/
• Selenium: http://selenium.thoughtworks.com/ (A movie demo)
  
• Samie: http://samie.sourceforge.net/