Automation of Burn-up and Burn-down charts using GScript and Entrance

I have always found that the burn-up and burn-down charts are very informative and fit to the iterative story-based development very well. Every project that I work on, I will try to figure out different ways to generate burn-up and burn-down charts.

Two months ago, I took the job on putting Platform on Sprints. After some consideration, I have decided to follow the setup that I have for the AF team, creating the stories in the form of JIRA issues. However, the chart generation that I had for AF team was still semi-manual, which means that it takes a couple of minutes to download, and a couple of minutes to update the stats every morning. The worst part is that when I get busy or sick, I will forget.

So my first action item was to figure out how to generate the same kind of charts with the push of a button. The idea seems to be easy enough:

1. Figure out the search criteria to retrieve all the JIRA issues of the backlog
2. Count the issues that are in different states
3. Update the data with the counts, and check it into Perforce.
4. Refresh the chart with the updated data
   

Number one and two were actually not that hard, because Guidewire GScript has a nice WebServices support. With a few tries, I was able to count the beans.

Here is an example of the data generated. I think you get the idea just looking at it.

Date,Day,Closed,Deferral Requested,In QA,Open Stories,Open New Features,Open Bugs
10/09/2009,41,55,0,1,13,7,40
10/12/2009,42,55,0,1,14,7,40
10/13/2009,43,56,0,0,14,7,40
10/14/2009,44,56,0,0,16,7,41
10/15/2009,45,56,0,0,21,8,42
10/16/2009,46,58,0,1,19,8,42
10/19/2009,47,58,0,2,28,8,42
10/20/2009,48,58,0,6,26,8,42
10/21/2009,49,58,0,6,26,8,42
10/22/2009,50,58,0,7,25,8,44

Number three took less time but a bit of research because Perforce Java library's API is not exactly straightforward.

It took me a while to figure out how to do the last one. After looking into JFreeChart and Google Chart API, I eventually turned to my dear friend, Tod Landis, who is also my partner at Entrance, and he quickly drafted an entrance script for me. Based on it, I was able to write a template that can be used for all teams within a few hours.

PLOT
very light yellow area,
light yellow filled circles and line,
DATALABELS
very light orange area,
light orange filled circles and line,
very light red area,
light red filled circles and line,
very light blue area,
light blue filled circles and line,
very light gray area,
dark gray filled circles and line,
very light green area,
dark green filled circles and line,
all AXISLABELS
WITH
ZEROBASED
TITLE "Sprint"
TITLE X "Days"
TITLE Y "Points"
SCALE Y 0 100 0
CLIP
legend
gridlines
collar
no sides
SELECT
`Open Bugs`,
`Open Bugs`,
date,
`Open New Features`,
`Open New Features`,
`Open Stories`,
`Open Stories`,
`In QA`,
`In QA`,
`Deferral Requested`,
`Deferral Requested`,
`Closed`,
`Closed`,
day
from report;

Please note this is the final PLOT script, there are other SQLs run before this to import the data into the MySQL database, sum up the data to produce a stacked chart, and even out the labels.

And I now have this chart generated automatically every morning with the help of a windows scheduler.

Cotta Asserts vs FEST Asserts

Background

Cotta-asserts is a public release of the JUnit 4 assertion adapter that came to as part of the Cotta implementation and my search for a better sematics for assertions in test.

FEST-Assert is a Java library that provides a fluent interface for writing assertions. Its main goal is to improve test code readability and make maintenance of tests easier.

• Cotta-asserts: http://cotta.sourceforge.net/assertions.html
• FEST-assert: http://fest.easytesting.org/assert/wiki/pmwiki.php

Bottom Line

The bottom line is that these two libraries are trying to solve the same problem in a different way, so they are both viable solution for writing assertions through fluent interface. Even though personally I like what I have built, I would not argument very hard about using one versus the other, same way as I would never argue between JUnit and TestNG, even though all my projects have been JUnit (And I know some people feel strongly about it).

I can also say the same thing about Java versus Ruby. When it comes to the bottom line, having a gelling team building high quality software that makes the user happy is all I care.

Why I like FEST asserts

Don't get me wrong, I love FEST asserts. Their API looks solid and really complete. Even now, Cotta Asserts still has some way to go to catch up, it is simply not high enough on my list, and I have not found anyone willing to contribute.

Why I like Cotta Asserts

With that said, I think it is worth it to write down the reason that I like Cotta Asserts.

First of all, Cotta Asserts is aimed to filling the gap of JUnit and Hamcrest, nothing more. It is just an API adapter that exposes the assertThat method in JUnit 4 and the matcher classes in Hamcrest. If one day this fluent API appears on the new JUnit release, I would be happily retire this little side project, and get back to the next item of my long list of things I would like to try as an open-source project.

Second, you can extend Cotta Assert however you want it to be, because all the classes are open to extension. You can extend AssertionFactory and add more methods to return other assert objects. You can make your AssertionFactory return different StringAssert.

For FEST-asserts, I always wonder why they made the classes final. It does not seem to serve any purpose. Then again, I never asked, so there could be some good reason behind it.

If you have been using assertThat method in JUnit, chances are that you already have a bunch of matchers lying around, and probably start having trouble remembering where they are. You just need to create your AssertionFactory and Asserts to leverage them and you don't have to throw anything away or even change anything.

With FEST-asserts, when you look at an API like this, something is just not right...

FEST-asserts requires you to add one assertThat method for each object type that you want to run asserts on, and one method for each custom condition. It can get confusing sometimes if you have different modules that do assertion on different objects in the module.

With Cotta Asserts, you only need one field declaration from the super test case class, and everything just goes from there. So it is really easy to set up, and really easy to customize it so that each module can have methods that only create asserts that are applicable to the module.

So here you are, happy testing!

Touché

(This one definitely falls in "Energized Work" category)

In an email thread with my old XP mentor and a long time friend, Rob Myers, about coordinating the next BayXP gathering, I mentioned that I have another baby coming soon.

Rob: "... about time!"

Shane: "... are you saying that I should get a life and stop messing with this thing called 'agile'?"

Rob:

"Au contrare my friend. Living 'agile' is supposed to provide you with the opportunity to have a life beyond the office. We should coach through example! :-)"

Shane:

"..."

"..."

"Touché"

"..."

"I think this deserves a blog post"

JUnit Assertion Adapter

I have been trying different assertion styles ever since I read this blog, like many others. If you google "assertThat" you can find many hits.

With the creation of Cotta project, I got a chance to really think hard on simple file operations and tests to see how I can make the API as polite as possible, and I think I have found a good answer.

With the default JUnit 4 assertion, you still don't get the benefit of a static typed language, in that you have to remember the class to use to create the matcher for a certain type.

So I created a JUnit assertion adapter that will allow you to type

ensure.that(yourValue)

and get the appropriate assertions object based on the type of the object being passed in. The type of the returned object will have just the assertion methods that are applicable to the value.

This also brings an additional benefit. For example, you can have a list instance and call

ensure.set(list)

and it will automatically convert the list to the set and return the set assertion object.

I have just made a release under the cotta project for general feedback

The simple document page is here: http://cotta.sourceforge.net/assertions.html
and the jar can be downloaded here:https://sourceforge.net/project/showfiles.php?group_id=171037

Phoenix First Two XP Sprints

Phoenix has finished its Sprint 5 and 6, which are the first two XP Sprints.

An XP team always goes through four stages, "forming, storming, norming, performing". The first Sprint felt like a storming stage, where we are trying to figure out the best way to get the code in without spending too much time on upfront design. At the same time, we are also getting used to paired programming.

Even though paired-programming has become an old trick for me, I still feel that my pairing skill has gotten worse during the past three years of working solo. The second Sprint felt a lot better, and I am hoping to keep this trend.

Items that worth noting:

• We modified the lava lamp to have a green light on when everything is good. Even though it is redundant, it has very positive effect among us. The only thing we might need to watch out is that someone mentioned that they could be fire hazard because the lamp gets very hot at the end of the day. So we are going to turn them off by the end of the day. This is when I found out that the X10 remote controller does not work, so they are back for replacement now.
• The lava lamps are helping us getting on the habit of treating broken tests as the highest priority. Due to the nature of Phoenix, we got some interesting test breakage already. We got tests that only break on the server, tests that only break on Linux, and a test that hung. One interesting discovery is that each time we are forced to figure out what is wrong and fix them, our tests ended up making better sense and being more like behavior driven, and I was planning on settling for hacks to keep the test passing!
• At the beginning of the project, we chose to create just enough stories to get us through the first Sprint, then created a few more for the second Sprint. Looking back, I think that is a good choice. The kind of stories that we create now are so much different but better from the earlier ones. I think that is because at the beginning, your system has literally nothing. It would take a very good story writer to come up with a list stories that really fit into the "INVEST" category of the story. I am not saying it is impossible, I just think that two Sprints of bad stories is not a bad price to pay to get the ball rolling as early as possible and avoid lots of hassle to learn and teach and debate about good stories vs bad stories.
  

Lava Lamp with CruiseControl

As we are getting Phoenix project under way, I am trying to get it started right by introducing more XP practices. The first three things that we are trying to do are Paired-Programming, Test-Driven Development, Continuous Integration.

Actually, Guidewire has already built an internal tool, ToolsHarness, to handle continuous integration, as I have written in "Managing Tests with ToolsHarness, Individually". The only difference that I want to introduce for Phoenix project is to fix broken tests AS SOON AS POSSIBLE.

What this means is that I want the testing status of our branch to show right in out faces, without us having to launch a browser, so that we know to take action the moment a test is broken.

I talked to the developer who manages ToolsHarness, and he wrote a servlet that serves information about broken tests and test status like this picture, except in one HTTP GET. Then I set up CruiseControl(version 2.8.2) with X10 publisher, following the setup described on this blog post "Bubble, Bubble, Build's In Trouble".

One thing about the normal lava lamp setup has always bugged me in the past, which is when the continuous integration server is in the "testing" state. When you have test broken, the red lava lamp will be on, and you just have to remind yourself that the fix is in and test is running. In some projects, I have used "project soundscape", so that when tests finishe but are still broken, you will know about it. But if you happen to step outside, you will miss it. Or if you just came in, you have to check the browser or ask others.

So this time, I have done it a little differently, taking advantage of the fact that CruiseControl is not the process running the tests. I bought two lava lamp, one kind of in the red color and the other in blue. I set it up so that when there are two independent lava lamps:

• Red Lava Lamp for broken tests: When there are broken tests, it will be on, otherwise, it will be off
• Blue Lava Lamp for testing status: When there are tests running, it will be on, otherwise it will be off

In this way, you have four state to display:

• Neither is on: All tests pass and the tests are up-to-date
• Blue is on and red is off: All tests pass so far, but there are tests running against newer changes
• Blue is off and red is on (see below): You have broken tests, and no code checked in to fix it
• Both blue and red are on (see below): You have broken tests and someone has cheked in new code (hopefully to fix it)
  

The setup is pretty straightforward, except CruiseControl 2.8.2 release is missing two crucial files, "lib/win32com.dll" and "lib/javax.comm.properties", for X10 publisher to work. That, and me missing a tiny but also crucial detail in the documentation, caused my three-hour-hair-pulling experience, and that was with Jeffrey coming to rescue through GTalk. I am going to submit the patch for the release script to include those two files, and documentation with the following checklist:

• You should provide all FOUR attributes related to X10 for the element, so that you are aware of them and make sure they are correct. These four attributes are as following:
  
  • "houseCode" and "deviceCode" are for X10 module configuration.
  
  
  • "port", with the value of COM1, COM2, etc., to match the place you plugin the COM module.
  
  
  • The last one is "interfaceModel", which you should really double check with the COM module that you have.


• Make sure "javax.comm.properties" is in your CruiseControl lib directory (should be there after 2.8.3)
• Make sure you copy "win32com.dll" from CruiseControl lib directory (should be there after 2.8.3) to your Java bin directory
  

In the end, I would like to say that I am a satisfied ci-guys customer!

Greate Article on Pair-Programming

All I can say is that this article says EXACTLY how I feel.

http://www.nomachetejuggling.com/2009/02/21/i-love-pair-programming/

One thing to add is James Shore's Programmer Man's Theme Song (see end of the post)

JIRA Story Wall

With the "shared dashboard" feature of JIRA, we have been experimenting a shared dashboard that can be served as a virtual story wall that can be useful to us. And here is one version.

AF stories are in the form of JIRA items, in this way, JIRAs created by other teams for bug fixese or support can be rolled into one backlog. Creating stories in the form of JIRA is not nearly as trivial and easy as creating stories on the index cards. But once you pass that phase and get yourself used to it, it does bring a lot of benefits of a digital media.

On the left, the first section shows the stories for the current Sprint with status and the person who is working on them. Each person is to finish the JIRAs assigned to him or her, before picking the ones assigned to the general bucket (AF General).

The section section shows the stories allocated for the next Sprint, grouped by assignees and components. The third section shows the full current backlog by component and priority. We used it a lot when trying to figure out what to work on next or what to push to next release. The last one is the backlog for the next milestone.

During the Sprint, some issues will come up. The most urgent ones will be pulled into the current Sprint to be dealt with right away. The others will either be added to the next Sprint, or add to the appropriate backlog. At the beginning of the Sprint, after counting the JIRAs already added to the Sprint, carrying over the ones from the past Sprint, we will select more JIRAs from the backlog by looking through the components.

On the right, the first section is the list of the JIRAs that the current user is working on (In Progress). It has been pretty useful to me to come in and get started right away by looking at this short list. However, I just learned today that everybody else is just looking at the JIRAs assigned to him or her in the current Sprint.

The JIRAs in the next list are the ones that have been marked as resolved by developers but not verified by QA. They are sorted based on the order that QAs would like to process them. QA team uses this to pick the JIRAs to verify during the Sprint.

The last section on the right contains the JIRAs that have not been added to any backlog. In this way, all the JIRAs will be looked at before adding to the backlog. One thing about using JIRA as story is that anyone can create a JIRA and assign it to your team, which mean your backlog can grow without you knowing it. With this extra step of adding newly created JIRA to the appropriate backlog, we are always aware of any new work coming our way.

Burn-up and Burn-down Charts

I have always thought Sprint reporting is a major communication tool to be used within the team as well as to the outside. It is the time for the team to take one step back, look at the project as a whole, comparing notes, and make continuous improvements. It is also the time for the team to report the progress and any difficulties encountered, so that the stake holders can make plan adjustments and provide help if needed.

Burn-up and burn-down charts are my favorite report, because they fit very well in the story based iteration model of the project development. Anyone understanding stories and iterations (not that it is always easy to learn) can understand these charts very easily. I also find that these chart can generate more questions and lead the team in the right direction.

Burn-down Chart

Burn-down chart is straightforward and easy to understand. It measures the burn rate of the story in the unit of the story points. It is really easy to understand different ways of predicting the outcome of the project by predicting the future velocity of the Sprints.

All the iteration tracking tools that I have tried have this support. This one is made by Pivotal Tracker.

For those who use JIRA or good-old story cards to track the iterations, it is not hard to produce this chart as well, with the worst part being figuring out where to use what formula. The following are from the other two projects, made with Microsoft Excel and Google spreadsheet. With customized tool, I get to explore different styles.

In the first one, the stories are divided into "must-have" and "everything else" category, and tracked at the same time. The prediction lines are shown in different color. In the second one, the progress is shown along with the burn-down, so that in the case where it is actually "burning up", it shows that it is not caused by losing velocity.

Burn-up Chart

For a project with just a single coach, burn-up chart can be a great help. It can explain a lot of concepts in the story based development, iterative development, and can help the coach recognize the patterns in the development and take actions to adjust the direction of the team.

I have found that burn-up chart always a bit harder to understand, and might look intimidating. So if you are introducing it for the first time, you should not just paste it in a report and email to others. It is best to show it in person and let the conversation start.

The first chart shows a project where development is fairly smooth, QA can just keep up with the story being finished. On the other hand, the project requirement is very volatile. The interesting thing to point out is that because the team is focusing on one Sprint at a time during the release, and one story at a time during the Sprint, the dramatic scope changes did not affect the development at all.





The second one is a quite typical burn-up chart, where the team discovers new cases as they go, and adding the understanding to the backlog in the form of the stories.

Sprint Burn-up Chart

I also found a burn-up chart for the Sprint is useful to figure out what happened during the Sprint. I think this is what is called "Sprint Signature" in the SCRUM book.

A Sprint burn-up chart should be used strictly internally, because only the team who have just been through the Sprint can look at it, talk about it and then draw conclusions. This should never be used for managerial purpose, in my humble opinion.

Staying Agile by Going off "Agile"

This is a blog post of a statement that I finally made after reading "The Decline and Fall of Agile"

I am going off "Agile"

No, I am not going to give up test-driven development. In fact, I am doing more of it by adopting more behavior-driven development, which is actually harder at certain cases. It helps me understand the code and verifies the design (I believe it does that rather than "drives" out the design nowadays but that is another post).

No, I am not going to give up aggressive refactoring. Every time, and I do mean EVERY TIME, I slack on it, I end up paying the price one way or another and kicking myself. I have been proud of every single line of code that I have produced (cannot say that for all the code that I have inherited and worked on), and they always serve me and my team well.

No, I am not going to give up on iterative development in the form of Iterations or Sprints. They help my teams focus, avoid distractions, and can still response to the request from outside the team with crystal clear transparency.

So what is it?

I am going to take "agile" off my vocabulary in all communications.

Rather than saying

"Not able to have QA accepting the stories as soon as they are finished is not agile"

I'll say "We need to get those finished stories accepted as soon as possible, so that we can close the feedback loop. When they are accepted, we know we are doing a good job. And when they are not, we can trace back to our thoughts as we were developing them and understand where it went wrong".

Rather than saying

"Not setting a goal at the begining of the Sprint and verifying them through the Sprint signature is not agile"

I'll say "We need to establish a way to provide feedback regarding our work and make continuous improvement to the way we work, so that we can provide better value to the people who pay us. One way we can do that is to look back at our progress in the past Sprint, talk about our experiences and thoughts, and come up with action items to make things better"

Apparently this will make conversation longer, because I have to present proof more than a good book (dozens of good books available as a matter of fact). Sometimes, I will have to wait, patiently, for the opportunity to present itself so that I can use as an example to persuade others to slow down, do it right, and do it well.

Why

I have been thinking along this line for a while. I read the "Good Agile and Bad Agile", felt annoyed because there is truth in what he is saying. From time to time, I get annoyed by the negative comments that don't even make sense to me. I wrote one post about "Things You Cannot Get Certified For", and argued hard on several news groups that I subscribe.

Very soon I got tired of it. Using the word "agile" has caused more distraction than its worth. Practices are sometimes picked upon literally and are attacked. Rather than looking at the value something is trying to bring, many seem to tend to look at the cost(time, tools, processes) first. It got attacked, it got debated, and at the end nothing is done and the bad things just keep going. And you get people from all over the world writing about how "agile" did not work for them and laughing at anyone who is interested in trying.

To add insult to injury, you can also hear usage of agile in the format like "Let's be agile about it, instead of insisting on ... ". It is really hard to argue in this situation, because you cannot just simply say "no, lets not be agile about it because we should insist on going through this three hour meeting to make sure that our stories are up to the standard".

I have been avoiding throwing agile around for a while and I think I am happy with the result. I also have been ignoring the bad usage of 'agile' out there so that I can stay healthy to focus on bringing agility to my teams. (I swear this is the last time). A month ago, I went through all my blog posts and took agile out of the labels and categories. This post will remain the only one with "Agile" as the label.

I have been thinking about writing a post like this and finally decided to do it after reading James' post "The Decline and Fall of Agile"

Other References

I am collecting references of others with similar ideas here:

• Why would anyone want to become agile
• Lean SCRUM
• Traps & Pitfalls of Agile Software Development - A Non-Contrarian View

Two Sprint Equations

What should be the order of items to do when installing a Sprint process from scratch? In the coaching days, we requested one to one ratio between the coaches and the rest of the developers plus a project manager, go all out for a couple of sprints, give everyone a chance to adjust to the process, before adjusting the process to the team.

For a single person, the strategy would have to be different. You would need to look at all the practices and make trade-offs with eyes on the big picture. The following two equations are what kept popping into my mind as I am installing the process to the two teams.

Value = Scope * (Feature Quality * Code Quality)

One comment to make here is that during the product development, these three factors are sometimes working against each other. A good business analyst or product manage is one that knows how and when to balance them. Only then, one with a strong personality can bring the best value to the product.

Scope is measurable, as the second section will show. Feature quality and code quality however are simply not something can be determined by objective measurement, not purely on it anyway. When pushed on something like a scope, the things that are not measurable get sacrificed, and everyone ends up paying for it sooner or later.

Scope = Velocity * Number of Sprints

Assuming that the quality of the product is controlled, the scope would be the next thing to look out for during a project. This is pretty easy to understand: the more the team can do without sacrificing the quality (both feature quality and code quality), the better.

Velocity is something that can only be affected by tuning the Sprint process of the team, but can never be demanded. What is left for this equation to work would be to adjust either the scope of the project, or the time of the project (number of sprints), and most of the time both. This is probably one of the commonly stated facts, at the same time it is probably also one of the most ignored fact.

Boosting velocity is the same as boosting productivity of the team, which is the job for the team lead. This is the purpose of a lot of XP practices: TDD, paired programming, co-location, shared ownership, continuous integration.

An Interesting Agile Team Paradox

I learned this on the latest BayXP meeting

A good self-organizing team is commonly one with a strong leadership

I think there are a lot to this interesting comment. I'd like to find out more about its origin and elaborate on it at latter time. I just want to write it down because I have almost forgot it twice already.

Building Java Project with Ruby - Part II

http://cotta.sourceforge.net/reports/

is now produced by this:

http://cotta.svn.sourceforge.net/viewvc/cotta/trunk/rakefile.rb?revision=200&view=markup

Building Java Project with Ruby

I probably would be stretching it if I say this is THE solution. But would you not look at this and think 'this just feels right'?

Next to do:

* Implement 'project.test'
* Implement 'project.report_coverage(dir_to_emma_files)'

Converting in Image in Ruby - Can it get any easier?

require 'RMagick'
Magick::ImageList.new('image.jpg').write('image.tif')

Ok, that is way too short. Let's add crop operation:

image = Magick::ImageList.new('image.jpg')
image.crop(0, 0, 64, 64).write('image.tif')

(I think I can add the first example to Cotta)

Five Sprints into SCRUM

We have kicked off Sprint 5 today for Application Framework team.

Guidewire development is following SCRUM methodology. However, through all these years, due to various reason, the ideas behind Sprints are not exactly followed. There are many reasons for this, some of which are actually good reasons. However, that does not mean it was the best decision, and some development teams are trying to bring back meaningful Sprints to the development process, including AF team.

So what have I done differently this time?

We ended up using JIRA to track our stories. There are many reasons for this. I think the first one is the kind of work we are doing right now. We are not yet doing active development, but rather fixing bugs for a point release and run performance testing. Since all the bugs are created in JIRA already, using JIRA to track items that are not bugs makes it easy to track all the items we need to do given any Sprint. On the weekly work-from-home day, which each Guidewire employee can choose freely, it is very convenience to go to JIRA to pick the next work to do.

I am still keeping a Sprint board by writing down the JIRAs on the story cards but it is not as effective as I would like it to be. I think one reason is that QAs are verifying the JIRAs on their own schedule. (And the reason for that is some QAs are not part of AF team, because AF work affects other application teams). I know it sounds strang, but that is the situation right now. We are talking about how to get away from this mode and have a real complete independent development teams but before that happens, we will just have to pull it through.

The purpose for Sprint board now is more for daily Sprint meeting, where we talk about what we have achieved yesterday and are planning to do today. I use it to help the team focus and work on only the blocker JIRA or the JIRAs scheduled for the Sprint. Old habits die hard but we are making progress in that direction. When we schedule too many for the Sprint, which has been the case for all the past Sprints, I use Sprint board to figure out what to push to the next Sprint. I have not bee doing this aggressively. Now that I have an idea of our current velocity, I'll do more now.

I am also changing the Sprint planning format. I am not going through the JIRAs one by one and ask question on them anymore, because the feedback has been that it takes a long time and becomes uninteresting. I think the first reason is that we are not sharing enough to make it a team conversation. Rather, it is me and whoever owns that part of the system talking with each other and figuring out the tasks to do. Even that, because I cannot pair on each and every JIRA, I am not able to track and check that each JIRA is estimated correctly and each JIRA is done within a Sprint. Without following them up and closing the feedback loop, all the work of creating tasks and track them become rather pointless.

So in the Sprint plannig, I now show the JIRA list scheduled, talk about them briefly in groups by the functional area, and track down the estimate after the meeting. I think I will change the estimation to be before the meeting next time, so that I would know how much to schedule for the Sprint.

Fix : ERROR: While executing gem ... (Zlib::BufError)

All you need to do is

gem update --system

Thank you dontrepeatyourself !!!

Guidewire Development Blog

I have developers from Guidewire customer commenting on my blog so I would like to mention that Guidewire now has its own development blog which I think you will find interesting.

http://guidewiredevelopment.wordpress.com

Enjoy!

Proposal for Agile 2008 Submitted

Even though I still have one more post to go for this topic on Enterprise Agile Testing, submission deadline for Agile 2008 is approaching. So I looked at other submissions, looked at what I have written, and submitted my proposal:

http://submissions.agile2008.org/node/3736

Feedbacks are welcome either through the submission page or my website http://www.shaneduan.com/contact.html

Enterprise Agile Testing Part III: Managing Tests with ToolsHarness, Individually

This is the third part of the Enterprise Agile Testing:

• Introduction
  
• Test fixtures like assertion, builder
• Test Environment Set Up with TestBase
• ToolsHarness, a continuous integration server farm that treats tests individually
• Active and stable branch, localizing the damage

Continuous Integration has proven to be one of the most important practices in agile software development. Every time that a developer checks in the code, the resulting code base is rebuilt and tests are run against it. The end result of the integration tells everyone on the project if the codebase is good enough for release. There are some prefer synchronous continuous integration through a push-button process over an asynchronous process through a tool like CruiseControl. But everyone agrees that it is something very useful.

the Difficulty of Holding the Line

With a tool or not, the most difficult part of installing such a process is probably holding the line of "zero broken tests". In my past consulting and coaching experience, it sometimes takes great effort and time to get the team into the habit of running all the unit tests before checking into the code, as well as making writing tests and test fixing as the highest priority, and keep tuning the test so that the whole process does not exceed ten minutes. Even that, not all the teams kept up with the practice after we left the project on a good note.

I recently got a chance to catch up Greg, one of the ex-ThoughtWorkers that I used to work with and respect. He showed interest in what I am writing and expressed his opinion, as I quote:

Our test suite is too large and too slow to run with every build. We are lucky to get results once a day.

- Not everyone cares about the unit tests to the same degree. Some people are too busy to track down failures right away. Not everyone sees the value in the unit tests, mostly because our coverage figures aren't high enough.
- Not everyone has the skills to write decent tests or design their code in a modular, testable fashion.

When I read that email, I became more motivated working on this third post, because it is exactly what I want to write about. In my previous job, I found out that the only way to make agile development work was to follow the XP practices, especially when it comes to continuous integration. In an enterprise environment (as defined by Introduction ), there seemed to be no middle ground between "green all the time" and "red all the time". It seemed that the moment the team fell off the status of zero broken test and couldn't recover quickly, they would be in a deep hole right away.

The first thing that I have learned at Guidewire, is that the above problems can be solved better with the help of a comprehensive continuous integration tool, ToolsHarness in this case. I am not saying this is a silver bullet, because there are still a lot of development practices and disciplines required. But I can certainly say that I am now seeing the light.

Test Farm for Parallel Testing

No matter how hard you try, eventually you will not have enough time to run all the tests that you would like before checking in code. When this becomes the case, test breaking will become the norm rather than exception, and the XP way of handling broken builds would not apply anymore. For a complicated system, the time to run the full tests will be huge. At the same time, agile software development dictates fast feedback time on code changes. The longer the turnaround time is, the more fraction there is on the iterative development - how is it even possible for the team to build on top of something not yet proven to be working and expect to have a high throughput?

Guidewire has a big testing farm, composed of dozens of machines, mostly on Linux and configured with H2 database. These machines are configured so that when a build is available, they will check out the test suites to run.

When a developer checks in a change list, ToolsHarness will first pull the source and do a full build to make sure that the projects still compile. Once the compiling finishes successfully, ToolsHarness will post the build for test. The tests used to be divided into different suites based on which test class they extends, for example, Database test, Metadata test, or Server test. With the introduction of TestBase, they are all converted to NewTestSuite and NewSmokeTestSuite. (The difference is that the tests in NewSmokeTestSuite are acceptance tests, which are full end-to-end test and require additional sample data.)

Based on the previous run of the test, the suites are divided evenly into several parts, so that each testing machine can check out each part and run it. In this way through parallel testing, it takes no more than 20 minutes to run a suite that would normally take hours to finish. This system is highly scalable, because all you need to do is adding more machines. With such a fast feedback loop, the developers can work on the large code base and still make medium size changes. The worst thing that can happen would be to revert the change that you made less than half an hour ago.

It is easy to run the tests against different databases and J2EE servers too. Some testing machines are configured with Oracle database or SQL database, some are configured on Windows platform, and some are configured using Tomcat or WebLogic. As I am writing this post, the tools pod (the team in charge of developing this server) is working on ‘customer build testing’, so that the test environment will be exactly as the production environment when running the acceptance tests.

Tracking Tests Individually

With a large team, you will have developers with different skill sets. While it is easy for an senior developers to be conscious about making small changes at a time, and be able to identify the problem based on the broken test, it normally takes a much longer time for a junior developer to fix them. Unless all your senior developers happen to be good coaches, you are going to be stuck with broken tests popping up here and there for a while.

When a testing machine is done with the test, it will post the test result back to the ToolsHarness. ToolsHarness will parse the XML file and store the result of each test into the database to track them. The benefit of this is so that developers can start tracking the tests individually. When a test is broken, ToolsHarness will make an educated guess based on the change list and changed package to assign it to the developer. If it turns out to be wrong, the developer can easily assign it to the right person.

When a developer logs into ToolsHarness, the first page, Desktop, contains a list of tests that have been assigned to him or her. In this way, you won’t be distracted as long as there are no tests assigned to you. The summary of all the tests is also on this page so that you know not to check-in anything when there are hundreds of tests broken, or give the senior developers a clue to check in others in fixing tests.

For each test, you can see the failure message in the form of the stack trace, the change lists associated with it, the history of the test to help you figure out the reason that the test is broken. You can also look into the log directory to see any additional generated file like server log and HTML page snapshots.

If you have written a broken test that you cannot yet fix, you can annotate it with KnownBreak, and it will show up properly in the ToolsHarness. If you have determined that a test in failing none-deterministically but you still cannot yet figure out why, you can mark it with NoneDeterministic, and it will show up as such in ToolsHarness. The key is to keep the noise of broken to minimum, if not zero, so that developers will get the notification accurately and fix them effectively.

Localizing the Damage through Branches

With aggressive refactoring, you will not be able to leave your platform code alone. Sometimes you know that the only way to be sure is to check in the code, let the continuous integration server do a full test on the changes. With this approach, you are going to risk putting the build into unstable state for a while before you can figure out the best solution. If the whole development team has to rely on a good build, they will either be out of commission for a while, or they are going to accumulate changes that will cause another wave of instability after you are done. And that is if you are lucky enough to quickly finish with the cycle of check in, revert, revert the revert to make more changes, check in, revert, ...

Sometimes, especially for the platform team and application framework team, you need to make big change in the code base. When it proved to break lots of tests in ToolsHarness, the best thing to do is to move forward by checking in more fixes, instead of reverting the change to do it again. The only problem with that, is that the code base comes unstable during the process. If you have a large team with others working on other areas at the same time, the number of broken tests could be disturbing. And as Greg pointed out at the beginning, not every team care about the tests in the same way. Some would prefer finishing the job at hand before tracking down the broken tests. The line for none-deterministic tests are much more blurry.

At Guidewire, the way to make it work for all the teams working on the same code base is through the branches. Each team works on an 'active' branch that they are free to do whatever they feel most productive. The only rule that they need to follow is to fix all the tests before pushing the change to the 'stable' branch, where every team is pulling change from on a daily basis. If it takes a team a couple of week to get into a stable state, then they will have to risk the merge conflicts. For a team that is diligent on fixing the test, their branch will be stable most of the time, and pushing would come much easier. It is exactly like what most agile books recommend on checking code -- check in as often as possible as long as the tests are passing -- except on the level of branches.

Sometimes you can still make a mistake and push a broken test (or more) into the stable. In this case, there is a 'merge' branch that you can use to fix the builds. Most of the time however, the fix is very easy, then all you need to do is to find out which team is next in line to push to stable, and manually integrate your fix into their branch. There are merge script written in Ruby to help the pull and push process. They are very robust and well tested, so that majority of the push and pull are merely a push-button process, i.e., you type in the command "merge.rb --pull", and you are good to go. We have a merge machine set up specifically for this job, so that the merger wouldn't have to give up his or her local resources.

Antidote

As many articles, books, blogs have pointed out, people are always the center of the agile development process. Even with a powerful tool like ToolsHarness, it is still up for the team to apply disciplines and agile practices. Because the team does not have to stop everything to fix any broken tests, it is actually easy for people to ignore the tests. Given enough time, enough code changes would have been checked in, making it much harder than it should be to fix the tests.

So the rule of the thumb is still the same: fixing any broken test as quickly as possible when they come up. The old tricks still apply, which including things like revert the changes that broke the build, make small changes, check-in often, monitoring email notifications, run tests before checking in, etc.