A New Beginning

Blog of the day

2011-05-11T13:12:00.001-07:00

Interesting article because it relates exactly to what we are going through. I am only wondering about the stage we are in

Top 10 Reasons for Slow Velocity

2010-07-28T10:04:00.000-07:00

http://www.svpg.com/top-10-reasons-for-slow-velocity/

My current challenge at work is to capture velocity reliably, but I am sure this post will come handy sometime down the road.

Automation of Burn-up and Burn-down charts using GScript and Entrance

2009-10-22T22:42:00.000-07:00

I have always found that the burn-up and burn-down charts are very informative and fit to the iterative story-based development very well. Every project that I work on, I will try to figure out different ways to generate burn-up and burn-down charts.

Two months ago, I took the job on putting Platform on Sprints. After some consideration, I have decided to follow the setup that I have for the AF team, creating the stories in the form of JIRA issues. However, the chart generation that I had for AF team was still semi-manual, which means that it takes a couple of minutes to download, and a couple of minutes to update the stats every morning. The worst part is that when I get busy or sick, I will forget.

So my first action item was to figure out how to generate the same kind of charts with the push of a button. The idea seems to be easy enough:

Figure out the search criteria to retrieve all the JIRA issues of the backlog
Count the issues that are in different states
Update the data with the counts, and check it into Perforce.
Refresh the chart with the updated data

Number one and two were actually not that hard, because Guidewire GScript has a nice WebServices support. With a few tries, I was able to count the beans.

Here is an example of the data generated. I think you get the idea just looking at it.

Date,Day,Closed,Deferral Requested,In QA,Open Stories,Open New Features,Open Bugs
10/09/2009,41,55,0,1,13,7,40
10/12/2009,42,55,0,1,14,7,40
10/13/2009,43,56,0,0,14,7,40
10/14/2009,44,56,0,0,16,7,41
10/15/2009,45,56,0,0,21,8,42
10/16/2009,46,58,0,1,19,8,42
10/19/2009,47,58,0,2,28,8,42
10/20/2009,48,58,0,6,26,8,42
10/21/2009,49,58,0,6,26,8,42
10/22/2009,50,58,0,7,25,8,44

Number three took less time but a bit of research because Perforce Java library's API is not exactly straightforward.

It took me a while to figure out how to do the last one. After looking into JFreeChart and Google Chart API, I eventually turned to my dear friend, Tod Landis, who is also my partner at Entrance, and he quickly drafted an entrance script for me. Based on it, I was able to write a template that can be used for all teams within a few hours.

PLOT
very light yellow area,
light yellow filled circles and line,
DATALABELS
very light orange area,
light orange filled circles and line,
very light red area,
light red filled circles and line,
very light blue area,
light blue filled circles and line,
very light gray area,
dark gray filled circles and line,
very light green area,
dark green filled circles and line,
all AXISLABELS
WITH
ZEROBASED
TITLE "Sprint"
TITLE X "Days"
TITLE Y "Points"
SCALE Y 0 100 0
CLIP
legend
gridlines
collar
no sides
SELECT
`Open Bugs`,
`Open Bugs`,
date,
`Open New Features`,
`Open New Features`,
`Open Stories`,
`Open Stories`,
`In QA`,
`In QA`,
`Deferral Requested`,
`Deferral Requested`,
`Closed`,
`Closed`,
day
from report;

Please note this is the final PLOT script, there are other SQLs run before this to import the data into the MySQL database, sum up the data to produce a stacked chart, and even out the labels.

And I now have this chart generated automatically every morning with the help of a windows scheduler.

Cotta Asserts vs FEST Asserts

2009-06-10T10:18:00.000-07:00

Background

Cotta-asserts is a public release of the JUnit 4 assertion adapter that came to as part of the Cotta implementation and my search for a better sematics for assertions in test.

FEST-Assert is a Java library that provides a fluent interface for writing assertions. Its main goal is to improve test code readability and make maintenance of tests easier.

Cotta-asserts: http://cotta.sourceforge.net/assertions.html
FEST-assert: http://fest.easytesting.org/assert/wiki/pmwiki.php

Bottom Line

The bottom line is that these two libraries are trying to solve the same problem in a different way, so they are both viable solution for writing assertions through fluent interface. Even though personally I like what I have built, I would not argument very hard about using one versus the other, same way as I would never argue between JUnit and TestNG, even though all my projects have been JUnit (And I know some people feel strongly about it).

I can also say the same thing about Java versus Ruby. When it comes to the bottom line, having a gelling team building high quality software that makes the user happy is all I care.

Why I like FEST asserts

Don't get me wrong, I love FEST asserts. Their API looks solid and really complete. Even now, Cotta Asserts still has some way to go to catch up, it is simply not high enough on my list, and I have not found anyone willing to contribute.

Why I like Cotta Asserts

With that said, I think it is worth it to write down the reason that I like Cotta Asserts.

First of all, Cotta Asserts is aimed to filling the gap of JUnit and Hamcrest, nothing more. It is just an API adapter that exposes the assertThat method in JUnit 4 and the matcher classes in Hamcrest. If one day this fluent API appears on the new JUnit release, I would be happily retire this little side project, and get back to the next item of my long list of things I would like to try as an open-source project.

Second, you can extend Cotta Assert however you want it to be, because all the classes are open to extension. You can extend AssertionFactory and add more methods to return other assert objects. You can make your AssertionFactory return different StringAssert.

For FEST-asserts, I always wonder why they made the classes final. It does not seem to serve any purpose. Then again, I never asked, so there could be some good reason behind it.

If you have been using assertThat method in JUnit, chances are that you already have a bunch of matchers lying around, and probably start having trouble remembering where they are. You just need to create your AssertionFactory and Asserts to leverage them and you don't have to throw anything away or even change anything.

With FEST-asserts, when you look at an API like this, something is just not right...

FEST-asserts requires you to add one assertThat method for each object type that you want to run asserts on, and one method for each custom condition. It can get confusing sometimes if you have different modules that do assertion on different objects in the module.

With Cotta Asserts, you only need one field declaration from the super test case class, and everything just goes from there. So it is really easy to set up, and really easy to customize it so that each module can have methods that only create asserts that are applicable to the module.

So here you are, happy testing!

Touché

2009-06-05T11:15:00.000-07:00

(This one definitely falls in "Energized Work" category)

In an email thread with my old XP mentor and a long time friend, Rob Myers, about coordinating the next BayXP gathering, I mentioned that I have another baby coming soon.

Rob: "... about time!"

Shane: "... are you saying that I should get a life and stop messing with this thing called 'agile'?"

Rob:

"Au contrare my friend. Living 'agile' is supposed to provide you with the opportunity to have a life beyond the office. We should coach through example! :-)"

Shane:

"..."

"Touché"

"..."

"I think this deserves a blog post"

JUnit Assertion Adapter

2009-06-03T23:04:00.000-07:00

I have been trying different assertion styles ever since I read this blog, like many others. If you google "assertThat" you can find many hits.

With the creation of Cotta project, I got a chance to really think hard on simple file operations and tests to see how I can make the API as polite as possible, and I think I have found a good answer.

With the default JUnit 4 assertion, you still don't get the benefit of a static typed language, in that you have to remember the class to use to create the matcher for a certain type.

So I created a JUnit assertion adapter that will allow you to type

ensure.that(yourValue)

and get the appropriate assertions object based on the type of the object being passed in. The type of the returned object will have just the assertion methods that are applicable to the value.

This also brings an additional benefit. For example, you can have a list instance and call

ensure.set(list)

and it will automatically convert the list to the set and return the set assertion object.

I have just made a release under the cotta project for general feedback

The simple document page is here: http://cotta.sourceforge.net/assertions.html
and the jar can be downloaded here:https://sourceforge.net/project/showfiles.php?group_id=171037

Phoenix First Two XP Sprints

2009-03-07T20:44:00.000-08:00

Phoenix has finished its Sprint 5 and 6, which are the first two XP Sprints.

An XP team always goes through four stages, "forming, storming, norming, performing". The first Sprint felt like a storming stage, where we are trying to figure out the best way to get the code in without spending too much time on upfront design. At the same time, we are also getting used to paired programming.

Even though paired-programming has become an old trick for me, I still feel that my pairing skill has gotten worse during the past three years of working solo. The second Sprint felt a lot better, and I am hoping to keep this trend.

Items that worth noting:

We modified the lava lamp to have a green light on when everything is good. Even though it is redundant, it has very positive effect among us. The only thing we might need to watch out is that someone mentioned that they could be fire hazard because the lamp gets very hot at the end of the day. So we are going to turn them off by the end of the day. This is when I found out that the X10 remote controller does not work, so they are back for replacement now.
The lava lamps are helping us getting on the habit of treating broken tests as the highest priority. Due to the nature of Phoenix, we got some interesting test breakage already. We got tests that only break on the server, tests that only break on Linux, and a test that hung. One interesting discovery is that each time we are forced to figure out what is wrong and fix them, our tests ended up making better sense and being more like behavior driven, and I was planning on settling for hacks to keep the test passing!
At the beginning of the project, we chose to create just enough stories to get us through the first Sprint, then created a few more for the second Sprint. Looking back, I think that is a good choice. The kind of stories that we create now are so much different but better from the earlier ones. I think that is because at the beginning, your system has literally nothing. It would take a very good story writer to come up with a list stories that really fit into the "INVEST" category of the story. I am not saying it is impossible, I just think that two Sprints of bad stories is not a bad price to pay to get the ball rolling as early as possible and avoid lots of hassle to learn and teach and debate about good stories vs bad stories.

Lava Lamp with CruiseControl

2009-02-23T12:35:00.001-08:00

As we are getting Phoenix project under way, I am trying to get it started right by introducing more XP practices. The first three things that we are trying to do are Paired-Programming, Test-Driven Development, Continuous Integration.

Actually, Guidewire has already built an internal tool, ToolsHarness, to handle continuous integration, as I have written in "Managing Tests with ToolsHarness, Individually". The only difference that I want to introduce for Phoenix project is to fix broken tests AS SOON AS POSSIBLE.

What this means is that I want the testing status of our branch to show right in out faces, without us having to launch a browser, so that we know to take action the moment a test is broken.

I talked to the developer who manages ToolsHarness, and he wrote a servlet that serves information about broken tests and test status like this picture, except in one HTTP GET. Then I set up CruiseControl(version 2.8.2) with X10 publisher, following the setup described on this blog post "Bubble, Bubble, Build's In Trouble".

One thing about the normal lava lamp setup has always bugged me in the past, which is when the continuous integration server is in the "testing" state. When you have test broken, the red lava lamp will be on, and you just have to remind yourself that the fix is in and test is running. In some projects, I have used "project soundscape", so that when tests finishe but are still broken, you will know about it. But if you happen to step outside, you will miss it. Or if you just came in, you have to check the browser or ask others.

So this time, I have done it a little differently, taking advantage of the fact that CruiseControl is not the process running the tests. I bought two lava lamp, one kind of in the red color and the other in blue. I set it up so that when there are two independent lava lamps:

Red Lava Lamp for broken tests: When there are broken tests, it will be on, otherwise, it will be off
Blue Lava Lamp for testing status: When there are tests running, it will be on, otherwise it will be off

In this way, you have four state to display:

Neither is on: All tests pass and the tests are up-to-date
Blue is on and red is off: All tests pass so far, but there are tests running against newer changes
Blue is off and red is on (see below): You have broken tests, and no code checked in to fix it
Both blue and red are on (see below): You have broken tests and someone has cheked in new code (hopefully to fix it)

The setup is pretty straightforward, except CruiseControl 2.8.2 release is missing two crucial files, "lib/win32com.dll" and "lib/javax.comm.properties", for X10 publisher to work. That, and me missing a tiny but also crucial detail in the documentation, caused my three-hour-hair-pulling experience, and that was with Jeffrey coming to rescue through GTalk. I am going to submit the patch for the release script to include those two files, and documentation with the following checklist:

You should provide all FOUR attributes related to X10 for the
element, so that you are aware of them and make sure they are correct. These four attributes are as following:
- "houseCode" and "deviceCode" are for X10 module configuration.
- "port", with the value of COM1, COM2, etc., to match the place you plugin the COM module.
- The last one is "interfaceModel", which you should really double check with the COM module that you have.

Make sure "javax.comm.properties" is in your CruiseControl lib directory (should be there after 2.8.3)
Make sure you copy "win32com.dll" from CruiseControl lib directory (should be there after 2.8.3) to your Java bin directory

In the end, I would like to say that I am a satisfied ci-guys customer!

Greate Article on Pair-Programming

2009-02-22T12:44:00.000-08:00

All I can say is that this article says EXACTLY how I feel.

http://www.nomachetejuggling.com/2009/02/21/i-love-pair-programming/

One thing to add is James Shore's Programmer Man's Theme Song (see end of the post)

JIRA Story Wall

2009-02-08T21:09:00.000-08:00

With the "shared dashboard" feature of JIRA, we have been experimenting a shared dashboard that can be served as a virtual story wall that can be useful to us. And here is one version.

AF stories are in the form of JIRA items, in this way, JIRAs created by other teams for bug fixese or support can be rolled into one backlog. Creating stories in the form of JIRA is not nearly as trivial and easy as creating stories on the index cards. But once you pass that phase and get yourself used to it, it does bring a lot of benefits of a digital media.

On the left, the first section shows the stories for the current Sprint with status and the person who is working on them. Each person is to finish the JIRAs assigned to him or her, before picking the ones assigned to the general bucket (AF General).

The section section shows the stories allocated for the next Sprint, grouped by assignees and components. The third section shows the full current backlog by component and priority. We used it a lot when trying to figure out what to work on next or what to push to next release. The last one is the backlog for the next milestone.

During the Sprint, some issues will come up. The most urgent ones will be pulled into the current Sprint to be dealt with right away. The others will either be added to the next Sprint, or add to the appropriate backlog. At the beginning of the Sprint, after counting the JIRAs already added to the Sprint, carrying over the ones from the past Sprint, we will select more JIRAs from the backlog by looking through the components.

On the right, the first section is the list of the JIRAs that the current user is working on (In Progress). It has been pretty useful to me to come in and get started right away by looking at this short list. However, I just learned today that everybody else is just looking at the JIRAs assigned to him or her in the current Sprint.

The JIRAs in the next list are the ones that have been marked as resolved by developers but not verified by QA. They are sorted based on the order that QAs would like to process them. QA team uses this to pick the JIRAs to verify during the Sprint.

The last section on the right contains the JIRAs that have not been added to any backlog. In this way, all the JIRAs will be looked at before adding to the backlog. One thing about using JIRA as story is that anyone can create a JIRA and assign it to your team, which mean your backlog can grow without you knowing it. With this extra step of adding newly created JIRA to the appropriate backlog, we are always aware of any new work coming our way.

Burn-up and Burn-down Charts

2009-01-21T21:35:00.000-08:00

I have always thought Sprint reporting is a major communication tool to be used within the team as well as to the outside. It is the time for the team to take one step back, look at the project as a whole, comparing notes, and make continuous improvements. It is also the time for the team to report the progress and any difficulties encountered, so that the stake holders can make plan adjustments and provide help if needed.

Burn-up and burn-down charts are my favorite report, because they fit very well in the story based iteration model of the project development. Anyone understanding stories and iterations (not that it is always easy to learn) can understand these charts very easily. I also find that these chart can generate more questions and lead the team in the right direction.

Burn-down Chart

Burn-down chart is straightforward and easy to understand. It measures the burn rate of the story in the unit of the story points. It is really easy to understand different ways of predicting the outcome of the project by predicting the future velocity of the Sprints.

All the iteration tracking tools that I have tried have this support. This one is made by Pivotal Tracker.

For those who use JIRA or good-old story cards to track the iterations, it is not hard to produce this chart as well, with the worst part being figuring out where to use what formula. The following are from the other two projects, made with Microsoft Excel and Google spreadsheet. With customized tool, I get to explore different styles.

In the first one, the stories are divided into "must-have" and "everything else" category, and tracked at the same time. The prediction lines are shown in different color. In the second one, the progress is shown along with the burn-down, so that in the case where it is actually "burning up", it shows that it is not caused by losing velocity.

Burn-up Chart

For a project with just a single coach, burn-up chart can be a great help. It can explain a lot of concepts in the story based development, iterative development, and can help the coach recognize the patterns in the development and take actions to adjust the direction of the team.

I have found that burn-up chart always a bit harder to understand, and might look intimidating. So if you are introducing it for the first time, you should not just paste it in a report and email to others. It is best to show it in person and let the conversation start.

The first chart shows a project where development is fairly smooth, QA can just keep up with the story being finished. On the other hand, the project requirement is very volatile. The interesting thing to point out is that because the team is focusing on one Sprint at a time during the release, and one story at a time during the Sprint, the dramatic scope changes did not affect the development at all.

The second one is a quite typical burn-up chart, where the team discovers new cases as they go, and adding the understanding to the backlog in the form of the stories.

Sprint Burn-up Chart

I also found a burn-up chart for the Sprint is useful to figure out what happened during the Sprint. I think this is what is called "Sprint Signature" in the SCRUM book.

A Sprint burn-up chart should be used strictly internally, because only the team who have just been through the Sprint can look at it, talk about it and then draw conclusions. This should never be used for managerial purpose, in my humble opinion.

Staying Agile by Going off "Agile"

2008-11-15T22:16:00.001-08:00

This is a blog post of a statement that I finally made after reading "The Decline and Fall of Agile"

I am going off "Agile"

No, I am not going to give up test-driven development. In fact, I am doing more of it by adopting more behavior-driven development, which is actually harder at certain cases. It helps me understand the code and verifies the design (I believe it does that rather than "drives" out the design nowadays but that is another post).

No, I am not going to give up aggressive refactoring. Every time, and I do mean EVERY TIME, I slack on it, I end up paying the price one way or another and kicking myself. I have been proud of every single line of code that I have produced (cannot say that for all the code that I have inherited and worked on), and they always serve me and my team well.

No, I am not going to give up on iterative development in the form of Iterations or Sprints. They help my teams focus, avoid distractions, and can still response to the request from outside the team with crystal clear transparency.

So what is it?

I am going to take "agile" off my vocabulary in all communications.

Rather than saying

"Not able to have QA accepting the stories as soon as they are finished is not agile"

I'll say "We need to get those finished stories accepted as soon as possible, so that we can close the feedback loop. When they are accepted, we know we are doing a good job. And when they are not, we can trace back to our thoughts as we were developing them and understand where it went wrong".

Rather than saying

"Not setting a goal at the begining of the Sprint and verifying them through the Sprint signature is not agile"

I'll say "We need to establish a way to provide feedback regarding our work and make continuous improvement to the way we work, so that we can provide better value to the people who pay us. One way we can do that is to look back at our progress in the past Sprint, talk about our experiences and thoughts, and come up with action items to make things better"

Apparently this will make conversation longer, because I have to present proof more than a good book (dozens of good books available as a matter of fact). Sometimes, I will have to wait, patiently, for the opportunity to present itself so that I can use as an example to persuade others to slow down, do it right, and do it well.

Why

I have been thinking along this line for a while. I read the "Good Agile and Bad Agile", felt annoyed because there is truth in what he is saying. From time to time, I get annoyed by the negative comments that don't even make sense to me. I wrote one post about "Things You Cannot Get Certified For", and argued hard on several news groups that I subscribe.

Very soon I got tired of it. Using the word "agile" has caused more distraction than its worth. Practices are sometimes picked upon literally and are attacked. Rather than looking at the value something is trying to bring, many seem to tend to look at the cost(time, tools, processes) first. It got attacked, it got debated, and at the end nothing is done and the bad things just keep going. And you get people from all over the world writing about how "agile" did not work for them and laughing at anyone who is interested in trying.

To add insult to injury, you can also hear usage of agile in the format like "Let's be agile about it, instead of insisting on ... ". It is really hard to argue in this situation, because you cannot just simply say "no, lets not be agile about it because we should insist on going through this three hour meeting to make sure that our stories are up to the standard".

I have been avoiding throwing agile around for a while and I think I am happy with the result. I also have been ignoring the bad usage of 'agile' out there so that I can stay healthy to focus on bringing agility to my teams. (I swear this is the last time). A month ago, I went through all my blog posts and took agile out of the labels and categories. This post will remain the only one with "Agile" as the label.

I have been thinking about writing a post like this and finally decided to do it after reading James' post "The Decline and Fall of Agile"

Other References

I am collecting references of others with similar ideas here:

Two Sprint Equations

2008-10-28T21:33:00.000-07:00

What should be the order of items to do when installing a Sprint process from scratch? In the coaching days, we requested one to one ratio between the coaches and the rest of the developers plus a project manager, go all out for a couple of sprints, give everyone a chance to adjust to the process, before adjusting the process to the team.

For a single person, the strategy would have to be different. You would need to look at all the practices and make trade-offs with eyes on the big picture. The following two equations are what kept popping into my mind as I am installing the process to the two teams.

Value = Scope * (Feature Quality * Code Quality)

One comment to make here is that during the product development, these three factors are sometimes working against each other. A good business analyst or product manage is one that knows how and when to balance them. Only then, one with a strong personality can bring the best value to the product.

Scope is measurable, as the second section will show. Feature quality and code quality however are simply not something can be determined by objective measurement, not purely on it anyway. When pushed on something like a scope, the things that are not measurable get sacrificed, and everyone ends up paying for it sooner or later.

Scope = Velocity * Number of Sprints

Assuming that the quality of the product is controlled, the scope would be the next thing to look out for during a project. This is pretty easy to understand: the more the team can do without sacrificing the quality (both feature quality and code quality), the better.

Velocity is something that can only be affected by tuning the Sprint process of the team, but can never be demanded. What is left for this equation to work would be to adjust either the scope of the project, or the time of the project (number of sprints), and most of the time both. This is probably one of the commonly stated facts, at the same time it is probably also one of the most ignored fact.

Boosting velocity is the same as boosting productivity of the team, which is the job for the team lead. This is the purpose of a lot of XP practices: TDD, paired programming, co-location, shared ownership, continuous integration.

An Interesting Agile Team Paradox

2008-10-24T21:04:00.000-07:00

I learned this on the latest BayXP meeting

A good self-organizing team is commonly one with a strong leadership

I think there are a lot to this interesting comment. I'd like to find out more about its origin and elaborate on it at latter time. I just want to write it down because I have almost forgot it twice already.

Building Java Project with Ruby - Part II

2008-08-17T00:52:00.000-07:00

http://cotta.sourceforge.net/reports/

is now produced by this:

http://cotta.svn.sourceforge.net/viewvc/cotta/trunk/rakefile.rb?revision=200&view=markup

Building Java Project with Ruby

2008-08-13T00:02:00.000-07:00

I probably would be stretching it if I say this is THE solution. But would you not look at this and think 'this just feels right'?

Next to do:

* Implement 'project.test'
* Implement 'project.report_coverage(dir_to_emma_files)'

Converting in Image in Ruby - Can it get any easier?

2008-07-10T10:04:00.000-07:00

require 'RMagick'
Magick::ImageList.new('image.jpg').write('image.tif')

Ok, that is way too short. Let's add crop operation:

image = Magick::ImageList.new('image.jpg')
image.crop(0, 0, 64, 64).write('image.tif')

(I think I can add the first example to Cotta)

Five Sprints into SCRUM

2008-06-16T21:23:00.001-07:00

We have kicked off Sprint 5 today for Application Framework team.

Guidewire development is following SCRUM methodology. However, through all these years, due to various reason, the ideas behind Sprints are not exactly followed. There are many reasons for this, some of which are actually good reasons. However, that does not mean it was the best decision, and some development teams are trying to bring back meaningful Sprints to the development process, including AF team.

So what have I done differently this time?

We ended up using JIRA to track our stories. There are many reasons for this. I think the first one is the kind of work we are doing right now. We are not yet doing active development, but rather fixing bugs for a point release and run performance testing. Since all the bugs are created in JIRA already, using JIRA to track items that are not bugs makes it easy to track all the items we need to do given any Sprint. On the weekly work-from-home day, which each Guidewire employee can choose freely, it is very convenience to go to JIRA to pick the next work to do.

I am still keeping a Sprint board by writing down the JIRAs on the story cards but it is not as effective as I would like it to be. I think one reason is that QAs are verifying the JIRAs on their own schedule. (And the reason for that is some QAs are not part of AF team, because AF work affects other application teams). I know it sounds strang, but that is the situation right now. We are talking about how to get away from this mode and have a real complete independent development teams but before that happens, we will just have to pull it through.

The purpose for Sprint board now is more for daily Sprint meeting, where we talk about what we have achieved yesterday and are planning to do today. I use it to help the team focus and work on only the blocker JIRA or the JIRAs scheduled for the Sprint. Old habits die hard but we are making progress in that direction. When we schedule too many for the Sprint, which has been the case for all the past Sprints, I use Sprint board to figure out what to push to the next Sprint. I have not bee doing this aggressively. Now that I have an idea of our current velocity, I'll do more now.

I am also changing the Sprint planning format. I am not going through the JIRAs one by one and ask question on them anymore, because the feedback has been that it takes a long time and becomes uninteresting. I think the first reason is that we are not sharing enough to make it a team conversation. Rather, it is me and whoever owns that part of the system talking with each other and figuring out the tasks to do. Even that, because I cannot pair on each and every JIRA, I am not able to track and check that each JIRA is estimated correctly and each JIRA is done within a Sprint. Without following them up and closing the feedback loop, all the work of creating tasks and track them become rather pointless.

So in the Sprint plannig, I now show the JIRA list scheduled, talk about them briefly in groups by the functional area, and track down the estimate after the meeting. I think I will change the estimation to be before the meeting next time, so that I would know how much to schedule for the Sprint.

Fix : ERROR: While executing gem ... (Zlib::BufError)

2008-05-18T11:56:00.000-07:00

All you need to do is

gem update --system

Thank you dontrepeatyourself !!!

Guidewire Development Blog

2008-02-24T14:21:00.000-08:00

I have developers from Guidewire customer commenting on my blog so I would like to mention that Guidewire now has its own development blog which I think you will find interesting.

http://guidewiredevelopment.wordpress.com

Enjoy!

Proposal for Agile 2008 Submitted

2008-02-24T14:18:00.000-08:00

Even though I still have one more post to go for this topic on Enterprise Agile Testing, submission deadline for Agile 2008 is approaching. So I looked at other submissions, looked at what I have written, and submitted my proposal:

http://submissions.agile2008.org/node/3736

Feedbacks are welcome either through the submission page or my website http://www.shaneduan.com/contact.html

Enterprise Agile Testing Part III: Managing Tests with ToolsHarness, Individually

2008-02-20T23:15:00.000-08:00

This is the third part of the Enterprise Agile Testing:

Introduction
Test fixtures like assertion, builder
Test Environment Set Up with TestBase
ToolsHarness, a continuous integration server farm that treats tests individually
Active and stable branch, localizing the damage

Continuous Integration has proven to be one of the most important practices in agile software development. Every time that a developer checks in the code, the resulting code base is rebuilt and tests are run against it. The end result of the integration tells everyone on the project if the codebase is good enough for release. There are some prefer synchronous continuous integration through a push-button process over an asynchronous process through a tool like CruiseControl. But everyone agrees that it is something very useful.

the Difficulty of Holding the Line

With a tool or not, the most difficult part of installing such a process is probably holding the line of "zero broken tests". In my past consulting and coaching experience, it sometimes takes great effort and time to get the team into the habit of running all the unit tests before checking into the code, as well as making writing tests and test fixing as the highest priority, and keep tuning the test so that the whole process does not exceed ten minutes. Even that, not all the teams kept up with the practice after we left the project on a good note.

I recently got a chance to catch up Greg, one of the ex-ThoughtWorkers that I used to work with and respect. He showed interest in what I am writing and expressed his opinion, as I quote:

Our test suite is too large and too slow to run with every build. We are lucky to get results once a day.

- Not everyone cares about the unit tests to the same degree. Some people are too busy to track down failures right away. Not everyone sees the value in the unit tests, mostly because our coverage figures aren't high enough.
- Not everyone has the skills to write decent tests or design their code in a modular, testable fashion.

When I read that email, I became more motivated working on this third post, because it is exactly what I want to write about. In my previous job, I found out that the only way to make agile development work was to follow the XP practices, especially when it comes to continuous integration. In an enterprise environment (as defined by Introduction ), there seemed to be no middle ground between "green all the time" and "red all the time". It seemed that the moment the team fell off the status of zero broken test and couldn't recover quickly, they would be in a deep hole right away.

The first thing that I have learned at Guidewire, is that the above problems can be solved better with the help of a comprehensive continuous integration tool, ToolsHarness in this case. I am not saying this is a silver bullet, because there are still a lot of development practices and disciplines required. But I can certainly say that I am now seeing the light.

Test Farm for Parallel Testing

No matter how hard you try, eventually you will not have enough time to run all the tests that you would like before checking in code. When this becomes the case, test breaking will become the norm rather than exception, and the XP way of handling broken builds would not apply anymore. For a complicated system, the time to run the full tests will be huge. At the same time, agile software development dictates fast feedback time on code changes. The longer the turnaround time is, the more fraction there is on the iterative development - how is it even possible for the team to build on top of something not yet proven to be working and expect to have a high throughput?

Guidewire has a big testing farm, composed of dozens of machines, mostly on Linux and configured with H2 database. These machines are configured so that when a build is available, they will check out the test suites to run.

When a developer checks in a change list, ToolsHarness will first pull the source and do a full build to make sure that the projects still compile. Once the compiling finishes successfully, ToolsHarness will post the build for test. The tests used to be divided into different suites based on which test class they extends, for example, Database test, Metadata test, or Server test. With the introduction of TestBase, they are all converted to NewTestSuite and NewSmokeTestSuite. (The difference is that the tests in NewSmokeTestSuite are acceptance tests, which are full end-to-end test and require additional sample data.)

Based on the previous run of the test, the suites are divided evenly into several parts, so that each testing machine can check out each part and run it. In this way through parallel testing, it takes no more than 20 minutes to run a suite that would normally take hours to finish. This system is highly scalable, because all you need to do is adding more machines. With such a fast feedback loop, the developers can work on the large code base and still make medium size changes. The worst thing that can happen would be to revert the change that you made less than half an hour ago.

It is easy to run the tests against different databases and J2EE servers too. Some testing machines are configured with Oracle database or SQL database, some are configured on Windows platform, and some are configured using Tomcat or WebLogic. As I am writing this post, the tools pod (the team in charge of developing this server) is working on ‘customer build testing’, so that the test environment will be exactly as the production environment when running the acceptance tests.

Tracking Tests Individually

With a large team, you will have developers with different skill sets. While it is easy for an senior developers to be conscious about making small changes at a time, and be able to identify the problem based on the broken test, it normally takes a much longer time for a junior developer to fix them. Unless all your senior developers happen to be good coaches, you are going to be stuck with broken tests popping up here and there for a while.

When a testing machine is done with the test, it will post the test result back to the ToolsHarness. ToolsHarness will parse the XML file and store the result of each test into the database to track them. The benefit of this is so that developers can start tracking the tests individually. When a test is broken, ToolsHarness will make an educated guess based on the change list and changed package to assign it to the developer. If it turns out to be wrong, the developer can easily assign it to the right person.

When a developer logs into ToolsHarness, the first page, Desktop, contains a list of tests that have been assigned to him or her. In this way, you won’t be distracted as long as there are no tests assigned to you. The summary of all the tests is also on this page so that you know not to check-in anything when there are hundreds of tests broken, or give the senior developers a clue to check in others in fixing tests.

For each test, you can see the failure message in the form of the stack trace, the change lists associated with it, the history of the test to help you figure out the reason that the test is broken. You can also look into the log directory to see any additional generated file like server log and HTML page snapshots.

If you have written a broken test that you cannot yet fix, you can annotate it with KnownBreak, and it will show up properly in the ToolsHarness. If you have determined that a test in failing none-deterministically but you still cannot yet figure out why, you can mark it with NoneDeterministic, and it will show up as such in ToolsHarness. The key is to keep the noise of broken to minimum, if not zero, so that developers will get the notification accurately and fix them effectively.

Localizing the Damage through Branches

With aggressive refactoring, you will not be able to leave your platform code alone. Sometimes you know that the only way to be sure is to check in the code, let the continuous integration server do a full test on the changes. With this approach, you are going to risk putting the build into unstable state for a while before you can figure out the best solution. If the whole development team has to rely on a good build, they will either be out of commission for a while, or they are going to accumulate changes that will cause another wave of instability after you are done. And that is if you are lucky enough to quickly finish with the cycle of check in, revert, revert the revert to make more changes, check in, revert, ...

Sometimes, especially for the platform team and application framework team, you need to make big change in the code base. When it proved to break lots of tests in ToolsHarness, the best thing to do is to move forward by checking in more fixes, instead of reverting the change to do it again. The only problem with that, is that the code base comes unstable during the process. If you have a large team with others working on other areas at the same time, the number of broken tests could be disturbing. And as Greg pointed out at the beginning, not every team care about the tests in the same way. Some would prefer finishing the job at hand before tracking down the broken tests. The line for none-deterministic tests are much more blurry.

At Guidewire, the way to make it work for all the teams working on the same code base is through the branches. Each team works on an 'active' branch that they are free to do whatever they feel most productive. The only rule that they need to follow is to fix all the tests before pushing the change to the 'stable' branch, where every team is pulling change from on a daily basis. If it takes a team a couple of week to get into a stable state, then they will have to risk the merge conflicts. For a team that is diligent on fixing the test, their branch will be stable most of the time, and pushing would come much easier. It is exactly like what most agile books recommend on checking code -- check in as often as possible as long as the tests are passing -- except on the level of branches.

Sometimes you can still make a mistake and push a broken test (or more) into the stable. In this case, there is a 'merge' branch that you can use to fix the builds. Most of the time however, the fix is very easy, then all you need to do is to find out which team is next in line to push to stable, and manually integrate your fix into their branch. There are merge script written in Ruby to help the pull and push process. They are very robust and well tested, so that majority of the push and pull are merely a push-button process, i.e., you type in the command "merge.rb --pull", and you are good to go. We have a merge machine set up specifically for this job, so that the merger wouldn't have to give up his or her local resources.

Antidote

As many articles, books, blogs have pointed out, people are always the center of the agile development process. Even with a powerful tool like ToolsHarness, it is still up for the team to apply disciplines and agile practices. Because the team does not have to stop everything to fix any broken tests, it is actually easy for people to ignore the tests. Given enough time, enough code changes would have been checked in, making it much harder than it should be to fix the tests.

So the rule of the thumb is still the same: fixing any broken test as quickly as possible when they come up. The old tricks still apply, which including things like revert the changes that broke the build, make small changes, check-in often, monitoring email notifications, run tests before checking in, etc.

Enterprise Agile Testing Part II : Test Environment Set Up with TestBase

2008-02-08T17:15:00.000-08:00

This is the second part of the Enterprise Agile Testing (Not exactly following my original order here):

Introduction
Test fixtures like assertion, builder
Test Environment Set Up with TestBase
ToolsHarness, a continuous integration server farm that treats tests individually
Active and stable branch, localizing the damage

Testing through Inversion of Control (IoC) Container

(For concept of IoC container, see Martin Fowler's article: Inversion of Control Containers and the Dependency Injection Pattern)

Ever since testing through dependency injection was formally named, it has become the most popular pattern for unit testing. You control the environment in which the class is tested by carefully constructing the classes in the dependency before injecting them into the class under test. In this style, a typical test is composed of three parts. They are named differently if you talk to different people, but the one that I like is what I learned when I presented the "Given, When, Ensure" notation of jBehave to BayXP meetings:

* Assemble: Construct the environment that the test is going to run.
* Act: Invoke the method(s) that you want to test.
* Assert: Assert that the tested method has caused the predicted change in the environment.

It is safe to say that anyone who has done enough testing won't have any problem with "Act" and "Assert". It is the "Assemble" that has been giving us trouble. The following is an illustration extending PicoContainer's diagram.

To test the class marked by the big arrow, you will need to create the world as this class sees, then invoke methods on class and assert changes caused by the invocation. Among the ways of constructing the world according to the class under test, Stub and Mocks are probably the only pattern that has been well documented. As indicated by the article, each solution has its own limitations. For a small to medium size application, these kind of tests are generally manageable. But if you have done enough enterprise application development (as defined in Introduction), then you probably have seen your fair share of mocks and stubs getting out of hand, as was the case for Guidewire tests until year 2007.

Testing with a Loaded Container

During the past year, Guidewire has been slowly converting its tests onto a home-grown JUnit extension framework. The framework does the heavy lifting of constructing the dependencies, so that by the time the test code is called, all the dependencies have already been set up properly. If you really want to, you can even access a full web container through the embedded Jetty server. By putting your class inside a full container, you get a lot of benefits that you won't normally with a bare bone unit test, and you without breaking a sweat.

The immediate benefit is that you no long deal with mocking, stubbing, guessing. When your test calls into a method, you can be sure that the class would be in the same state as it is called in the real world (It might still not be in the one that you want in your test but that is a separate issue). Without mocking and stubbing, you don't need to walk on the egg shells any more as you change the class responsibilities and collaborations. You can call into a real messaging manager through the container, enable a message destination, commit an entity, and test that the message of the changes appear. All the codes paths match exactly the real world, so that your won't have any integration surprises down the road.

Because all the required validations are turned on, you are forced to create realistic data. With realistic data, your test becomes more realistic. You can put your test under debug mode at any time and get a good sense of what the data will be like in a real server. If you make a mistake and forget to set a non-nullable field, your test will blow up right away.

With a loaded container, you feel more confidence in the class that you are designing. Because you can see easily how this class fits into the whole world, you can make sure it becomes a good citizen by doing just its job, no more, no less.

This framework is extremely flexible, making it very powerful. You can modify the testing environment by annotating your test and registering your own annotation handlers. In this way, you can add additional set up code without even creating your own super test base, a typical case of favoring composition over inheritance. You will see many annotations that we have built already in the past half year.

Performance Improvement Considerations

Of course, all these are much easier said than done. And we are sort of going against the conventional wisdom of unit testing here. The first question most readers will raise would probably be "Loading the whole container for a simple unit test??? How can your test perform!?" Please trust me when I say that I had the same concerns. But after adapting to it for half a year, I think this is definitely a good solution.

First of all, performance is overrated. No, I am just kidding. The first thing that I would like to say is that if you are a TDD veteran, in that you know how to design your class such that you can manage your own dependencies well most of the time, then kudos to you and you can use the @RunLevel annotation to tell the framework not to do any set up that for you (see below)

I was actually not totally joking. I would like to argue that for an enterprise application (as described in Introduction), it is not uncommon that some part of the system is not designed as cleanly as it could have been. As a result, you have to choose between making the test run fast through the kind of mock that no one knows what is going on, or making the test run a bit slower but reflects the real system. Since design validation is the whole purpose of tests, I vouch for testing the right thing with a bit of sacrifice on the speed.

In addition, the test framework has a set of performance considerations in place to make sure that overall the test performs well.

Run Level

Guidewire applications have the notion of run level as a way to bring the system online in stages. You can annotate each test with the desired run level to have just the things you need set up before the test. The following is the list of run levels that I have used.

* NONE: This is just like good-old jUnit test.
* Shutdown: At this level, you have all the system configuration read in and meta data loaded. You can run any test that does not touch database
* No Daemon: This is the default value. At this level, you have the database connection initialized and the schema updated. You can run any test that hits the database.
* Multiple User: At this level, you have a full blown application server with background batch process running. This is typically used by QA for acceptance testing.

Database Tests

By default, all tests are using H2 as the embedded databases which greatly improves the test performance. I have been a big fan on in-memory database since HSQLDB. DBFixture is the proof.

During the development, the database schema changes all the time. Guidewire products have an upgrader built in place to compare the database schema and automatically issues SQL statements to upgrade the database to the right schema. However, the upgrade process can take time. To save time, a backup copy is created after the upgrade finishes so that the database can be restored as necessary (See @ChangesSchema). There is one implementation for each database that we officially support so all the tests can run on all databases if we choose to.

For each table there is also a shadow table that stores the default data set up by the test environment. Before each test run, the data in each table is restored from the shadow. In this way, different tests won't step on each other's toes and end up causing other tests to fail. For performance reason, the data is only restored once for each test classes, because it is easier to make sure that the test methods in the same test class don't affect each other's data.

Server Mode for Web Testing

The QA acceptance tests are written in GScript. When running in browser mode, it uses Selenium to drive the browser to connect to the server and run tests. However, when you have enough tests, the slowness of the browser really shows. Guidewire applications are built on top of JSF framework, where the generated HTML source is driven by the page model on the server. With the exactly same script, we can run them in server mode, where the scripts are run against the page models in the server session. Without the browser layer, HTTP connection, HTML generation and parsing, the test run is cut down dramatically again.

Functional Considerations

The meta data layer of Guidewire applications is extremely extend-able and configurable, and the SQL being executed in the database layers is generated dynamically based on the metadata configuration and the database set up. It would not be practical to mock out the whole thing. The test framework provides a fixed out-of-the-box container for each test and locks it down so that the test or the code under test wouldn't accidentally try to change those dependencies. But the developers can modify the test environment through annotations. The following are the typical annotations:

@IncludeModules for Configuration Testing: With this annotation, you can specify a list of directory where the test should load the additional configuration from. In this way, you can configure the test environment (registering additional plugin, registering additional SOAP interface, extend the basic data model, add additional web pages, etc.). This is great when you want to test different configuration cases, and still leave the base configuration simple and fast.

@ChangesTime for Time-based Testing: Sometimes your test is date sensitive. With this annotation, you get a hook to change the system date on the fly before you creates the data you want so that timestamp meets your condition.

@ChangesSchema for upgrade testing: With this annotation, your test can run wild and make a havoc of the database schema. At the end of your test, the schema will be restored from the backup automatically. This is very useful for upgrader related tests.

Testing Annotations

These are the additional annotations telling the test framework how you want your test to run:

@ProductUnderTest: You can write a test, put it in a common module and tell the test framework which product you want this test to run. For example, we need to make sure that the base data model can pass the validation for all applications. We can write a test that will start the validation without being dependent on which product it is. With this annotation, the same test can be run with data model from each products. Think dependency injection on production is a good way to go? Why not apply it to test?

@TestInDatabase: From time to time, you have to implement something that is a little different for different databases, or a feature that is only applicable to one database (Oracle AWR report, for example). With this annotation, you can tell the test framework which database this test should be run against. By default, all tests are running in H2 database only for performance reason.

@DoNotRunInHarness: This is for push-button tests that cannot be run automatically. For example, we have a test that pings map point web services and make sure that we can parse the result properly. Map point ended up telling us not to ping their staging server continuously. So this test is disabled in the testing server.

Testing Semantics

There are also other productivity improvements. Your test case can now implement beforeClass(), afterClass(), beforeMethod() and afterMethod() to be run in the way, well, as the name indicated. After answered enough question about when setUp() and tearDown() are run, I think it is a nice change.

Because jUnit holds on to ALL the test instances, each fields in the test class is actually a memory leak as far as the test concerns. The test framework automatically null out all the fields (with some configurable exceptions) at the end of the test case when all the tests methods are done.

Other Considerations

This kind of test writing is also supported by our other development practices, namely ToolsHarness and Branching strategy, which I will cover in detail in later posts.

With your tests covering more code, the tests could very well break for the wrong reason. With the ToolsHarness, we were able to exam each test failure easily, locate and isolate the problems easily and the development won't grind to a halt every time there is a broken test. With the test farm provided by the ToolsHarness, our test can run concurrently so we can have better tolerance on the speed of individual test.

With the branching strategy, we are making sure that the platform code is in a good enough state before it is released to the application team.

Appendix: Things to watch out for

At the same time we creating a path to make test easier to write, we also put ourselves on a slippery slope that could lead us further and further away from effective unit testing. Sometimes it is much easier to write a test that covers a lot of than to set up the environment so that only the code you want to be tested will be tested. Why is that bad? Here is an example:

As I am writing this post, I am wrapping up a feature called "Field Level Encryption" by adding upgrade support from an earlier version of the application. It was extremely tempting to do the following:

...
// the column is length 6 nullable, alter it to leng 3 and not nullable
String[] sqls = getDbCatalogSupport().alterColumn(table, column).withLength(3).withNullability(false).getSql()
DatabaseTestUtil.updateInTx(sqls)

// Insert data that need to be updated
DatabaseTestUtil.updateInTx("insert into px_test_encryption (id) values (1)")

// run the upgrader to make sure it does not fail
new Upgrader(database).upgrade()

// run the schema checking to make sure everything is up-to-date
List error = new DatabaseSchemaVerifier(getDbCatalogSupport.buildSchema()).verifyAll()
assertThat().list(error).isEmpty()

Object[] row = assertThat().sql("select encrypted_field from px_test_encryption where id = 1", new Class[] {String.class}).hasOneRow()
assertThat().array(row).is("tluafeddefault") // null column should be updated with encrypted default value.

I am very sure that we can all agree that this is very concise and expressive. Change the database schema, insert data, run the upgrade, make sure that the schema is now up-to-date and that the row is updated correctly, just like it should be, right?

Not quite...

The problem with this test lies in the "upgrade()" and "verifyAll()" method calls. They are both very comprehensive and cover a lot of area. As a result, this test runs for a long time (over a minute). At the same time, someone could check-in a code with bug in either the upgrade code, or schema verification that has nothing to do with encryption, and this test will be broken. In an enterprise environment, you only need a small portion of tests like this to generate enough noise. And eventually developers will be so tire of spending time on a broken test only to find out that three other people are also looking at it and it will be fixed by one of them. You will start delaying looking at broken tests and they will stay broken for a long time, other changes will be applied on top of the changes that broke the test, you will have a hard time fixing them, you will start hate tests, you will write less, the quality of the product will go down...

So, for the sake of everybody, lets spent more time making the test as fine-grained as possible

...
// the column is length 6 nullable, alter it to leng 3 and not nullable
String[] sqls = getDbCatalogSupport().alterColumn(table, column).withLength(3).withNullability(false).getSql()
DatabaseTestUtil.updateInTx(sqls)

// Insert data that need to be updated
DatabaseTestUtil.updateInTx("insert into px_test_encryption (id) values (1)")

// run the upgrader to make sure it does not fail
new Upgrader(database).encryptDecrptUpgrade()

... (Some other code to verify just this schema)...

Object[] row = assertThat().sql("select encrypted_field from px_test_encryption where id = 1", new Class[] {String.class}).hasOneRow()
assertThat().array(row).is("tluafeddefault") // null column should be updated with encrypted default value.

However, this is not to say a test like the original one does not provide some value for being comprehensive. We do have upgrade tests like this for specific kind of upgrades, the ones that our customers are going to go through. Those tests will load the database from a backup so that the schema matches the ones that we release to our customer, then we run upgrader through it and verify that the schema is up-to-date. Each test also has an opportunity to insert additional data before the upgrade and do additional verification after the upgrade. In this way, when our customers get our newer build, rest assured the upgrade will not blow up horribly.

Enterprise Agile Testing Part I : Introduction

2008-01-12T20:03:00.000-08:00

The idea of "Enterprise Agile Testing" has been in my head for several months now, result of what I have learned at Guidewire and based on my previous XP experience at ThoughtWorks. I am planning on a proposal to Agile 2008 on this topic. Before I can choose my proposal topic, I need to write everything down first, kind of like a project engagement.

Actually, project engagement is not a bad analogy. I must define what my proposal is about and what it not about --- what is out of scope if you will. My approach is to write a series of blog posts, each cover a specific topic and look back to see what I end up with when I am done. If I approach it as if I like writing a book or even like that agile 2007 paper, I might never finish it.

The Enterprise in Agile Testing

Enterprise here means large scale software development. The large scale can come about through a large code base or a large team. Here I am ignoring the controversial topic of whether or not large code base or large team are problems that should be avoided in the first place. They exist, I just want to point out two things that result from them with regarding testing in such an environment.

First, with a large code base, a tester cannot clearly hold onto the code's design in his or her head, let alone the intention of the test. Instead, agile testing in enterprise environments requires a comprehensive testing framework. This framework must do more than what JUnit does out-of-the box, so that anyone (including you) can come back to a test at any time and understand it.

Second, with large team, it is pretty much impossible to ensure everyone is aware of how important testing is. This is not to say that you should give up on a large team's continuous improvement on treating testing seriously and writing better tests, but I have found out that the line of "Zero Test breakage" is extremely hard to hold. As a result, some middle ground must be reached between complete awareness and total ignorance. Only in this way is it possible to see results in testing improvement efforts.

Agile

Since Rob and Jim pulled me away from the EJB madness and introduced me to the wonderful world of XP in 2001, "Enterprise" has slowly restored its place in my vocabulary. At the same time, the term "Agile" is getting closer and closer to my list of red flag words.

Agile here refers to the situation where the code is constantly under changes. This can happen because the requirements keep changing, or the development is done iteratively through story driven development. Constant change makes the ability to write concise tests more important. Additionally, the tests status of project must be treated as more than a binary state if testing is to keep pace with development. In this way, the development would not be paralyzed because there will almost always a test broken here and there, and the turn-around time for testing the checked-in code is not as short as 10 minutes.

Testing

So you have a large project code base that you keep changing, with a team of members with mixed skills. You need to ensure that the code (including the tests) you write is of high enough quality so that two moths from now you can still read them, understand what they do, understand why they do that, and change them. At the same time, you want to give others the time and tools to adjust to test infected development and hopefully test-driven development eventually.

Content and Structure

So, the above is the introduction. The items that are in my mind are as following, I'll update the links as I post them.

Test utilities like assertion, builder
TestBase with annotations for test environment configuration
ToolsHarness, a continuous integration server that treats tests individually
Active and stable branch, localizing the damage

Team Estimation Game - By Steve Bockman

2008-01-08T23:19:00.001-08:00

Well, Steve didn't have time to put it up his website, so I figure I'll write up what I have learned here.

I ran into Steve during the agile 2007 conference so I invited him to do the session for us at BayXP. Last October, Steve came down and hosted a session of the Team Estimation Game. This is the first time I learned it I really liked it.

Given a set of defined stories, the purpose of this game is to come to the consensus about the relative estimation of the work to be done for each story. If you don't have a set of stories to try it out, you can use the sample produced by Steve here.

So once you have all the story cards, you first estimate their complexity relative to each other:

Place Story Cards in pile on table.

First player places top card on playing surface.

Next player places top card on playing surface relative to first card.

Next player can either:
* Play top card from pile, or
* Move a card on the playing surface, or
* Pass
Repeat Step 4 until
a) no more cards remain in pile, and
b) no player wishes to move a card

Here is an example of the result:

After the first stage is finished, you can now go on to assign points to them so that you can track your team velocity:

6) As a team, choose estimation units and values.

With the agreed units:

7) Place an estimate at the top of each column.
8) Change estimates until all team members agree.

We did it during the meeting and it went very smoothly. The interesting thing is that Steve told us the result is not too much different from the ones that he has conducted elsewhere with other groups.

I am definitely going to try it on my next story estimation session.

References:

* Planning poker, which is another popular and useful estimation game.
* Steve's slides: http://www.shaneduan.com/estimation/Team_Estimation_Game.ppt
* Planning Poker Cards you can buy here.

Paired Programming

2007-11-08T16:58:00.000-08:00

I posted this internally at Guidewire today in response to a "meta-analysis of empirical studies done on analyzing the effectiveness of Pair Programming." shared by colleague. Then I thought this is actually a pretty good short summary of my point on Paired Programming nowadays:

As useful as they are, my problem with these analysis is that they are trying to isolate the behaviors that work best when applied together. This applies to TDD, team development, short iteration, continuous integration, etc.

Unlike engineering, software development is actually a discovery process, during which most of the value comes when discoveries are put together. For example, when two heads get together, the value is not only about performing complex programming tasks correctly, sometimes it is actually more about realizing that there are other parts in the system that can help with the current task, or even not to attempt because someone has already tried.

On the other hand, programming is a highly concentrated task. Once interrupted, it always takes a while to get back to the mode, and we all know how frequent interruption is during a normal work day. With paired programming, you can actually keep the flow even when one person is interrupted for a short period of time. There are practices that you need to do to make sure that this is really the case. For example, it it turns that the the person is going to help another pair for a long period of time, the right thing to do is to switch the pair, etc.

Also the practices of XP work the best when they are working together. For example, rotating your pair routinely help the effectiveness of the paired programming. If you do TDD during the paired programming, it will help the two people to focus and lower the overhead of communication.

One last thing that I have found out about paired programming is that it helps the team communication. Paired programming forces the two people to talk constantly, which creates the "omni-communication". I know some people find it annoying. But I have sit in such environment on many occasions and once you get used to it, the kind of information you can pick up without any effort is very surprising. Many project managers I talked to also reported that because of the "omni-communication" they pretty much eliminated the "status meeting" and replaced it with "issue resolving meeting".

Thoughts of the Day

2007-11-07T11:00:00.001-08:00

This person posted a blog with the following content, then posted another blog saying he just deleted it and made the apology. Several weeks later, he posted another blog saying he is quitting.

I don't know the details, but I swear to agile god that I would rather quit than have this happen on my team.

It doesn’t bring any benefit , Pardon I asked stunned to hear the response I reeled in surprise as I took in what I was being told I mentally backed up a bit so in effect the team lead on a high value project was telling me not to worry and more to the point it was unlikely that anyone at least anyone on his team would ever worry about refactoring a piece of code I had come across because it worked and it was unlikely we would ever need to change it!

I wanted to say but its wrong , its bad code but I didn’t I stood there and said oh ok I understand and I walked away and sat back down and carried on letting my fingers fly across the keyboard trying hard to block out the bad code screaming at me form the other file.

Green Day Blog

2007-10-15T08:57:00.000-07:00

Something hilarious: http://www.youtube.com/watch?v=T4SDQu7Izrk

And something useful:

Cotta 1.2 Release

2007-09-19T08:27:00.000-07:00

I have got quite a few good feedback from Brian who is using it ('in anger'!) at Guidewire. So I am putting together a 1.2 release, even though the change list is really small.

This also include some JavaDoc updates. Even though I am not a believer myself on JavaDoc, I have come to realize that it is still a recognized way of communication and sometimes preferred over email or forum.

(I have been writing drafts about the work that I have been doing at Guidewire, just never been able to publish any of them... Not good if I want to keep this blog alive)

WebGen, Cool!

2007-09-07T08:03:00.000-07:00

I finally run into WebGen today when checking out rSpec. So far by the look of it, it is exactly what I have been looking for for the last two years! Of course, that means that one third of my work with BuildMaster will be out of commission, but this is such a good thing!

I supposed this was bound to happen sooner or later.

I thought it was a bit odd that I didn't find anything interesting when I had the idea of generating a static HTML website based on specified template and using wiki style format like Textile. I had several simple websites to generate at that moment (my homepage, jBehave at the time, Cotta, and BuildMaster), so I settled for creating one of my own with some simple Ruby script as the base. It worked out well because I have been able to update the content very easily. Earlier this year, I also set up the selenium Ruby project very easily.

It didn't come without a price, when I released BuildMaster 0.7.0 . I just settled for the fact that there is simply not one that is easily found. I also try my best to enhance it to be more powerful and flexible but I just never got enough time.

Now with WebGen, I can finally take something off my plate and still get what I want. Since I wrote those websites with Textile syntax already, I don't see a big content format conversion either.

My first reaction was that "Man I wasted a lot of time because the lack of information". Then I realized that I learned pretty much everything I know about Ruby and rSpec while working on this module, and that the Cotta layer that I extracted out of it is still very valuable. So even though I do need to do a better research next time, this time it worked out for the best.

Agile 2007 Experience Report

2007-08-19T19:51:00.000-07:00

It is finally over. Julian and I have been working hard on this experience paper for agile 2007, and I have just come back from the trip to Washington DC for the presentation.

The title of the paper is "Large Build Teams: Help or Hindrance?". If you google it, you should see an older version.

When I got the copy of the experience reports at the agile 2007 registration booth, it sure feels good to flip through the pages and locate the fruit of my hard work in print. Now I really admire the authors who can keep coming up with good books.

The presentation itself went very well too, although I wish I had more time to make a good point of what I have learned in this project. Oh well, those who are interested can certainly read the report and contact me directly.

How Old is the Concept of "Inversion of Control"?

2007-08-13T12:29:00.001-07:00

Think this article by Martin Fowler, probably most quoted, is the first one talking about this concept? Well, then we were both wrong, because I have just found out this article, "The Dependency Inversion Principle", published as early as 1996!

Isn't that something?

With Wing Bark, No Quack

2007-08-02T18:03:00.000-07:00

I have heard tons of insane software development project stories, and this one beats all of them added up...

I only hope the story is not true...

Atlassian and Cenqua

2007-08-02T09:16:00.000-07:00

Two of my favorite companies from down under is now one. I love it.

Atlassian taught me that wiki can be cool and professional, and that Bugzilla is not as good as a bug tracking system can get.

Cenqua simply saved me from the CVS madness.

Thank you both!

Sad Day for Polite Software like IDEA

2007-07-18T09:08:00.000-07:00

IDE Showdown - Evangelists duke it out at Cologne JUG

My hats off for organizing this meeting. However, based on the conclusion that the article has drawn, I mark this as a sad day for the polite software like IDEA. Apparently we still live in a society where the quantity of the work, i.e. feature list and 'advanced features', rather than the quality of the work, i.e. how you implement the same feature in polite way, matters.

And I am also annoyed every time someone says "Eclipse is more than just a Java IDE", or "Eclipse is more than just an IDE", as if it is a good reason for it to be poorly designed and most annoying to use. You know, that is not even a good excuse.

I can rant on for days. But I have been ranting for so long since Borland decided to stop the development of JBuilder purely out of fear for Eclipse (hence my reason to quit) several years ago, that I am just tired. All that I feel up to do right now, is to quote Alan Cooper in his great book “The Inmates Are Running the Asylum”:

It is abundantly clear to most of us that common folk don't know the difference between a token ring and a mood ring. We need these creator of "faster, stronger, better" innovations to be sure that the creations actually improve our work and lives -- not simply drive us crazy. We need the technology to work in the same way average people think. We need a revolution to restore our sanity.

Update: I also think that IDEA is wrong by trying to fight on the open source battle ground. Of course you are going to lose! What is so wrong by saying

Our business model is to produce a killer softerware full time and get paid well enough for it to be sustainable. Because of our business model, we have the purest agenda and vision for our IDE, i.e., make Java developers do their own job in the most efficient way. If any other FREE IDE can beat us in this market, we will cease to exist. But so far, it looks like we are the best. And we ARE supporting open source in our own way, by giving out open source licenses.

Squeak - My New Language of the Year

2007-07-09T08:21:00.000-07:00

When someone took the time and write a tutorial like this, the least that I should do is to read it through and learn it to show my respect.

I am not totally sold on the ""Learn at least one new language every year" (p. 14 on Pragmatic Programmer book). Not that I don't think it is a great idea. I just think for one it would be very hard to do since it normally takes one (me at least) several years to learn to fully appreciate the language. Another reason is that I think it is more important to contribute back to the society of what you have learned than keep learning for learning's sake. I know I have not contribute enough as it is, given the pressure to put food on the table, making my way to afford a house in bay area, take time to enjoy life, and such.

But when all you need to do is to download the Virtual Machine and Image, and read the tutorial, and you are up every night feeding the baby anyway, why not.

Update: As it just turned out, this post showed up in case you wonder what are the choices are out there if you want to learn smalltalk.

Second Update: Free (and Legal) online Smalltalk books

Yak shaving

2007-07-08T03:51:00.000-07:00

I have found this Yak Shaving to be true in many occasions, especially when starting the BuildMaster project.

I think that is why Agile development works. You use one iteration to focus on one level of the work to build a "Walking Skeleton" as described in "Crystal Clear". At the same time, you keep updating the story backlog about all the work that you discovered on the way so that you don't lose anything and can prepare for future iterations at the same time.

Reference: Really Bad Powerpoint

2007-07-08T03:18:00.000-07:00

Julian sent me this link, "Really Bad Powerpoint", as a guideline as we are developing our presentation for the Agile 2007.

I do find the content convincing, especially about preparing full notes but not handing them out, and use less words on the slides.

I don't think I agree with the idea of no transition, though. I have had good experience using them, and I have seen others using them with good effect.

Bad Interaction Design - Installing Acrobat Reader

2007-06-12T22:59:00.000-07:00

It just so happened that two day after I was impressed with Firefox' good interaction design, I was utterly annoyed by the installing process of Acrobat Reader.

What is so wrong about downloading an exe file that is an installer (using any download manager that I want), run it, and choose the location that I want to install the program???

Oh no... You need to install the nice "Acrobat Get", which is either an ActiveX control for IE or Firefox plugin, which of course means that you have to set the security setting on your browser. Then you have to wait for that thing to come up, start downloading, and silently install the program at the place where it sees fit (which happens to be my C drive that has only 1.5 GB left).

So I ended up with even smaller space on C drive (40 GB free space on D drive). And guess what, my computer hung during the process and I had to do a hardware shutdown (holding on to power button for 5 seconds).

Update: Tom just commented and provided this link to download the installer.

ftp://ftp.adobe.com/pub/adobe/reader/win/8.x/8.1/enu/AdbeRdr810_en_US.exe

Thanks! I love internet!

Test Doubles

2007-06-10T16:13:00.000-07:00

This is a nice article that covers the topic very well. Stub, Mock, etc., are all part of the test double.

Pay attention to the very first paragraph. Going out of this context is how mocks are getting abused.

Good Interaction Design: Closed Tab List

2007-06-06T13:58:00.001-07:00

I was very delighted to discover the "Recently closed tab" feature in firefox!

If you accidentally close a tab, you can now recover it through the menu "History-->Recently Closed Tabs". And guess what, your history is available as well!

Now that is a good interaction design.

Remote vs Collocated Team

2007-06-03T23:09:00.001-07:00

There is a heated but surprisingly friendly discussion about Remote vs Collocated Team going on right now at Yahoo! agile-usability group. I have just started paying attention (and try to follow it) this weekend because the daily digest I get each day is full of it now.

It is pretty interesting to read. Well, if you have done XP coach or consulting at a firm that is not so hot on everything about agile, you might appreciate it more.

It is also interesting to point out that the very first post started with this:

Maybe this is not a hotly debated topic any more since ...

rSpec 1.0.4

2007-06-01T22:01:00.000-07:00

Busy with BuildMaster, selenium-ruby, and Entrance, I have not looked at rspec for a while.

Until the Code Kata with rSpec we did at BayXP last week.

Boy have they changed...

And all for the better I'll say, after converted one file last night. Personally I liked the 'specify' more like 'it'. Actually, I thought they would go with 'should' since I type that at the beginning of the every specification name.

But that is small thing. With the new version, it would be a lot easier to come up with new matchers, re-use behaviors.

Update: Check out the rspec report I have after converting to this version! It is actually cool to see the broken tests report, more so than the passing report that rspec has itself

http://buildmaster.rubyforge.org/rspec/

Different Perspective: Singleton as Dependency Management

2007-05-26T14:28:00.000-07:00

I used to loath the singleton pattern, mostly because of the pain it has caused me whenever I take over some old code and try to write test on it. Interesting enough, Guidewire has used it in a way that is actually kind of cool.

If you have tried to manage component dependencies before the IoC container came out, you are probably fully aware of the pain you have to go through when a new dependency is introduced. Either you end up with a HUGE interface, or you have to change the signature of a handful classes and make some of them have longer parameter list in the process.

Here those dependencies are manager as "Dependencies" singleton interface. It is a set of static methods to be precise. So if a class decides it needs to use metadata layer to build the query and database layer to execute the query (like the archiving project that I am working on), it can just grab the instance by calling "PLDependencies.getExtractController()" and "DatabaseDependencies.getDatabase()". The dependencies are set up during the server initialization (through dependency registrations) and never changes.

There are some behavior changes associated with this design. Two on top of my head:

* No mocking/stubbing. Nothing is stubbed out within the application. We use H2 for testing and the database is reset using restore. So the tests run fairly fast. After seeing some horrible mocking code, I like this style more and more.
* Be aware of the dependencies. With dependency injection, you get used to understanding the dependency relationships by looking at the construction and method parameter list. With this design, surprises can really happen. You could very well call into something that does not take any parameter only to find out that it actually does a lot of things.

It is not the perfect design, but once you understand the pattern and learn not to abuse it, you can actually manage the dependencies quiet easily.

Test Coverage

2007-05-15T11:19:00.000-07:00

http://www.artima.com/weblogs/viewpost.jsp?thread=204677

The above link got passed around at Guidewire the other day. It reminded me of the consulting days where we had to figure out the right answer for the right stage of the agile enablement.

This also reminded me about another interesting thing is this "90%, 50% rule" from Adam. He told me that he started paying more attention to the line coverage of the projects that he has been on. What he noticed is that the line coverages seem always end up at 90% or 50%. Normally 90% are the projects that developers are doing test-driven development, and 50% are the ones that people recognize that writing tests is an essential part of software development but not necessarily in the TDD fashion.

I do remember that all my projects in ThoughtWorks had the around 90% coverage. I wonder how much we have at Guidewire or my open source projects. The reason that I don't know is because at Guidewire the QA tests affects the coverage as well so it is not the same thing. My open source projects are all done in the TDD fashion so I never bothered to spend time create something that I don't really use.

Build vs Buy

2007-04-29T17:27:00.000-07:00

It has always been a constant debate on Build v.s. Buy. After working for an elite consulting company, and now a revolutionary product company, I start thinking that there probably is no simple answer.

http://www.addsimplicity.com/adding_simplicity_an_engi/

Maybe the truth is that this is the wrong question to ask, that the real question is "How to verify the return of the investment in IT"

GScript - You don't Have to Use Ruby to be Cool

2007-04-22T17:49:00.000-07:00

So why do you think Ruby is cool?

If you are like me who work mainly on Java platform, then you probably like Ruby because how much less code you write in Ruby to achieve the same result. Now, if you think about it, not all of them are because of the type-free concept.

With the way that generics goes (what it was and what it is), we can see that the design of Java language comes with some tough trade-offs. After all, I don't suppose you want to piss off the big vendors who keep the money rolling.

Well, what if we can start fresh?

Check this one out...

var list = new ArrayList({one, two})
// this creats a map of beans with ID as key.
var idMap = list.partition(\bean -> bean.ID)

A few notes:

The last statement uses closure, in GScript syntax. For anyone familiar with Ruby, this call would be something like "list.partition {|bean| bean.id}"
The expression "bean.ID" is actually a "bean.getID()" method call, which is declared as property getter in GScript
Even though "list" is a type of List, you can add "Enhancement" to this interface so that you can have additional behavior built on top of existing public methods. This gives you some of the benefit of "mixing" without sacrificing the type safety. If you use apache commons library extensively, you probably have a list of enhancement that you can write already.
The type is inferred so you really need to type once. (Note that the type of variable idMap will be inferred as Map<Key, KeyableBean>, a generic map instead of just a Map)
Create the list like the way you mean it instead of the Arrays.asList syntax.

You have just caught a glimpse (and I do mean "glimpse") of GScript, the language that Guidewire developed on top of Java. You want a rule engine that can be configured? What about a rule engine that can be programed and deployed without bouncing the server?

There are talks about making this public, by which time you will be able to see the full power of it. Before that, you can consider coming in for an interview. :)

Free Broadband using Sewer System from Google!

2007-04-01T10:40:00.000-07:00

As expected, Google has something cool this time every year.

The last several ones that I have enjoyed was:

Or you can read all of them here.

Label Blog Post!

2007-03-30T12:00:00.000-07:00

If you are a blogger user, did you know that you can put labels on your posts???!!!

I have been jealous of WordPress users for a long time but I am just too busy/lasy to set up a server of my own.

Until the other day I noticed a blog post with labels hosted on blogspot. Well, I guess that says how sloppy I have been on writing this blog.

Again, thank you Google!

So Close

2007-01-13T17:00:00.000-08:00

Working on several things at the same time and they are all getting very close but just not quite. I have to say this puts a bit of stress on me.

At guidewire, I am working on a feature that involves analyzing the domain graph so that we can do things like record purging, archiving, data relationship analysis, etc. I think I have the main graph finally worked out (even have a primitive UI to show for it). I have the tests running and one of them can generate a Wiki content so that the document is always up-to-date. However, at the edge of the graph where the domain objects inside the graph has references to or from objects outside the graph, it gets a little complicated. I am keeping down a list on the wiki and it is getting more and more complete..... Just not yet.

I have arranged next sprint (two-week period) to start on next week so I was really hoping that I could finish this before I kick it off, because it will make a lot of stories easier to write. But for various reason I just couldn't quite just make it. I think I'll have to spend next Tuesday (Monday being a holiday) to make sure that the stories for the next sprint is good to go and that QA is on board with what I want to do. Adam has got me an account with Pivotal's tracker. Things are all good, just that being one-man team is not easy because you still have to do the same amount tracking if you want to still get the benefits for agile development.

At home, I have been busy with mainly two projects, BulidMaster and Selenium-Ruby.

For BuildMaster, I finally figured out how to use RubyGem to archive the file, and why reading image file with Cotta cannot read the full content (you have to call io.binmode even if you open it with 'wb' argement). I am getting very close to have Cotta support archive and zip so that we can just do 'dir.archive_to(target_file).zip' and get a zip file. I need to redesign the BuildMaster website so that project releasing, website building and cotta are presented as three separate pieces.

For Selenium, that is something that I am still not ready to share but I think it will be interesting as well. I will need to modify the selenium server and again I am getting very close.

Yet Another Keyboard Layout?

2007-01-10T08:21:00.000-08:00

Interesting... Although switching between QWERTY and Dvorak, and telling people about Dvorak is already hard enough. If anything, I would be glad to see the day that Dvorak prevail.

Nevertheless, the comparison applet is still something very interesting.

This Made My Day

2007-01-09T21:19:00.000-08:00

Enough said...

Bye bye Eclipse! I know I am supposed to say "thank you and miss you" and stuff, but I really don't want to lie.

Domain Based Web Testing

2007-01-04T00:19:00.000-08:00

This has been in my mind for some time and with the creation of Selenium-Ruby I finally have someplace to put it:

http://selenium.rubyforge.org/doc/domain-based-web-testing.html

The idea of domain driven is also behind the BuildMaster and Cotta project.

fix svn tagging in sourceforge

2006-12-15T20:44:00.000-08:00

I have been having tagging problem with Subversion in Sourceforge. It took me several days to realize that this is not going to be resolved by itself. Once again, Google helped me found the solution:

You need to point your working directory to a different repository URL, through command like:

svn switch --relocate https://svn.sourceforge.net/svnroot/cotta/trunk https://cotta.svn.sourceforge.net/svnroot/cotta/trunk

You just need to replace 'cotta' to the unix name of your project.

So Much for Java Generics

2006-12-04T22:22:00.000-08:00

I always thought that Java generics is supported through compiler is a bit weird, and after 5 hours of email and spike, I finally know why.

Don't get me wrong, I love generics. Even though with TDD I can catch my mistakes most of the time, but with generics I can see my mistakes even before I run the tests. I won't say no to that.

Anyway, it all started with me starting a prove-of-concept project with another friend. So I am setting up libraries and figured I'll use jMock's constraint framework, after I convert it to support generics. Hence started my journey.

I checked out jmock code and was surprised to find out that there is a jmock2 directory. By the document that has been written, it is a way to stay with jMock style but don't lose the type safety. Cool! What is more, the constraint framework has been extracted into something called hamcrest.

Through email, Nat pointed me to the project in sourceforge (duh): http://sourceforge.net/projects/hamcrest. It looks very promising, even have the adapter for JUnit3, JUnit 4, and TestNG. It is apparent the intention of this project.

After I calmed down with all the excitement, I noticed one thing strange: The Matcher interface, the one interface that matching framework is evaluating matching logic, takes the "Object" type, rather than the generic type. What this means is that all the matching instance will have to do a type cast.

"That was silly", thought I, "Surely they know better than this". So I modified the interface in hamcrest, ran all the tests and proudly submitted my patch to Joe.

Turns out that the issue is not hamcrest itself, but jMock.

If you have two methods of the same name that you want to mock, say
public void test(int value);
public void test(String value);

And you provide the expectation
mock.expect("test").with(eq(1))

Upon the invocation of a.test("arg"), it will try to find the match by checking the matcher returned by eq(1) as well as that returned by eq("arg"). When comparing to eq(1) it will fail because from the instance there is no way you can figure out what the generic type is, and invoking by force will cause a class cast exception because there is a class cast injected by the compiler.

So in the above invocation, instead of having something like "unexpected invocation error", you end up with a weird ClassCastException error.

Bloody hell.

SCRUM at Guidewire

2006-11-29T00:18:00.000-08:00

A month ago, I have joined guidwire, a local start-up building insurance products.

At guidwire, we are using SCRUM. So every new hire will get a book of "Agile Software Development with SCRUM". Somehow I missed it but I found it soon enough and got a copy right away.

I don't know why I never got a chance to read this book and I am really glad that I did. SCRUM is very much like the project management part of what I have been doing for the last several years. This includes the details like format of SCRUM meeting (daily stand-ups), feature backlog (master story list), goal driven team development.

On google video, there is a presentation done by Ken himself: http://video.google.com/videoplay?docid=-7230144396191025011&q=ken+scrum.
In it, he talked about SCRUM and XP complements each other. I didn't understand it before, and now I think it is really true.

Typing Contest : Dvorak vs. QWERT

2006-11-22T17:15:00.000-08:00

After switching to Dvorak for almost a year, I decided to switch back to QERTY, for two reasons.

I was pairing at least 6 hours a day and was driving everyone mad with the layout switching
I wanted to see how fast I was with QWERTY.

Before I switched back, I timed myself in typing out the page "Dvorak Follow-Up". Now that I am fluent with QWERTY again (I should have done that earlier but I was too lazy), I just typed the page again using QWERTY.

And the result is...(drum...)...

DVORAK!

Time used with Dvorak: 9:28minutes.
Time used with QERWT: 9:55 minutes

I cannot say that I am not surprised. After all, I have been typing QWERTY almost my entire life and I only used Dvorak for a year. But here you go.

On the other side, 30 second is very small. I could just be tired tonight.

I also finished the "Joy and Pain with Fitnesse" in 29:30 minutes using QWERTY. If I ever switch to Dvorak again, I might do another contest.

Scrum et al. at Google Vedio

2006-11-15T23:36:00.000-08:00

http://video.google.com/videoplay?docid=-7230144396191025011

I thought it was interesting on the part that SCRUM works with XP. I always thought SCRUM was too much on the management side, and now I think I know why.

Too bad most of the guys on agilechina cannot see it. Copyright issue I suppose.

Code Snippet, with Color!

2006-11-09T01:51:00.000-08:00

I did it!

Now build master can grab code snippet from another file and apply syntax highlighting using syntax.

Next step is to figure out a way to specify the snippet validations.

I am dead tired right now and I have a 9am meeting tomorrow. So I just updated two (actually one and half) pages to show it off.

I also fixed the document source and history link at the end of each page so you should be able to see the source file now.

Last Day as ThoughtWorker

2006-10-17T22:12:00.000-07:00

Last day as ThoughtWorker, writing this blog that I have been putting off until the last minute, I find myself lost in thoughts in all the things that I have experienced since I joined ThoughtWorks 30 months ago.

It has been a nothing short of life-changing experience. As a matter of fact, it is with these changes that I have decided that in order for me to push even further, I will have to go out and look for good opportunities that can help me, rather than waiting for them to happen. As I going through the projects, my expectation on what I can do on the project has changed dramatically.

When I first joined, I was just happy if I can merrily write my tests, be happy about the 40-hour week and that the CEO is not trying to cash in one million dollars every quarter when the company stock is not going anywhere (like some company that I have worked for). Now on my last day, I have found myself comfortable coaching agile software development and influencing project management. I have experienced all the things a senior developer can do in a project, and even have had a short taste of being an IM and PM. I have been to China, twice, to lead projects. I have been on more and more discussion and presentations that involve more than "TDD" or "patterns". I am even pulling an effort with several others right now to write a book that we hope can be published.

Like my fellow Borlanders showing me the way to be a good developer, my fellow ThoughtWorkers have helped me understand what it takes to have a success project, besides technical execellence. I created this blog for my experience in ThoughtWorks. I have grown fond of it over the time and I am going to keep it up for my experience in applying agile practices.

So to wrap it up:

First several books that I read as a ThoughtWorker:

Enterprise Application Architecture
Domain Driven Design
pragmatic Project Automation

Most recent books that I read as a ThoughtWorker:

Tipping Point
Blink
Build to Last
Blue Ocean Strategy
Crystal Clear

Things I enjoyed doing in ThoughtWorks:

Talk to Roy (don't worry about subjects)
Meet old timers, they will show you why the idea that "employee managed company" has a chance in ThoughtWorks. (If someone was surprised that you were not one, it is a compliment)
Meet at least one ThoughtWorker from each office from other country, especially the noisy ones.
Request traveling to other country for a project.

Things that would be fun to do in ThoughtWorks:

Tell Obie that ruby won't fly because it does not have an IDE like IntelliJ and it is not type safe. (well, maybe not anymore since he has been fighting that battle for a while and getting very good at it)
Tell Hammant that you love Spring
Buy beer around the table and ask for airplane takeoff and landing stories

Things that I did not enjoy doing:

On beach for more than two weeks
Travel weekly with connection flights each way.
Stuck in an airport for a delayed flight for 7 hours
Stuck in an airplane on the ground for 3 hours
Saying goodbyes
On your last day, be notified that you won't be covered by medical insurance anymore (So much for chilling out before the next job), even though you have accumulated over four weeks of vacation, and you could technically take your vacation before you send in the two weeks' notice, getting covered and accumulate additional vacation at the same time. The reason? You are not worth risking the liability anymore, now that you are not going to make money for the company and all. (Not in that word but I get the message)

I would have put boring period of the project in there but the following are what I have done each time I got bored:

Learned Dvorak
Created open source projects: http://www.shaneduan.com/opensource.html
Read books

Things You CANNOT Get Certified For

2006-09-29T11:12:00.000-07:00

So...

I was chatting through IM with a friend of mine about something that I read recently (Crystal Clear). And the conversation, as hard as I tried, just went downhill from there. I got really agitated in the end, even after I realizing that half the time it is one of those flag words that got on my nerve.

Looking back at the log, I realized that I thought he knew what I have been doing at ThoughtWorks for the last two and half years, and he probably thought I had no idea about the topic in this book. During the whole conversation, we were on different depth level of the topic. I guess this is one more case proving that you should watch out for "Barriers for Effective Listening" and ask "Why do you ask".

With all that behind, I was still stunned at the fact that no matter how well so many people try to protect a good idea, there are still people out there trying to profit from it by coming up with bogus stuff in its name. And one very good example is "certification". Because apparently they succeeded in making my friend think that is what it is all about, another certification.

So here is a list of things that you cannot get certified for. Instead, only your peers who work with you day in and day out can rate you, subjectively.

Doing TDD even when you are under the pressure to delivery. Everyone can pass a test and do a little practice in their own pleasure.
Have the courage to speak up when there are things that you think is wrong. "Do you think you should speak up when you see something wrong?" "Yes." "Good! You are certified!"
Sit-Together. How can you certify that, just by sitting together for a week? It is one thing to say "yeah, it is a good idea", it is another thing to go to the extreme length of getting the tools yourself and start taking cubicles apart. (Yes I have met someone who really did it).
Code Co-ownership. I am sure everyone can check that checkbox to get certified. But how can you certify a person's willingness to learn as much as the codebase, make sure that the design is as clear as it can be, and pass on the knowledge to others as soon as possible?

Sorry but "certification" is such a binary rating system, that it just does not fit into what it is in the real world.

Then of course, it is always good to get some education on the topic and have a proof that you have finished them successfully.

But that hardly gives anyone the authorization to say that "It is nice, but it is really really hard to do a business-driven situation".

Not when the purpose of the whole thing is "Deliver the business value in whatever the best possible way".

(BTW, did you realize that if you mistype blogspot.com as blogpsot.com, it is an actual website selling ads?)

BuildMaster Surgery - Moving to Cotta

2006-09-23T15:08:00.000-07:00

I have been struggling with the File operations for a while, before I fully understand what is going on.

After I released Cotta project, the way File operations works in Ruby is like a hideous stain on top of a shining gem, always bugged me, especially when there are these nice classes like Pathname and StringIO already exist in Ruby.

Finally, my curiosity of making it better got the best of me, and I ported BuildMaster over to Cotta Style.

I have decided to call the files CottaFile and CottaDirectory so that I don't have to worried about the name space issue. Since BuildMaster does quite a bit of shell operations, I added method "shell" to the cotta instance so that the in memory version can log them for tests.

All things are happening behind the scene. The following is a code snippet demonstrate the new API:


   #system implementation is injected here
   cotta = BuildMaster::Cotta.new(BuildMaster::InMemorySystem.new)
   file2 = cotta.file('dir2/file.txt')
   file2.exists?.should_equal false
   # parent directories are created automatically
   @file.save('my content')
   @file.copy_to(file2)
   file2.exists?.should_equal true
   file2.load.should_equal 'my content'
   file2.read {|file| puts file.gets}

The API still need to be fine tuned but they provide all the operations that BuildMaster needs for the moment. Hopefully, someday something like this can be part of the ruby library.

BuildMaster Surgery - Moving to rSpec

2006-09-23T13:48:00.000-07:00

This is the first part of the BuildMaster surgery that I have just finished (well, 95%) that have been keeping me up late every day. I will write the second part as a separate blog.

Moving BuildMaster to rSpec turned out to be not as smoothly as I hoped. Hopefully this blog will help anyone who is interested on similar task. This is what I have figured out so far so if there is a better way of doing it do let me know.

Syntax Change

If you have tons of test/unit scripts already, changing the code to use the rSpec syntax could be boring. They do have "Test::Unit Migration" support. However, I didn't get to take full advantage. It just generated "migration error" with no detail information for me. So I had to use some regular expressions to get the most of the change done, which I have blogged in "Going rSpec".

Specification Sharing

The "Context API" document on rSpec teaches you how to call helper methods in your specification code. However, it does not show you how you can make two context share the specifications, which is very useful if you want to make sure that two different implementation of the same interface shares the exact same behaviors.

Brain from Pivotal showed me that you can achieve this by using 'extend' (instead of include). So you can write the common behavior specifications in one file:

module CottaSpecifications

def register_cotta_file_specifications
  setup do
  # this one uses the @system that will be set up by each implementation spec setup
  @file = BuildMaster::CottaFile.new(@system, Pathname.new('dir/file.txt'))
end

specify 'file can be created with system and pathname' do
  @file.name.should_equal 'file.txt'
end
end

Then you can include those common behavior in your specification file for the implementation:

require 'cotta_file_specifications'
module BuildMaster
context 'Cotta file' do
  # extending the module so that you can call the methods
  extend CottaSpecifications
  setup do
  # setup the implementation for specifications
  @system = InMemorySystem.new
end

# this call registers all the spefications
register_cotta_file_specifications
end
end

Hopefully the above code makes sense to you. If not, let me know.

Running All Specifications

Again, thanks Brain to explain the reason and provided me with a solution.

Somehow, all the registered contexts are run at the end of each context registration. So my code for loading all the tests will now make the same context run again and again:

If you have context A and B in tc_a.rb and tc_b.rb and you require each file one by one. After "require 'tc_a'", the specifications in Context A will run. After "require 'tc_b'", the specifications in Context B will run, this time ALONG with all the specification in Context A. If you have C, D, etc., then specifications of Context A will run again and again at the end of each following require.

So Brain wrote his own specification runner, which will hold off the execution until all the contexts are loaded:

require 'rubygems'
require 'spec'
#require 'diff/lcs'
dir = File.dirname(__FILE__)
#require "#{dir}/../test/common_test_case"
require 'test/unit'
Test::Unit.run = true

args = ARGV.dup
unless args.include?("-f") || args.include?("--format")
  args << "--format"
  args << "specdoc"
end
#args << "--diff"
args << $0
$context_runner  = ::Spec::Runner::OptionParser.create_context_runner(args, false, STDERR, STDOUT)

def run_context_runner_if_necessary(system_exit, has_run)
  return if system_exit && !(system_exit.respond_to?(:success?) && system_exit.success?)
  return if has_run
  exit context_runner.run(true)
end

at_exit do
  has_run = !context_runner.instance_eval {@reporter}.instance_eval {@start_time}.nil?
  run_context_runner_if_necessary($!, has_run)
end

To use this, all I needed to do was to require this file at the beginning of my "ts_buildmaster.rb" file.

And it generates nice output too:

...
Directory object with cotta for in memory system
...
- dir should return sub directory
- dir should return a directory from a relative pathname
- should get file in current directory
- should create dir and its parent
- should delete dir and its children
- should do nothing if dir already exists
- should list dirs

Directory object with cotta for physical system
...
- dir should return sub directory
- dir should return a directory from a relative pathname
- should get file in current directory
- should create dir and its parent
- should delete dir and its children
- should do nothing if dir already exists
- should list dirs
...

Tricky Business of Being a ThoughtWorks Consultant

2006-09-19T22:56:00.000-07:00

I have been thinking about this for quite a while and the topic came up again recently. So I figure I might as well write it down. Keep in mind that these are my personal opinion as ONE ThoughtWorker based on my situations.

Blog or not to blog

Historically, my blog thoughts come through several ways, all of which is under the condition that requires me to blog at airport or late at night (like now).

1. Feeling Pain

As the book "Project Retrospective" has pointed out, when you feel the pain of doing something, it is very important to stop for long enough period of time to reflect about what you are doing. As true as it is, which makes me a true believer of retrospective, I still have the same amount, if not more than usual, of billable work that I need to do. When I do come up with an idea of improving the process, I only create more work for me to talk with others to sell the idea, one more thing on my coaching list, and a task to implement the fix.

So at the same time that I have learned a new aspect on software development, I am also burned out more from my day's work, which makes it a lot less appealing to write a blog about it. Sometimes an idea has to go through several round to mature, so I hold off the blog. Then when it is mature, I have got used to it and don't feel the excitement and urge writing it down anymore.

Yes probably it would be ideal if I can slow down my other work so that I can keep my energy up everyday, which brings back to the title "consultant", as in, like it or not, I am being paid to live up to my previous promises as much as possible.

2. Learning something new

When I got to know about something new, I always find a way to apply it so that I can learn more. What that means is my afterhour will pretty much be occupied by those activities rather than blog about my incomplete thoughts about it. To name a few:

Behaviour-driven development: Liz from UK office did a presentation on jBehave in the China office. After that, I start visiting Behaviour-driven website to learn more about it. When we created Cotta project, we decided to give it a try and it turned out to be very well. But the IDE integration sucked very much and jBehave is a bit rough around the edges (like loading all behaviors), so we joined jBehave project, created Eclipse and IntelliJ plugin and fine-tuned the API, during which process the site building part of BuildMaster became mature.

Can you imagine me keep a log of all these activities while I am working on them in my afterhour? I didn't think so. Although now I realize that I have different take on BDD from Dan North and Dave Astels, maybe I should write something about it, well, after I finish this thing about build and release that I learned from last project, and my thoughts on how BuildMaster can rock the world.

3. Getting Into Interesting Internal Discussion

From time to time I met people, ThoughtWorkers or not, who are not afraid of expressing their opinion and discuss about it. A lot of the time the outcome of the discussion is that "We now know what we don't know" (in a good way, kind of like your feeling after finish reading "the World is Flat" from what I have been told). So is there a point of blogging that? Maybe, maybe not. How to blog it, what is a good balance to strike, I think this is the time when I feel that technical writers are being paid for a reason.

Discuss or Not to Discuss

A good project is a project that is under "Total Quality Control", i.e., every single detail of the process is constantly under review to see if there are any room for improvement. Since ThoughtWorks has got itself out of the staffing business, ThoughtWorkers are generally taking the lead on the projects.

Whenever a new ThoughtWorker joins the project, it is more than welcome, as part of ThoughtWorks culture, for him or her to question every detail of how this project is run. Sometimes debates will rise during the day so that everyone can understand not only the way it is but also the history behind it.

On the other hand, ThoughtWorks is brought in as "consulting firm", being paid to say what the right way is to run a process. So when the new member joins and debates start occurring, it can be potentially perceived by the client as the incompetence of the existing project lead, incompetence of the new member, mismatch of the personality between the parties, or weird management style of ThoughtWorks.

This happened to me several times, and luckily, my clients have been open enough to raise the question thus giving me a chance to explain. It happens a lot less now because the first impression that I am trying to make is that "there is generally no silver bullet". At the same time I am giving out suggestions, sometimes strong ones, I also tell the reason behind it and that those reasons are revalidated constantly.

Then that makes me look like the kind of consultants that I have had contact in the past, with who are all talks and never tells the client anything concrete, the exact kind that I loath and want to stay way from.

Hence the "Tricky Business".

There are a few more thoughts on this but they are not as well thought. Plus it is 12:30am now and I did plan on have a full night sleep for a change, well, after I finish ironing my shirt for tomorrow.

Going rSpec

2006-09-18T23:37:00.000-07:00

After played with rSpec for a while, I have decided to convert BuildMaster on top of it. However, the test2spec script did not work for the project, so after a few experiment, the following regular expression helped me converting the majority of the syntax. I still had to convert the class definition to context definition manually, though.

1. Converting assert_equal(a, b) to b.should_equal a

replace "assert_equal$([^,]+),\s*([^\n]+)$" with "$2.should_equal $1"

2. Converting "test_..." methods to "specify ..."

replace "def test_([^\s]+)" to "specify '$1' do"

Speed of Unit Test

2006-09-05T22:26:00.000-07:00

I have heard more than once from people saying that their unit tests are long because they have thousands, or tens of thousands of them.

Some would blame it on the complicated application architecture, or the language (like Java).

But if these guys can use a script language to run 132 tests, 219 assertions in just over half a second, maybe there is something to the idea of faster tests after all.

I am not saying it is easy, just that it is a goal worth pushing for.

One Week in SLC

2006-09-01T14:25:00.000-07:00

The past four days just flew by and I am now sitting at the airport waiting for my flight back. I was hoping to fly my wife over so that we can visit the mountains around here during the weekend but it turned out to be much more expensive. That is OK, I still need to scout out the beach park where I am going to host the BBQ party next weekend.

I held on to a story card and paired with different developer each day. The domain that this story affects turned out to be the worst part of the application so each day I did a bit of clean-up and refactoring. The story dragged on a bit but I managed to make steady progress.

Since this is a .NET project, there are SQL server and IIS server to run on the laptop. After figuring out how to start and stop the service from the command, I wrote two Ruby driver class for them in the hotel room and checked them into BuildMaster.

So now I have a script that I can call to stop both service with:

services.rb stop

and all it does is:

services = [
BuildMaster::IisDriver.new, BuildMaster::SqlServerDriver.new
].each do |service|
service.send ARGV[0]
end

Back on the Road

2006-08-29T06:23:00.000-07:00

After almost three months of local project, which I intend to log sometime soon, I am back on the road again.

This time I have my Verizon national access card that allows me get connected to do things like writing this blog at the Oakland airport. I could request the extra battery but with only 3-hour travel time I doubt the weight is worth the benefit.

Next project is an ASP.NET application and have quite a few ThoughtWorkers on it. I am afraid that is as much as I am allowed to say. I would like to whine about the painful download/install process I have been through in order to run the application, but I don't think anyone, like .NET or not, would care.

This project is on a 4-10 schedule which means once I get myself comfortable with the code and my role, I will be able to spend only 3 nights in the hotel and sleep at home 4 nights per week. Not that I am tired of traveling but there is something to "sleeping in your own bed".

A Good Read: Eight Barriers to Effective Listening

2006-08-28T10:08:00.000-07:00

http://www.sklatch.net/thoughtlets/listen.html

Richard Watt hosted an OpenSpace session in the last HOD. I was thinking about checking it out at the time. But other things came up and I forgot about it.

I cam in the office today preapring for the next project and saw the printout at the printer. Figure I better grab a copy and blog it before that happens again. (Also need to buy "Getting Things Done").

Tying Up Loose End - Cotta

2006-08-26T00:13:00.000-07:00

So I have rolled off another project and I am going on the next project right away next week.

Typing up a few the loose end at the moment, since I won't be able to have a couple of weeks beach time like I did last time.

I have started with Cotta because this is most close to being ready. Based on the feedback and my experience with Ruby file handling, I have finalized the core API. Release candidate is now available. Even though I still prefer setting all the API right on the first release, my busy schedule just won't allow it. I think I'll release 1.0 and leave it like that, until I get a chance using it or get some feedback from the field.

Of course, SourceForge is done, so in the meanwhile, only the website is available.

More blog coming up with last project, and other loose ends.

Why Lotus WebAccess is not My Favorite Webmail Client

2006-08-22T18:12:00.000-07:00

I am on a machine that has 1.99GHZ dual CPU, 2GB of RAM, and 75GB of harddrive. Yet for every scroll up and down action in my inbox view, it sends a separate request. So even when I know the exact date an email was sent, it still takes forever to locate it.

And I noticed this is on the log out screen today:

* Secure: Delete all traces of my personal use of Domino Web Access and Web pages from this computer but keep program elements (boosts performance when next person logs on).

* More secure: Delete all traces of Domino Web Access and all other Web pages in the Temporary Internet Files folder.

Right... I understand what they are trying to do. But what's up with the "Secure" and "More Secure" notion? Apparently, the "Secure" is not so "Secure" after all when there is a "More Secure", isn't it?

Benevolent Dictator

2006-08-11T19:45:00.000-07:00

This came up during a chat with someone while I was visiting the office.

This is a term that has its own history (http://en.wikipedia.org/wiki/Benevolent_dictator), but is now applied to a kind of management style for software development. In opensource development, Codehaus is probably the most famous for this management style (project despot). You should read "Codehaus Manifesto" if you want to understand it fully.

In Agile software development, the developers have (or should have) the absolute power on deciding how and when a business value can be delivered. While this brings lots of benefits, which we won't get into here, it does bring certain risks at the same time.

Different developers might have different opinion on design of certain part of the system, or no one have strong opinion over several sound solutions for the same problem. This is when benevolent dictator steps in, makes a decision in the team's place, and move on.

Of course, this is not nearly as easy as it sound. I did a little dig (very little, mind you) on my previous projects and pulled together the following criteria:

* The power should be used when necessary but ONLY when necessary.
* Any decision is a team's decision. Everyone takes the credit, everyone accepts the responsibility.
* Decision is not final in the sense that objections are noted and tracked, so that the wrong ones can be amended as early as possible.
* The title of "benevolent Dictator" is better granted.

At last, here is a PDF file that I have on my bookmark, which I think is related: "Agile Project Management.pdf"

Ask "Why Do You Ask"

2006-08-09T23:34:00.000-07:00

As a traveling consultant working for a company pushing for agile, I keep on being asked for similar question repeatedly. "What do you do in stand-up?" "Why is TDD good enough to spend time on?" "What do you mean the velocity is 20 this week, 25 last week, and 18 the week before???"

Naturally, I trid to come up with the best answer based on the experience I had had, the books and articles I had read, the other conversation that I had had with other people, and the understanding that I had gained about the one who asked the question. Slowly, my answer to the above questions became bigger and bigger, with more details, more angles, as if I was getting closer and closer to the truth.

WRONG...

What I have realized in the past year, was that the best way to answer these questions, actually, are:

Give a 30-second brief answer. Stop, and ask: "Why Do You Ask?"

Different people have different situation, and are seeking the solution to different problem when asking the same question. In this case, giving a by-the-book answer, no matter how pragmatic or complete it is, does not necessarily solve that problem.

Here is a true case.

When I was traveling in China, I stayed late in the office one day to do some market research and spike on BuildMaster project. Another ThoughtWorker, who was doing some of his own thing, turned to me and asked "So what have you been working on lately?"

Since he expressed interest in DbFixture before, and that I knew he was on beach at the moment, I thought he was looking for something interesting to work on. So immediately I went on talking about the problem that I was trying to solve and the "grand" vision that I had with BuildMaster, Cotta and jBehave. I went on for 5 minutes (if not more) and realized none of them gained any interest. Going out on a limb, and out of a bit frustration, I finally asked the million-dollar question.

Turned out that he had an idea of starting an open-source project to help WebServices testing and would like to get me involved because of all the xFixture projects that I had been working on.

Granted, it was not exactly a straightforward question. Yet more and more I start paying attention to the motivations behind the questions rather than the questions themselves, and even sometimes observing the conversations others have.

And the rest is history.

I can now control myself much better nowadays instead of "blah blah..." for minutes.

Otherwise, I might as well say "42"

Why Eclipse is not My Favorite IDE

2006-08-07T22:22:00.000-07:00

Need to find a way to exclude certain folders in my Eclipse Ruby project so that it does not show up in the Open Resource dialog. Searched on line and find this in Eclipse bug database:

There is now technology in place to support exclusion filters in the workspace.
You can create an exclusion filter in I20060110 by creating a linked resource
to the "null" file system.  The linked resource will hide any file system
resource with the same name, thus acting as an exclusion mechanism.

Maybe it is now a good time to reconsider the investment in Komodo

---

Update: Thanks for Robert for pointing this one out and I stopped pounding my head (a lot less at least):

apparently you can right click on the directory, select "Properties", and check the "Derived" check box, which will do the trick.

Loading All Test Cases in Ruby

2006-08-07T08:56:00.000-07:00

Now that I finally understand that '$:' in '$:.unshift' is a variable reference, and that 'require ...' is literally a method call loading the file, I realized why there are no utilities in ruby to load all the test cases for you:

create a file 'ts_all.rb' at the root directory of the test with the following content:

# test/unit module that will pick up all loaded testcass automatically
require 'test/unit'

# find where the root directory of the tests is
root = File.dirname(__FILE__)

# do a name matching and iterate through all tc_*.rb files
Dir.glob("#{root}/**/tc_*.rb") do |file|
# load the file (assuming naming convention)
require "#{file}"
end

That is it! Executing 'ruby ts_all.rb' will just run all the tests.

BuildMaster 0.7.0 Released

2006-07-23T02:40:00.000-07:00

Since it has already helped me a lot on setting up my personal website (http://www.shaneduan.com), I have decided to burn some midnight oil to release 0.7.0.

Now the whole BuildMaster site can be downloaded from here http://rubyforge.org/frs/download.php/11919/buildmaster-0.7.0-docs.zip. Source is also included so that it can serve as an example itself.

Still need to work on a better way to convert absolute path reference (for link, css and images references). And need to support a site descriptor like Maven or XSite so that the XHTML writing for the skin won't be so intimidating.

Geek Joke

2006-07-06T08:30:00.000-07:00

Heard this one yesterday from Miles:

"C++ should actually be called ++C..."

Cotta Power

2006-07-02T20:37:00.000-07:00

So I got one feedback already for Cotta project, neat!

So the million-dollar question is: "How is Cotta different from other files system project like Apache commos VFS, other than support a lot less file system types?"

My first response is, "Ouch, that hurt!"

Second response would be, "Hmm... I think we should think through how we structure our website so that people would not try to compare it with the wrong kind of library"

From the beginning, the main reason behind Cotta project is to make file operations easy to write and test (or verify the behaviours, in jBehave term). So the main focus, which is also the power, of Cotta, is as below in two parts. We are still bouncing our thoughts around it, and the final documentation should be published on cotta project website.

Domain Driven

In my humble opinion, the current File class in Java, or the FileObject in Apache common VFS, or the Url in the VFS system of Borland JBuilder OpenAPI (BTW, it is nicely designed as well), all has one fundamental problem. For this or that reason, they still don't match fully to the domain, which is, in this case, NORMAL file system operation.

The number one problem I have is: "A file is NOT a directory". Namely, you want to list the children of a DIRECTORY, and the parent of a FILE should be a DIRECTORY. You should be able read/write only the content of a FILE, not a DIRECTORY. When your code has a path string, it should already know that it is a DIRECTORY or a FILE.

Also, when I delete a DIRECTORY, it should be GONE, instead of telling me that it is not empty. If I want a safety measure, my filesystem implementation should do that for me. My list can go on for a while if you let me...

Cotta was implemented in that way. As it was written, we have already got a project that does file operations, so we laid out the expected behaviours of the classes one by one, and implemented the code one by one. When it comes down to physical file system operations, there has been no guesses. The resulting API is compact, which is exactly what we intended.

Test Driven

Or Behaviour-Driven to be exact. When I write a class that operates on filesystem, I'd like to be able to verify the behaviour of that class. Even being a seasoned BDD developer, I still like to spend less time writing more functionality. And cutting the time spend on setting up and tearing down environment is always a good choice, hence the birth of XmlFixture (part of XPE, but you'll have to download the package to see the document), DbFixture, and now Cotta.

When it comes to behaviour verification, a common pattern is to go through "Dependency Injection". With Cotta, the domain class just need to work on a "FileSystem" instance being passed in, or one of the Cotta objects (TFileFactory, TFile, TDirectory) that can be created from the FileSystem. During the verification phase, the filesystem is replaced with a "InMemoryFileSystem".

Cotta also has FileSystem decorators that can be used to verify the behaviour on file system like read only (ControlledFileSystem), or file system with disk problems (CatastrophicFileSystem).

What Next?

The idea of making file system implementation for JAR file, FTP etc., was an after thought. I forgot to check with Apache common VFS again to see if I am re-inventing the wheels (as it turns out, I am, more or less). I think the next thing for us to do is to focus more on the two aspects that I have laid out above, and make it clear on the website about the real market value of this project.

Why dive into the Red Ocean when you have already discovered a Blue one, right?

Cotta Project in Business

2006-07-01T00:58:00.000-07:00

Finally we are in bunnies, well, sort of.

A project to ease the pain of using file/directory related operations. See detail at the front page that is still under construction: http://cotta.sourceforge.net

A Cool Story

2006-06-26T23:21:00.000-07:00

I thought something like this might happen when I started BuildMaster, and I sure enjoyed seeing it unfold. Thank Nat for following up on each one of us, and thank Joe for putting it all together.

You can read it here: http://buildmaster.rubyforge.org/history

BuildMaster 0.6.0 Released

2006-06-20T23:00:00.000-07:00

Finally! I get to write this blog about this little project that I have been working on. It is amazing how much you can get done when you are commuting by BART instead of driving.

It is a Ruby project that helps on the aspect of project releasing management, and website management.

The URL is http://buildmaster.rubyforge.org

Architectural Central Approach?

2006-06-02T12:16:00.000-07:00

Another post that I didn't post.

"... Picking story using an architecture central approach ..."

My head is hurting...

Power of Whiteboards

2006-05-23T11:47:00.000-07:00

Just finished a kick-off of a project in Beijing, before I came back to my base office, San Francisco, for the next assignment.

The project is a short one, with one week as QuickStart + Iteration Zero, four weeks of development, and one week of bug fixing + final release.

Needless to say, the time was short and there were lots of things to do in four days, gathering the requirements, breaking them down to stories, and coming up with a release plan.

As it turned out, the whiteboards became very useful. We found two of them available, and pulled up the blinds to make the glass wall a third one.

One board at the end of the room, half filled with goal of the week, and the other half as the parking area. With the goals listed, we were able to keep ourselves focused. And we were also free to write topics or thoughts down on the parking area as we discuss, so that we don't get side tracked. From time to time, we check these items to cross out the things that have been taken care of.

The other whiteboard stands right by the door, with a little space in front of it, for requirements discussion. We use it to write down the role and goals, and draw process diagram on it again and again until we get it right. Then we divide it into stories with numbers mapped to different part of the diagram and took digital photos, so that we can easily see how the stories fit into the whole pictures.

The glass window turned into a over-all design board, with cards indicating internal components, external components, screens, and running list for the notes on each of the above. For example, there are WebServices that we need to call as part of the application. So we had a card with the WebServices name on it. During the requirement discussion, whenever we realized that we need to use the WebServices to retrieve or update some information, we will pull the card over, put it as part of the process diagram on the discussion board, and start writing down the kind of interface we need from the WebServices. In the end, we can just take a look at this card and come up with a list of dependencies that this application has on the WebServices.

At the end of the week, looking at the three boards full of arrows, blocks, notes, I just cannot imagine how we can do without them.

A Sample of Practices Interdependencies

2006-04-23T05:27:00.000-07:00

More than once, I have got the kind of inquiry as "I'd like to adopt agile(XP) practices, which ones do you think that I should try first. It is always a hard question.

One main reason is that I very much intend to say "every team member should have the XP book by hand, the principle up on the wall, be open-minded and try to adopt the whole practices as a whole". It might sound intimidating but here is the reason:

All These Practices are Related
All these practices are supporting each other. I know it has been spelled out since day one that XP came out, but I still think it is a point that gets forgotten sometimes, especially when the team is under pressure.

In one of the training engagement that I was involved in, after some training sessions of several practices that we think the client needed, in an effort of making sure that the student understand that these practices are much better working together, we created a little dependency diagram using Graphviz.

We did it by creating the graphics file as following (detail omitted):

digraph g {
"Collaboration" [label="Collaboration"];
"Collocation" -> "Time-Boxed Iterations and Releases";
"Collocation" -> "Continuous Improvement";
...
"Test First" -> "Collocation";
"Test First" -> "Collaboration";
"Test First" -> "Automated Testing";
...
"Time-Boxed Iterations and Releases" ->"Collocation";
"Time-Boxed Iterations and Releases" ->"Collaboration";
...
}

The result was pretty staightforward:

Of course, in the presentation that we gave it looks a lot nicer with bi-directional arrows and stuff. But the idea is the same, you cannot pull one out without affecting the whole picture.

On the other hand, if you want to adopt the practices, trying them one by one will probably only make things slower and harder.

A Geek's Version of a Joke

2006-04-18T05:28:00.000-07:00

If you have Ruby and Watir installed, here is my "Geek's Version of 'How far do you want to go for a deep joke'". Enjoy


require 'watir'

ie = Watir::IE.start("http://www.sbpoet.com/2006/01/how_far_would_y.html")
while link = ie.link(:text, /(blond)|(joke)|(this one is great)|(>CLICK<)/)
  link.focus
  link.click
end

Personally I don't find it THAT great, but like Steve Boswell always says, "I digress".

Less Leads to More?

2006-04-08T20:08:00.000-07:00

Just found out that I never published this...

Sometimes, less does leads to more.

After I came to Xi'an, I started working on a distributed project. The development was splitted between Xi'an China and Bangalore India office.

Based on the observations, I decided to cut the development team in China down as well so that we can be more focused during the communication between India and China teams. So rather than 7-8 developers, we only have 4 developers, including me and Jake to work on the project. It worked out pretty well. We have got to know with each other and built a relationship that is good enough that we are free to use phone/IM/skype/iChat/campfire whenever any one of us needed something.

During the weekend, I had a chance to chat with another ThoughtWorker who is working as PM on another project and he gave me another example (I don't remember the example now...)

Building China Office

2006-04-08T18:53:00.000-07:00

Flew in China on Feb. 22nd, I got myself busy right away. The experience is still very exciting and sometimes the tension is killing me.

So these are what we have been doing in China office:

Building the China Website

We were finally able to translate the ThoughtWorks website into Chinese and started hosting it on our China domain (http://www.thoughtworks.com.cn/index.html). Arguing about how to translate a sentence or even just a term has been very much fun and educational at the same time.

On the principle of "Fixing the Windows", we improved the staging environment by reducing the downtime from 30 minutes every hour to 5-6 minutes every hour. Tests were also written for two bugs that we have found during the translation test.

Building the Awareness

ThoughtWorkers in China have been pretty active in getting in touch with the local communities. We have made arrangement with several universities in Xi'an to give the student some exposure of agile software development.

Other ThoughtWorkers working on projects in the other cities are also presenting to the local universities and communities.

As for me, I am going to give presentation to BEA user group meeting this month at Shanghai (http://dev2dev.bea.com.cn/usergroup/20060460.html), titled "Agile in Practice" (originally "XP in Practice").

I think this is an interesting topic to give presentation to. Whenever after someone giving the presentation about agile development, there was always a long session of questions and answers. This presentation can serve as a follow-up by targeting those questions.

With the help from other ThoughtWorkers, I have collected feedback from the projects that we have worked on in China, grouped and categoried them, so that we can show what agile is from yet another perspective. The more I work on it, the more it reminds me of the book "XP Installed".

So I am concentrate more on the photo shots and project samples. It already has over 30 slides and over 10 megabytes in size. Maybe I will have to cut it a bit to fit into the 60 minutes that I have for this presentation.

Distributed Project

Of course, the one and only activity that leads directly to revenue in ThoughtWorks is working on a billable project. Since the first day that I entered Xi'an office, I have been assigned to a distributed project.

The client's headquarter is in UK, and this project involves their offices in Spain and France. So we have two BAs working in Europe gathering requirement. The development team was originally in our Bangalore office. Due to the lack of resource, another small development team was formed in Xi'an office and some of the work has been shifted here.

As you can imagine, the communication has been identified as a risk from day one. We have taken various measures to make sure that the developers understand the code base as well as the requirement, and that everyone is update to date on the project status. This includes Yahoo! IM configuration so that everyone can tell which story each other is working on, daily standup meeting through iChat between Xi'an and Bangalore, and daily BA meeting conference, etc. Of course, Campfire made by 37 signals helped a lot on this issues, gaining more of my already established respect for them.

Just within a month, there has already been a change of direction from the customer, which proved one more time that change is constant and planning is an on-going activity. The expats in the China team left, except me, Jake who knows the code base, and Sagar who knows the business requirements.

Performance Testing Project

2006-02-09T22:15:00.000-08:00

This is the project that we engaged for the last two weeks. Even though it did not happen there are things that I have learned through out the process.

Business

The project came to us as a returned business. Basically, when party B did a contract for party A to build a system, they are required to find a third party to do a performance testing for the system so that they can prove that this system will work with the estimated workload.

This came to us as a returned business. It sounded like a routine contract that will just benefit all parties. Party B gets to finish its contract and get paid, party A will get to know better about the non-functional behavior of the system. As I have learned, the rate for performance testing contract is normally higher than other contracts. So it would have been a good short contract for me (working with a long time ThoughtWorker, Matthew Short) before I head back to China. And ThoughtWorks doesn't have to put me on beach.

For reasons that I cannot publicly post, the contract fell through even though technically we have a pretty good idea of who we need to work with, what we need to produce and what to do to get the data.

So now everyone lose.

This is my first fall-through project. Not totally surprised since I cannot imagine that all engagements that we have turn into a project, but still I am still a bit uneasy to the fact that despite all efforts, we just cannot influence the other parties to prevent them from taking actions that jeopardizes the project.

Or can we, I wonder?

Technically

I have done some performance testing for various projects, but always as part of the project iterative tuning process. It mainly focused on the profiling part rather than the performance benchmarking as a whole.

In the last project in China, the client who hired us had a contract with the customer, which put the specific performance requirement as part of it. It looked pretty vague to be, when reading through that part of the contract, that they are just numbers that can be easily met as long as you are free to choose the hardware, or could be almost impossible if the hardware is fixed.

In preparation for project by referring to the one that we did before, I have learned about performance testing methodology, Transaction Processing Performance Council and their benchmarks (http://www.tpc.org.)

Because the application is going to be deployed on a Linux system (while the last one was on a Windows system), with a little research and the great help from Barrow Kwan, we were able to have a Linux at our disposal, run resource monitoring programs, process the data into csv files, port them over to Microsoft Excel and generate the performance chart.

I guess I didn't totally lose on this after all.

Three Announcements from ThoughtWorks

2006-02-07T10:10:00.000-08:00

Internally, ThoughtWorks has lots of interesting emails about announcements and discussions. Here are three announcements that I got this week that I think worth logging.

Selenium on OpenQA
Selenium has moved to OpenQA (http://www.openqa.org/selenium/). For those who don't know, Selenium is a web testing technology written to various degree by a couple of ThoughtWorkers and friends. It turns your browser into a web testing tool so that you can test your web application in any browser on any platforms. I have used to in some projects and I strongly suggest you check it out.

CruiseControl Better and Greater
I have been using CruiseControl on every single project that I have worked on in ThoughtWorks. And I have to say that they have come a long way. Now the latest version of CruiseControl launches web server along with the CruiseControl server, and has a "build" button on the reporting page!

Event Announcement: CI and Testing
Greetings Continuous Integration Testers,

This email is being sent to announce an upcoming event for everyone interested in continuous integration and the type of automated testing associated with it.

Jeffrey Fredrick and Paul Julius are cohosting an event that will focus on these topics. The event will use Open Spaces to structure conversation, understanding and innovation.

What: Open Space event discussing all aspects of CI and Testing, together
Where: Chicago, IL
When: Early April (final date tbd)
Who: Everyone interested in CI and Testing
Cost: Free

We'll be inviting people for all manner of projects and places. In fact, feel free to pass this invitation along to anyone that you think will be interested.

For us to finalize the details of time and place we need to get a feel for how many people are likely to attend. If you are interested in attending please join the CITCON mailing list at:

http://groups.yahoo.com/group/citcon

and post an introductory message. In your message it would be useful if you could indicate any topics of special interest and also how likely you are to attend.

Sincerely,
Jeffrey Fredrick (jtf@agitar.com)Paul Julius (pj@thoughtworks.com)

Hey Look! A Delete Button!

2006-01-23T13:42:00.000-08:00

This just proves that it sometimes takes a while for even the best to see the light.

Finally GMail has a "Delete" button! Now I just need to un-train myself to click it instead of that drop-down list.

Finally Yahoo Mail logs in using SSL by default! Now I just need to stop clicking on that link on the login page.

曲突徙薪无恩泽，焦头烂额为上客 -- An IT Phenomena

2006-01-20T10:08:00.000-08:00

This is a thought that I had during my two week vacation.

Sitting at in-law's home, I grabbed a book about Chinese idioms for my nephew to kill some time, and a story caught my attention.

The story started when a guy built a beautiful house, and invited his friends and neighbors over for a house warming party. As everyone else admiring the house and congratulating the host, one guest, however, made a comment, "The chimney is too straight up and the fire woods are too close to the stove. If you don't make the chimney into an S shape and move the logs away from the fire, chances are that the house will catch fire before long."

The host, however, was not very happy to hear these, thinking this guest is saying words of such bad luck on such a good day.

As the luck went, the house did catch fire a month later. All the neighbors came to help. Eventually the fire was put out, leaving everyone with burns and bruises.

The host was very grateful. He opened the kitchen immediately. He had everyone sitting in the best seat, and treated them with the best food and drinks.

During the meal, another guest, who also came to the house-warming party, brought up the topic, "A month ago during your house warming party, there was a guest that predicted that this would happen. Had you listen to him at that time, chances are that your house will not catch fire and we would not be sitting here enjoying the feast. Don't you find it backwards, when the one who recommended the modification on the chimney and moving away the log was not being appreciated, while the rest of us with burns and bruises are being treated as the best guests (曲突徙薪无恩泽，焦头烂额为上客)?"

The host came to realize the mistake that he has made. He immediately sent for that guest, and treated him with the best seat in the house.

As I work through projects and in different organizations, in ThoughtWorks or before ThoughtWorks, I noticed that the person who is good at putting out the fires are always the one that gets the limelight. The person could be a manager who knows how to talk to the customer to cut the scope in the last minute, or find ways to persuade the developers to work overtime, or he/she could be a developer who is good at digging through a bunch of messy code and come up with a hacking patch for a last minute bug.

On the other hand, there don't seem to have enough attention to the sources of all these crises. People tend to pay most attention to how to resolve these crises rather than how to prevent these crises. So advice was not given any thought and measures were not taken because "if it aint broke, don't fix it".

And there are more crises.

And it is more important to keep the ones who knows how to put out fires with burns and bruises.

The Job of Project Manager

2005-12-14T22:35:00.000-08:00

I always believe a real good project manager is not one who always resolves crises, the same reason that a real good developer is not one who knows how to fix his own bugs. The reason is simple: it takes more skill to prevent problem than that to resolve problems.

I think a good project manager is one that make sure everything runs smoothly. For example, all the necessary preparations are done before each meeting and all the action items come out of a meeting are being followed in time and properly. Each of the team member knows what the project vision is, knows how to do his or her own job, and is motivated to do the job.

Sounds too vague huh? I agree. And the reason for this bolg is that I finally have something concrete.

For various reason, we are now having an agile development team with a project manager who is not experienced in an agile project. In China, it is very common to have a project manager with technical background, and it is very common for the project manager to help out the development work rather than managing the project.

In order to help the new PM managing this project (rather than checking out the source code and database schema), I asked for Renee, another ThoughtWorker who were the manager of this team to write down a list of responsibilities. It turns out to be very helpful for everybody. She also directed me to this on-line document (Agile Project Management), which is a good read as well.

The following is the list:

Here is a summary of the key responsibilities of a PM on an Agile project:

VISION - Establishing a guiding vision for the project and continuously reinforcing it through works and actions.
Ensure the team and client understand the vision.
Clear up any questions about the vision.
Includes timeline, release planning, scope.
TEAMWORK & COLLABORATION - Facilitate collaboration and teamwork through relationships and community.
Establish a good relationship with the team and client.
The team and client will come to you with issues if they trust you.
SIMPLE RULES - Establish and support the teamÂs set of guiding practices.
Examples include TDD, standups, pair programming, etc..
OPEN INFORMATION - Provide open access to information
Release plan
Story list
Progress, velocity
Staffing changes
LIGHT TOUCH - Apply just enough control to foster emergent order.
Demanding control will only cause the team to lose order.
Empower team members to do their jobs
AGILE VIGILANCE - Constantly monitor and adjust.
Hold retrospectives on team progress

An Agile PM needs to be full time on the project. A part-time PM cannot remove obstacles, encourage collaboration, or provide vision. Here are some addition day to day responsibilities:

Lead stand-ups
Track daily progress
Manage release plan
Communicate with customer leads
Supporting the team and helping the customer to understand project decisions, e.g., why we want to do J2EE without EJB
Listening to the developers and team, helping to remove obstacles
Risk identification and mitigation

The Job of Architect

2005-11-10T07:13:00.000-08:00

In the ThoughtWorks project that I have been working on, there are just one lead and several developers. Even though the lead is normally called "Architect" because that is the term that we use externally, he or she is much difference from what people normally think of Architects.

Rather than drawing all kinds of fancy diagrams and leave to the developers to implement the design (More than once I have heard of this theory that the developers don't have to be highly skilled as long as they do what is told).

In my previous projects, I have been very impressed with the work of our project lead. Highly technical and very experienced, the lead pairs with the team and working out the design decisions with the team. I felt that this is the best way for each member of the team to learn and grow and this is the best way to come to the right decision for the project.

On this project, being an architect turns out to be much more than just leading the development work. The current project involves three different servers posting their interfaces through WebServices and communicate to each other through these interfaces. WebMethods is the data exchange platform so that more services can be added, and there is a CA server for certificate verifications. Because of this, there are a lot of coordination required among the development groups. In addition, the requirement is not clear so I need to help the BA on the story list creation.

So between the development groups technical discussions and requirements gathering, I have not been too active on programming. The worst day was last Tuesday, where I had meeting back to back from the morning to the evening. I have been trying to play catch-up, looking through the code after work. But now I realize that with the code changes everyday, I will end up killing myself and still not able to keep myself up-to-date. So I am now delegating the design decisions to the team and pay more attention to their development practices like pair programming, test driven development, rotation, etc.

The good thing is that with the BA on track with the requirements gathering and story list creation, I get to concentrate more and more on the development.

Two More about China Environment

2005-11-03T09:10:00.000-08:00

So here are two more things that I thought worth mentioning.

All the three hotel rooms that I have been staying at have a central control panel by the bed. So to go to sleep you can just press one button, which will turn off all the lights and sets up the sign "Don't disturb" at the door. Pretty cool eh?

Although they don't have an alarm clock.

The connection in China to US is BAAAAAAAAAAD. I did a download speed test the other day and here are the data:

ThoughtWorks gforge 0.6kb
sourceforge 1.9kb
rubyforge 2.3kb
java.net 2.7kb
codehaus 2.3kb

Doesn't that remind you the good-old days of dial-up? :)

BTW, the connection within China is pretty fast.

Things to be Prepared for in China

2005-10-30T07:47:00.000-08:00

First of all I think I need to say that I am still greatly enjoying working with great people in China and enjoying great food here.

Originally I put the title as "Things that I don't like in China". But later I thought about all these stuff I realized that they probably won't change in a short period of time. So the only solution is to watch out and try not to get bothered. (Granted, the city we are staying is not as a big city like Beijing or Shanghai).

Private Space

I know it always happens in a place where there are lots of people and limited spaces. However, from time to time, I still get surprised.

For example, now I have learned that when the elevator reaches your floor, you need to step right in from of the door and get ready to block the door so that you can push your way out easily. Otherwise, there will be people pushing their way in so quickly that you might have a hard time getting out.

Another case is that when talking to the front desk don't get offended if someone else just budge in and start asking question to the person who you are talking to at the moment. And don't get surprised if that person will switch attention to him or her right away and let you stand there waiting for the answer.

Limited Services

Traveling in United States, I have gotten used to stepping in a hotel and have iron board, steaming iron ready in the room. When I travel for over a week, I know I can always find a place to wash my clothes, most of the time inside the hotel that I am staying at. However, that is not the case here. Both hotels I stayed at in Shijiazhuang don't have iron and board. And I had to buy an iron because the second hotel has only the old style ones that burn shirts easily. Laundry has been a problem because I cannot find any place to wash them myself.

The other guys found two apartments to stay in. I was hoping to be able to do laundry at their places. However, the apartment services here is hardly any good. The washing machine in one apartment does not work, and the other one is missing a water pipe. What's more, there has been some repairing or construction going on and they have been out of hot water for over a week. Calling the owner or the property management has been in vain. The guys have been pretty cool about it by taking the shower at the shower room of the swimming pool downstairs. Still, I don't find it satisfying.

Task Management

In our working environment, there seems to exist a tendency of setting the task without setting the date. At the beginning the reply we here very often was "ok, let me discuss with them about this". In a project that has only 3 months and still working on requirements at the third iteration, this kind of got me nervous. However, all we needed to do was to ask for a date, either for the answer or a follow-up date about the date.

One Week in China

2005-10-13T06:23:00.000-07:00

This moment exactly one week ago, I checked in the Great Wall Sheraton Hotel in Beijing, and began my life as a consultant traveling oversea.

One week is not a long time, but it has already been a blast. The ThoughtWorkers in China get along very well (not that the ones in US didn't) and there is always constant conversation during the day as we cranking off the story cards. Because of this tight relationship (the other four guys actually live together in two apartments of the same complex), conversations are very open. For example, the CruiseControl server is in the same room, on a Windows machine, so the guys set it up so that every time there is a broken build, it will play a 20 second clip saying in Chinese something like

"What the !@#$% are you doing that for... Would you ever stop? I have already told you that I cannot handle it anymore, but you keep on doing it, doing it, doing it. You just don't care if I can stand it or not. If you do it one more time I probably will have to stab you..."

The original text can be found here and it is definitely much funnier to hear it in person. Because it is long and loud, it easily gets annoying, and with the good team relationship whoever broke the build definitely gets a lot peer pressure to fix the build.

There is also a bonus for me on this project. That is I get to get involved in a lot of project management stuff. With the client speaking Chinese, I am helping the manager whenever we need to talk to the client or whenever they need to talk to us about project plan and arrangement. The current crucial task is the story list. Because there is not a full-time BA, and that we are only developing one part of the system, and because that the time is limited, we are trying to finalize the story list, so that we can track the project progress with higher accuracy. So I am also going to play the role of BA for a little bit whenever there is a translation needed.

On Monday, we had a kick-off meeting. Because this is the first time that the customers were involved in iteration planning, and that this is the second time that they were presented with Agile Software Development, we decided to give them a second presentation of agile software development, focusing on the benefit, role and responsibility of the customers, and we were able to invite the deputy director. I believe the presentation was well received, because the deputy directory got excited and spoke at the end that we should receive the full support from their side. The full-time customers that have been assigned to us also turned out to be good natured and knowledgeable to the domain.

One more reason that I love this team is also because that the guys are very comfortable with me adding the Dvorak keymap to each laptop that I touched. We set Alt+Shift+8 for QWERT and Alt+Shift+9 for Dvorak and it works out pretty good. Someone even showed interest in learning Dvorak. Hehee... Maybe sometime we can finally say bye-bye to QWERT in the China office.

It is also interesting to learn the Chines meaning of the terminologies that I have been working with for the last 6 years. The following is a list that is on top of my mind:

Development: 开发
Iteration: 迭带
Document: 文档
Release: 发布
Deployment: 部署
Class: 类型
Method: 方法
Agile Development: 敏捷开发
Project Management: 项目管理
Test: 测试
Run Test: 跑测试
Constructor: 构造函数

And learning Wubi (五笔), a Chinese input method, is fun as well.

Arrived at ShiJiaZhuang

2005-10-07T08:03:00.000-07:00

After a 12-hour flight, half hour Customs Service (The 'Paul and Jamie Buckman' behind me sure made it seem like 2 hours), one night hotel, 2-hour train ride with half hour delay, I finally arrived the city of the project, ShiJiaZhuang (石家庄).

I have finally met three of the four ThoughtWorkers in China office, Li Mo, Li Chaoqun and Li Xiao. (Yes, Li is a common name in China), and I got a chance to talk with Li Mo on the train. Sid's comment of ThoughtWorkers was "young and super smart" (sounds like the early days of ThoughtWorkers?) and talking with them sure confirmed it.

What I have gathered so far has already shown that projects in China is definitely different from the states, unfortunately I cannot reveal much. As much as I want to keep this blog on my experience in ThoughtWorks, the special condition in China might force me to log lots of them on an internal server. I guess we shall see.

Walking off, and Going to China Project

2005-09-24T18:14:00.000-07:00

This is one of the times that I am proud to be a ThoughtWorker.

We have been working with the current client for over a half year (I joined three months ago). The agreement has always been that this is a transformation (formerly known as enablement) project, that, given the clients' background of waterfall approach and ThoughtWorks' experience in agile development, should rely highly on ThoughtWorks' input. That is, for a start, during the scoping phase form a small group (6) and have one on one ratio of ThoughtWorker and the employee from the client.

But apparently someone thinks that if they pay us they own us. Before the pilot project even starts (hence gets the requirement), there is already order from high up to use one of those so called "J2EE framework server". And during the initial phase, there are going to be 9 people instead, with only 3 from ThoughtWorks. All these decisions are made without even consult ThoughtWoks (you would think that should be the reason to hire a "consultant"). And no reason has been given for these decision.

So as the current finishes this month. The manager decided not to renew it and walk off from a project that is not built on a solid ground of trust and understanding, even thought the rate on this project is fair and we might have a good account. This decision got the support from the headquarter and the four of us are already on the list of available consultants for October.

As for me, I am really glad of the decision being made, because that shows that we are serious about being good at what we do, and the we care about have happy consultants on the project.

Another good reason for me to be happy is that the moment my name come up, there is already a good project for me to go -- China. Because of this and last transformation project that I worked on, and the fact that I speak Madrine, I turn out to be a good fit for this project that needs someone to come in and help out. Time is short so I am busy working on the visa and travel plans.

At this moment, my goal of China trip is the following, I am still updating it:

Get to know the ThoughtWorkers in China: I have heard of great things about the people that we hire there. I need to spend as much time as I can to get to know them and exchange our experiences to learn from each other.
Get to know the market in China: Despite the great potential in Xi'an, I still think there should be a lot of opportunities in Beijing and Shanghai. Agile software development in China is getting a lot of attention, if we don't grasp the market and build our good names, there will be someone else very soon.
Get to know the IT environment: There might be a lot of things that I am taking for granted here that is not as available. Programming in a non-English speaking environment might bring different dynamics.

Deployment Script with Watir (Take THAT, Websphere!)

2005-09-21T19:04:00.000-07:00

So, after some struggling with Websphere on deploying an EAR file (Don't ask me question about why we still bother, please... ) and not able to get a hand on a simple deployment script (I think there should be one but I just didn't have the time to look further), I have decided to automate the web process that I have been doing, using WATIR.

Task
Call me innocent, but I really thought that depolying an EAR file on an application server would be easy, given the 'comprehensive' file structure and all the deployment descriptors.

But no..., you have to click through this long 'wizard' (ooohhh...) process every single time. Not only you have to do it manually, it also takes quite a lot of time. And I happen to be one of the impatient guys who thinks that it is a machine's job, then a human should not suffer.

Watir
In case you haven't heard of it, Watir (http://wtr.rubyforge.org/) is a ruby library used for web testing. I am still not sure how exactly it works, but the result is that you can control an internet explorer instance using Ruby script. So basically almost anything you do with IE, you can write a script and automate it.

Code
So long story short, I was finally able to write a script that uploads the file to the server, click through the steps, and save the configurations. Needless to say, I am very impressed the progess that Watir has made, functionality as well as the documentations.

The script is actually not that long. Here it is (with comments):


require 'watir'
 
class IeLauncher
  def IeLauncher.open(*args)
    result = ie = Watir::IE.start(*args)
    if block_given?
      begin
        result = yield ie
      ensure
        ie.close
      end
    end
    return result
  end
end
 
# Ignore the above, they are just basic set up.
 
# Launch IE, go to the specified URL
IeLauncher.open('http://localhost:9060/ibm/console/') do |ie|
  # Log in, my local server doesn't have a password
  ie.text_field(:name, 'username').set('admin')
  ie.button(:value, 'Log in').click
 
  # Only one login is allowed so you will have to log out last one
  if ie.contains_text('Login Conflict')
    ie.radio(:name, 'action', 'force').set
    ie.button(:value, 'OK').click
  end
 
  # Apprantly when you deploy an application, you need to 'save' it.
  # This recovers it
  if ie.contains_text('Recover prior changes')
    ie.radio(:name, 'action', 'recover').set
    ie.button(:value, /OK/).click
  end
 
  # This get the frame that has the navigation tree and look for the right node
  navigation = ie.frame('navigation_tree')
  navigation.link(:text, 'Applications').click
  navigation.link(:text, 'Enterprise Applications').click
  detail_frame = ie.frame('detail')
 
  # Uninstall the existing deployment
  if detail_frame.contains_text('DukesBank')
    dukesbank_checkbox = detail_frame.checkbox(:value, /DukesBank/)
    dukesbank_checkbox.click
    detail_frame.button(:value, 'Uninstall').click
    detail_frame.button(:value, 'OK').click
  end
 
  # Install
  detail_frame.button(:value, 'Install').click
  detail_frame.radio(:name, 'radioButton', 'local').click
 
  # Upload the file (Note, file path cannot have space)
  detail_frame.file_field(:name, 'localFilepath').set('c:\Work\Dukes\DukesBankEAR\build\artifacts\DukesBankEAR.ear')

  # Clicking through the wizards
  button = detail_frame.button(:value, 'Next')
  while button.exists?
    button.click
    button = detail_frame.button(:value, 'Next')
  end
  detail_frame.button(:value, 'Finish').click
  detail_frame.link(:text, 'Save to Master Configuration').click
  detail_frame.button(:value, 'Save').click
  navigation.link(:text, 'Enterprise Applications').click
end

Feedback from the Client

2005-09-09T22:52:00.000-07:00

I have meant to post this a while ago, but with the traveling my daily schedule is still not settled. So I never found a good time that I can just quiet down and write some blog.

This week is a short week due to the past long weekend. So we were able to convince the client that it is better for us to work from our home office so that we don't need to spend two days traveling for a four-day week. And once again, I am able to continue my routine of blogging.

After my last enablement project (now the term is "transforming" because "enablement" has been used along with "training". I will explain that later), the client team had a project retrospective, discussing what they have learned during the project.

With their still fresh memory, they were able to extrapolate the following, with which they gave the presentation to the rest of the organization. They are nice enough to let me publish this. I hope this can be useful for some of you who are still wondering about agile software engineering, or at least some of the "weird practices" that I have been defending from time to time. This is the first-handed note. Hopefully I can help them revise the format for more formal presentations.

Agile: One Developer's Perspective

What methodology did we use in the past?

It doesn't have a name, as far as I know, but here are some features of it
big upfront requirements-gathering, big upfront design, with some formal and some informal documentation
one developer per application, or a couple of developers working in different tiers of an application
integration & testing in a big bang at the end
manual testing

Some problems we had with this approach

Big upfront...

late-breaking requirements changes combined with inflexible deadlines led to periodic frenzies of coding that eroded the integrity of the big upfront design
sometimes we specified & designed more than we could actually implement with the time and resources we had
integration late in the development cycle led to periodic frenzies of coding that eroded the integrity of the big upfront design
disagreements about how open we should be to changes

One developer, or a couple of developers working in different tiers of an application...

we didn't always take advantage of possibilities for using some common code or common approaches across applications
difficulty scaling up for larger efforts
the culture of code ownership by individuals was sometimes at odds with our "esprit de corps"

Integration & testing in a big bang at the end; manual testing...

huge burden placed upon QA (when we had QA at all)
difficulties detecting the ramifications of maintenance changes, so we became timid about upgrading applications and we ended up with multiple versions of the same module in production
we often had as many opinions about how close we were to being finished as we had team members
we did not know how to thoroughly test componentry; we could only test built apps

For me personally, the biggest problem was that I was unhappy. I felt that our team could be great, but we just couldn't get there, and I didn't know how to be "part of the solution".

Enter Thoughtworks, with a philosophy and a toolbox

Some things I was immediately enthusiastic about...

Eclipse IDE (Great no matter what methodology you embrace)
Automated testing
Daily standup meeting
Frequent discussions as a group about what is working and what isn't, so that we can make incremental corrections in our direction
Realistic attitude towards estimating: as the project progresses, we can get more accurate in our predictions of a delivery date, but we can't expect perfect accuracy at the beginning.
steady pace, avoiding individual heroics
Simplicity--the art of maximizing the amount of work not done--is essential. (From "Principles behind the agile manifesto" http://agilemanifesto.org/principles.html)

Some things I was skeptical but open-minded about...

TDD
Test-driven development requires that a unit test be written for each tiny piece of functionality before you write any application code
The unit tests are written by the developers, not QA
Although "test" appears in its name, test-driven development is not so much about testing as it is about enforcing a development discipline that makes refactoring possible
The unit tests serve as documentation of the developer's understanding of requirements
"radical co-location"
continuous integration
fully automated build and test process that allows a team to build and test their software many times a day
everybody using the same code base, checking in and updating their code very frequently
at first, I was incredulous that checking in and building as frequently as once per day would even be possible; TWers viewed once a day as a minimum

Some things I was deeply skeptical about...

paired programming
code ownership is to be avoided
evolutionary design through refactoring

Martin Fowler's definition

What is Refactoring?
Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations. Each transformation (called a 'refactoring') does little, but a sequence of transformations can produce a significant restructuring. Since each refactoring is small, it's less likely to go wrong. The system is also kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring.

very short iterations
less emphasis on formal documentation
"embrace change"

A year later
Life on our team is completely different than before. From my perspective, it is much better.
Did the philosophy and the toolbox make a difference?
Yes, but not all by themselves. We had to have...

the right attitude
one-on-one mentoring from experienced agile developers
effective leadership
discipline

Attitude
We could not have made changes as extensive as this without

positive attitudes
team spirit
perseverance

one-on-one mentoring from experienced agile developers

from the agile manifesto: "value individuals and interactions over processes and tools"
TDD feels initially so unnatural, that it is essential to have a patient, experienced person at your elbow
without discipline, the agile philosophy can degenerate into "a license to hack", so it is important to have constant reminders about the discipline until it is second nature

effective leadership

the project manager may be an exception to the general principle that individual heroics should not be required
protected the group from distractions
keeps both the big picture and myriad details in mind
inspires confidence

discipline

do not write even the smallest method without writing a test first; see the test fail, then write the minimum amount of code to make it pass
update from the repository frequently, run the entire test suite before each check-in of your own
builds & test suites run continuously; if your check-in breaks the build, STOP and fix
to maintain design and code integrity, coding decisions are made in pairs...if the pair can agree, go forward

If the discipline is not observed, then we can't "embrace change" and "evolve the design through refactoring" because we can't be confident about extensive changes.

Notes

Below is a summary of BTM/70, a methodology in use at a job I had 20 years ago at a company. It is probably an example of what the "value individuals and interactions over processes and tools" principle is a reaction against. All that I remember about BTM/70 is that there were big binders of sample forms, and every step of the way, one had to photocopy the appropriate forms and fill them out. I do not recall any specific benefits.
The life cycle of an BTM/70 project consists of eight (8) phases ranging from Systems Requirements to System Implementation. Coupled with BTM/70 is ET/70, a Project Control tool set that is used for project planning and tracking. ET/70 assists in milestone planning, helps establish project target dates, supports activity tracking, and detail time reporting of each staff member down to the task level, within each activity, of each phase of the project