Tricky Business of Being a ThoughtWorks Consultant

I have been thinking about this for quite a while and the topic came up again recently. So I figure I might as well write it down. Keep in mind that these are my personal opinion as ONE ThoughtWorker based on my situations.

Blog or not to blog

Historically, my blog thoughts come through several ways, all of which is under the condition that requires me to blog at airport or late at night (like now).

1. Feeling Pain

As the book "Project Retrospective" has pointed out, when you feel the pain of doing something, it is very important to stop for long enough period of time to reflect about what you are doing. As true as it is, which makes me a true believer of retrospective, I still have the same amount, if not more than usual, of billable work that I need to do. When I do come up with an idea of improving the process, I only create more work for me to talk with others to sell the idea, one more thing on my coaching list, and a task to implement the fix.

So at the same time that I have learned a new aspect on software development, I am also burned out more from my day's work, which makes it a lot less appealing to write a blog about it. Sometimes an idea has to go through several round to mature, so I hold off the blog. Then when it is mature, I have got used to it and don't feel the excitement and urge writing it down anymore.

Yes probably it would be ideal if I can slow down my other work so that I can keep my energy up everyday, which brings back to the title "consultant", as in, like it or not, I am being paid to live up to my previous promises as much as possible.

2. Learning something new

When I got to know about something new, I always find a way to apply it so that I can learn more. What that means is my afterhour will pretty much be occupied by those activities rather than blog about my incomplete thoughts about it. To name a few:

Behaviour-driven development: Liz from UK office did a presentation on jBehave in the China office. After that, I start visiting Behaviour-driven website to learn more about it. When we created Cotta project, we decided to give it a try and it turned out to be very well. But the IDE integration sucked very much and jBehave is a bit rough around the edges (like loading all behaviors), so we joined jBehave project, created Eclipse and IntelliJ plugin and fine-tuned the API, during which process the site building part of BuildMaster became mature.

Can you imagine me keep a log of all these activities while I am working on them in my afterhour? I didn't think so. Although now I realize that I have different take on BDD from Dan North and Dave Astels, maybe I should write something about it, well, after I finish this thing about build and release that I learned from last project, and my thoughts on how BuildMaster can rock the world.

3. Getting Into Interesting Internal Discussion

From time to time I met people, ThoughtWorkers or not, who are not afraid of expressing their opinion and discuss about it. A lot of the time the outcome of the discussion is that "We now know what we don't know" (in a good way, kind of like your feeling after finish reading "the World is Flat" from what I have been told). So is there a point of blogging that? Maybe, maybe not. How to blog it, what is a good balance to strike, I think this is the time when I feel that technical writers are being paid for a reason.

Discuss or Not to Discuss

A good project is a project that is under "Total Quality Control", i.e., every single detail of the process is constantly under review to see if there are any room for improvement. Since ThoughtWorks has got itself out of the staffing business, ThoughtWorkers are generally taking the lead on the projects.

Whenever a new ThoughtWorker joins the project, it is more than welcome, as part of ThoughtWorks culture, for him or her to question every detail of how this project is run. Sometimes debates will rise during the day so that everyone can understand not only the way it is but also the history behind it.

On the other hand, ThoughtWorks is brought in as "consulting firm", being paid to say what the right way is to run a process. So when the new member joins and debates start occurring, it can be potentially perceived by the client as the incompetence of the existing project lead, incompetence of the new member, mismatch of the personality between the parties, or weird management style of ThoughtWorks.

This happened to me several times, and luckily, my clients have been open enough to raise the question thus giving me a chance to explain. It happens a lot less now because the first impression that I am trying to make is that "there is generally no silver bullet". At the same time I am giving out suggestions, sometimes strong ones, I also tell the reason behind it and that those reasons are revalidated constantly.

Then that makes me look like the kind of consultants that I have had contact in the past, with who are all talks and never tells the client anything concrete, the exact kind that I loath and want to stay way from.

Hence the "Tricky Business".

There are a few more thoughts on this but they are not as well thought. Plus it is 12:30am now and I did plan on have a full night sleep for a change, well, after I finish ironing my shirt for tomorrow.

Going rSpec

After played with rSpec for a while, I have decided to convert BuildMaster on top of it. However, the test2spec script did not work for the project, so after a few experiment, the following regular expression helped me converting the majority of the syntax. I still had to convert the class definition to context definition manually, though.

1. Converting assert_equal(a, b) to b.should_equal a

replace "assert_equal\(([^,]+),\s*([^\n]+)\)" with "$2.should_equal $1"

2. Converting "test_..." methods to "specify ..."

replace "def test_([^\s]+)" to "specify '$1' do"

Speed of Unit Test

I have heard more than once from people saying that their unit tests are long because they have thousands, or tens of thousands of them.

Some would blame it on the complicated application architecture, or the language (like Java).

But if these guys can use a script language to run 132 tests, 219 assertions in just over half a second, maybe there is something to the idea of faster tests after all.

I am not saying it is easy, just that it is a goal worth pushing for.

One Week in SLC

The past four days just flew by and I am now sitting at the airport waiting for my flight back. I was hoping to fly my wife over so that we can visit the mountains around here during the weekend but it turned out to be much more expensive. That is OK, I still need to scout out the beach park where I am going to host the BBQ party next weekend.

I held on to a story card and paired with different developer each day. The domain that this story affects turned out to be the worst part of the application so each day I did a bit of clean-up and refactoring. The story dragged on a bit but I managed to make steady progress.

Since this is a .NET project, there are SQL server and IIS server to run on the laptop. After figuring out how to start and stop the service from the command, I wrote two Ruby driver class for them in the hotel room and checked them into BuildMaster.

So now I have a script that I can call to stop both service with:

services.rb stop

and all it does is:

services = [
BuildMaster::IisDriver.new, BuildMaster::SqlServerDriver.new
].each do |service|
service.send ARGV[0]
end

Back on the Road

After almost three months of local project, which I intend to log sometime soon, I am back on the road again.

This time I have my Verizon national access card that allows me get connected to do things like writing this blog at the Oakland airport. I could request the extra battery but with only 3-hour travel time I doubt the weight is worth the benefit.

Next project is an ASP.NET application and have quite a few ThoughtWorkers on it. I am afraid that is as much as I am allowed to say. I would like to whine about the painful download/install process I have been through in order to run the application, but I don't think anyone, like .NET or not, would care.

This project is on a 4-10 schedule which means once I get myself comfortable with the code and my role, I will be able to spend only 3 nights in the hotel and sleep at home 4 nights per week. Not that I am tired of traveling but there is something to "sleeping in your own bed".

A Good Read: Eight Barriers to Effective Listening

http://www.sklatch.net/thoughtlets/listen.html

Richard Watt hosted an OpenSpace session in the last HOD. I was thinking about checking it out at the time. But other things came up and I forgot about it.

I cam in the office today preapring for the next project and saw the printout at the printer. Figure I better grab a copy and blog it before that happens again. (Also need to buy "Getting Things Done").

Tying Up Loose End - Cotta

So I have rolled off another project and I am going on the next project right away next week.

Typing up a few the loose end at the moment, since I won't be able to have a couple of weeks beach time like I did last time.

I have started with Cotta because this is most close to being ready. Based on the feedback and my experience with Ruby file handling, I have finalized the core API. Release candidate is now available. Even though I still prefer setting all the API right on the first release, my busy schedule just won't allow it. I think I'll release 1.0 and leave it like that, until I get a chance using it or get some feedback from the field.

Of course, SourceForge is done, so in the meanwhile, only the website is available.

More blog coming up with last project, and other loose ends.

Why Lotus WebAccess is not My Favorite Webmail Client

I am on a machine that has 1.99GHZ dual CPU, 2GB of RAM, and 75GB of harddrive. Yet for every scroll up and down action in my inbox view, it sends a separate request. So even when I know the exact date an email was sent, it still takes forever to locate it.

And I noticed this is on the log out screen today:

* Secure: Delete all traces of my personal use of Domino Web Access and Web pages from this computer but keep program elements (boosts performance when next person logs on).

* More secure: Delete all traces of Domino Web Access and all other Web pages in the Temporary Internet Files folder.

Right... I understand what they are trying to do. But what's up with the "Secure" and "More Secure" notion? Apparently, the "Secure" is not so "Secure" after all when there is a "More Secure", isn't it?

Benevolent Dictator

This came up during a chat with someone while I was visiting the office.

This is a term that has its own history (http://en.wikipedia.org/wiki/Benevolent_dictator), but is now applied to a kind of management style for software development. In opensource development, Codehaus is probably the most famous for this management style (project despot). You should read "Codehaus Manifesto" if you want to understand it fully.

In Agile software development, the developers have (or should have) the absolute power on deciding how and when a business value can be delivered. While this brings lots of benefits, which we won't get into here, it does bring certain risks at the same time.

Different developers might have different opinion on design of certain part of the system, or no one have strong opinion over several sound solutions for the same problem. This is when benevolent dictator steps in, makes a decision in the team's place, and move on.

Of course, this is not nearly as easy as it sound. I did a little dig (very little, mind you) on my previous projects and pulled together the following criteria:

* The power should be used when necessary but ONLY when necessary.
* Any decision is a team's decision. Everyone takes the credit, everyone accepts the responsibility.
* Decision is not final in the sense that objections are noted and tracked, so that the wrong ones can be amended as early as possible.
* The title of "benevolent Dictator" is better granted.

At last, here is a PDF file that I have on my bookmark, which I think is related: "Agile Project Management.pdf"

Ask "Why Do You Ask"

As a traveling consultant working for a company pushing for agile, I keep on being asked for similar question repeatedly. "What do you do in stand-up?" "Why is TDD good enough to spend time on?" "What do you mean the velocity is 20 this week, 25 last week, and 18 the week before???"

Naturally, I trid to come up with the best answer based on the experience I had had, the books and articles I had read, the other conversation that I had had with other people, and the understanding that I had gained about the one who asked the question. Slowly, my answer to the above questions became bigger and bigger, with more details, more angles, as if I was getting closer and closer to the truth.

WRONG...

What I have realized in the past year, was that the best way to answer these questions, actually, are:

Give a 30-second brief answer. Stop, and ask: "Why Do You Ask?"

Different people have different situation, and are seeking the solution to different problem when asking the same question. In this case, giving a by-the-book answer, no matter how pragmatic or complete it is, does not necessarily solve that problem.

Here is a true case.

When I was traveling in China, I stayed late in the office one day to do some market research and spike on BuildMaster project. Another ThoughtWorker, who was doing some of his own thing, turned to me and asked "So what have you been working on lately?"

Since he expressed interest in DbFixture before, and that I knew he was on beach at the moment, I thought he was looking for something interesting to work on. So immediately I went on talking about the problem that I was trying to solve and the "grand" vision that I had with BuildMaster, Cotta and jBehave. I went on for 5 minutes (if not more) and realized none of them gained any interest. Going out on a limb, and out of a bit frustration, I finally asked the million-dollar question.

Turned out that he had an idea of starting an open-source project to help WebServices testing and would like to get me involved because of all the xFixture projects that I had been working on.

Granted, it was not exactly a straightforward question. Yet more and more I start paying attention to the motivations behind the questions rather than the questions themselves, and even sometimes observing the conversations others have.

And the rest is history.

I can now control myself much better nowadays instead of "blah blah..." for minutes.

Otherwise, I might as well say "42"

Why Eclipse is not My Favorite IDE

Need to find a way to exclude certain folders in my Eclipse Ruby project so that it does not show up in the Open Resource dialog. Searched on line and find this in Eclipse bug database:

There is now technology in place to support exclusion filters in the workspace.
You can create an exclusion filter in I20060110 by creating a linked resource
to the "null" file system.  The linked resource will hide any file system
resource with the same name, thus acting as an exclusion mechanism.

Maybe it is now a good time to reconsider the investment in Komodo

---

Update: Thanks for Robert for pointing this one out and I stopped pounding my head (a lot less at least):

apparently you can right click on the directory, select "Properties", and check the "Derived" check box, which will do the trick.

Loading All Test Cases in Ruby

Now that I finally understand that '$:' in '$:.unshift' is a variable reference, and that 'require ...' is literally a method call loading the file, I realized why there are no utilities in ruby to load all the test cases for you:

create a file 'ts_all.rb' at the root directory of the test with the following content:

# test/unit module that will pick up all loaded testcass automatically
require 'test/unit'

# find where the root directory of the tests is
root = File.dirname(__FILE__)

# do a name matching and iterate through all tc_*.rb files
Dir.glob("#{root}/**/tc_*.rb") do |file|
# load the file (assuming naming convention)
require "#{file}"
end


That is it! Executing 'ruby ts_all.rb' will just run all the tests.

BuildMaster 0.7.0 Released

Since it has already helped me a lot on setting up my personal website (http://www.shaneduan.com), I have decided to burn some midnight oil to release 0.7.0.

Now the whole BuildMaster site can be downloaded from here http://rubyforge.org/frs/download.php/11919/buildmaster-0.7.0-docs.zip. Source is also included so that it can serve as an example itself.

Still need to work on a better way to convert absolute path reference (for link, css and images references). And need to support a site descriptor like Maven or XSite so that the XHTML writing for the skin won't be so intimidating.

Geek Joke

Heard this one yesterday from Miles:

"C++ should actually be called ++C..."

Cotta Power

So I got one feedback already for Cotta project, neat!

So the million-dollar question is: "How is Cotta different from other files system project like Apache commos VFS, other than support a lot less file system types?"

My first response is, "Ouch, that hurt!"

Second response would be, "Hmm... I think we should think through how we structure our website so that people would not try to compare it with the wrong kind of library"

From the beginning, the main reason behind Cotta project is to make file operations easy to write and test (or verify the behaviours, in jBehave term). So the main focus, which is also the power, of Cotta, is as below in two parts. We are still bouncing our thoughts around it, and the final documentation should be published on cotta project website.

Domain Driven

In my humble opinion, the current File class in Java, or the FileObject in Apache common VFS, or the Url in the VFS system of Borland JBuilder OpenAPI (BTW, it is nicely designed as well), all has one fundamental problem. For this or that reason, they still don't match fully to the domain, which is, in this case, NORMAL file system operation.

The number one problem I have is: "A file is NOT a directory". Namely, you want to list the children of a DIRECTORY, and the parent of a FILE should be a DIRECTORY. You should be able read/write only the content of a FILE, not a DIRECTORY. When your code has a path string, it should already know that it is a DIRECTORY or a FILE.

Also, when I delete a DIRECTORY, it should be GONE, instead of telling me that it is not empty. If I want a safety measure, my filesystem implementation should do that for me. My list can go on for a while if you let me...

Cotta was implemented in that way. As it was written, we have already got a project that does file operations, so we laid out the expected behaviours of the classes one by one, and implemented the code one by one. When it comes down to physical file system operations, there has been no guesses. The resulting API is compact, which is exactly what we intended.

Test Driven

Or Behaviour-Driven to be exact. When I write a class that operates on filesystem, I'd like to be able to verify the behaviour of that class. Even being a seasoned BDD developer, I still like to spend less time writing more functionality. And cutting the time spend on setting up and tearing down environment is always a good choice, hence the birth of XmlFixture (part of XPE, but you'll have to download the package to see the document), DbFixture, and now Cotta.

When it comes to behaviour verification, a common pattern is to go through "Dependency Injection". With Cotta, the domain class just need to work on a "FileSystem" instance being passed in, or one of the Cotta objects (TFileFactory, TFile, TDirectory) that can be created from the FileSystem. During the verification phase, the filesystem is replaced with a "InMemoryFileSystem".

Cotta also has FileSystem decorators that can be used to verify the behaviour on file system like read only (ControlledFileSystem), or file system with disk problems (CatastrophicFileSystem).

What Next?

The idea of making file system implementation for JAR file, FTP etc., was an after thought. I forgot to check with Apache common VFS again to see if I am re-inventing the wheels (as it turns out, I am, more or less). I think the next thing for us to do is to focus more on the two aspects that I have laid out above, and make it clear on the website about the real market value of this project.

Why dive into the Red Ocean when you have already discovered a Blue one, right?

Cotta Project in Business

Finally we are in bunnies, well, sort of.

A project to ease the pain of using file/directory related operations. See detail at the front page that is still under construction: http://cotta.sourceforge.net

A Cool Story

I thought something like this might happen when I started BuildMaster, and I sure enjoyed seeing it unfold. Thank Nat for following up on each one of us, and thank Joe for putting it all together.

You can read it here: http://buildmaster.rubyforge.org/history

BuildMaster 0.6.0 Released

Finally! I get to write this blog about this little project that I have been working on. It is amazing how much you can get done when you are commuting by BART instead of driving.

It is a Ruby project that helps on the aspect of project releasing management, and website management.

The URL is http://buildmaster.rubyforge.org

Architectural Central Approach?

Another post that I didn't post.

"... Picking story using an architecture central approach ..."

My head is hurting...

Power of Whiteboards

Just finished a kick-off of a project in Beijing, before I came back to my base office, San Francisco, for the next assignment.

The project is a short one, with one week as QuickStart + Iteration Zero, four weeks of development, and one week of bug fixing + final release.

Needless to say, the time was short and there were lots of things to do in four days, gathering the requirements, breaking them down to stories, and coming up with a release plan.

As it turned out, the whiteboards became very useful. We found two of them available, and pulled up the blinds to make the glass wall a third one.

One board at the end of the room, half filled with goal of the week, and the other half as the parking area. With the goals listed, we were able to keep ourselves focused. And we were also free to write topics or thoughts down on the parking area as we discuss, so that we don't get side tracked. From time to time, we check these items to cross out the things that have been taken care of.

The other whiteboard stands right by the door, with a little space in front of it, for requirements discussion. We use it to write down the role and goals, and draw process diagram on it again and again until we get it right. Then we divide it into stories with numbers mapped to different part of the diagram and took digital photos, so that we can easily see how the stories fit into the whole pictures.

The glass window turned into a over-all design board, with cards indicating internal components, external components, screens, and running list for the notes on each of the above. For example, there are WebServices that we need to call as part of the application. So we had a card with the WebServices name on it. During the requirement discussion, whenever we realized that we need to use the WebServices to retrieve or update some information, we will pull the card over, put it as part of the process diagram on the discussion board, and start writing down the kind of interface we need from the WebServices. In the end, we can just take a look at this card and come up with a list of dependencies that this application has on the WebServices.

At the end of the week, looking at the three boards full of arrows, blocks, notes, I just cannot imagine how we can do without them.

A Sample of Practices Interdependencies

More than once, I have got the kind of inquiry as "I'd like to adopt agile(XP) practices, which ones do you think that I should try first. It is always a hard question.

One main reason is that I very much intend to say "every team member should have the XP book by hand, the principle up on the wall, be open-minded and try to adopt the whole practices as a whole". It might sound intimidating but here is the reason:

All These Practices are Related
All these practices are supporting each other. I know it has been spelled out since day one that XP came out, but I still think it is a point that gets forgotten sometimes, especially when the team is under pressure.

In one of the training engagement that I was involved in, after some training sessions of several practices that we think the client needed, in an effort of making sure that the student understand that these practices are much better working together, we created a little dependency diagram using Graphviz.

We did it by creating the graphics file as following (detail omitted):

digraph g {
"Collaboration" [label="Collaboration"];
"Collocation" -> "Time-Boxed Iterations and Releases";
"Collocation" -> "Continuous Improvement";
...
"Test First" -> "Collocation";
"Test First" -> "Collaboration";
"Test First" -> "Automated Testing";
...
"Time-Boxed Iterations and Releases" ->"Collocation";
"Time-Boxed Iterations and Releases" ->"Collaboration";
...
}

The result was pretty staightforward:



Of course, in the presentation that we gave it looks a lot nicer with bi-directional arrows and stuff. But the idea is the same, you cannot pull one out without affecting the whole picture.

On the other hand, if you want to adopt the practices, trying them one by one will probably only make things slower and harder.

A Geek's Version of a Joke

If you have Ruby and Watir installed, here is my "Geek's Version of 'How far do you want to go for a deep joke'". Enjoy

require 'watir'

ie = Watir::IE.start("http://www.sbpoet.com/2006/01/how_far_would_y.html")
while link = ie.link(:text, /(blond)|(joke)|(this one is great)|(>CLICK<)/)
  link.focus
  link.click
end

Personally I don't find it THAT great, but like Steve Boswell always says, "I digress".

Less Leads to More?

Just found out that I never published this...

Sometimes, less does leads to more.

After I came to Xi'an, I started working on a distributed project. The development was splitted between Xi'an China and Bangalore India office.

Based on the observations, I decided to cut the development team in China down as well so that we can be more focused during the communication between India and China teams. So rather than 7-8 developers, we only have 4 developers, including me and Jake to work on the project. It worked out pretty well. We have got to know with each other and built a relationship that is good enough that we are free to use phone/IM/skype/iChat/campfire whenever any one of us needed something.

During the weekend, I had a chance to chat with another ThoughtWorker who is working as PM on another project and he gave me another example (I don't remember the example now...)

Building China Office

Flew in China on Feb. 22nd, I got myself busy right away. The experience is still very exciting and sometimes the tension is killing me.

So these are what we have been doing in China office:

Building the China Website

We were finally able to translate the ThoughtWorks website into Chinese and started hosting it on our China domain (http://www.thoughtworks.com.cn/index.html). Arguing about how to translate a sentence or even just a term has been very much fun and educational at the same time.

On the principle of "Fixing the Windows", we improved the staging environment by reducing the downtime from 30 minutes every hour to 5-6 minutes every hour. Tests were also written for two bugs that we have found during the translation test.

Building the Awareness

ThoughtWorkers in China have been pretty active in getting in touch with the local communities. We have made arrangement with several universities in Xi'an to give the student some exposure of agile software development.

Other ThoughtWorkers working on projects in the other cities are also presenting to the local universities and communities.

As for me, I am going to give presentation to BEA user group meeting this month at Shanghai (http://dev2dev.bea.com.cn/usergroup/20060460.html), titled "Agile in Practice" (originally "XP in Practice").

I think this is an interesting topic to give presentation to. Whenever after someone giving the presentation about agile development, there was always a long session of questions and answers. This presentation can serve as a follow-up by targeting those questions.

With the help from other ThoughtWorkers, I have collected feedback from the projects that we have worked on in China, grouped and categoried them, so that we can show what agile is from yet another perspective. The more I work on it, the more it reminds me of the book "XP Installed".

So I am concentrate more on the photo shots and project samples. It already has over 30 slides and over 10 megabytes in size. Maybe I will have to cut it a bit to fit into the 60 minutes that I have for this presentation.

Distributed Project

Of course, the one and only activity that leads directly to revenue in ThoughtWorks is working on a billable project. Since the first day that I entered Xi'an office, I have been assigned to a distributed project.

The client's headquarter is in UK, and this project involves their offices in Spain and France. So we have two BAs working in Europe gathering requirement. The development team was originally in our Bangalore office. Due to the lack of resource, another small development team was formed in Xi'an office and some of the work has been shifted here.

As you can imagine, the communication has been identified as a risk from day one. We have taken various measures to make sure that the developers understand the code base as well as the requirement, and that everyone is update to date on the project status. This includes Yahoo! IM configuration so that everyone can tell which story each other is working on, daily standup meeting through iChat between Xi'an and Bangalore, and daily BA meeting conference, etc. Of course, Campfire made by 37 signals helped a lot on this issues, gaining more of my already established respect for them.

Just within a month, there has already been a change of direction from the customer, which proved one more time that change is constant and planning is an on-going activity. The expats in the China team left, except me, Jake who knows the code base, and Sagar who knows the business requirements.

Performance Testing Project

This is the project that we engaged for the last two weeks. Even though it did not happen there are things that I have learned through out the process.

Business

The project came to us as a returned business. Basically, when party B did a contract for party A to build a system, they are required to find a third party to do a performance testing for the system so that they can prove that this system will work with the estimated workload.

This came to us as a returned business. It sounded like a routine contract that will just benefit all parties. Party B gets to finish its contract and get paid, party A will get to know better about the non-functional behavior of the system. As I have learned, the rate for performance testing contract is normally higher than other contracts. So it would have been a good short contract for me (working with a long time ThoughtWorker, Matthew Short) before I head back to China. And ThoughtWorks doesn't have to put me on beach.

For reasons that I cannot publicly post, the contract fell through even though technically we have a pretty good idea of who we need to work with, what we need to produce and what to do to get the data.

So now everyone lose.

This is my first fall-through project. Not totally surprised since I cannot imagine that all engagements that we have turn into a project, but still I am still a bit uneasy to the fact that despite all efforts, we just cannot influence the other parties to prevent them from taking actions that jeopardizes the project.

Or can we, I wonder?

Technically

I have done some performance testing for various projects, but always as part of the project iterative tuning process. It mainly focused on the profiling part rather than the performance benchmarking as a whole.

In the last project in China, the client who hired us had a contract with the customer, which put the specific performance requirement as part of it. It looked pretty vague to be, when reading through that part of the contract, that they are just numbers that can be easily met as long as you are free to choose the hardware, or could be almost impossible if the hardware is fixed.

In preparation for project by referring to the one that we did before, I have learned about performance testing methodology, Transaction Processing Performance Council and their benchmarks (http://www.tpc.org.)

Because the application is going to be deployed on a Linux system (while the last one was on a Windows system), with a little research and the great help from Barrow Kwan, we were able to have a Linux at our disposal, run resource monitoring programs, process the data into csv files, port them over to Microsoft Excel and generate the performance chart.

I guess I didn't totally lose on this after all.

Three Announcements from ThoughtWorks

Internally, ThoughtWorks has lots of interesting emails about announcements and discussions. Here are three announcements that I got this week that I think worth logging.

Selenium on OpenQA
Selenium has moved to OpenQA (http://www.openqa.org/selenium/). For those who don't know, Selenium is a web testing technology written to various degree by a couple of ThoughtWorkers and friends. It turns your browser into a web testing tool so that you can test your web application in any browser on any platforms. I have used to in some projects and I strongly suggest you check it out.

CruiseControl Better and Greater
I have been using CruiseControl on every single project that I have worked on in ThoughtWorks. And I have to say that they have come a long way. Now the latest version of CruiseControl launches web server along with the CruiseControl server, and has a "build" button on the reporting page!

Event Announcement: CI and Testing
Greetings Continuous Integration Testers,

This email is being sent to announce an upcoming event for everyone interested in continuous integration and the type of automated testing associated with it.

Jeffrey Fredrick and Paul Julius are cohosting an event that will focus on these topics. The event will use Open Spaces to structure conversation, understanding and innovation.

What: Open Space event discussing all aspects of CI and Testing, together
Where: Chicago, IL
When: Early April (final date tbd)
Who: Everyone interested in CI and Testing
Cost: Free

We'll be inviting people for all manner of projects and places. In fact, feel free to pass this invitation along to anyone that you think will be interested.

For us to finalize the details of time and place we need to get a feel for how many people are likely to attend. If you are interested in attending please join the CITCON mailing list at:

http://groups.yahoo.com/group/citcon

and post an introductory message. In your message it would be useful if you could indicate any topics of special interest and also how likely you are to attend.


Sincerely,
Jeffrey Fredrick (jtf@agitar.com)Paul Julius (pj@thoughtworks.com)

Hey Look! A Delete Button!

This just proves that it sometimes takes a while for even the best to see the light.

Finally GMail has a "Delete" button! Now I just need to un-train myself to click it instead of that drop-down list.

Finally Yahoo Mail logs in using SSL by default! Now I just need to stop clicking on that link on the login page.

曲突徙薪无恩泽，焦头烂额为上客 -- An IT Phenomena

This is a thought that I had during my two week vacation.

Sitting at in-law's home, I grabbed a book about Chinese idioms for my nephew to kill some time, and a story caught my attention.

The story started when a guy built a beautiful house, and invited his friends and neighbors over for a house warming party. As everyone else admiring the house and congratulating the host, one guest, however, made a comment, "The chimney is too straight up and the fire woods are too close to the stove. If you don't make the chimney into an S shape and move the logs away from the fire, chances are that the house will catch fire before long."

The host, however, was not very happy to hear these, thinking this guest is saying words of such bad luck on such a good day.

As the luck went, the house did catch fire a month later. All the neighbors came to help. Eventually the fire was put out, leaving everyone with burns and bruises.

The host was very grateful. He opened the kitchen immediately. He had everyone sitting in the best seat, and treated them with the best food and drinks.

During the meal, another guest, who also came to the house-warming party, brought up the topic, "A month ago during your house warming party, there was a guest that predicted that this would happen. Had you listen to him at that time, chances are that your house will not catch fire and we would not be sitting here enjoying the feast. Don't you find it backwards, when the one who recommended the modification on the chimney and moving away the log was not being appreciated, while the rest of us with burns and bruises are being treated as the best guests (曲突徙薪无恩泽，焦头烂额为上客)?"

The host came to realize the mistake that he has made. He immediately sent for that guest, and treated him with the best seat in the house.

As I work through projects and in different organizations, in ThoughtWorks or before ThoughtWorks, I noticed that the person who is good at putting out the fires are always the one that gets the limelight. The person could be a manager who knows how to talk to the customer to cut the scope in the last minute, or find ways to persuade the developers to work overtime, or he/she could be a developer who is good at digging through a bunch of messy code and come up with a hacking patch for a last minute bug.

On the other hand, there don't seem to have enough attention to the sources of all these crises. People tend to pay most attention to how to resolve these crises rather than how to prevent these crises. So advice was not given any thought and measures were not taken because "if it aint broke, don't fix it".

And there are more crises.

And it is more important to keep the ones who knows how to put out fires with burns and bruises.

The Job of Project Manager

I always believe a real good project manager is not one who always resolves crises, the same reason that a real good developer is not one who knows how to fix his own bugs. The reason is simple: it takes more skill to prevent problem than that to resolve problems.

I think a good project manager is one that make sure everything runs smoothly. For example, all the necessary preparations are done before each meeting and all the action items come out of a meeting are being followed in time and properly. Each of the team member knows what the project vision is, knows how to do his or her own job, and is motivated to do the job.

Sounds too vague huh? I agree. And the reason for this bolg is that I finally have something concrete.

For various reason, we are now having an agile development team with a project manager who is not experienced in an agile project. In China, it is very common to have a project manager with technical background, and it is very common for the project manager to help out the development work rather than managing the project.

In order to help the new PM managing this project (rather than checking out the source code and database schema), I asked for Renee, another ThoughtWorker who were the manager of this team to write down a list of responsibilities. It turns out to be very helpful for everybody. She also directed me to this on-line document (Agile Project Management), which is a good read as well.

The following is the list:

Here is a summary of the key responsibilities of a PM on an Agile project:

• VISION - Establishing a guiding vision for the project and continuously reinforcing it through works and actions.
• Ensure the team and client understand the vision.
• Clear up any questions about the vision.
• Includes timeline, release planning, scope.
• TEAMWORK & COLLABORATION - Facilitate collaboration and teamwork through relationships and community.
• Establish a good relationship with the team and client.
• The team and client will come to you with issues if they trust you.
• SIMPLE RULES - Establish and support the teamÂ’s set of guiding practices.
• Examples include TDD, standups, pair programming, etc..
• OPEN INFORMATION - Provide open access to information
• Release plan
• Story list
• Progress, velocity
• Staffing changes
• LIGHT TOUCH - Apply just enough control to foster emergent order.
• Demanding control will only cause the team to lose order.
• Empower team members to do their jobs
• AGILE VIGILANCE - Constantly monitor and adjust.
• Hold retrospectives on team progress

An Agile PM needs to be full time on the project. A part-time PM cannot remove obstacles, encourage collaboration, or provide vision. Here are some addition day to day responsibilities:

• Lead stand-ups
• Track daily progress
• Manage release plan
• Communicate with customer leads
  
• Supporting the team and helping the customer to understand project decisions, e.g., why we want to do J2EE without EJB
• Listening to the developers and team, helping to remove obstacles
• Risk identification and mitigation

The Job of Architect

In the ThoughtWorks project that I have been working on, there are just one lead and several developers. Even though the lead is normally called "Architect" because that is the term that we use externally, he or she is much difference from what people normally think of Architects.

Rather than drawing all kinds of fancy diagrams and leave to the developers to implement the design (More than once I have heard of this theory that the developers don't have to be highly skilled as long as they do what is told).

In my previous projects, I have been very impressed with the work of our project lead. Highly technical and very experienced, the lead pairs with the team and working out the design decisions with the team. I felt that this is the best way for each member of the team to learn and grow and this is the best way to come to the right decision for the project.

On this project, being an architect turns out to be much more than just leading the development work. The current project involves three different servers posting their interfaces through WebServices and communicate to each other through these interfaces. WebMethods is the data exchange platform so that more services can be added, and there is a CA server for certificate verifications. Because of this, there are a lot of coordination required among the development groups. In addition, the requirement is not clear so I need to help the BA on the story list creation.

So between the development groups technical discussions and requirements gathering, I have not been too active on programming. The worst day was last Tuesday, where I had meeting back to back from the morning to the evening. I have been trying to play catch-up, looking through the code after work. But now I realize that with the code changes everyday, I will end up killing myself and still not able to keep myself up-to-date. So I am now delegating the design decisions to the team and pay more attention to their development practices like pair programming, test driven development, rotation, etc.

The good thing is that with the BA on track with the requirements gathering and story list creation, I get to concentrate more and more on the development.

Two More about China Environment

So here are two more things that I thought worth mentioning.

All the three hotel rooms that I have been staying at have a central control panel by the bed. So to go to sleep you can just press one button, which will turn off all the lights and sets up the sign "Don't disturb" at the door. Pretty cool eh?

Although they don't have an alarm clock.

The connection in China to US is BAAAAAAAAAAD. I did a download speed test the other day and here are the data:

• ThoughtWorks gforge 0.6kb
• sourceforge 1.9kb
• rubyforge 2.3kb
• java.net 2.7kb
• codehaus 2.3kb

Doesn't that remind you the good-old days of dial-up? :)

BTW, the connection within China is pretty fast.

Things to be Prepared for in China

First of all I think I need to say that I am still greatly enjoying working with great people in China and enjoying great food here.

Originally I put the title as "Things that I don't like in China". But later I thought about all these stuff I realized that they probably won't change in a short period of time. So the only solution is to watch out and try not to get bothered. (Granted, the city we are staying is not as a big city like Beijing or Shanghai).

Private Space

I know it always happens in a place where there are lots of people and limited spaces. However, from time to time, I still get surprised.

For example, now I have learned that when the elevator reaches your floor, you need to step right in from of the door and get ready to block the door so that you can push your way out easily. Otherwise, there will be people pushing their way in so quickly that you might have a hard time getting out.

Another case is that when talking to the front desk don't get offended if someone else just budge in and start asking question to the person who you are talking to at the moment. And don't get surprised if that person will switch attention to him or her right away and let you stand there waiting for the answer.

Limited Services

Traveling in United States, I have gotten used to stepping in a hotel and have iron board, steaming iron ready in the room. When I travel for over a week, I know I can always find a place to wash my clothes, most of the time inside the hotel that I am staying at. However, that is not the case here. Both hotels I stayed at in Shijiazhuang don't have iron and board. And I had to buy an iron because the second hotel has only the old style ones that burn shirts easily. Laundry has been a problem because I cannot find any place to wash them myself.

The other guys found two apartments to stay in. I was hoping to be able to do laundry at their places. However, the apartment services here is hardly any good. The washing machine in one apartment does not work, and the other one is missing a water pipe. What's more, there has been some repairing or construction going on and they have been out of hot water for over a week. Calling the owner or the property management has been in vain. The guys have been pretty cool about it by taking the shower at the shower room of the swimming pool downstairs. Still, I don't find it satisfying.

Task Management

In our working environment, there seems to exist a tendency of setting the task without setting the date. At the beginning the reply we here very often was "ok, let me discuss with them about this". In a project that has only 3 months and still working on requirements at the third iteration, this kind of got me nervous. However, all we needed to do was to ask for a date, either for the answer or a follow-up date about the date.

One Week in China

This moment exactly one week ago, I checked in the Great Wall Sheraton Hotel in Beijing, and began my life as a consultant traveling oversea.

One week is not a long time, but it has already been a blast. The ThoughtWorkers in China get along very well (not that the ones in US didn't) and there is always constant conversation during the day as we cranking off the story cards. Because of this tight relationship (the other four guys actually live together in two apartments of the same complex), conversations are very open. For example, the CruiseControl server is in the same room, on a Windows machine, so the guys set it up so that every time there is a broken build, it will play a 20 second clip saying in Chinese something like

"What the !@#$% are you doing that for... Would you ever stop? I have already told you that I cannot handle it anymore, but you keep on doing it, doing it, doing it. You just don't care if I can stand it or not. If you do it one more time I probably will have to stab you..."

The original text can be found here and it is definitely much funnier to hear it in person. Because it is long and loud, it easily gets annoying, and with the good team relationship whoever broke the build definitely gets a lot peer pressure to fix the build.

There is also a bonus for me on this project. That is I get to get involved in a lot of project management stuff. With the client speaking Chinese, I am helping the manager whenever we need to talk to the client or whenever they need to talk to us about project plan and arrangement. The current crucial task is the story list. Because there is not a full-time BA, and that we are only developing one part of the system, and because that the time is limited, we are trying to finalize the story list, so that we can track the project progress with higher accuracy. So I am also going to play the role of BA for a little bit whenever there is a translation needed.

On Monday, we had a kick-off meeting. Because this is the first time that the customers were involved in iteration planning, and that this is the second time that they were presented with Agile Software Development, we decided to give them a second presentation of agile software development, focusing on the benefit, role and responsibility of the customers, and we were able to invite the deputy director. I believe the presentation was well received, because the deputy directory got excited and spoke at the end that we should receive the full support from their side. The full-time customers that have been assigned to us also turned out to be good natured and knowledgeable to the domain.

One more reason that I love this team is also because that the guys are very comfortable with me adding the Dvorak keymap to each laptop that I touched. We set Alt+Shift+8 for QWERT and Alt+Shift+9 for Dvorak and it works out pretty good. Someone even showed interest in learning Dvorak. Hehee... Maybe sometime we can finally say bye-bye to QWERT in the China office.

It is also interesting to learn the Chines meaning of the terminologies that I have been working with for the last 6 years. The following is a list that is on top of my mind:

• Development: 开发
• Iteration: 迭带
• Document: 文档
• Release: 发布
• Deployment: 部署
• Class: 类型
• Method: 方法
• Agile Development: 敏捷开发
• Project Management: 项目管理
• Test: 测试
• Run Test: 跑测试
  
• Constructor: 构造函数

And learning Wubi (五笔), a Chinese input method, is fun as well.

Arrived at ShiJiaZhuang

After a 12-hour flight, half hour Customs Service (The 'Paul and Jamie Buckman' behind me sure made it seem like 2 hours), one night hotel, 2-hour train ride with half hour delay, I finally arrived the city of the project, ShiJiaZhuang (石家庄).

I have finally met three of the four ThoughtWorkers in China office, Li Mo, Li Chaoqun and Li Xiao. (Yes, Li is a common name in China), and I got a chance to talk with Li Mo on the train. Sid's comment of ThoughtWorkers was "young and super smart" (sounds like the early days of ThoughtWorkers?) and talking with them sure confirmed it.

What I have gathered so far has already shown that projects in China is definitely different from the states, unfortunately I cannot reveal much. As much as I want to keep this blog on my experience in ThoughtWorks, the special condition in China might force me to log lots of them on an internal server. I guess we shall see.

Walking off, and Going to China Project

This is one of the times that I am proud to be a ThoughtWorker.

We have been working with the current client for over a half year (I joined three months ago). The agreement has always been that this is a transformation (formerly known as enablement) project, that, given the clients' background of waterfall approach and ThoughtWorks' experience in agile development, should rely highly on ThoughtWorks' input. That is, for a start, during the scoping phase form a small group (6) and have one on one ratio of ThoughtWorker and the employee from the client.

But apparently someone thinks that if they pay us they own us. Before the pilot project even starts (hence gets the requirement), there is already order from high up to use one of those so called "J2EE framework server". And during the initial phase, there are going to be 9 people instead, with only 3 from ThoughtWorks. All these decisions are made without even consult ThoughtWoks (you would think that should be the reason to hire a "consultant"). And no reason has been given for these decision.

So as the current finishes this month. The manager decided not to renew it and walk off from a project that is not built on a solid ground of trust and understanding, even thought the rate on this project is fair and we might have a good account. This decision got the support from the headquarter and the four of us are already on the list of available consultants for October.

As for me, I am really glad of the decision being made, because that shows that we are serious about being good at what we do, and the we care about have happy consultants on the project.

Another good reason for me to be happy is that the moment my name come up, there is already a good project for me to go -- China. Because of this and last transformation project that I worked on, and the fact that I speak Madrine, I turn out to be a good fit for this project that needs someone to come in and help out. Time is short so I am busy working on the visa and travel plans.

At this moment, my goal of China trip is the following, I am still updating it:

• Get to know the ThoughtWorkers in China: I have heard of great things about the people that we hire there. I need to spend as much time as I can to get to know them and exchange our experiences to learn from each other.
• Get to know the market in China: Despite the great potential in Xi'an, I still think there should be a lot of opportunities in Beijing and Shanghai. Agile software development in China is getting a lot of attention, if we don't grasp the market and build our good names, there will be someone else very soon.
• Get to know the IT environment: There might be a lot of things that I am taking for granted here that is not as available. Programming in a non-English speaking environment might bring different dynamics.
  

Deployment Script with Watir (Take THAT, Websphere!)

So, after some struggling with Websphere on deploying an EAR file (Don't ask me question about why we still bother, please... ) and not able to get a hand on a simple deployment script (I think there should be one but I just didn't have the time to look further), I have decided to automate the web process that I have been doing, using WATIR.

Task
Call me innocent, but I really thought that depolying an EAR file on an application server would be easy, given the 'comprehensive' file structure and all the deployment descriptors.

But no..., you have to click through this long 'wizard' (ooohhh...) process every single time. Not only you have to do it manually, it also takes quite a lot of time. And I happen to be one of the impatient guys who thinks that it is a machine's job, then a human should not suffer.

Watir
In case you haven't heard of it, Watir (http://wtr.rubyforge.org/) is a ruby library used for web testing. I am still not sure how exactly it works, but the result is that you can control an internet explorer instance using Ruby script. So basically almost anything you do with IE, you can write a script and automate it.

Code
So long story short, I was finally able to write a script that uploads the file to the server, click through the steps, and save the configurations. Needless to say, I am very impressed the progess that Watir has made, functionality as well as the documentations.

The script is actually not that long. Here it is (with comments):

require 'watir'
 
class IeLauncher
  def IeLauncher.open(*args)
    result = ie = Watir::IE.start(*args)
    if block_given?
      begin
        result = yield ie
      ensure
        ie.close
      end
    end
    return result
  end
end
 
# Ignore the above, they are just basic set up.
 
# Launch IE, go to the specified URL
IeLauncher.open('http://localhost:9060/ibm/console/') do |ie|
  # Log in, my local server doesn't have a password
  ie.text_field(:name, 'username').set('admin')
  ie.button(:value, 'Log in').click
 
  # Only one login is allowed so you will have to log out last one
  if ie.contains_text('Login Conflict')
    ie.radio(:name, 'action', 'force').set
    ie.button(:value, 'OK').click
  end
 
  # Apprantly when you deploy an application, you need to 'save' it.
  # This recovers it
  if ie.contains_text('Recover prior changes')
    ie.radio(:name, 'action', 'recover').set
    ie.button(:value, /OK/).click
  end
 
  # This get the frame that has the navigation tree and look for the right node
  navigation = ie.frame('navigation_tree')
  navigation.link(:text, 'Applications').click
  navigation.link(:text, 'Enterprise Applications').click
  detail_frame = ie.frame('detail')
 
  # Uninstall the existing deployment
  if detail_frame.contains_text('DukesBank')
    dukesbank_checkbox = detail_frame.checkbox(:value, /DukesBank/)
    dukesbank_checkbox.click
    detail_frame.button(:value, 'Uninstall').click
    detail_frame.button(:value, 'OK').click
  end
 
  # Install
  detail_frame.button(:value, 'Install').click
  detail_frame.radio(:name, 'radioButton', 'local').click
 
  # Upload the file (Note, file path cannot have space)
  detail_frame.file_field(:name, 'localFilepath').set('c:\Work\Dukes\DukesBankEAR\build\artifacts\DukesBankEAR.ear')

  # Clicking through the wizards
  button = detail_frame.button(:value, 'Next')
  while button.exists?
    button.click
    button = detail_frame.button(:value, 'Next')
  end
  detail_frame.button(:value, 'Finish').click
  detail_frame.link(:text, 'Save to Master Configuration').click
  detail_frame.button(:value, 'Save').click
  navigation.link(:text, 'Enterprise Applications').click
end

Feedback from the Client

I have meant to post this a while ago, but with the traveling my daily schedule is still not settled. So I never found a good time that I can just quiet down and write some blog.

This week is a short week due to the past long weekend. So we were able to convince the client that it is better for us to work from our home office so that we don't need to spend two days traveling for a four-day week. And once again, I am able to continue my routine of blogging.

After my last enablement project (now the term is "transforming" because "enablement" has been used along with "training". I will explain that later), the client team had a project retrospective, discussing what they have learned during the project.

With their still fresh memory, they were able to extrapolate the following, with which they gave the presentation to the rest of the organization. They are nice enough to let me publish this. I hope this can be useful for some of you who are still wondering about agile software engineering, or at least some of the "weird practices" that I have been defending from time to time. This is the first-handed note. Hopefully I can help them revise the format for more formal presentations.

Agile: One Developer's Perspective

What methodology did we use in the past?

• It doesn't have a name, as far as I know, but here are some features of it
• big upfront requirements-gathering, big upfront design, with some formal and some informal documentation
• one developer per application, or a couple of developers working in different tiers of an application
• integration & testing in a big bang at the end
• manual testing

Some problems we had with this approach

Big upfront...

• late-breaking requirements changes combined with inflexible deadlines led to periodic frenzies of coding that eroded the integrity of the big upfront design
• sometimes we specified & designed more than we could actually implement with the time and resources we had
• integration late in the development cycle led to periodic frenzies of coding that eroded the integrity of the big upfront design
• disagreements about how open we should be to changes

One developer, or a couple of developers working in different tiers of an application...

• we didn't always take advantage of possibilities for using some common code or common approaches across applications
• difficulty scaling up for larger efforts
• the culture of code ownership by individuals was sometimes at odds with our "esprit de corps"

Integration & testing in a big bang at the end; manual testing...

• huge burden placed upon QA (when we had QA at all)
• difficulties detecting the ramifications of maintenance changes, so we became timid about upgrading applications and we ended up with multiple versions of the same module in production
• we often had as many opinions about how close we were to being finished as we had team members
• we did not know how to thoroughly test componentry; we could only test built apps

For me personally, the biggest problem was that I was unhappy. I felt that our team could be great, but we just couldn't get there, and I didn't know how to be "part of the solution".

Enter Thoughtworks, with a philosophy and a toolbox

Some things I was immediately enthusiastic about...

• Eclipse IDE (Great no matter what methodology you embrace)
• Automated testing
• Daily standup meeting
• Frequent discussions as a group about what is working and what isn't, so that we can make incremental corrections in our direction
• Realistic attitude towards estimating: as the project progresses, we can get more accurate in our predictions of a delivery date, but we can't expect perfect accuracy at the beginning.
• steady pace, avoiding individual heroics
• Simplicity--the art of maximizing the amount of work not done--is essential. (From "Principles behind the agile manifesto" http://agilemanifesto.org/principles.html)

Some things I was skeptical but open-minded about...

• TDD
• Test-driven development requires that a unit test be written for each tiny piece of functionality before you write any application code
• The unit tests are written by the developers, not QA
• Although "test" appears in its name, test-driven development is not so much about testing as it is about enforcing a development discipline that makes refactoring possible
• The unit tests serve as documentation of the developer's understanding of requirements
• "radical co-location"
• continuous integration
• fully automated build and test process that allows a team to build and test their software many times a day
• everybody using the same code base, checking in and updating their code very frequently
• at first, I was incredulous that checking in and building as frequently as once per day would even be possible; TWers viewed once a day as a minimum

Some things I was deeply skeptical about...

• paired programming
• code ownership is to be avoided
• evolutionary design through refactoring

Martin Fowler's definition

What is Refactoring?
Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations. Each transformation (called a 'refactoring') does little, but a sequence of transformations can produce a significant restructuring. Since each refactoring is small, it's less likely to go wrong. The system is also kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring.

• very short iterations
• less emphasis on formal documentation
• "embrace change"

A year later
Life on our team is completely different than before. From my perspective, it is much better.
Did the philosophy and the toolbox make a difference?
Yes, but not all by themselves. We had to have...

• the right attitude
• one-on-one mentoring from experienced agile developers
• effective leadership
• discipline

Attitude
We could not have made changes as extensive as this without

• positive attitudes
• team spirit
• perseverance

one-on-one mentoring from experienced agile developers

• from the agile manifesto: "value individuals and interactions over processes and tools"
• TDD feels initially so unnatural, that it is essential to have a patient, experienced person at your elbow
• without discipline, the agile philosophy can degenerate into "a license to hack", so it is important to have constant reminders about the discipline until it is second nature

effective leadership

• the project manager may be an exception to the general principle that individual heroics should not be required
• protected the group from distractions
• keeps both the big picture and myriad details in mind
• inspires confidence

discipline

• do not write even the smallest method without writing a test first; see the test fail, then write the minimum amount of code to make it pass
• update from the repository frequently, run the entire test suite before each check-in of your own
• builds & test suites run continuously; if your check-in breaks the build, STOP and fix
• to maintain design and code integrity, coding decisions are made in pairs...if the pair can agree, go forward

If the discipline is not observed, then we can't "embrace change" and "evolve the design through refactoring" because we can't be confident about extensive changes.


Notes

• http://www.martinfowler.com/articles/newMethodology.html#id2249560
• http://loki.lokislabs.org/weblog/archives/2003/11/15/radical_collocation_is_a_good_thing.php

Below is a summary of BTM/70, a methodology in use at a job I had 20 years ago at a company. It is probably an example of what the "value individuals and interactions over processes and tools" principle is a reaction against. All that I remember about BTM/70 is that there were big binders of sample forms, and every step of the way, one had to photocopy the appropriate forms and fill them out. I do not recall any specific benefits.
The life cycle of an BTM/70 project consists of eight (8) phases ranging from Systems Requirements to System Implementation. Coupled with BTM/70 is ET/70, a Project Control tool set that is used for project planning and tracking. ET/70 assists in milestone planning, helps establish project target dates, supports activity tracking, and detail time reporting of each staff member down to the task level, within each activity, of each phase of the project

Start Traveling

Starting traveling last week. I realize that since Cysive I have forgotten how tiring it can get sometimes. Fortunately, I have got a pile of book that I have been wanting to read but never "find" the time between work, hiking and gaming. Since I can do none of the above on the 8-9 road trip twice a week, a book surely comes in handy.

BTW, Robin told me about this interesting Dvorak post on slashdot (http://ask.slashdot.org/askslashdot/05/07/10/1645229.shtml?tid=227&tid=4)

Dvorak Follow-Up - Why I Think It Is Great

Two weeks into trying out Dvorak hardly makes me into an expert on the keyboard layout, and chances are that my opinion about it will change as the time go by. So I was trying to focus more on how I managed to learn it rather than any conclusion that I have drawn.

However, I realized that since I already made the comment that "It is great", I'd better explain it in the whole aspect, even though it is just a snapshot of what I am thinking and it might very well change.

I think it is great because I am amazed at how easy it was to learn. With vows at the left hand home row, it somehow makes it very easy to remember. Most of the common key combinations stay on the home row, which makes very easy to type. Here is a good example (he calls it Dvorak poetry) that Alex sent me:

Shane Duan nodded to his sis as he situated his sedan in the shade

If you want to know what it feels like, here is the key this sentence mapped to on a QWERTY input:

:jald Hfal lshhdh ks jg; ;g; a; jd ;gkfakdh jg; ;dhal gl kjd ;jahd

I think it is great because in Dvorak, you can type the whole sentence without your hands leaving the home row. And I believe that is the case with Dvorak that you stay on the home row most of the time. I think that is the main reason why it is so easy to remember. You just need to learn the home row and you can pick up the rest as you go.

No I am not ready to commit my life to Dvorak. The website that I have referred to (http://www.mwbrooks.com/Dvorak/) has very though references on both side of the story, especially here (http://www.mwbrooks.com/Dvorak/dissent.html). And this link (http://www.utdallas.edu/~liebowit/keys1.html) referred form this link in the comment, is making a very compelling case. However, just like the author said: you don't have to believe either side of the argument. I am hoping through a first hand experience, I can make a convincing case either way.

Of course, the speed will always be the hot topic here. However, from what I have concluded so far, I am afraid there is no easy answer. As a fast typer, I felt that the speed comes from not only the ability to map a letter to a key without thinking, but also that to map the key combinations. What this means is that probably there is no much difference between the inputs as far as a fast typist concern because he or she will simply just remember the same key combos. If that is true, I might be the wrong person to try Dvorak, since I type 8 hours a day for living.

With that being said, I think that a casual typer probably would benefit more from Dvorak input. However, since all the keyboards you can see during everyday life are labeled with QWERTY, one tip to learn is not to look at the keyboard. Ironically, if you can do that, you are probably already a good typer.

I also have to be honest with you. I am a geek, and I love trying different stuff. Just the fact that I am able to say "I tried it for a year and I think..." rather than "I read it on this website one day that it seems..." is already something worth doing for me.

The last thing is really rather subjective. I never had any good feeling for QWERTY typing. I simply do it because it was the only choice. However, I somehow enjoyed Dvorak. It could be the combination of the fact of typing mostly on home row (With kjd as the keys for "the" just makes more sense) and the fun of trying something new.

Two Weeks Into Dvorak

It has been two weeks since I have switched to Dvorak and I really think it is something worth writing about.

The topic of Dvorak just came out of a lunch conversation. I was talking about the fact how sometimes we keep doing what used to useful without realizing that the circumstances have changed. A small example was how Java IDEs forget that maybe the traditional windows mapping doesnot apply anymore (who opens a file through file chooser, or print the code anymore). And IntelliJ's found of F keys making its useful features much less accessible, and that by some miracle Eclipse made the right choice of having all ctrl+alt+key reserved to refactoring related actions.

Then the topic switched to even the key board layout tha comes with every computer was not designed to make your typing fast and easy and how I heard of another kind of keyboard that is supposed to do just that. As it turned out, Adam knows exactly what it is because he has tried it before.

With Adam's reference, I was able to enable my laptop (window XP) with Dvorak keymap and find the related sites. After reading through it, I figure I'll give it a try.

I am a fast typer. I got my hands on a type writer and a typing training book when I was a kid. So I literally learned typing before I even knew all the words that I was typing. Then in graduate school for a year I got addicted to IRC and another year on text-based MUD game, both of which require intense typing in that the faster you type the more fun you can get out of it.

So you should believe when I tell you this: Dvorak is great!

I started with a online tutorial (http://www.gigliwood.com/abcd/). I do about three sessions per day, 15 - 30 minutes per session (when I am getting tired or bored and staring mixing keys). At work I still have to use the QWERTY because it would be too slow otherwise.

After a week, I felt that I can finish the lessons of the home row without problem, but my training on the other rows is not making good progress. I felt that it is because my old habbit and this half Dvorak is not working togethir. Since one major advantage of Dvorak is supposed to be the fact that your fingers stay on home row most of the time, I have decided to swithch to it full time, with my Fitnesse blog as the first target.

Well, that weekend was not too easy. It is as if I am learing to walk again. For a while I was even wondering if it is really worth the effort. Two-day is hardly a long enough period to give up, so I just lowered my target on how much I can write in a hour so that I won't be too frustrated. I also noticed that most of the time I am having trouble because I am too used to the old way that for key combination like "ing", "tion", or even "th" my figer will go without even my brain had a chance to process them in Dvorak context. So I concentrated on several key binding that is giving my trouble.

Now after two weeks, I would recommend anyone who wants to learn typing to try it out, AFTER you have read the pro's and con's on the websit and make your decision.

Memorial Weekend - A Sample of Quality of Life with XP

First of all, the criteria of the quality of life are rather subjective. If you do enjoy working on your project at home, then all I can say is that you are lucky. For me, there are so many other things besides work and software engineering that I am just too interested in to give up.

One good thing that comes with XP is that you have the control of your time. Because you always know how far you are to finish the project, your project management is fairly predictable. As a result, you can deliver the project on time, and your personal life is not affected.

So, even though my work is still busy as hell, I was still able to put it behind and enjoy my long weekend. For some person it might be really obvious, but my old buddies at JBuilder team would only know too well how obsessed I can get.

Without further a due, here it is:

Stinson Beach Hiking
This is by far the best hiking place that I have been to around bay area. The reason is simple: This is the only place that you can hike on the beach, on the grass field, and the woods in one afternoon.

Brentwood Cherry Picking
Eat all that you can pick, and buy home the ones that you cannot. Just make sure that you need to pace yourself otherwise your stomache might not be able to take it.

Ninja Gaiden
I don't understand why they want to associate this game with Dead or Alive 3, because it is totally different kind of game. Besides the normal similar quest that is quite similar to Final Fintasy X, it requires the player to analyze the enemy's attacking patterns and come up with counter measures. The last big boss was quite disappointing, though, because I just chased him around like a mad man and that killed him.

Texas Hoedown
Yes I also do not believe that there is luck in poker. But, I do believe they should put the number on those chips so that I don't have to remember them.

Joy and Pain with Fitnesse

(This blog is entered using Dvorak input, which I am going to write a blog about after I either master it or give it up)

Last Wednesday (May 18th.) I gave a presentation to the BayXP (http://groups.yahoo.com/group/bayxp/) titled "Joy and Pain with Fitness". I am putting the presentation material here, and try to explain the examples.

Joy and Pain with Fitnesse

Presentation of what we have learned during our usage with Fitnesse, an acceptance test framework, and how we have benefited from it. This presentation is intended on purpose to avoid raising philosophical issues, software engineering methodology, or what the correct usage of the Fitnesse should be. Rather, it concentrates on the retrospective of our experience with Fitnesse. It is more important to understane the reason behind

Overview
Fitness' own description can be found at its website "OneMinuteDescripiton", with the following in particular in our project:

1. Fitnesse is an acceptance testing framework. Its targeting audience is not only the developer but also the user who is going to write and run acceptance tests, like domain experts, QA, manager, etc. (And it has done a great job to achieve this goal).
2. Fitnesse is written in Java, which makes it easy for developer to assist the acceptance test writing.

What We Have Achieved
Just like an application product, we slowly developed a set of Fitnesse fixtures that allow everyone on the team, including the QAs and BAs, to set up the system state, simulate the events processing, and assert the end state of the system by writing XML document assertions and database assertions.

In order to document how to use the fixtures, we also created a "Fixture Tutorial" page, which contains all the fixtures that are available, with detailed descriptions. Each page is a test by itself, which means that people can change them and run them. This is very similar to the ones on Fitnesse page, e.g. JdbcSample, except ours are related directly to our domain.

How Fitnesse Works
The following is the outline of what we have learned in our project. It is quite possible that there are useful features that we have not discovered so you should make sure to check out Fitnesse website.

1. Fixtures are loaded through reflection by very simple naming convention. For example, if the header of the table is "Insert Person Data", then the class "InsertPersonData" will be loaded and instantiated. The packages in which Fitnesse should look for the classes can be specified through "Import" fixtures.
2. A page called "SuiteSetup" will run at the beginning of the suite and will not run for a single test. A page called "SetUp" will run at the beginning of each test all the time. The same rule applies to "SuiteTearDown" and "TearDown".
3. The following are the four most useful features to us in this project:
1. Column Fixture: The column name without the question mark is a public field in the fixture that Fitnesse will set value to. The column name with the question mark is a public method in the fixture that Fitnesse will call and compare to the value in the table.
2. Row Fixture: Great for doing assertion on a set of data. For example, if you want to assert that the person has three specific phone numbers (555-2222, 555-4444, 555-9999), you just need write the Fitnesse page with these numbers and implement a row fixture that simply returns all the phone numbers this person has. Fitnesse will do a fantastic job of comparing these two sets and show you the difference.
3. Action Fixture: The first column of each row is mapped to a public method. This is especially useful to run a sequence of actions and check the result between the actions. This fixture is being utilized by jWebFit.
4. If you use standard fixtures, all data type conversions are handled seamlessly. As explained in "DataTypesInFixtures".

Why We Love Fitnesse
We believe that tests are good and that we as XP and Agile developers should do everything we can to encourage developers to write unittests and QA/BA to write acceptance tests. In this project, we would never have achieved this level of tests without Fitnesse. To quote from our QA:

It is great to have a tool that allows me writing tests for our system. I have been quite uncomfortable with matching logic. With these many tests and assertions (over 23,000) I am feeling a lot better now.

Because it is easy and fun to write test, many things just follow naturally. Stories are written with test in mind, making them easy to verify. Acceptance tests are also written early, sometimes even ahead of the iteration. Consider this is the team that didn't know what a story card is and didn't have any automated tests, we are really satisfied with the result.

For BA and QA, or any non-technical member of the team, Fitnesse is great because it has very clean UI with neat user interaction design. It is much less itimidating than a programming language. You just feel that you are writing a Wiki page with tables that are runnable, instead of a program with limited way of writing comments. The pages are much more readable than the JUnit tests.

For developers, there are already fixtures helping them writing unit tests. With Fitnesse written in Java and loading fixtures through simple reflections, it is quite easy to hook up the Fitnesse with the existing testing fixtures, thus greatly reduces the code duplications athe cost of the Fitnesse fixtures.

Why We Hate Fitnesse
I believe it is as important to explain the struggle we had with Fitnesse as well as the success, because this way the reader can learn to use this tool effectively. It is also possible that in the limited amount of time and with project development to bring business value as our ultimate goal, we might not have learned the way that Fitnesse was originally designed for. And I'd very much like to be directed to the right direction.

No matter how many times that I say it is very easy to write Fitnesse, you still have to spend time writing it. Also because it is not the developers who write and run the Fitnesse, you get the same issues that you have for any project. What this means is that developers are taking requirement request. It is only natural that you want to help, and the better a developer you are, the more likely you are going to put down your story and write fixture instead. Before you know it, your velocity is down and you are only overwhelmed with more and more fixture requests. There is another member of BayXP who encountered the similar problem.

The target audience of Fitnesse are the none-technical member of the team so it is doing its best job to achieve that. This achievement, however, comes with the cost of the usability for developers. A developer who has been infected by TDD is a developer who has been spoiled by JUnit. With Fitnesse we have to again and again wait patiently for the suite to finish. While waiting, we have no idea how many have been run, how many there are in total, how long it is going to take. When there is a failure, we have to wait for all the tests to finish before we can see the report. (You can copy the URL and open it in another browser window but then you have to worry about running two acceptance test processes at the same time on the same machine, which can give you some trouble sometimes.) Even with the report, you still cannot open that line of code with one click. And it is a lot harder to put breakpoint at the place that you want in debug a problem.

In the spirit of Continuous Integration, we run some of our Fitnesse tests through CruiseControl. The best way we were able to figure out is to launch the server, then use the Fitnesse runner connecting to it. With the way Fitnesse is implemented, we ended up with a server thread serving the HTML, a client thread reading it, and another thread monitor the communication between these two. Somewhere it is not managed correctly and it is causing the process to hang. We spent a lot of time trying to figure out the reason but this problem only occurs on one of the Solaris machine (which happened to be our release machine), occurs 10% of the chance on our Linux machine (which is our build machine) and never on the windows machine. The closest thing that we found online is this post about how an earlier similar bug was fixed. We moved to an Linux machine for the release and bounce build machine every now and then, and moved on.

Fitnesse also tries too hard, in my humble opinion, to be everything. Rather than a simple webapp, it is a stand alone web server. This means that you have another server to manage, another account to manage and another port to open. It also has its own Wiki engine so it is another set of rule to remember (after played with 5 different rules among Wiki, Twiki, Confluence, Rubywiki, one really gets tired.) Fitnesse dose not extend JUnit framework. The whole suite is actually run as one test, which means memory will increase as the test is running because of the HTML page.

Patterns
As we work with Fitnesse, we have developed the following patterns that helped us to use it successfully.

Just like any product feature request, every Fitnesse fixture request needs to be considered and discussed. We include them in our iteration planning meeting, we estimate them, and we pair up to implement them.

As we develop the fixtures, we write the tutorial pages first, which serves as our tests for the fixtures themselves.

Fitnesse pages are saved in their original format in TXT files. We launch the server with the correct arguments so that it does not manage the history of the pages. Then we check the files into our version control system.

For each project, we use one singleton class to manage the data that we want the fixtures to pass to each other, and make sure that all fixtures are stateless. In this way, the fixtures can stay simple and we were able to optimize the tests to make all the test run under 5 minutes.

For each project, we designated one suite as our acceptance tests area. Once a tests pass, we move it in there so that it never fails. In this way, the QA can feel free to write any tests they want and still can check in.

Always remember, it is nice and cool to do everything, but it is worth the effort only when the benefit outweighs the cost. We didn't have Fitnesse tests for a project that involve Peoplesoft because it uses views that are oracle specific. We resort to jWebUnit tests for another webapp because it was not as easy to use jWebFit (However, with the helper classes that we developed later on for web tests, we are taking another look at it to see how much it would cost now. The QAs is getting a hang of automated testing and starts feeling bad about not able to write them for Web apps).

References

• QA: Quality Assurance
• BA: Business Analyst role in a development team. (http://www.xprogramming.com/xpmag/BizAnalysis.htm)
• Fit: The framework for integration test that Fitnesse is based on (http://fit.c2.com/)
• Other nice tools for web testing
• Watir: http://wtr.rubyforge.org/
• Selenium: http://selenium.thoughtworks.com/ (A movie demo)
  
• Samie: http://samie.sourceforge.net/

ThoughtBlog

Adewale Oshineye sent me an email because there was a link from my profile page on ThoughtWorks linking to this blog, to ask me if I want to add mine to the ThoughtBlog. The reason that I have not done it is because I want to start publishing this blog only when I really know what kind of blog I am going to keep here, or even if I think it is worth it to keep a blog about my experience in ThoughtWorks.

But I figure, help is already here it really does not make sense to turn it down so I replied yes. Then I figure it is time to subscribe to the blog feed to see what my fellow ThoughtWorkers are writing about, especially with ThunderBird it is so easy.

So already I have learned something new today! I thought I have known all that is to know about writing assertions (assertEquals, assertNotEquals, assertTrue, assertFalse, simple, eh?). Boy was I wrong. Joe's blog "Flexible JUnit assertions with assertThat()" sure opened up my eye.

When I was working for JBuilder, I wrote a test framework for unit-testing some part of JBuilder modules. When a test fail, it is always desirable to see as much information as possible in the error message regarding information like project configuration, run configuration, etc. However, JUnit requires you to provide a message before the assertion happens, which means that all the tests will need to spend a lot of time constructing the messages, even though they might not be needed. So I wrote this LazyAssert that only constructs the message if the assertion fails, and ended up duplicating a lot of Assert APIs. I was quite bugged by that result but concluded that there was nothing I can do.

And now I know the right way.

My hats off to Joe for sharing it with us. And again to my fellow ThoughtWorkers.

Update of an example:

Check this out:

before:

junit.framework.AssertionFailedError
...(Click for full stack trace)...
at com.company.product.person.dao.jdbc.JDBCAffiliationDAOTest.testInsertAffiliation(JDBCAffiliationDAOTest.java:92)
...
at org.jmock.core.VerifyingTestCase.runBare(Unknown Source)
...

after:

junit.framework.AssertionFailedError:
Expected: timestamp is after 2005-05-16 15:21:34.871
but got : Mon May 16 15:21:34 PDT 2005

...(Click for full stack trace)...
at com.company.product.test.BaseAsserts.assertThat(BaseAsserts.java:31)
at com.company.product.test.BaseAsserts.assertThat(BaseAsserts.java:23)
at com.company.product.person.dao.jdbc.JDBCAffiliationDAOTest.testInsertAffiliation(JDBCAffiliationDAOTest.java:93)
...
at org.jmock.core.VerifyingTestCase.runBare(Unknown Source)
...

Fellow ThoughtWorkers

One thing that I found different working at ThoughtWorks, are the fellow ThoughtWorkrs. Here are three examples:

Names
I am the kind of person that cannot remember people's name unless I get to know them personally. So even though I greatly enjoyed the book Patterns of Enterprise Application Architecture, I only know the name of the main author Martin Fowler. Well, after working at Yabe project, I picked it up one day again to look up a pattern. Guess how surprised for me to find out that of the people that I worked with, three are on the cover (David Rice, Edward Hieatt, Robert Mee). Even though only Dave works for ThoughtWorks, I think this still shows that this company is well connected.

Arguments
ThoughtWorkers are all opinionated and love a good argument. Someone told me this the first week that I joined, and I have seen it again and again. I think this is good because it will bring out the best from all sides so that we can always think of the way we look at things or do things. Of course, I can understand that it is not for everybody, even though I enjoyed every argument.

Humorous
Here is a good example. During a discussion about the practical use of loosely coupled architecture like web services, EJB, or CORBA, Dan North made the following comment that cracked me up:

You mean you don't use CORBA?? Are you mad??

How on earth do you instantiate an object without first looking up a Factory Factory using IIOP to register as a Listener for a Factory to call a Factory Method via reflection, if you don't have CORBA?

Bloody amateurs.