Walking off, and Going to China Project

This is one of the times that I am proud to be a ThoughtWorker.

We have been working with the current client for over a half year (I joined three months ago). The agreement has always been that this is a transformation (formerly known as enablement) project, that, given the clients' background of waterfall approach and ThoughtWorks' experience in agile development, should rely highly on ThoughtWorks' input. That is, for a start, during the scoping phase form a small group (6) and have one on one ratio of ThoughtWorker and the employee from the client.

But apparently someone thinks that if they pay us they own us. Before the pilot project even starts (hence gets the requirement), there is already order from high up to use one of those so called "J2EE framework server". And during the initial phase, there are going to be 9 people instead, with only 3 from ThoughtWorks. All these decisions are made without even consult ThoughtWoks (you would think that should be the reason to hire a "consultant"). And no reason has been given for these decision.

So as the current finishes this month. The manager decided not to renew it and walk off from a project that is not built on a solid ground of trust and understanding, even thought the rate on this project is fair and we might have a good account. This decision got the support from the headquarter and the four of us are already on the list of available consultants for October.

As for me, I am really glad of the decision being made, because that shows that we are serious about being good at what we do, and the we care about have happy consultants on the project.

Another good reason for me to be happy is that the moment my name come up, there is already a good project for me to go -- China. Because of this and last transformation project that I worked on, and the fact that I speak Madrine, I turn out to be a good fit for this project that needs someone to come in and help out. Time is short so I am busy working on the visa and travel plans.

At this moment, my goal of China trip is the following, I am still updating it:

• Get to know the ThoughtWorkers in China: I have heard of great things about the people that we hire there. I need to spend as much time as I can to get to know them and exchange our experiences to learn from each other.
• Get to know the market in China: Despite the great potential in Xi'an, I still think there should be a lot of opportunities in Beijing and Shanghai. Agile software development in China is getting a lot of attention, if we don't grasp the market and build our good names, there will be someone else very soon.
• Get to know the IT environment: There might be a lot of things that I am taking for granted here that is not as available. Programming in a non-English speaking environment might bring different dynamics.
  

Deployment Script with Watir (Take THAT, Websphere!)

So, after some struggling with Websphere on deploying an EAR file (Don't ask me question about why we still bother, please... ) and not able to get a hand on a simple deployment script (I think there should be one but I just didn't have the time to look further), I have decided to automate the web process that I have been doing, using WATIR.

Task
Call me innocent, but I really thought that depolying an EAR file on an application server would be easy, given the 'comprehensive' file structure and all the deployment descriptors.

But no..., you have to click through this long 'wizard' (ooohhh...) process every single time. Not only you have to do it manually, it also takes quite a lot of time. And I happen to be one of the impatient guys who thinks that it is a machine's job, then a human should not suffer.

Watir
In case you haven't heard of it, Watir (http://wtr.rubyforge.org/) is a ruby library used for web testing. I am still not sure how exactly it works, but the result is that you can control an internet explorer instance using Ruby script. So basically almost anything you do with IE, you can write a script and automate it.

Code
So long story short, I was finally able to write a script that uploads the file to the server, click through the steps, and save the configurations. Needless to say, I am very impressed the progess that Watir has made, functionality as well as the documentations.

The script is actually not that long. Here it is (with comments):

require 'watir'
 
class IeLauncher
  def IeLauncher.open(*args)
    result = ie = Watir::IE.start(*args)
    if block_given?
      begin
        result = yield ie
      ensure
        ie.close
      end
    end
    return result
  end
end
 
# Ignore the above, they are just basic set up.
 
# Launch IE, go to the specified URL
IeLauncher.open('http://localhost:9060/ibm/console/') do |ie|
  # Log in, my local server doesn't have a password
  ie.text_field(:name, 'username').set('admin')
  ie.button(:value, 'Log in').click
 
  # Only one login is allowed so you will have to log out last one
  if ie.contains_text('Login Conflict')
    ie.radio(:name, 'action', 'force').set
    ie.button(:value, 'OK').click
  end
 
  # Apprantly when you deploy an application, you need to 'save' it.
  # This recovers it
  if ie.contains_text('Recover prior changes')
    ie.radio(:name, 'action', 'recover').set
    ie.button(:value, /OK/).click
  end
 
  # This get the frame that has the navigation tree and look for the right node
  navigation = ie.frame('navigation_tree')
  navigation.link(:text, 'Applications').click
  navigation.link(:text, 'Enterprise Applications').click
  detail_frame = ie.frame('detail')
 
  # Uninstall the existing deployment
  if detail_frame.contains_text('DukesBank')
    dukesbank_checkbox = detail_frame.checkbox(:value, /DukesBank/)
    dukesbank_checkbox.click
    detail_frame.button(:value, 'Uninstall').click
    detail_frame.button(:value, 'OK').click
  end
 
  # Install
  detail_frame.button(:value, 'Install').click
  detail_frame.radio(:name, 'radioButton', 'local').click
 
  # Upload the file (Note, file path cannot have space)
  detail_frame.file_field(:name, 'localFilepath').set('c:\Work\Dukes\DukesBankEAR\build\artifacts\DukesBankEAR.ear')

  # Clicking through the wizards
  button = detail_frame.button(:value, 'Next')
  while button.exists?
    button.click
    button = detail_frame.button(:value, 'Next')
  end
  detail_frame.button(:value, 'Finish').click
  detail_frame.link(:text, 'Save to Master Configuration').click
  detail_frame.button(:value, 'Save').click
  navigation.link(:text, 'Enterprise Applications').click
end

Feedback from the Client

I have meant to post this a while ago, but with the traveling my daily schedule is still not settled. So I never found a good time that I can just quiet down and write some blog.

This week is a short week due to the past long weekend. So we were able to convince the client that it is better for us to work from our home office so that we don't need to spend two days traveling for a four-day week. And once again, I am able to continue my routine of blogging.

After my last enablement project (now the term is "transforming" because "enablement" has been used along with "training". I will explain that later), the client team had a project retrospective, discussing what they have learned during the project.

With their still fresh memory, they were able to extrapolate the following, with which they gave the presentation to the rest of the organization. They are nice enough to let me publish this. I hope this can be useful for some of you who are still wondering about agile software engineering, or at least some of the "weird practices" that I have been defending from time to time. This is the first-handed note. Hopefully I can help them revise the format for more formal presentations.

Agile: One Developer's Perspective

What methodology did we use in the past?

• It doesn't have a name, as far as I know, but here are some features of it
• big upfront requirements-gathering, big upfront design, with some formal and some informal documentation
• one developer per application, or a couple of developers working in different tiers of an application
• integration & testing in a big bang at the end
• manual testing

Some problems we had with this approach

Big upfront...

• late-breaking requirements changes combined with inflexible deadlines led to periodic frenzies of coding that eroded the integrity of the big upfront design
• sometimes we specified & designed more than we could actually implement with the time and resources we had
• integration late in the development cycle led to periodic frenzies of coding that eroded the integrity of the big upfront design
• disagreements about how open we should be to changes

One developer, or a couple of developers working in different tiers of an application...

• we didn't always take advantage of possibilities for using some common code or common approaches across applications
• difficulty scaling up for larger efforts
• the culture of code ownership by individuals was sometimes at odds with our "esprit de corps"

Integration & testing in a big bang at the end; manual testing...

• huge burden placed upon QA (when we had QA at all)
• difficulties detecting the ramifications of maintenance changes, so we became timid about upgrading applications and we ended up with multiple versions of the same module in production
• we often had as many opinions about how close we were to being finished as we had team members
• we did not know how to thoroughly test componentry; we could only test built apps

For me personally, the biggest problem was that I was unhappy. I felt that our team could be great, but we just couldn't get there, and I didn't know how to be "part of the solution".

Enter Thoughtworks, with a philosophy and a toolbox

Some things I was immediately enthusiastic about...

• Eclipse IDE (Great no matter what methodology you embrace)
• Automated testing
• Daily standup meeting
• Frequent discussions as a group about what is working and what isn't, so that we can make incremental corrections in our direction
• Realistic attitude towards estimating: as the project progresses, we can get more accurate in our predictions of a delivery date, but we can't expect perfect accuracy at the beginning.
• steady pace, avoiding individual heroics
• Simplicity--the art of maximizing the amount of work not done--is essential. (From "Principles behind the agile manifesto" http://agilemanifesto.org/principles.html)

Some things I was skeptical but open-minded about...

• TDD
• Test-driven development requires that a unit test be written for each tiny piece of functionality before you write any application code
• The unit tests are written by the developers, not QA
• Although "test" appears in its name, test-driven development is not so much about testing as it is about enforcing a development discipline that makes refactoring possible
• The unit tests serve as documentation of the developer's understanding of requirements
• "radical co-location"
• continuous integration
• fully automated build and test process that allows a team to build and test their software many times a day
• everybody using the same code base, checking in and updating their code very frequently
• at first, I was incredulous that checking in and building as frequently as once per day would even be possible; TWers viewed once a day as a minimum

Some things I was deeply skeptical about...

• paired programming
• code ownership is to be avoided
• evolutionary design through refactoring

Martin Fowler's definition

What is Refactoring?
Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations. Each transformation (called a 'refactoring') does little, but a sequence of transformations can produce a significant restructuring. Since each refactoring is small, it's less likely to go wrong. The system is also kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring.

• very short iterations
• less emphasis on formal documentation
• "embrace change"

A year later
Life on our team is completely different than before. From my perspective, it is much better.
Did the philosophy and the toolbox make a difference?
Yes, but not all by themselves. We had to have...

• the right attitude
• one-on-one mentoring from experienced agile developers
• effective leadership
• discipline

Attitude
We could not have made changes as extensive as this without

• positive attitudes
• team spirit
• perseverance

one-on-one mentoring from experienced agile developers

• from the agile manifesto: "value individuals and interactions over processes and tools"
• TDD feels initially so unnatural, that it is essential to have a patient, experienced person at your elbow
• without discipline, the agile philosophy can degenerate into "a license to hack", so it is important to have constant reminders about the discipline until it is second nature

effective leadership

• the project manager may be an exception to the general principle that individual heroics should not be required
• protected the group from distractions
• keeps both the big picture and myriad details in mind
• inspires confidence

discipline

• do not write even the smallest method without writing a test first; see the test fail, then write the minimum amount of code to make it pass
• update from the repository frequently, run the entire test suite before each check-in of your own
• builds & test suites run continuously; if your check-in breaks the build, STOP and fix
• to maintain design and code integrity, coding decisions are made in pairs...if the pair can agree, go forward

If the discipline is not observed, then we can't "embrace change" and "evolve the design through refactoring" because we can't be confident about extensive changes.


Notes

• http://www.martinfowler.com/articles/newMethodology.html#id2249560
• http://loki.lokislabs.org/weblog/archives/2003/11/15/radical_collocation_is_a_good_thing.php

Below is a summary of BTM/70, a methodology in use at a job I had 20 years ago at a company. It is probably an example of what the "value individuals and interactions over processes and tools" principle is a reaction against. All that I remember about BTM/70 is that there were big binders of sample forms, and every step of the way, one had to photocopy the appropriate forms and fill them out. I do not recall any specific benefits.
The life cycle of an BTM/70 project consists of eight (8) phases ranging from Systems Requirements to System Implementation. Coupled with BTM/70 is ET/70, a Project Control tool set that is used for project planning and tracking. ET/70 assists in milestone planning, helps establish project target dates, supports activity tracking, and detail time reporting of each staff member down to the task level, within each activity, of each phase of the project