Monday, September 25, 2006

Stability Can Be Stressful

Last week I got the bright idea to add RAID-1 (mirroring) to the pile of moving parts on my intranet / backup server in the home network.

Adding RAID to the kernel and configuring root-on-raid went quite smoothly. It was probably because of this that I got the bright idea to do LVM-on-raid on a Thursday evening instead of over a long weekend.

Migrating LVM started by creating md1 on hdc4 via mkraid. This was then added to LVM through vgextend. (hdd is not yet in the case and that does complicate things a bit.) Now I needed to move the data from hda4 onto md1. This is where things began to go very badly...

For reference there appears to be a bug in pvmove 2.02.05. pvmove will move data from one physical volume in a volume group onto another physical volume. In my case, I wanted to move from hda4 to md1 so that I could then add hda4 to md1 thus mirroring the data across both hda4 and hdc4.

Keep in mind that the whole point of the operation is to add stability and reliability to the system...

Due to the (alleged) bug I got kernel panics and a solid HDD activity light. No chance to reboot, complete system hang on any attempt. The only solution was hardware reset.

Now you have to realize that this box is the backup- and file-sever for my home network. It has backups of all of the other hosts and some data that isn't found anywhere else in the network. As it turns out the most critical (and irreplacable) logical volumes live(d) physically on hda4.

After the hard reboot I decided that even though I had cross-host backups it might be wise to do another of these critical bits. Unfortunately, pvmove was still trying to do its job even after the reboot and any attempt to read the in-move filesystems resulted in yet another kernel Oops and solid HDD activity.

Hours transpire during which I try any number of things to cancel the pvmove (including pvmove --abort) to no avail. I finally discovered that there is a version 2.02.06 of pvmove and immediately upgraded. To my horror I could no longer even identify the in-move logical volumes. Evil ioctl error messages accompanied any attempt. Fortunately, at this point an --abort did work and I was finally able to see everything.

I decided then and there that pvmove is not my friend and I don't want to associate with it any longer. I did what any sensible person would do... I created parallel filesystems for each one I wanted to move and used my old, reliable friend rsync followed by lvremove of the old volumes and lvrename of the parallels. And you know what? It all worked perfectly.

So this is a quick post covering about three and a half hours of sheer terror. Now that its all done (and hdd to be added very soon) I'm quite happy with the results. I feel much more confident that I can loose any one of the four drives and not loose a bit of data. I'm glad to be where I am. I just wish the road had been a bit smoother...

(And, oh yea, I'm rsync'ing the latest backups to my dev box every night and have no plans to stop...)

Monday, September 11, 2006

Wired to Win

"Wired to Win"... These were some of the last words spoken by our worship pastor Rick Pearson before he went home to Christ last summer. Rick was only 23 when he was hit suddenly and fataly with leukemia. Through it all he never lost faith and he never questioned God's will.

Rick's diagnosis and treatment wasn't cheap. His family racked up a lot of bills along the way. In September of last year his sister and some friends setup a benefit concert. Through that they raised not only enough money to pay the debt but also to continue to contribute to several awsome organizations Rick supported: Compassion International, Passion, Big Stuff and Mountain Lake Church as well as the Leukemia and Lymphoma Society.

At Mountain Lake Church Rick touched hundreds of people, maybe even a few thousand, in our community. Rick's Legacy is now touching thousands around the world. We may wonder why such a young, talented man of God would be called home at a seemingly untimely hour. We just attended the Second Annual Rick Pearson Memorial Benefit Concert and heard there about the amazing things done with the funds raised by last year's event. Perhaps Rick's task was just too big to be done here on Earth.

Like so many others influenced by the events in Rick's life I wasn't there when the words were spoken. In my mind I imagine the conversation went something like this: "If I get well and am with you all for many more years, I win. If I don't and go to Heaven, I win. No matter what happens, I'm wired to win."

Wednesday, September 06, 2006


For a long time now I've been following a software development methodology with no name. A few months ago a colleague and I were discussing such things and he challenged me to formalize my thoughts. I present it to you, today for comment and consideration.

The Three M's (or M-cubed or M^3 or, if you like, The M-Factor) are (or is) quite simple:
  • Make it Work
  • Make it Better
  • Make it Right
There is nothing intricate, mysterious or complex here. It is just as you read it. In fact, most of you reading are probably responding with "well, duh" or something similar. And so you should. Any develompent methodology should be simple and non-intrusive and I believe that M^3 captures that sentiment.

Make it Work

Many teams fall into an analysis paralysis. You've heard the term. We've all been guilty of it at some point. The spirit of Make it Work is to put all of that aside and just get something done. Do whatever it takes to have something you can touch and see. At this point I don't care if you have scriptlets in your JSPs or hard-coded magic values in your Java classes. Just get something (anything) done so that what you have in your head can be shared with others. In some cases you may find that what the feature you're working on isn't needed at all or may be needed in a different way than you imagined. Best to find that out early, after a quick hack than later after a lengthy, yet perfect, implementation.

The challenge, of course, is to not stop at Make it Work. Too many times we see junior (and not so junior) developers quite satisfied with solutions at this phase. Resist the urge to move on to the next thing before ensuring that the current thing is done correctly.

Make it Better

This is where you'll spend most of your time in the process. Revise your Make it Work results to get rid of the truly embarrasing things first (e.g. -- scriptlets, hard-coded values and whatever else). If you're not following Test Driven Design then now is the time to start writting those unit tests. (If you are doing TDD then you already have at least a scaffold of tests as a result of Make it Work and I applaud you.) Next look for opportunities for pluggable strategies. Then consider repackaging for reuse. Apply design patterns. Etc... In other words, iterate through Make it Better as many times as necessary to reach a point of "good enough". The challenge, of course, is knowing when enough is enough.

Many people will get hung up in Make it Better. How many times have we seen the tinkerer refactor the same bit of code so many times that the end result looks nearly exactly like the original? Coding in circles isn't very productive... Refactoring is a good thing. A very good thing. Just don't get caught up in the excitement of it all and loose track of the fact that you have to deliver something eventually.

Make it Better is also the place where you will respond to changes in your requirements. Not wholesale changes along the lines of "you're building a house but we want a boat" but those "oh, it works *that* way..." kinds of changes. In Make it Better your code is fluid and can easily respond to small to middlin' changes that are, themselves, responses to the evolution of the solution.

Make it Right

Once you have finally satisfied yourself that the result is good enough (or when someone tells you to quit refactoring for refactoring's sake) you have reached Make it Right. This is the exact opposite of our initial Make it Work. Now we want to do one last pass over the many iterations of our second phase to make sure that we have taken every reasonable (though not every possible) opportunity to make our solution "right". Is it as extensible as it should be? Is it pluggable where appropriate? Etc...

It is important to realize that Make it Right is not the same as Make it Perfect. There is no such thing as perfect software. Your Make it Better iterations will eventually reach a point where further refinement adds no benefit. At that point you're ready to enter Make it Right where you can "dot the I's and cross the T's".

Does M^3 fit with other methodologies? Of course it does. At its heart M^3 is about agility. It is about getting something done as quickly as possible, improving and adapting and putting a nice shine on the final solution.

So, there you have it; the methodology I've been following without realizing it for probably twenty years or so. In a lot of ways M^3 mirrors our natural learning process. Since much of development is the process of learning what the customer really wants it only makes sense that our development process follow a similar approach.

(Thanks goes to Scott, my friend and co-worker, who challenged me to actually think this through and formalize my thoughts.)