From like minded friends over at the Semantic Foundry posted a fantastic video about all things System theory-ish. So I thought I would repost.
There’s an amazing section at 13:20 from Russell Ackoff about the truth behind some behaviors of large organisations. Pure gold.
I quoted Conway’s law, not so long ago on this blog:
Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.
But now I think about it, it doesn’t go deep enough, I would add this:
Further to this, any organization with a strong hierarchy (defined broadly) will usually have a communication structure that represents the mental state of the highest node of the hierarchy.
What this means very simply is, in a strong hierarchy, if you have a leader with strong silo-ing of the faculties of the brain, so someone who doesn’t balance well between creative and logic, then you will have a company that separates itself into Silo’s of creativity and logic. If you have a leader that does not learn, then the communication structure will represent that in-ability to not learn by never improving. Conversely if you have a leader that’s for example well organized, or has a strong sense of social justice, then the communication structure of the organisation will be well organized, with good channels from top to bottom to allow the flow of communication.
Amazing video. How have we gone so long, understanding this, without implementing it.
Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.
In order to understand something, sometimes it’s easier to understand it by defining what the opposite is.
Recently, I was trying to explain to someone the fundamental principle behind continuous improvement was. Do you ever run into the scenario where you find yourself in a conversation and you hear:
“We need to build an X, it will solve all our problems and make our code bug free.”
Conversations like this, immediately sound off alarm bells in my head, because what’s essentially being suggested is a magic potion, a concoction to cure all ills, an elixir of eternal life or just plain old powered tiger scrotum.
The overuse of hyperbole and setting truly unrealistic expectations in software development only results in continuous disappointment, and slowly kills a team spirit and stakeholder trust.
Aim for steady incremental continuos improvement with small achievable micro goals, know when to pat yourself and your team on the back when they have achieved them, and make sure going off solo on a magical quest to slay a dragon get’s stigmatised and not rewarded.
I really like stories.
In fact, I more than like them. Stories are an amazingly versatile tool for passing on wisdom, knowledge, ideas and illustrating principles in a way everyone can understand. They can be especially useful in web development and I try to use them as often as possible for just this purpose, even if the story is not about development itself (indeed sometimes it’s better if it’s not).
So I was delighted when I came across a story on Aza Raskin’s blog that illustrates beautifully the principle of continuous deployment using the topic of airplane construction.
Read it. Absorb it. Pass it on to your team, and make sure you’re solving the right problem…
For the last few weeks, my team has been working on some pretty fundamental changes for Nokia maps on the web.
But that’s the short term. For the last six months my team has been working hard on getting a suite of automated tests up and running, and I’m proud to say we now have close to 1000 Jasmine unit tests and 200 cucumber acceptance scenarios running on every commit.
So I was delighted, when I heard that this week, one of the new changes had broken 52 of our acceptance tests.
Why should I be delighted?
Doesn’t that mean the team now has to go away and fix 52 acceptance tests? Well, yes, and that was certainly the way the team saw it, until I said:
“Now imagine we didn’t have those tests. What would have happened?”
And then we all started to realise, the power of automated testing.
So if we didn’t have automated testing, what would have actually happened?
Well, someone would have gone into the code. Found the piece of code that needed changing. And changed it. Probably tested it in the one place they knew about. Committed it. And moved on. Completely oblivious to the fact that they’d broken the application in 51 other different places.
Then the build would have gone to QA, the QA’s would have found the 52 different breakages, raised 52 new bugs, and sent them back to the developers.
Then the developers would have gone into the code, tried to fix those 52 bugs, and probably caused another 50 regressions. Which would then go to the QA team, be raised and go back to the developers for fixing. Eventually this hugely inefficient cycle would have continued until the build gets to an extremely brittle stability, a bit like building a house of cards, and it would have gone live.
However, with the tests, those bugs were caught. Immediately. And fixed. Immediately. Those 52 broken acceptance tests, saved us potentially months and months of rework.
So the moral of the story is, when you reduce the feedback cycle of a bug, with the saftey net of automated tests, it allows you to develop in confidence, giving you warning to what you’ve broken and giving you the opportunity to fix it then and there (however having the discipline to do this is another story). In Lean, this is referred to as ‘building integrity and quality into the product from the beginning of the cycle’, rather than trying to add it in at the end with a torturous QA cycle.
I thought I would share the slide deck for a short talk I held on the truth about innovation in agile environments.
I can’t count the number of times I think I’ve been in this situation. You get that sinking feeling in your stomach. Your neck muscles tense. You clutch at the last shreds of hair you have and rub your forehead with your palms. You look up, and bearing down on you is a mountain. You know the peak well, you know it’s piled high, higher than the clouds you can see. Welcome to spagetti code mountain.
Never ending loops, endless switch statements, functions calling infinite functions, cryptic variables, duplication after duplication, premature optimization, you name it, it’s there.
I’ve been in this situation before, but everytime I come to it, I find myself starting from scratch how to deal with this problem. The feeling of hopelessness when opening up your editor to find a x-thousand line file is overwhelming. My first reaction is almost instinctive to every programmer, control-a, delete and start again. However, this approach is flawed. When looking at code you have to remember, that the person who wrote it, was doing his best in the circumstances he was in, with the knowledge he had. Maybe the specification changed alot, maybe he was going through a tough divorce, maybe he hadn’t been trained in the particular skills need to tackle this problem. Whatever the circumstance, the fact is that there are lessons and behaviours in that messy code that you want to keep and it’s your job to tease them out and clean them up.
The last two weeks, i’ve been getting my hands dirty again, and I though I would share my methodology for refactoring legacy code. So how do you start?
First, you’ll never grasp everything at the beginning, but it’s good to get an idea of the depth of the piece of code you have, how many tentacles it has into different parts of the codebase, which of those tentacles in turn have tentacles? Imagine you were hired to clean a house, you would first go into the house and look in each room. While doing this cursory check you would look for the messiest rooms, the biggest rooms, rooms that come off other rooms, which rooms you could clean easily versus which would take a large effort for a minimal gain. Make a note in a pad with a pen, draw some scratchy boxes with the names of important functions and how they link together, it helps to record your thoughts before you get stuck into the chaos of someone else’s code.
Next, try to take a behaviour and understand it, preferably the most important or most broken one. Ultimately you should approach it from the user’s point of view, ‘As a user when I do X, then Y should happen.’ so start from X and trace it back, what does X do, what does it fire, how does it get there?
This stage is just a spike, so start experimenting. If I comment out this line, does it stop doing the thing I expect it to? Don’t spend too long on it, just to get a feel that you can and have some sort of control of the codebase, and even though it’s still monstrous you can begin to make it make some sense and get a feel for how it’s put together.
So, I just want to say now YOU ARE NOT READY TO REWRITE ANY CODE YET. DON’T EVEN THINK OF PERMANENTLY MODIFYING the code base.
When you’ve got a feel for the path through the code you are going to focus on, then you can prepare for the next stage. If you have made any experimental changes, REVERT THEM NOW. As they will only serve to confuse you later on.
The next stage is tests. Every time you modify the codebase, it’s like walking a tightrope. Below the tightrope is a deep chasm with spikes and rivers with hungry snapping crocodiles in, which if you fall into you will struggle to climb out of. Every automated test you write is like creating a string in a net designed to catch you when you fall, allowing you to climb out faster.
Your first test should be acceptance tests. Take the user flow you’ve just spiked, go get a QA and a Designer/IA, sit down to hammer out some acceptance criteria and write a test in Jbehave or Cucumber or one of the many other BDD testing solutions available to the modern programmer, check it works and add it to your automated build. There, you’ve just created your basic safety net, it’s not indestructible and is pretty low down, but it saves you from the crocodiles.
Now personally, opinions differ on what to do next, but my preference is at this stage to do some tidy up.
Now what I mean by tidy up, is very specific, it’s changes to the code that don’t affect the actual execution of the code. So formatting basically, fixing whitespace issues, line breaks, tab indents for example are a good thing to cover at this stage. Removing unhelpful comments is also good, and finally, I like to go through, method by method, variable by variable and check for dead code that’s never called (a good editor is vital at this stage to make sure you don’t remove something referenced in anther file in your project). You can also start to mark things that you think are worth refactoring. It’s a bit like going into the room you are about to clean and removing all the objects that don’t belong there and will be thrown away, as there’s no point in cleaning something that’s going in the rubbish bin.
When that’s done, run the tests, if you’ve not changed anything that affects the execution of the code, everything should be fine, behave how you expect it to and pass so you can now commit your tidied code.
Finally we can start to think about making some changes, but not so fast my tightrope walking friends. You forgot your old buddy the unit test. For every method you touch, write a unit test to test the existing functionality. Your unit tests are another safety net under the tighrope, but this net is much closer to the tightrope, so if you fall off you can litterally just jump back up onto the rope and carry on, not like the acceptance tests, where you have to climb back up the wall of the chasm.
Then once your methods test is in place, then you can think about breaking it and refactoring that method. First thing I like to do is go through and look at all the variable names, do they make sense, can I by reading the method understand exactly what it’s doing? Is it good english? Does it make sense? Having a pair is vital at this stage to bounce language off and make sure what you write is understandable for someone else.
Now, your finally ready to do the majority of the damage, and you can begin to decimate your method and reduce it to the bare essentials (still readable obviously) eliminating all the code detritus and reducing the entropy. Once your happy, run both your set of tests, commit it and move to the next method. Congratulations you made it over your first pass of the tightrope and you didn’t end up lost in the chasm of code chaos.
Happy refactoring!
Ever hear of the Two Standard Deviation rule (or the 2SD rule for short)?
The rule states:
There’s a 95% chance that a normally distributed variable will fall within 2 standard deviations (plus or minus) of it’s mean.
I was recently in a meeting where we were trying to predict velocity of a team. The team had collected 6 sprints worth of data, where they had completed, 20, 16, 5, 6, 4 and 12 story points (why the distribution was like this is another story).
Now the mean of the sprints is ((20 + 16 + 5 + 6 + 4 + 12)/6) = 10.5, which at first glance is what you might expect the velocity of the next sprint to be. But I wanted to go a level deeper and get a better idea of how accurate that was, so I used the 2SD rule.
Firstly calculating the standard deviation, I decided to write a small JS function to do it for me, if anyone can write a better one, I am always happy to see it…
(function(data) {
var average, standardDeviation, i;
for(i = 0, average = 0; i < data.length; i++) {
average += data[i];
}
average = average/data.length;
for(i = 0, standardDeviation = 0; i < data.length; i++) {
standardDeviation += Math.pow(data[i]-average, 2);
}
standardDeviation = Math.sqrt(standardDeviation/data.length-1);
return {average : average, standardDeviation : standardDeviation};
}([20, 16, 5, 6, 4, 12]));
So now we have 2 key pieces of data, the mean 10.5 and the standard deviation 5.9, lets apply the 2SD rule.
So standard deviation is 5.9 points multiplied by 2 is 11.8, making my minimum velocity -1.3 (is it possible to have a negative velocity?) and my maximum velocity 22.3.
The outcome of this is that I can say I am now 95% sure that the velocity will lye between -1.3 and 22.3 points for the next sprint.
Ok, that’s maybe not so useful, but it could be if I had more data and a slightly smaller bell curve (but it looks like we’re going through a Satir change curve at the moment).