I recently gave a talk on unit testing with Jasmine. Below are the slides.
Update : I have uploaded the files I used for my live demo to github : Jasmine-rhino they include a sample project and a runner script ./run.sh which will easily integrate with Jenkins.
Want to warn people when they commit large amounts of changes to your repo to keep commits more atomic?
Use the bash script below in your post-commit hook on SVN to count the number of files modified, added and deleted in one commit, then give a warning if the number is too high.
#!/bin/sh
REPOS="$1"
REV="$2"
MAXFILESCHANGED ="5"
LINECOUNT=$(/usr/bin/svnlook changed -r $REV $REPOS | wc -l)
if [ "$LINECOUNT" -gt "$MAXFILESCHANGED" ]
then
echo "Whooaaaa there, crazy horse."
echo "Your commit was a success, but you made alot of changes."
echo "Try to keep your commits as small as possible."
echo "Thanks for your diligence." 1>&2
exit 1
fi
Committing changes more atomically reduces risk and chance of critical failure by pushing smaller units of consumer value through the pipeline serially rather than in parallel, helping to increase throughput.
Someone just came up to me and told me to load maps.nokia.com to see how fast it had got after our new server migration.
I opened up a new tab, typed the first few keys of the URL and bam the site was there - super fast! Wow! I check the network tab, 1.47seconds - amazing!
Then something strange happened, I cleared my cache and loaded the page again, no surprise, it took a bit longer without a primed cache. The strange part was when I reloaded again to get the super fast version. It was no longer super fast, it was a second or so slower than the un-primed version.
That’s weird, I thought.
I loaded again, no change. More weird.
I noticed that all the requests in the network tab had gone from being cached responses to 304’s so there was a server hit after all - but we have far future expires headers - this shouldn’t happen?
After some googling, I discovered my answer on stackoverflow.
When you use CTRL-R to reload a page, most browsers will send a 304 even to cached resources.
So as it turns out, the page was always caching resources and was always returning in 1.47 seconds, but ONLY if you reload the page by re-entering the url. Mystery solved.
This took some time to figure out so just wanted to share. Many thanks to Ben Barnard for providing final sounding board and Xargs wisdom.
find . -type f -mtime +10 | grep -v \.svn/ | sed 's/ /\\ /g' | xargs svn delete
Explanation:
find . -type f -mtime +10
This executes the find command, firstly on the current directory (.) then specifies type of file, in this case regular i.e. not symbolic link (-type f) then specifies the last modified time which is in 24 hour periods, in this case ten 24 hour periods - 10 days (-mtime +10).
grep -v \.svn/
Here the input is then piped into Grep, which removes any files with “.svn” in the path with the option (-v \.svn/).
sed 's/ /\\ /g'
Then, in our case we had spaces in some of the directory names, this broke the final SVN command, so we have to replace them using sed to be escaped.
xargs svn delete
Finally we pipe that output into xargs which calls svn delete with our cleaned up list and removes all files older than 10 days from svn. A final commit back to the repo and we are done.
Ok, so earlier, I started investigating how to write a geohash algorithm in Javascript. I’m sure you’ve been waiting in eager for the results. Well dear readers, now is the time.
Below, is a geohash algorithm, based on the Maidenhead Locator system but with a few tweaks.
Firstly to get more accuracy with less characters, I replaced the complicated Maidenhead codes for different characters with one set of 64 different characters for each character. They range from the lowercase alphabet to the uppercase, then the base 10 numbers, and the greater than and less than symbols.
With this method, effectively the globe is split into 4096 squares (64×64) and then the longitude and latitude are interleaved together, so every odd character is Longitude and every even character is Latitude. The process is then recursive, based on the level of precision you set. So after the first square is picked - 2 characters, the selected square is then split again into 4096 squares and stored as another 2 characters, this goes on so on and so forth.
A 1 degree arc of the earth’s surface is roughly 111,000 meters long, which means you need 5 decimal places to get around 1 meter worth of accuracy with your location. The following code takes a 5 decimal place longitude and effectively transforms it into a geohash and back again.
The result it, we can take the latitude, 47.64932, convert it into a geohash of kwGK and then back again to 47.64932.
In part 3, I will looking at creating a formal proof of this algorithm, possibly using induction proofs.
var lat,
latMin,
latMax,
base64A,
reduce,
calculateBoundaries,
reverseGeoHash,
geoHash,
reversed;
latMin = -90;
latMax = 90;
base64A = "<>0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
lat = 47.64932;
reduce = function(boundaries, bucketSize, value, precision, geohash) {
boundaries = calculateBoundaries(boundaries, bucketSize, geohash);
var bucket = Math.floor((value - boundaries.min)*(bucketSize/(boundaries.max-boundaries.min)));
return precision-1 > 0 ? base64A[bucket] + reduce(boundaries, bucketSize, value, precision-1, base64A[bucket]) : base64A[bucket];
};
calculateBoundaries = function(boundaries, bucketSize, geohash) {
if(geohash.length > 0) {
var lastGeoHash = base64A.indexOf(geohash[geohash.length-1]);
var range = boundaries.max-boundaries.min;
boundaries.max = (lastGeoHash+1)*(range/bucketSize)+boundaries.min;
boundaries.min = (lastGeoHash*(range/bucketSize))+boundaries.min;
}
return boundaries;
};
reverseGeoHash = function(boundaries, bucketSize, geohash) {
for(var i = 0; i < geoHash.length; i++) {
boundaries = calculateBoundaries(boundaries, bucketSize, geohash[i]);
}
return Math.round((((boundaries.max-boundaries.min)/2)+boundaries.min)*100000)/100000;
};
geoHash = reduce({min:latMin, max:latMax}, base64A.length, lat, 4, 0);
reversed = reverseGeoHash({min:latMin, max:latMax}, base64A.length, geoHash);
console.log(geoHash, reversed, lat);
Sometimes you find yourself with a random computing science seeming problem that you just want to roll up your sleeves and get you hands dirty with. I had one of those moments, about 4 months ago.
Now, four months may seeeem like a really long time, but as a classic lean saying say’s, results are not the point. It may have taken four months, but in those four months I learned some valuable lessons about computer programming. I thought it would be worth sharing them.
The first part of the process is to frame the problem. On maps.nokia.com we have some functionality that means whenever you drag the map, the url bar updates with the new latitude/longitude.
Which is fine.
Unless.
You live in China.
In China you are not allowed to display latitude/longitude. Instead you must hide it from the sight of your viewers, incase they see it and realise the world is not the distorted place they’ve been told it is by their government. Go figure. So, my problem was how to save the location, via the url bar, without using the lat/long.
Googling this topic leads you to three likely looking solutions.
The geohash appears to be a relatively new (2008) discovery, invented by Gustavo Niemeyer, you can find the implementation at Geohash.org.
I spent sometime replicating this technique in Javascript, essentially, it involves, systematic storing of deltas (or I guess diff’s for people who know source control). The starting premise is that for example, a latitude exists between -90 and 90 degrees. The first delta identifies whether it is in the first or second half of that range. So if I took the latitude, 52.51607, I would programatically ask, is that between -90 and 0 or 0 and 90, clearly it would be the latter so my first delta would be 1, if it had been -52.51607 then my delta would be 0. Doing this recursively I can then create a pattern of 1’s and 0’s that slowly drill down to my desired latitude. Then, once i’ve done that, I can take that binary string and convert it to integers between 0 and 32 and then using base32 represent it as a string.
It means for the coordinate pair 57.64911,10.40744, you end up with a geohash of u4pruydqqvj which is a saving of 6 characters (including points and the comma delimiter). Not bad. But what are the other options?
Working for Nokia, I have luxury of being able to tap into a vast treasure trove of both Microsoft and Nokia patents without the fear of being sued. Now, I’m not saying I think software patents are a good thing, but it was a relief when I found this patent which is described as compact text encoding of Longitude/Latitude was owned by Microsoft rather than Apple. It seemed to be exactly what I was looking for.
This technique takes the long/lats and converts them into non-negative numbers. So for example 47.64932 converts to 18,583,657 to be honest, don’t ask me why this works, I followed the formulas in the paper and boom, got the right number, but the theory behind converting to a non-negative number is lost on me. Then with this integer, you can generate a base-n (so in this case they used base32 again) string.
So with the example co-ordinate pair, 47.6493, -122.12926 you end up with a hash of ry7cx4tp95, a saving of nine whole characters! Pretty damn impressive, but there was one final method I wanted to research, mainly because it was named after a place in my homeland.
Developed in 1980 in Maidenhead, England, this system was designed by VHF managers and used by amateur radio enthusiasts. The Maidenhead Locator system was created with the use case of being able to transmit a location within 12km accuracy in a simple 6 character string using morse code.
It essentially works on the idea of converting the world map into a grid, based on degrees and assigning them a letter or a number. Each character in the string and precisely positioned and different locations have different boundaries. An example of a Maidenhead Locator system hash is:
BL11bh16
Stolen from wikipedia, below is a run down of what each of the characters does.
Interesting, a system invented in the 80’s that achieves a small, human readable hash of a lat/long with only 8 characters but alas with one large draw back, which is that it’s not as accurate as I need.
Having invested a large amount of time replicating in Javascript each of these techniques, I realised that what I need was a hybrid system, using the Maidenhead locator concept as a basis. In part 2, I will explore the javascript I need to write a geohash which matches the precision of the Geohash and Microsoft patent solution with the number of characters from the Maidenhead system.
In order to understand something, sometimes it’s easier to understand it by defining what the opposite is.
Recently, I was trying to explain to someone the fundamental principle behind continuous improvement was. Do you ever run into the scenario where you find yourself in a conversation and you hear:
“We need to build an X, it will solve all our problems and make our code bug free.”
Conversations like this, immediately sound off alarm bells in my head, because what’s essentially being suggested is a magic potion, a concoction to cure all ills, an elixir of eternal life or just plain old powered tiger scrotum.
The overuse of hyperbole and setting truly unrealistic expectations in software development only results in continuous disappointment, and slowly kills a team spirit and stakeholder trust.
Aim for steady incremental continuos improvement with small achievable micro goals, know when to pat yourself and your team on the back when they have achieved them, and make sure going off solo on a magical quest to slay a dragon get’s stigmatised and not rewarded.
I really like stories.
In fact, I more than like them. Stories are an amazingly versatile tool for passing on wisdom, knowledge, ideas and illustrating principles in a way everyone can understand. They can be especially useful in web development and I try to use them as often as possible for just this purpose, even if the story is not about development itself (indeed sometimes it’s better if it’s not).
So I was delighted when I came across a story on Aza Raskin’s blog that illustrates beautifully the principle of continuous deployment using the topic of airplane construction.
Read it. Absorb it. Pass it on to your team, and make sure you’re solving the right problem…
For the last few weeks, my team has been working on some pretty fundamental changes for Nokia maps on the web.
But that’s the short term. For the last six months my team has been working hard on getting a suite of automated tests up and running, and I’m proud to say we now have close to 1000 Jasmine unit tests and 200 cucumber acceptance scenarios running on every commit.
So I was delighted, when I heard that this week, one of the new changes had broken 52 of our acceptance tests.
Why should I be delighted?
Doesn’t that mean the team now has to go away and fix 52 acceptance tests? Well, yes, and that was certainly the way the team saw it, until I said:
“Now imagine we didn’t have those tests. What would have happened?”
And then we all started to realise, the power of automated testing.
So if we didn’t have automated testing, what would have actually happened?
Well, someone would have gone into the code. Found the piece of code that needed changing. And changed it. Probably tested it in the one place they knew about. Committed it. And moved on. Completely oblivious to the fact that they’d broken the application in 51 other different places.
Then the build would have gone to QA, the QA’s would have found the 52 different breakages, raised 52 new bugs, and sent them back to the developers.
Then the developers would have gone into the code, tried to fix those 52 bugs, and probably caused another 50 regressions. Which would then go to the QA team, be raised and go back to the developers for fixing. Eventually this hugely inefficient cycle would have continued until the build gets to an extremely brittle stability, a bit like building a house of cards, and it would have gone live.
However, with the tests, those bugs were caught. Immediately. And fixed. Immediately. Those 52 broken acceptance tests, saved us potentially months and months of rework.
So the moral of the story is, when you reduce the feedback cycle of a bug, with the saftey net of automated tests, it allows you to develop in confidence, giving you warning to what you’ve broken and giving you the opportunity to fix it then and there (however having the discipline to do this is another story). In Lean, this is referred to as ‘building integrity and quality into the product from the beginning of the cycle’, rather than trying to add it in at the end with a torturous QA cycle.
I thought I would share the slide deck for a short talk I held on the truth about innovation in agile environments.