Lateral Thinking with SQL Spatial Visualisation

Reading Time: 2 minutes

It just occurred to me that SQL’s geometry type can be used like a canvas to visualise nonspatial data in SQL Server – to make graphs etc.

Sure there are other tools, more appropriate tools (like Excel) but you don’t always have them to hand or set up. For instance all I wanted was a quick visualisation of the number of calls one of our customers was taking over time.

declare @result nvarchar(max)=N'select geometry::STGeomFromText(''LINESTRING('

select @result=@result+convert(nvarchar(14),convert(float,dy))+' '+CONVERT(nvarchar(14),ct)+', '
from
(select DATEADD(Day, DATEDIFF(Day, 0, timestamp),0) dy,COUNT(*) ct from dummydata.call
group by DATEADD(Day, DATEDIFF(Day, 0, timestamp),0)) T1 order by dy asc

select @result=substring(@result,1,LEN(@result)-1)+N')'',0)'
exec sp_executesql @result

Which produces, using SQL Server Management Studio’s Spatial visualiser…

Visualisation of non-spatial data using geometry

This is perfectly good enough to give me a general idea of what the data looks like.
The reason for the shape is that what is shown is test data – one can clearly identify the periods where testing was taking place!

Point data is, of course, easier – one can simply select the points directly out of the data set.

select geometry::STGeomFromText('POINT('+CONVERT(varchar(14),CONVERT(float,dy))+' '+CONVERT(varchar(14),ct)+')',0)
from
(select DATEADD(Day, DATEDIFF(Day, 0, timestamp),0) dy,COUNT(*) ct from dummydata.call
where timestamp>'2011-01-01'
group by DATEADD(Day, DATEDIFF(Day, 0, timestamp),0)) T1 order by dy asc

Or if we want to get really silly, we can visualise the data as the number of calls taken for each day of the week in a nice bar chart…

select geometry::STGeomFromText('POLYGON(('
	+CONVERT(VARCHAR(14),dow)+' 0,'
	+CONVERT(VARCHAR(14),dow)+' '+CONVERT(varchar(14),ct)+','
	+CONVERT(VARCHAR(14),dow+99)+' '+CONVERT(varchar(14),ct)+','
	+CONVERT(VARCHAR(14),dow+99)+' 0,'
	+CONVERT(VARCHAR(14),dow)+' 0))',0),dnam
from (select (DATEPART(dw,timestamp)-1)*100 dow,LEFT(DATENAME(dw,timestamp),3) dnam, COUNT(*) ct from cnc.call 
	group by DATEPART(dw,timestamp),DATENAME(dw,timestamp) 
	) T1 order by T1.dow

Which produces this…

Using SQL Server Spatial to Draw a Bar Chart

It’s Easy Not To Pick a Crap Password

Reading Time: 4 minutes

Most people are rubbish at picking secure passwords – “pr0gn0s1s”, for instance, is a crap password. Taking an English word or a name and then changing a few of the letters just doesn’t cut it. Why? Well, the complex explanation is given in XKCD Comic but if “bits of entropy” means nothing to you, it works like this.

Most hacking attempts aren’t a done by a smart but misguided geek, they’re done by other computers and they’re dictionary based. That doesn’t always mean an English dictionary, rather a list of words that are commonly used in passwords. A computer program works through the dictionary trying each phrase in turn. Unsurprisingly hackers are well aware that people change O to zero etc. so all those combinations are tried too. Using the password “pr0gn0s1s” will take maybe a millisecond more to crack than simply “prognosis”. Changing letters to numbers is almost no help in making a password more secure.

So how do we avoid this?

1. Use Multiple Unrelated Words

XKCD makes a good point, you can just use multiple unrelated words, spelt entirely normally. By unrelated I mean that “DavidBeckham” is 2 words but would be a monumentally terrible password. “hollowpoolbutton” is much better and much easier to remember. “The Hollow Crown” is a series of Shakespeare plays on BBC TV, “Crown Pools” is a swimming pool in Ipswich, “Poole” is the town where Jenson Button (a racing driver) wasn’t born (but I thought he was until I just Googled him). So this is an easy password for me to remember.

This might not sound like it’s better than “pr0gn0s1s” but it is, massively so. There are more than 200,000 words that could follow “hollow” and another 200,000 that could follow “pool”. That’s more than 40 billion possibilities. A bit better than the handful of different substitutions that can be made in “prognosis”.

Adding some capital letters, a number, a special character may make it slightly better still, but the main strength of the password comes from it being made up of three unconnected sequences.

2. Insert Something Foreign Into a Word

Another way that sounds rather counter-intuitive but is surprisingly effective is to add something into the middle of a word. Instead of “pr0gn0s1s” you could use “progn9osis”. Now it’s not an English word any more and it’s not an obvious change. Personally I’m not happy with a single insertion, there aren’t that many letters on the keyboard or letters in “prognosis”. Dump an entire other word in, however, and it makes a big difference. It is very improbable that “prognmintosis” would ever be tried by a hacker.

Although, point of order, now I’ve written that exact phrase on a web page about choosing a strong password, you probably shouldn’t use “prognosis” and “mint” in the same password any more.

3. Use The First Letters of a Passage of Text

The last simple method I’ll mention is to find a phrase, a bit of poetry, prose or song fragment and take the first letter of each word. So if you’re a big Samuel Taylor Coleridge fan you might select;

“The naked hulk alongside came,
And the twain were casting dice;”

from the most excellent (but rather scary) Rime of the Ancient Mariner. This would make a password of “tnhacattwcd” which scores well on all levels. It feels like a secure password, it actually is a secure password and it’s easy to remember.

Why Do Systems Insist On Password Rules?

We’ve all been there, “Sorry, you can’t use this password because it doesn’t contain a capital letter, a lowercase letter, a special character a number and a fractal equation.”

XKCD clearly demonstrates that this is a bag of arse. You can do all those things (except maybe the fractal equation) and still have a rubbish password, or you can do none of those things and have a really strong one.

One reason is that some people really are unbelievably crap at passwords. They’ll use their own names, pet names or other information that a lot of people would know and could easily guess. At least if they’ve got a number or some punctuation in the password they stand some chance of their new credit card not being immediately hacked by their 5 year old kid.

A second reason is that, although the world of computers contains some of the cleverest people on the planet, it also contains a lot of people who aren’t. Some people who really should know better actually believe that these things are required for a strong password.

Then there are managers who just want to feel safe. Despite having the situation explained to them multiple times, they still feel that a password needs to have these elements to be strong. At an emotional level we can have some sympathy, it does feel like “pr0gn0s1s” is a stronger password than “PencilDaquiri” but it isn’t. Unfortunately, when a manager has those emotions, the wrong password rules tend to propagate into systems.

The last thing I want to mention is user expectation. There are situations where everyone in the production of the system is fully aware of the facts, but they’re producing a system that the users need to trust. We’ve established already that people have an emotional connection, if they feel a password is complicated they also feel it is secure.

People might feel, because they had to try 3 times to meet the password strength requirements, that because there are fancy traffic lights telling them how “strong” their password is, that a web site must be really secure. Behind that, the password might written in plain text into a database that itself has poor security and is directly exposed to the entire Internet.

It doesn’t matter if the a rival site has a much better password policy and a much better and more secure way of storing those passwords, if the user doesn’t feel that the site is secure, they’ll use the one that it is actually less secure.

Sadly, I think that all adds up to us being stuck with these rather silly password rules. Fortunately if you’re in the habit of setting secure passwords then it doesn’t actually make them much more difficult to remember. It’s just irritating.

Talk to Our Resellers: It’s Not Good Enough

Reading Time: 2 minutesMe: So what sort of order of magnitude are we looking at for pricing?
Exhibitors: Oh well I see you’re British I can put you in contact with our reseller in London…

This is just not good enough for any business that thinks itself truly international. I didn’t ask for a quote I asked for the order of magnitude, something roughly in the area. If a company can’t provide any idea then there’s something very wrong…

  1. It implies that the company has no direct presence in the UK, the further implication being that UK is not a core region hence support will be lacking and any UK specific portions of the product will be poorly maintained or just plain poor fullstop.
  2. It implies geographical pricing. Today’s business is global, I can and have bought COTS software from other regions. This business model doesn’t work, get over it.
  3. When a quick Google of the resellers also fails to reveal pricing one wonders if the product is actually available boxed, or whether it always comes with a cheesy sales drone and a whack of “consultancy” or “project management” attached. Further, there is an implication that without value added services it won’t actually work.
  4. Nobody likes talking to cheesy sales drones, not even IT Professionals.

So this is how it should go :-
Me:  So what sort of order of magnitude are we looking at for pricing?
Exhibitor: Err, well,  in the USA we either licence per device at $5pa each or you can buy a server licence for $2000pa. Obviously there will be some regional variation in the UK but they’re the sorts of figures you’ll be looking at. We have a number of partners in the UK, some just ship boxes but if you really want to get the most out of our software I’d recommend talking to blah… blah…

Agile is Good For Your Social Skills

Reading Time: < 1 minuteI used to work for a software house that used the waterfall model. Despite being a strong proponent of continual customer contact I actually hated doing it because all I seemed to be doing was noting issues and reporting them to project management where they disappeared into a mire of beurocracy. I was very rarely able to give customers good news.

Using an Agile methodology however SEED is able to accomodate change. I actually enjoy talking to customers now because we’re able to have open and constructive discussions. This is good for everyone:

  • The customer feels lie they’re being listened to – largely because they are
  • I and the team feel better because we’re not in a conflict scenario with the customer
  • Ultimately the customer gets a system that more reflects their needs, which benefits their business and ours.

Code Complexity – A Complex Matter

Reading Time: 2 minutes

Eats Shoots and Leaves?
Eats Shoots and Leaves?

“Oh we didn’t use your code in the end,” said an ex-colleague some time after I’d left a previous employer, “nobody could understand it.” I was a little taken aback because I’d thought the code was particularly clear given the complexity of what I was doing.

I could understand that some less experienced developers might have had trouble with it, but to someone experienced I didn’t think it would be an issue.  This left me feeling distinctly sheepish – should I have written the code in the way I did? Instead of going for ultimate performance could I have simplified it, got enough of a performance gain and achieved a greater degree of maintainability?

I felt sheepish because code that nobody understands is a big development problem for 2 reasons. The first is that at some point someone will need to understand it – that’s going to take time and development time is expensive. The second is that there is a danger that someone who doesn’t understand it (even if they think they do) modifies it and makes a mistake. Often people under pressure to fix bugs have to make guesses and sometimes they’re wrong – it might appear to fix the issue but it may actually unleash a greater demon that’s much, much harder to find.

Sometimes code does need to be complex: the earth for instance is far from round and it’s hard to avoid some pretty hideous maths if you need to work with its geometry. That’s excusable but every effort should be made to lay the code out clearly and comment well. What’s not acceptable is needless complexity through bad design, bad coding style, bad layout, lack of commenting or even in some cases developers showboating.

A good developer is not the one who writes really clever, complex code that nobody else understands. It’s the one who solves complex problems by writing simple, easy to understand code. Maintainability is the key, that’s what business needs to succeed.

The Art of Making Multithreading Issues Worse

Reading Time: 3 minutes

It also took me ages to find that these were dead

I recently spent days looking for a bug – a thread safety bug – that I should have found in minutes. The reason it took so long is that someone had found it before me and attempted to fix it, twice, but each time had in fact made matters worse.

What made matters even worse than that is a phenomenon that will be all too familiar to you if you handle multithreaded code a lot – the customer could make it happen every time but on the bench, back at the office with all the debug tools? Practically impossible to reproduce.

So the code started off like this, and here I’ve extracted the essence of the problem, this is not the real code. In reality the mistake was the same, but heavily obfuscated right from the start.

class AlarmManager
{
    private Dictionary alarms = new Dictionary();

    public Guid StartAlarm()
    {
        Alarm newAlarm = new Alarm();
        newAlarm.AlarmId = Guid.NewGuid();
        alarms.Add(newAlarm.AlarmId, newAlarm);
        newAlarm.AlarmWorkflow = WorkflowManager.CreateAlarmWorkflow(newAlarm.AlarmId);
        return newAlarm.AlarmId;
    }

    public void DeleteAlarm(Guid alarmId)
    {
        Alarm toRemove;
        if (alarms.TryGetValue(alarmId, out toRemove))
        {
            toRemove.AlarmWorkflow.StopAndRemove();
            alarms.Remove(alarmId);
        }
    }
}

Two schoolboy errors in there. Someone spotted the first, Dictionaries are not thread safe, so they added some locking…

class AlarmManager
{
    private Dictionary alarms = new Dictionary();

    public Guid StartAlarm()
    {
        Alarm newAlarm = new Alarm();
        newAlarm.AlarmId = Guid.NewGuid();
        lock (alarms)
        {
            alarms.Add(newAlarm.AlarmId, newAlarm);
        }
        newAlarm.AlarmWorkflow = WorkflowManager.CreateAlarmWorkflow(newAlarm.AlarmId);
        return newAlarm.AlarmId;
    }

    public void DeleteAlarm(Guid alarmId)
    {
        Alarm toRemove;
        lock (alarms)
        {
            if (alarms.TryGetValue(alarmId, out toRemove))
            {
                toRemove.AlarmWorkflow.StopAndRemove();
                alarms.Remove(alarmId);
            }
        }
    }
}

Fixed? No. This is where we hit the real problems. Someone noticed that the line toRemove.AlarmWorkflow.Stop(); was throwing null reference exceptions, so rather than investigate how toRemove.AlarmWorkflow came to be null they simply put a null check in…

public void DeleteAlarm(Guid alarmId)
{
    Alarm toRemove;
    lock (alarms)
    {
        if (alarms.TryGetValue(alarmId, out toRemove))
        {
            if (toRemove.AlarmWorkflow != null)
                toRemove.AlarmWorkflow.StopAndRemove();
            alarms.Remove(alarmId);
        }
    }
}

The reason it was occasionally null is in StartAlarm. That’s where the bug is – the new alarm is added to the dictionary and the lock released before it’s finished initialising. So if it’s deleted by another thread immediately after it’s started, the threads can interleave in a way where the alarm is removed from the dictionary with the workflow being null, then the workflow is assigned and started. As the workflow is managed by the WorkflowManager, there’s still a reference to it, hence it continues to run.

Now the issue got compounded further, because someone spotted that exceptions were still being thrown during the delete, from the StopAndRemove method. This is where my simplification falls down a bit because the real reasons for the exception are somewhat complex involving events and another access to the alarms dictionary, suffice to say however that this was not the way to solve the problem…

public void DeleteAlarm(Guid alarmId)
{
    Alarm toRemove;
    lock (alarms)
    {
        if (alarms.TryGetValue(alarmId, out toRemove))
        {
            try
            {
                if (toRemove.AlarmWorkflow != null)
                    toRemove.AlarmWorkflow.StopAndRemove();
            }
            catch 
            {
            }
            alarms.Remove(alarmId);
        }
    }
}

These two attempted fixes are part of a mindset of patching it up rather than fixing the root cause. In certain extreme circumstances patching it up may be acceptable. I’ve had to do it, but it must be highlighted that this is what has been done and that it may actually be masking the root cause or indeed causing further knock-on issues.
I find the #warning preprocessor directive useful in such circumstances.

The solution is trivial…

class AlarmManager
{
    private Dictionary alarms = new Dictionary();

    public Guid StartAlarm()
    {
        Alarm newAlarm = new Alarm();
        newAlarm.AlarmId = Guid.NewGuid();
        lock (alarms)
        {
            newAlarm.AlarmWorkflow = WorkflowManager.CreateAlarmWorkflow(newAlarm.AlarmId);
            alarms.Add(newAlarm.AlarmId, newAlarm);
        }
        return newAlarm.AlarmId;
    }

    public void DeleteAlarm(Guid alarmId)
    {
        Alarm toRemove;
        lock (alarms)
        {
            if (alarms.TryGetValue(alarmId, out toRemove))
            {
                alarms.Remove(alarmId);
                toRemove.AlarmWorkflow.Stop();
            }
        }
    }
}

When Customers Bite

Reading Time: 2 minutes

Hungry Tom's if File, North Yorkshire

Good software can flunk if it’s badly demonstrated. This is common sense but something I learnt only recently was the importance of data in the user’s perception of what a good software is.
When I did the sales demonstrations I’d used demonstration data. It worked, the buyers placed the order and I was then asked if I wouldn’t mind demonstrating the system to the end users.
Part of the bid involved importing the customer data and I was about half way done so I thought it would be cute to show the end users their own data.

In retrospect that was a big mistake.

When replacing a core system, something that the users spend most of their time with, they’re naturally going to be a little cautious. I was very aware of this so I attempted to mitigate it by saying that there may be a few little holes as the data import wasn’t finished. Sure enough one of them spotted a data error which I attempted to brush aside, but they wouldn’t let it go and it was followed by an avalanche of questions – unsurprisingly the users knew where the complex data was – the bits that I hadn’t yet imported. End user confidence utterly nosedived and optimistic caution went to out of hand rejection. It was a very difficult session, especially when you have people saying directly to you – and I quote, “Your system is shit. We can’t use this shit.”

The fact that I had demonstrated everything that they needed to do, had demonstrated that it was easier and clearer with our system than the one they had and that we gave them a heap load of extra functionality stood for nothing because the data behind it was poor.

About the same time a failing government project to replace the many disparate control room systems in the Fire Service with one package was being lambasted in the press. The entire history of the venture pretty much reads like a manual on how to screw up project – critical mistakes were being made left, right and centre. All these reasons are complex and somewhat intangible. What the press had picked up on was that a supposedly national system only “knew about” one small area of the country – Wakefield I think. That’s tangible, that was something people could understand. That’s what people were laughing about and of course it was a problem, but it had precious little to do with the failure of the project.

So a hard lesson for me to learn that people just don’t seem able, or perhaps willing, to separate the concepts of data and functionality when it comes to software. It took months of hard work after that bad demonstration to get the users on board.
It is very important to get them on board early in a project and keep them informed, you want them to feel like they’re involved – part of the process – then acceptance of the system is much easier. Until you’re confident that their data is 100% though, use demo data.

What C#’s For is For…

Reading Time: < 1 minuteEver wondered why C# uses such a silly format for its for loop?

I mean why not do it like Pascal?

 
    var loopVar:int;
    for loopVar := 0 to 10 do
    begin
        writeln('something erudite');
    end;

Whereas C, C#, and C++ use constructs like…

    int i;
    for(i = 0; i < 10; i++)
    {
        Console.WriteLine("something erudite");
    }

The reason is not always taught in C / C# /C++ 101 and it’s really quite useful. C’s for loop is split into 3 sections.

  1. Initialisation. Here you set up the start conditions for your loop.
  2. Comparison. The loop will continue to iterate whilst the condition here evaluates to true.
  3. Loop action. An action that is performed every time the loop iterates.

You do not have to initialise an ordinal type, you do not have to use a numeric comparison, you do not have to increment or decrement anything. Today I needed to walk an exception stack, I used a for loop, roughly (but not exactly) like this.

catch(Exception ex)
{
    for (Exception subject = ex; null != subject; subject = subject.InnerException)
    {
        if (subject is ArgumentNullException)
        {
            ///do stuff
        }
    }
}

This is just a simple example, there are all sorts of places where the C style for construct comes in useful – a friendly word of advice though, don’t go nuts. If the operations that you need to perform in order to run the loop don’t neatly fit into the for loop construct, then you should probably do it a different way, else your code is likely to become difficult to read and buggy.

The Power of the Random Other Coder

Reading Time: 2 minutes

I'm not paranoid...
Enough to make you paranoid!

If you’re a junior engineer and you have a problem you can easily ask someone more senior. But what if you’re the most senior person in the room?

Well if the problem isn’t one of not having the technical expertise but just not being able to find what’s wrong then the answer couldn’t be more simple – ask anyone.

I have a reputation for being able to find problems with other people’s code but the truth is that 90% of the time they actually find the errors themselves. The simple process of having to explain what one was trying to achieve and how he was trying to achieve it often reveals problems with the logic or problems with the code.

In fact it can be better to ask people who, although capable of understanding what you’re doing, are from a different technical area or who are considerably less experienced – the key is that you have to explain what you’re doing to them and the more explaining you have to do, the better.

At my previous company I found that the technical director was a particularly good person to ask. He had been a programmer but wasn’t really used to the way we worked in the modern production environment. He provided a very intelligent but completely different slant on things. OK, so sometimes it was frustrating when he disagreed with a perfectly good way that I was doing something, but he spotted a fair few little mistakes more often and that was very helpful.

Tom Recommends

Reading Time: < 1 minuteOccasionally, in all the stuff I use, I find something that I really like. So why not tell people about it?

Polycell Crack Free Ceilings is great. It’s thick, gloopy paint that hides a multitude of sins and not just on ceilings. It’s expensive, but combined with a little lightweight filler the time it saves is well worth it. As an aside though, this doesn’t mean that I recommend all the Polycell miracle solutions, some of which I’ve found not to work for me.

Dawn Simulator Alarm Clocks – any of them really – I happen to have the Lumie Bodyclock Starter 30. I’m not a SAD sufferer, but nonetheless I find that not being suddenly jolted awake makes me far more alert and puts me in a far better mood in the morning.

Ultrafire WF-606a – a pocket size LED torch, runs on 2 AA size batteries. It’s very bright and very rugged. A thoroughly sound piece of kit, far better in my opinion than the equivalent MagLite and much cheaper, too.