Looking for Rhino (real ones, not mocks)

Whilst I was looking for something else I found an explanation I wrote years ago to try to explain to senior management how testing had changed and become critical in the Agile environment. It’s not the easiest thing to try to get across: the business wants features that it can sell. I don’t think I did a bad job, so here it is.

Having now spent a few months with this project I’m very conscious that, at the moment, we don’t have much of a testing strategy. It looks as if we’re trying to offload all responsibility for testing onto the testing department.
This rather implies that our primary testing strategy is manual system testing.
Failed system tests are expensive – and if they impact promised delivery dates they can be a major problem. It would be better if we could have a good idea before we entered a system test whether or not it’s going to pass and that is a software quality management problem.

(Technical) testing should start in the software design. The questions the developers should be asking are;

1. How am I going to achieve this task?
2. How am I going to test it?

As an example, user interfaces are notoriously difficult to test, but if one of the MVC style design patterns is used it means that the view (the actual UI) can be separated from the rest of the code. This means you can write a program to test the logic behind the UI without having to try to actually click buttons programmatically.

There are any number of places within the design of software where adding a little thought to how it can be (easily or preferably automatically) tested is of significant advantage.
The more automated testing we can get in the better. If we can get to a position where a substantial portion of the code is automatically built and tested every night that would be great. It means that we, as developers, get continuous feedback about the state of the software.
We can then go into a system testing phase with a much higher level of confidence that it will pass.

This does mean however that we need to start investing in testing, which can be a difficult message to get across. If we don’t however then as the software and the functionality grows and the code-base becomes more difficult to maintain we will be taking ever increasing risks with the future of the product.

It’s a good idea to get this baked in now.

I would suggest that a feature is not finished unless the issue of how it is going to be tested is solved – and that means the automated tests and / or the manual tests written. If there’s no automated testing there needs to be an explanation of why.

The more bodges, work-arounds and spaghetti code there is, the harder it becomes to maintain and the longer it takes to develop each new feature. A product which leads the market can very quickly fall behind because every time someone tries to do something they have to try to unravel all the spaghetti, they then inevitably end up piling on more spaghetti just to make it work and making it worse for the next edit. Technical debt snowballs.

As with all things there’s a balance, I’m conscious we need to get features to market fast, but as the product matures the importance of testing will increase. Automated unit testing ensures that each building block of the project actually does what we think it should do. It’s not a silver bullet for solving all technical debt, but it’s a good starting point. Making code testable enforces certain good behaviours that will increase the longevity of the product.

I do not believe that the current development strategy is sustainable. I am therefore intending to phase in a plan that will, in time, see all new development covered by automated testing and will start to retrofit automated testing into the existing code-base. Inevitably this will slow down the speed we can get features to market, by a known and controllable overhead. Our business model is based on repeat business. If we fail to get a grip of technical debt the competition will overtake us in the mid term and it will invalidate that model.

I was writing some unit tests: largely out of completeness I wanted to test that you couldn’t insert two records with the same primary key. The code is simple enough.

    var car = db.Cars.OrderBy(g => Guid.NewGuid()).First();
    var pool1 = db.Pools.OrderBy(g => Guid.NewGuid()).First();
    var driverName1 = Guid.NewGuid().ToString().Trim('{', '}');
    var created1 = PoolAllocatedCar.Create(car, pool1, driverName1);
    db.PoolAllocatedCars.Add(created1);

    var pool2 = db.Pools.OrderBy(g => Guid.NewGuid()).First();
    var driverName2 = Guid.NewGuid().ToString().Trim('{', '}');
    var created2 = PoolAllocatedCar.Create(car, pool2, driverName2);
    db.PoolAllocatedCars.Add(created2);

    Assert.Catch<Exception>(()=> db.SaveChanges());

Car Id is the sole primary key of the PoolAllocatedCars table.

Being bit lazy I guessed it was quicker for me to run the code and find out what exception SaveChanges() it threw rather than trawl through the docs and work out what it should be.

The problem: it didn’t throw an exception. So I added some debug to find out what happened, the result is disappointing to say the least.

---===*** FAILED TO NOTICE PK CLASH ***===---

In Memory:
    Car Id [24] Pool Id [49] Driver [bda1d05c-8dae-4648-ab42-736eb8c44b71]
    Car Id [24] Pool Id [08] Driver [9dae73a0-5b8d-45bc-9d0a-f8b73141aa2c]

In Database:
    Car Id [24] Pool Id [08] Driver [9dae73a0-5b8d-45bc-9d0a-f8b73141aa2c]

It would appear that Entity Framework simply overwrote the first record with the second without giving any indication that there was ever a primary key clash.

Now I’m sure that somewhere in the documentation there’s a warning or a note about this but I haven’t found it yet…

Update: It Gets Worse

I was taken aback by the above, that it could be deemed acceptable to treat the explicit addition of a second object with the same primary key as an implicit update with no warning to the user.

I guess then I shouldn’t have been surprised that it even does this after an explicit call to SaveChanges()

The following test gives exactly the same result as the first. This is a massive gotcha.

    var car = db.Cars.OrderBy(g => Guid.NewGuid()).First();
    var pool1 = db.Pools.OrderBy(g => Guid.NewGuid()).First();
    var driverName1 = Guid.NewGuid().ToString().Trim('{', '}');
    var created1 = PoolAllocatedCar.Create(car, pool1, driverName1);
    db.PoolAllocatedCars.Add(created1);

    db.SaveChanges();

    var pool2 = db.Pools.OrderBy(g => Guid.NewGuid()).First();
    var driverName2 = Guid.NewGuid().ToString().Trim('{', '}');
    var created2 = PoolAllocatedCar.Create(car, pool2, driverName2);
    db.PoolAllocatedCars.Add(created2);

    Assert.Catch<Exception>(()=> db.SaveChanges());

The good news is that if you use a different DbContext it throws an exception :- in fact on the Add, not the SaveChanges.

I’m having trouble getting my head around this: I can’t see the logic. If you thought there was a chance that you’d want to update an object after you’d added it to the table then you should keep your own reference to it. If that’s a problem because of scope then your design is probably wrong.

I’d barely got into the corridor before I found myself being bundled into the wall by a rotund sales guy. “What the fuck do you think you’re doing?” he barked in my face, “If you ever pull a stunt like that again I’ll making fucking sure we put you out of business. Consider this a warning.” he stuttered a little and I could see the fear in his eyes: he was not just angry, but terrified. “And I don’t give second warnings. Understand?” he eventually added and with that marched angrily away.

I’d got maybe 15 metres further before another salesperson came up beside me and sniped, “If that’s how you’re going to behave you can forget ever partnering with us.” before he too shuffled off.

What had I done to deserve such attention? Insulted their mothers? Murdered a puppy? No, just give a sales presentation.

One of the way the UK government keeps track of technology is through seminars, they take various forms but this was essentially a day-long event where vendors of a particular type of technology were invited to present their products to a bunch of interested parties within government.

Somehow – despite being a tiny little firm – we’d managed to blag our way onto the list. I’d been to these events before, as the “tech guy” that answers the difficult questions so I knew the format. What I hadn’t done before was actually sit through a whole day of them.

They were dismal. The scripts were poorly thought out, badly structured, omitted important details and contained much that was irrelevant. The presenters were worse. Monotonous at best, incoherent and rambling at worst. By the second or third presentation I was genuinely ashamed that this is supposedly the best of what my industry can offer.

By the second session after lunch half the audience were asleep and I was reasonably convinced that the other half were dead.

When we eventually took to the stage my opening words after introducing myself were “I’m sure you’ll be delighted to hear that this is the only Powerpoint slide I’m going to use.” I then talked for a few minutes about who we are and what the product was whilst my colleague set up a fully working system. We put it up on the projector, handed our various devices to the audience and spent the rest of our time role playing live scenarios that demonstrated the product features.

It was a bold move and quite a risk, there were some very senior civil servants in the room who could have taken umbrage, but they didn’t, they did the exact opposite. At the insistence of one of them our slot was extended to allow for an additional 5 minutes of questions. My colleague and I both ran out of business cards in seconds and took to writing our details on the backs of other peoples. From a sales point of view it could not have gone better for us.

That’s all I’d done. I’d hadn’t bad-mouthed the competition, said anything that was untrue or in any way behaved unprofessionally. We turned up, we told jokes and tried to be as engaging as we could whilst demonstrating a live system that could be set up in less than 5 minutes.

That is the kind of behaviour that the competitors needed to stop. For 30 years they’ve been telling government that software systems are big, complex and expensive. That they take years to develop, months to install and configure and that they require huge, expensive racks of equipment.

Then two guys turn up with a laptop and a bag of kit and prove that none of this is true.

The issue is more complex than one of straight deception, though. These organisations were formed at a time when these things were all true. The organisations themselves have grown to provide a heavyweight framework. They have little momentum and colossal inertia.

They also have huge problems with legacy code-bases. The requirements of government tend to evolve slowly, hence re-writes are rare. Instead old code lumbers on and on and velocity nosedives as technical debt increases [more on that in a minute] and new development techniques, tools, frameworks and languages accelerate the pace that can be achieved within the industry in general.

These factors have reciprocated a framework within government that has grown to expect and indeed facilitate it. The tendering process results in excessively detailed documents that emphasise the minutia and neglect the actual business requirement for the system. These are often produced by committees and are stitched together from requirements drawn from multiple sources – or in other words they’re actually just wish-lists and can at times can be incoherent.

This is a thoroughly unhealthy relationship, it’s not just wasteful, it’s actively counterproductive. It stifles technical innovation and guarantees mediocrity.

More than that though it damages the engineering quality of the products as developers rush to strap on extra features and to modify existing ones in order to meet the letter of the requirement where in reality the product was designed to solve the overall business need and does so perfectly adequately.

This is often a panic scenario too. In order to gain a higher score on the tender a business will commit to what is it at the absolute limit of what it can achieve – often a little more than. The result is the in order to meet the delivery, development have to cut every corner possible. Inevitably this piles on the technical debt.

As the development team crashes from tender to tender its velocity plummets, the business throws more developers at the problem and you end up where we are today. The products are bloated, resource hungry, hideous to configure, often still include archaic technologies that (now) require special environments and are generally impossible to demonstrate anywhere but a dedicated, static suite specifically for the purpose.

That is why 2 people turning up to a sales seminar with a laptop and a bunch of hand-held devices and doing a fully interactive live demonstration caused such a furore. Yes I do believe that everyone else there on that day was at least one of: incompetent, phoning it in or just plain burnt out. None of them however could have done a live demonstration even if they’d wanted to.
To the competitors watching it must have seemed like we’d taken the bar for a sales presentation, which was set barely higher than the crash-mat, and smashed it through the roof. There was no way they could compete.

The reality however is that we weren’t a real threat to them. If we wanted to win tenders then we had to tick boxes. We could undercut them only a handful of times before the same thing happened to us as happened to them 30 years ago. Thankfully we recognised that and stopped responding to additional tenders.

I have to admit that I don’t know what the answer is. As a business or an individual one can make purchasing decisions however we like. If we see something that doesn’t meet what we thought our requirements were, but if we thought about the problem in a different way is actually a far better solution, we can buy that and change the way we work.

Government is different, a democracy must always be accountable to the people. The government must be able to justify why it chose supplier X over supplier Y. A qualitative response is always going to be open to challenges. It is far more robust from a point of view of the accountability of government to take a quantitative approach: supplier X was able to meet more of the requirements than supplier Y. It is also more legally robust, many of the suppliers are large companies that are not afraid of litigation if they believe it might give them a competitive advantage.

One could perhaps say that this adds weight to the argument that the government is trying to do too much and that many of these key areas should be privatised, where a more effective purchasing strategy could be implemented. This however has its own problems: ultimately it’s still public money and all this is doing is to move the point of accountability higher up. The same end could be achieved by just increasing the threshold at which a tendering process is required.
I’m also not particularly keen to live in a country that has a private Police Force and I rather suspect I’m not alone.

The counter-argument is that all such systems should be developed in-house by the government itself. In an ideal world this would be the answer. Unfortunately the world isn’t ideal and the list of reasons why this is a terrible idea is really rather long.

However we spin the model and try to look at it a different way, it’s difficult to get away from the idea of a tendering system that’s essentially scored quantitatively. There have been schemes to try to improve the situation such as to get people from the software industry in ahead of the tenders being written, however their influence has to be carefully managed in case of any allegations of impropriety and they are inevitably associated with the very organisations that will be responding to the tender. Thus this is has not proved and effective means for improving the situation.

I rather dislike finishing an article without at least suggesting something that I think might improve the situation, but I have to admit that I’m struggling with this one.

—=== END ===—

I’ve had a number of comments since writing this article. I can split them largely into 3 categories.

The saddest is the number if people who’ve also reported being directly threatened by larger firms within the industry.

I hadn’t realised how much technical jargon I’d used. Basically “velocity” is the speed at which you can make changes to a product and “technical debt” is what happens when you keep making changes without the proper procedures:
eventually you end up with a big bird’s nest and it gets harder and harder to make changes and it all gets more and more out of control.

A few people within the industry have commented that I’ve made some generalisations and simplifications. This is true, it’s a trade-off between writing a fastidiously accurate article and writing one that makes the point sufficiently well but is engaging enough to appeal to a wider audience.

Tom Fosdick is a Software Architect specialising in Control Room systems for the Emergency Services. His current engagement is with Motorola Solutions, although this article refers to an engagement with another organisation.