Understanding Notes and its Developers.

This was originally written in May 2002 back when Lotus Notes was slightly more relevant than it is today. I’ve reproduced it here in its original form for posterity and because it got a bit of attention back then.

How many times have you heard it said that Notes is no good... that it’s a useless piece of software that’s unstructured and only supports sloppy apps? The developers don’t know what they’re doing, they’re stupid and don’t understand how to program or work with real programming languages or databases. I’ve heard it a lot – an awful lot – in fact I’ve even said it a bit too. But how is it that this bunch of talentless code monkeys can work with such a slow and unwieldy product and yet keep getting employed to do yet more work? I’ll tell you...

Is Notes really that bad? #

The short answer is no. In fact to be honest I’m going to have to stray from my usual ‘sit on the fence – social sciences background’ and say that it’s the long answer too. I’ll even go as far as to say it’s a great product. Apart from its heritage as a great RAD tool with good support for open standards, in-built authentication, privacy and signing not to mention its replication abilities it still remains by far the best software in its field. To be honest that’s helped along by the fact that it’s pretty much the only software in its field of groupware-messaging-application development platform cum web application server with a web browser thrown in for good measure.

You see, Notes really is unique in offering such a varied and wide feature set – nothing else tries. Microsoft’s approach is nowhere near (although before you flame me – read on. I’m not a prophet looking to start a religious war – and I happen to really appreciate Microsoft’s approach too). In order to do what Notes or Domino does, you’d need Outlook, Exchange, SQL, IIS, the CryptoAPI and Visual Studio – that’s a lot. What I would certainly accept is that once all that’s set up and you’re conversant in its usage then you really are onto a winner – it’s superbly fast (both to use and develop for) and its great fun too. But back to the point – Notes does it out of the box.

Why do so many people think Notes is no good? #

The short answer is that it’s not a good word-processor; it’s not a good RDBMS; it’s not a great pure application server. Actually it’s not the best at anything it does except when you combine the lot and you find it’s the best jack of all trades. The fact is, I’ve seen some blinding Notes and Domino apps. I’ve used pretty much every API going in Domino, and it really can do some cool stuff – and fast too. The reason that so many Domino projects are slow, late or just plain bad is that it was either the wrong product for the job or the developers are just useless.

Yes but Notes is slow isn’t it? #

I’ve recently been working on the SQL side of a Notes to SQL reporting project where one of the many things we had to do (fairly obviously) was get all of the data out of one bucket and into another. Of course, in Notes there isn’t really any notion of field-size. I mean – there’s text and rich text (64k and 2gb or something silly), but there no char(16) or varchar(255). It’s funny – the SQL guys would look at me and laugh about how useless Notes was for not enforcing a size, but I remember in my early days as a Notes Developer thinking exactly the same about how stupid MS Access was for only allowing 255 characters into a field. Anyway, how do you deal with it in SQL?

Before you answer – add to this the fact that the tables are stupidly wide (because Notes has no conception of joins) and all ‘related’ data is actually stored on every document and that making every column a varchar(8000) is likely to produce at least 50% of rows going over the 8k limit. We came to the fairly straightforward conclusion that the best and safest way to deal with it was to use ‘text’ columns.

Sure enough we got all the data in and it worked great. But guess what; queries ran like a dog – and I’m not talking about the racing variety. I mean the friendly old ones who are really affectionate – even though they are blind in one eye, have a gammy leg and smell a bit. In fact, not only were they slow but you couldn’t even make useful selections or do anything useful in native T-SQL because you can’t declare ‘text’ variables.

Am I finished yet? #

No. The reason for all this is that SQL imposes limits on things so that it can arrange rows in memory pages. This turns out to be a really good thing for scanning a column in a table; it’s a simple question of pointer arithmetic to move between rows and columns and be able to compare it to something else. Text fields, on the other hand, move the data outside of the page and just leave a pointer behind. In C terms you have to dereference every pointer in every row to do anything at all, then, the chances are that you’ll have to load something from disk too – at this point, you’re yawning and the processor’s sweating. For a cool discussion about this see what Joel Spolsky says about SQL and XML. Notes works the same way (as the XML) - more or less. Yes it has summary items and views and all that, but when it comes down to it, that’s what it has to do.

Now I’m finished. The point of all this is to say that when it comes to selecting, sorting, massaging and manipulating big rows of characters, SQL is no better than Notes (and is frankly worse) at it. The answer to the original question is that Notes is slow when it’s being used as a relational database - so don’t use it as a relational database.

When is Notes the right tool? #

Well it’s really good at so many other things. To be honest though, in my travels from one technology to the next the features that you really don’t want to have to write from scratch are replication and security. I could happily live without every other feature in Notes – I’ve pretty much done all the other things you can do in Notes in other ways and not missed it. These things though, are great.

You may already have the infrastructure and licensing for Notes in which case you will most likely have gone through IT auditing and approval for new products. This reduces the TCO for your new app and makes support more straightforward. All this, and that’s not to mention the easy integration with existing Notes apps.

If you want a knowledge management system, or an app that lets people work remotely and disconnected from everyone else with nothing but a 9600 modem, Notes is right. If you want to allow people to collaborate across networks or corporate boundaries, if you want off-the-shelf document or even field level encryption and security, Notes is right. If you want an enterprise sales system for procuring and brokering big fat multi-million pound transactions with up to the minute dynamic reporting – I’d suggest something else.

So what about the developers? – You said they were useless. #

They’re not – at least, not intrinsically and certainly not all of them. Just as with any craft though you get good and bad developers; it’s just that Notes developers come from a different planet to the average red-bull swilling, pizza eating, minus-hygiene, bad-beard beer-monster programmer.

You see, they all too often come from a business background or ‘fall into’ development rather than study it and move into it – they probably studied social geography or something. They tend to be bright people who know the inner workings of a company well and are best positioned to help work out a new ‘system’. What happens then is that they start to learn ‘programming’ by creating forms and views in a GUI and think that it constitutes a ‘database’. But guess what? They hand it back to ‘the business’ after a day or so and ‘the business’ loves it. Suddenly ‘the business’ can do all the things that it couldn’t before. “But”, they say, “could we also have this form and that field, and make it so that you can press this button and do that”. I know – I used to do just that.

I’d build a ‘database’ (unable to distinguish between the storage and the application), and deploy it only to find that it didn’t really validate any data – or have any nice drop down lists let alone security. I soon learned about ACLs and a few @Formulas (think Excel Macro language if you’re not of a Lotus persuasion). I also got to grips with @DbLookup (like Excel VLOOKUP() or a nested select within one row in SQL) which meant that I could reference data outside the current document.

So far – as a non-Notes person you might be wondering what could go so disastrously wrong. I’ll tell you: the root of all evil in Notes is that there’s no such thing as a join. Sure – I just introduced @DbLookup, but that only works in a Form (more or less), it doesn’t work in views. The end result of this is that if you reference data in Notes, then to see it appear in a view you also need to store the referenced data too; you’re not just storing the foreign key but the whole row that the foreign key relates to. This means that if the reference data changes then you need to hope that it’s not too long before the document referencing it is refreshed.

As to why you can’t join, it comes down to performance and data not being in ‘pages’ etc. As for the consequences, well I’ve seen it too many times to remember: messy data, broken links, frustrated and overworked developers and an angry and unsympathetic business.

The way to avoid this is to think carefully about where your application might go and to choose the platform based on those criteria; don’t be afraid to mix and match.

The way to recover from this situation is basically to evaluate what the business really needs, evaluate the costs and benefits analysis of what you could deliver and make a decision that may or may not involve Notes. Is that too woolly? Of course – I did social sciences remember?

So what are you actually saying? #

It’s true that some Notes Developers are stupid, but I know a load of really smart ones too. They have a different outlook of the world – they see things in terms of lookups and documents, forms and views. They know that in order to get decent performance out of Notes and have some pseudo-relational behaviour they have to cheat a bit and store foreign data. But Notes is just plain different to Visual Basic, Forte and Java 2, HTML and ASP or Delphi for that matter. To be honest it’s more akin to MS Access in that it’s a product that can be customised heavily and treated as a development tool in its own right. But the consequence is that Notes Developers don’t easily cross the border to find that you get Forms everywhere or that a View is like a SQL Index. A Document is like a row, a Script Library like a DLL and an Agent is just an entry point or a public static main.

Notes Developers can get results pretty quickly – especially at the beginning of a project where they can knock up a few forms and stuff. That’s why they get more work.

Many of them also know a lot of LotusScript and Java and HTML and JavaScript and are really good at it. Many think sideways, upside-down and are not constrained by any box, but they also rarely know anything about life beyond Notes. Show them an RDBMS and they wince, talk about Object Oriented Programming and some will look confused.

I guess what I’m trying to say is that you shouldn’t judge Notes and the Developers that it spawns in terms of your own standards – like I said, they’re from another planet and don’t think about things in the same way as you do – but that doesn’t make them stupid.

• Next: Secure Yourself