Domino and SSL: Come with Me If You Want to Live

Wed Sep 24 15:38:51 EDT 2014

Tags: nginx
  1. Setting up nginx in Front of a Domino Server
  2. Adding Load Balancing to the nginx Setup
  3. Arbitrary Authentication with an nginx Reverse Proxy
  4. Domino and SSL: Come with Me If You Want to Live

Looking at Planet Lotus and Twitter the last few weeks, it's impossible to not notice that the lack of SHA-2 support in Domino has become something of A Thing. There has been some grumbling about it for a while now, but it's kicked into high gear thanks to Google's announcement of imminent SHA-1 deprecation. While it's entirely possible that Google will give a stay of execution for SHA-1 when it comes to Chrome users (it wouldn't be the first bold announcement they quietly don't go through with), the fact remains that Domino's SSL stack is out-of-date.

Now, it could be that IBM will add SHA-2 support (and, ideally, other modern SSL/TLS features) before too long, or at least announce a plan to do so. This is the ideal case, since, as long as Domino ships with stated support for SSL on multiple protocols, it is important that that support be up-to-date. Still, if they do, it's only a matter of time until the next problem arises.

So I'd like to reiterate my position, as reinforced by my nginx series this week, that Domino is not a suitable web server in its own right. It is, however, a quite-nice app server, and there's a crucial difference: an app server happens to serve HTML-based applications over HTTP, but it is not intended to be a public-facing site. Outside of the Domino (and PHP) world, this is a common case: app/servlet servers like Tomcat, Passenger, and (presumably) WebSphere can act as web servers, but are best routed to by a proper server like Apache or nginx, which can better handle virtual hosts, SSL, static resources, caching, and multi-app integration.

In that light, IBM's behavior makes sense: SSL support is not in Domino's bailiwick, and nor should it be. There are innumerable features that Domino should gain in the areas of app dev, messaging, and administration, and it would be best if the apparently-limited resources available were focused on those, not on patching things that are better solved externally.

I get that a lot of people are resistent to the notion of complicating their Domino installations, and that's reasonable: one of Domino's strengths over the years has been its all-in-one nature, combining everything you need for your domain in a single, simple installation. However, no matter the ideal, the case is that Domino is now unsuitable for the task of being a front-facing web server. Times change and the world advances; it also used to be a good idea to develop Notes client apps, after all. And much like with client apps, the legitimate benefits of using Domino for the purpose - ease of configuration, automatic replication of the config docs to all servers - are outweighed by the need to have modern SSL, load balancing/failover, HTML post-processing (there's some fun stuff on that subject coming in time), and multiple back-end app servers.

The last is important: Domino is neither exclusive nor eternal. At some point, it will be a good idea to use another kind of app server in your organization, such as a non-Domino Java server, Ruby, Node, or so on (in fact, it's a good idea to do that right now regardless). By learning the ropes of a reverse-proxy config now, you'll smooth that process. And from starting with HTTP, you can expand to improving the other protocols: there are proxies available for SMTP, IMAP, and LDAP that can add better SSL and load balancing in similar ways. nginx itself covers the first two, though there are other purpose-built utilities as well. I plan to learn more about those and post when I have time.

The basic case is easy: it can be done on the same server running Domino and costs no money. It doesn't even require nginx specifically: IHS (naturally) works fine, as does Apache, and Domino has had "sitting behind IIS" support for countless years. There is no need to stick with an outdated SSL stack, bizarre limitations, and terrible keychain tools when this problem has been solved with aplomb by the world at large.


Edit: as a note, this sort of setup definitely doesn't cover ALL of Domino's SSL tribulations. In addition to incoming IMAP/SMTP/LDAP access, which can be mitigated, there are still matters of outgoing SMTP and requests from the also-sorely-outdated Java VM. Those are in need of improvement, but the situation is a bit less dire there. Generally, anything that purports to support SSL either as a server or a client has no business including anything but the latest features. Anything that's not maximally secure is insecure.

Commenter Photo

Richard Moy - Wed Sep 24 16:55:35 EDT 2014

Jesse,

This is a good discussion, though I think IBM should still fix the problem.  We are in the process of implementing your nginx/Domino SHA-2 solution and will have it work by today. Thanks for your postings. Everything can sit on the same server if that is a concern for people.

I agree with you that Domino is a poor Web server. Is relatively slow compare to nginx.  We are looking at transitioning Domino to become only as a secure app server to deliver our JSON API.  This is possible since we are strictly using  Javascript/JSON RESTful API  model for our applications. With a single page MVC app, all the view/controller is created dynamically through separate http requests. And even better, consider Domino only as a secure database server.  

Commenter Photo

Henning Heinz - Wed Sep 24 18:46:53 EDT 2014

Does not work for me. This software costs money. IBM charges for their "maintenance" and while they in my opinion have significantly reduced their efforts, pricing has only been going up over the years.

If a free as in beer software can do a task better then in my opinion the vendor hasn't done a good job. Maybe Domino is a "poor web server" but I cannot accept this as an excuse for poor performance. Proprietary software always had the claim to be better, more secure, more feature rich, future proven and with better support. Now for a complex product like the Domino server this probably can be a difficult task but I am sorry to say that I don't have the impression that IBM even tries to keep this product up-to-date within its basic feature set.

I have been using Nginx for many years (or LightHttp). I use many tools with (and without) Domino. In most cases because I have to, in some cases because I want to. I even have a solution for (hopefully) securely serving JSON data (although I do like Domino Data Services which seem to rely on the Apache Wink project).

While we are at tools. The ability to maintain everything at a single place (e.g. the Domino Directory) still is a very sexy idea. Our Windows guys really think of voodoo when they see me fiddling with config files in a SSH session.

Commenter Photo

Russell Maher - Wed Sep 24 19:31:58 EDT 2014

This is really a good point you're making and I have to admit that I have had the same opinion on this matter for quite a while although for perhaps a different reason.  My reasons have to do with remediating web application vulnerabilities. I value Domino and we have no plans to leave it however the product simply does not have the functionality built in to adequately address some vulnerabilities found by standard scanning tools.  It's all about managing risk.  If you have your Domino server just sitting behind a firewall there is substantially more risk to your data than if you were to place it behind other devices/servers designed to better address web-related vulnerabilities.  That level of risk may be totally acceptable to you but is foolish to not understand that the risk profile is very different if you're running a Domino server that is only protected by a firewall.

Commenter Photo

Jesse Gallagher - Wed Sep 24 19:35:48 EDT 2014

You'll get little argument from me: proprietary software is indeed supposed to be all of those things. Domino is not, though, at least on this topic. When it comes to HTTPS, though, I think the negligence has the benefit of pointing to a better setup. The bundled IHS on Windows is a step in this direction, but I think it's better still to use this as an opportunity to improve the normal way of doing things with open-source alternatives.

As for what the increasing pricing gets you, that's another conversation that I imagine the community in general will have down the line.

Commenter Photo

Nathan T. Freeman - Wed Sep 24 22:04:04 EDT 2014

"Proprietary software always had the claim to be better, more secure, more feature rich, future proven and with better support."

I'm sorry, but what kind of propaganda have you been drinking? Relying on proprietary vendors to solve your business problem because you paid them is like voting Democrat because they're the "party of peace." You have to use a special kind of reality distortion field to keep believing that.

Some vendors will sometimes solve your business problems if your goals align. If you count on that, you're a fool.

Commenter Photo

Henning Heinz - Thu Sep 25 05:14:26 EDT 2014

Well it is not so important what I think about proprietary software. My point is that if a vendor charges me for their product I need to feel comfortable. And of course in some ways it has to be better than other solutions (free or proprietary). Or I should at least get the impression it is but marketing and IBM is a difficult topic.

If not, I am not going to use it.

For me it still makes a difference if a product is just a bit old fashioned or abandoned. This SSL problem falls into the abandoned category.

For my own purposes I don't use IBM software anymore (neither for mail nor for applications/development) but part of my work still is much about IBM Notes and Domino. There are still people who like the platform and my job is to keep them happy. Loyal customers are a good thing. IBM should handle them with care.

Commenter Photo

Fredrik Malmborg - Thu Sep 25 05:20:19 EDT 2014

That is very clever put together Jesse!

Your text makes me step back a bit and think differently on the issue. It is hard to kill your darlings,

It would be very nice to have SHA-2 support in Domino, but there are definitely better things to put resources on. If there had been no alternative solutions, things would have been different.

Thank you

Commenter Photo

Andy Dennis - Thu Sep 25 06:17:36 EDT 2014

It makes perfect sense to me. Domno's SSL stack has always been slow, and offloading the requests to the Proxy Server, means that not only can you use SHA-2 Support, you will also increase the response of you SSL application. It's a great application server, but that HTTP Stack has been left behind..

Commenter Photo

Michael Bourak - Thu Sep 25 06:24:48 EDT 2014

Dare I say that this kind of setup AppServer <--> reverse proxy with SSL handling <--> Internet is a best practice FOR ALL app servers (JEE or not) since years...not only Domino. Domino allows us to do simpler if we want...but its clearly not a good practice for many reasons including performance, security and so on.

Commenter Photo

Craig Wiseman - Thu Sep 25 09:38:48 EDT 2014

I'm curious about the statement, " In addition to incoming IMAP/SMTP/LDAP access, which can be mitigated..."

I'm assuming you mean by putting a *separate* *additional* server that has a real TLS security in front of each of these services. That's not a workable solution, especially for a NOT cheap vendor supported product.

Nothing is exclusive, or eternal, but it's fair to expect the vendor of an expensive product to do basic maintenance on it.

cpw...

Commenter Photo

Darren Duke - Fri Sep 26 09:27:56 EDT 2014

Hang on a minute....time to play devils advocate.

So the gist of this push is that it's "more correct" (for want of a better phrase) to reverse proxy a Domino server. OK, for arguments sake let's say it is. Then shouldn't the recommendation be to use a F5 appliance? It's top of the line, best of breed, load balancing proxy? Based on this conversation the answer is "yes". What we're forgetting here is context, not everyone has $75,000 to drop on an F5 (I have no idea what they actually cost, so yes, exaggeration for effect), nor the deep pockets required for the professional services to get the thing to work.

In the same way most of use don't have $75,000 I would surmise many don't have Linux skills, and yes, you *can* do this with only a modicum of Linux knowledge. But how many Domino Admins who are already overworked are going to realize that ShellShocked has maybe made their new shiny, secure, bullet proof reverse proxy a wobbly bowl of jello/jelly? Not many.

Domino is already an overly complex piece of bloatware and now we are advocating adding more complexity and bloat? Again, back to context, you *can* do a reverse proxy (I do it all the time), but you SHOULD NOT HAVE TO. IBM sells their kit with this functionality, then it's not an aberration for it's customers to:

  1. expect it to work out of the box
  2. expect it to adhere to common and recent security standards
  3. pay a pretty penny in annual software and support AND get software support and not new, soon to be released vaporware

To give IBM a free pass here is a disservice to every Domino customer out there.

 

Commenter Photo

Jesse Gallagher - Fri Sep 26 11:17:14 EDT 2014

@Craig I'm not saying IBM shouldn't fix those protocols as well, only that they haven't, and it may be required to put real servers in front of them as well. However, the benefit for those isn't as great as it is for HTTP, nor is it as pressing, so I haven't bothered actually doing it myself yet.

@Darren Ha, no, I doubt most people have any use for an F5 device. And if there's concern about learning Linux, it's also not required: nginx runs on Windows, as do Apache, IIS, and IHS. I'm demonstrating nginx on Linux because my servers are Linux already.

Probably the best-case scenario would be for IBM to fix the SSL stack generally (to cover non-HTTP protocols) and then improve the IHS integration but using it by default on all platforms and modifying it to pull its configuration from the Directory or other NSF store (or just have Domino write out config files based on an NSF, which would be easier and hackier). It would be great for Domino to maintain its "one-stop shop" capabilities in all areas. However, is that likely, given that it appears that IBM's attention to Domino is such that they can only do a handful of minor-to-medium improvements per year? The reason why I focus on this as an area to fix using third-party software is that it's something that is practical to do now and results in benefits outside what a basic fix from IBM would bring anyway.

Commenter Photo

Richard Moy - Fri Sep 26 20:01:27 EDT 2014

@Darren,  I totally agree with you.  However, we did get nginx working with Domino with relative ease on CentOS and I will be blogging this weekend when I have time.  If I am correct, the advantage of having a separate reverse proxy as you know it allows the server as a hold to handle more https request then if the Domino server was running https directly. This is a poor man F5.  

We have decided that this will be our standard configuration from now on to have nginx be the front face web server regardless if we are running https or not.  Please note that https is the main protocol of concern for us. Now we have other alterative motives for having nginx there.

Commenter Photo

Nathan T. Freeman - Sun Sep 28 01:35:27 EDT 2014

I don't care what anybody says, Microsoft charges good money for Windows and therefore it should never crash. And when it does crash, it's Microsoft's job to fix the product because I and millions of other customers paid good money for it. It's ridiculous for you to put forth this "Linux solution" and try to give a free pass to Microsoft. Customers don't have time and money to invest in learning how to implement Linux and they shouldn't have to pay a consultant to do it either.

When customers pay Microsoft good money for Windows they have every right to expect it to work out of the box and not be vaporware.

Quit apologizing for Microsoft by offering low-cost solutions. The only right answer is to pound your shoe on the podium and demand that Microsoft fix these goddamn security protocols once and for all or you'll quit paying them! Proposing alternatives is elitist! We should all just be able to run Microsoft operating systems and they should do whatever we say and quit bothering me with your empirical evidence!

Commenter Photo

Darren Duke - Sun Sep 28 10:06:03 EDT 2014

To Nathan's point then everyone should stop running AS400/i5/iSeries/Power i too.

Commenter Photo

Mike McP - Mon Sep 29 10:06:49 EDT 2014

We pay a boatload for IBM maintenance and support of Domino.  Next year, we probably won't be paying a boatload for IBM Domino support, largely due to this issue.

 

How long would it take for IBM to fix this issue?  If they put a developer on it as his only task for 2 months, could the issue be rectified?  If so, IBM will be losing money just our business lone.  Most of the time, I don't understand the rationale behind anything IBM does.

New Comment