As I've tweeted I have spent the last couple of days (and the weekend) helping out a customer that exceeded the hard 64 gb database size limit in Lotus Domino. Before discussing how we solved the problem and got the customer back in business I would like you to think about how situations like this could be avoided. And avoiding it is key as once you exceed the size you're doomed.

First --- how and why database platform would EVER allow a database to cross a file size that makes it break. Why doesn't Domino start to complain at 50gb and make the warnings progressively harder to ignore as the database gets closer to 64gb. Why doesn't it refuse data once it reaches 60gb? I find it totally unacceptable that a software product allows a database to exceed a size it knows it cannot handle.

Now I know that there are considerations for such a warning and that it could be done in application code (e.g. database script, QueryOpen event) but it really isn't something an application developer should think about. Also it should be applied to backend logic as well and really doesn't lend itself to a UI computation. I also know that DDM or similar could warn about it but it still doesn't change my stance. The 64gb limit is a hard limit and reaching, and exceeding it, shouldn't depend on me configuring a specific piece of functionality.

Second -- having the option of keeping the view index in another location/file than the database would have helped. This has been brought up a number of times including at Lotusphere Ask-The-Developers sessions. One could argue that externalizing the view index from the database would just have postponed the problem but the view index takes up a substantial amount of disk for databases of this size.

Now on to how we saved the data.

The bottom line in this is that the customer was lucky. VERY lucky. The customer uses Cisco IP telephones and keeps a replica of the database in question on a secondary server for phone number lookup using a Java servlet. Due to the way the way the servlet is written only as single, very small, view was built on the secondary server. This is turn meant that the database that had exceeded 64 gb on the primary server was "only" 55 gb on the secondary server. The database on the primary server was toast and gave out very interesting messages if attempting the access or fixup the database:

**** DbMarkCorruptAgain(Both SB copies are corrupt)

1. Take the database offline (restrict access using ACL)
2. Purge all view indexes (using Ytria ViewEZ)
3. Create a database design only copy to hold archived documents
4. Delete all views to avoid them accidentally being built
5. Build a very simple view to prepare for data archiving
6. Write a LotusScript to archive documents (copy then delete) from the database
7. Use Ytria ScanEZ to delete deletion stubs from the database (this works for them because the database isn't replicated to user workstations or laptops)
8. Do a compact to reclaim unused space
9. Make the database available on the primary server

So what are the main take aways from this?

1. UI check -- in the future all databases that I develop will have a database script check on the database size to try and prevent situations like this
2. DAOS -- enable DAOS for databases to keep attachments out of the database and keep the size down
3. Monitoring -- monitor databases either using DDM or other tools to try and prevent sitations like this

And so concludes a story from the field. 4 days later where my hair have turned gray from watching copy/fixup/compact progress indicators the customer is back in and happy once again. Whew!!

Recent Blog Posts

236

Authentication vs. Authorization Wed, May 8th 2013 7:39a Mikkel Heisterberg When ever I talk to customers and partners about single-sign-on (SSO) and the concepts of "authentication" I'm quite often baffled by the level of misunderstanding, misconception and lack of knowledge about just how "authentication" works. Now the reason I put "authentication" is quotes is that when we talk about authentication it's really not just authentication we're talking about. When we talk about confirming the identity of a user and confirming that the user is allowed to access a [read] Keywords: acl domino ibm notes application database password server websphere

269

Just arrived - thank you IBM Wed, May 8th 2013 6:10a Mikkel Heisterberg [read] Keywords: ibm

103

Websphere Application Server WIM LDAP adapter log trace Thu, May 2nd 2013 12:50a Mikkel Heisterberg When debugging LDAP login issues for Websphere Application Server (WAS) you're actually debugging the WIM (Websphere Identity Manager) part of WAS. The actual login piece is part of the adapters (database, ldap, file) which is the repository specific piece that WIM delegate the actual authentication to. The best debug string to use is "com.ibm.ws.wim.adapter.ldap.*=finest" as it limits the debugging to the LDAP piece of WIM. [read] Keywords: ibm application database server websphere

138

Setting up LDAP failover for Websphere Application Server Wed, May 1st 2013 2:16a Mikkel Heisterberg As you may know LDAP is crucial to Websphere Application Server (WAS) when using it for IBM Connections so it makes good sense to configure failover for LDAP. If the LDAP server becomes unavailable you can no longer log in (actually you can't even log into ISC) and WAS can have a hard time reconnecting to the LDAP. Failover is set up using either the ISC Federated Security UI or by editing wimconfig.xml directly (or using wsadmin commands). Using wimconfig.xml have some advantages as you can se [read] Keywords: connections ibm application security server websphere xml

186

Looking up a datasource from an IBM Connections event handler Tue, Apr 30th 2013 7:33a Mikkel Heisterberg For a customer project I'm working on these days I'm writing an event handler for IBM Connections Profiles to integrate two profile systems in real-time using the IBM Connections 4.0 Event SPI. Pretty powerful stuff in case you've never looked into it. In IBM Connections an event handler is basically just a Java bean which you register in events-config.xml to be called when certain events occur such as a profile being updated, the photo set, the photo removed etc. In this event handler I n [read] Keywords: connections ibm ldd lotus application database interface java profile server websphere xml

160

Fixing IBM Connections help for IE users Mon, Apr 15th 2013 5:13a Mikkel Heisterberg At a customer site they were actually using the IBM Connections help documents (a first I know) but it didn't work for the users in Internet Explorer. After some research it turned out to be due to a missing compatability statement in the generated HTML documents (this statement is present in HTML generated for other features). I've previously reported this issue to IBM but it still hasn't been fixed in version 4.0 CR3 so I took it upon me to find a solution. The solution turned out to be sim [read] Keywords: connections ibm microsoft

260

Hiding the Social Mail username and password from socialmail-discovery-config.xml Fri, Apr 5th 2013 12:38a Mikkel Heisterberg It's been bothering me a while that the username and password for our LDAP user was visible in clear text in our socialmail-discovery-config.xml. After going looking for a solution by using very specific searching I found a solution where you can hide the username and password and - stupid as I am - it's actually right there in the install docs. Stupid is as stupid does. The solution is to remove the authentication data from the socialmail-discovery-config.xml and replace the and tags with a [read] Keywords: connections domino ibm ldd lotus exchange exchange password websphere wiki xml

631

There's a new sheriff in town Thu, Mar 28th 2013 2:33a Mikkel Heisterberg Last week was the annual BLUG event this time in Leuven, Belgium, and as always Theo and team created an amazing event. The BLUG event is now the biggest user group in the a World with a staggering 325 attendes and it really makes BLUG a mini-Connect event attracting the top names from IBM as well as the top speakers from all over the World. This year was no exception and the attendees were gifted with 18 IBM Champions covering everything from Domino to Websphere, XPages to widgets and social to [read] Keywords: atlantic collaboration connections domino ibm inotes notes xpages community linkedin twitter websphere widgets

299

Finally! Mon, Mar 25th 2013 12:25p Mikkel Heisterberg [read] Keywords:

200

Calendar integration example using OnTime Group Calendar API Mon, Mar 25th 2013 12:09p Mikkel Heisterberg The Solution To remedy this they decided to use the OnTime Group Calendar API to integrate the two systems using web services using an intermediate Enterprise Service Bus (ESB). The OnTime Group Calendar API web services are hosted directly on IBM Domino and performs extremely well. After implementing the solution the insurance agents only need to maintain their calendar in Notes as it will reflect their true calendar showing both internal, external and personal appointments and meetings. Th [read] Keywords: domino ibm inotes notes notes client application enterprise integration mobile server