As I've tweeted I have spent the last couple of days (and the weekend) helping out a customer that exceeded the hard 64 gb database size limit in Lotus Domino. Before discussing how we solved the problem and got the customer back in business I would like you to think about how situations like this could be avoided. And avoiding it is key as once you exceed the size you're doomed.
First --- how and why database platform would EVER allow a database to cross a file size that makes it break. Why doesn't Domino start to complain at 50gb and make the warnings progressively harder to ignore as the database gets closer to 64gb. Why doesn't it refuse data once it reaches 60gb? I find it totally unacceptable that a software product allows a database to exceed a size it knows it cannot handle.
Now I know that there are considerations for such a warning and that it could be done in application code (e.g. database script, QueryOpen event) but it really isn't something an application developer should think about. Also it should be applied to backend logic as well and really doesn't lend itself to a UI computation. I also know that DDM or similar could warn about it but it still doesn't change my stance. The 64gb limit is a hard limit and reaching, and exceeding it, shouldn't depend on me configuring a specific piece of functionality.
Second -- having the option of keeping the view index in another location/file than the database would have helped. This has been brought up a number of times including at Lotusphere Ask-The-Developers sessions. One could argue that externalizing the view index from the database would just have postponed the problem but the view index takes up a substantial amount of disk for databases of this size.
Now on to how we saved the data.
The bottom line in this is that the customer was lucky. VERY lucky. The customer uses Cisco IP telephones and keeps a replica of the database in question on a secondary server for phone number lookup using a Java servlet. Due to the way the way the servlet is written only as single, very small, view was built on the secondary server. This is turn meant that the database that had exceeded 64 gb on the primary server was "only" 55 gb on the secondary server. The database on the primary server was toast and gave out very interesting messages if attempting the access or fixup the database:
**** DbMarkCorruptAgain(Both SB copies are corrupt)
So thank God they had the secondary server otherwise the outcome of the story would have been less pleasant because using the secondary server we were able to:
Take the database offline (restrict access using ACL)
Purge all view indexes (using Ytria ViewEZ)
Create a database design only copy to hold archived documents
Delete all views to avoid them accidentally being built
Build a very simple view to prepare for data archiving
Write a LotusScript to archive documents (copy then delete) from the database
Use Ytria ScanEZ to delete deletion stubs from the database (this works for them because the database isn't replicated to user workstations or laptops)
Do a compact to reclaim unused space
Make the database available on the primary server
Whew! They are now back in business after building views in the database. They were lucky - VERY lucky. If they hadn't had that secondary replica the data would probably have been lost to much distress. To them and me.
So what are the main take aways from this?
UI check -- in the future all databases that I develop will have a database script check on the database size to try and prevent situations like this
DAOS -- enable DAOS for databases to keep attachments out of the database and keep the size down
Monitoring -- monitor databases either using DDM or other tools to try and prevent sitations like this
And so concludes a story from the field. 4 days later where my hair have turned gray from watching copy/fixup/compact progress indicators the customer is back in and happy once again. Whew!!
Trusting certificates in WebSphere Application Server
Fri, Nov 29th 2013 3:33a Mikkel Heisterberg If you make SSL connections from a WebSphere Application Server based application the server (or rather the cell) needs to trust the certificate of the server you are connecting to. This is very easy to do in WAS and is easily done using the Integrated Solutions Console (ISC). The way to establish the trust is as follows:
Log into the WebSphere Application Server Integrated Solutions Console (ISC)
From the lefthand navigator select Security/SSL certificate and key management
In the list of rela [read] Keywords: connections
On the IBM Connections SPI's
Wed, Nov 6th 2013 1:25a Mikkel Heisterberg Earlier this week I had a requirement to interact with the IBM Connections user storage from a servlet running within WebSphere Application Server meaning that I needed to obtain the currently logged in users email address from the username (i.e. the principal name in JEE speak). As I saw it there were three options - 1) reproduce the entire Federated Repositories configuration in the servlet config, 2) use an IBM Connections API if available or 3) try and figure out if there was a WebSphere API [read] Keywords: connections
New plugin appdev values for Notes 9.0.1
Tue, Oct 29th 2013 3:36a Mikkel Heisterberg With the release of Notes 9.0.1 we need new values to tweak our configuration to continue developing plugins for Notes. The new values for rcp.base_version and install_id are below. I've also updated the Configure Eclipse 4.2 for Notes 9 page.
rcp.home=framework [read] Keywords: notes
List of OpenSocial fixes in 9.0.1?
Tue, Oct 29th 2013 2:58a Mikkel Heisterberg As reported all over the Yellow-bubble we've now got a Domino 9.0.1 server and a Notes 9.0.1 client to install. All is good and the list of fixes reported by developerWorks is 267. The list however doesn't mention OpenSocial at all. This is sad as this is one of the new features from 9.0 and getting an idea as to what have been fixed would have been nice (plus I know that lots have been fixed).
Hope that IBM will release more information as to what was fixed for OpenSocial. [read] Keywords: domino
Elusive IBM Connections 4.5 CR2 upgrade issue solved
Tue, Oct 15th 2013 3:03a Mikkel Heisterberg We've had a lot of problems trying to upgrade one of our internal test environments
from IBM Connections 4.5 CR1 to CR2. The only symptom was the we were unable to move
past the screen in the update wizard (updateWizard.sh / updateWizard.bat) where we select
to install or remove fixes. The UI would simply refuse to move past this screen -
no message, no log, no nothing. To try "something else" I got the idea to try and run
the update installer in silent mode (updateSilent.bat / updateSil [read] Keywords: connections