Nevermind.dk

Watch out...Load Compact -REPLICA may kill your server

I have run ".Load Compact -REPLICA" on two servers where both server where dead afterwards. The names.nsf database was corrupted in the process.and the server could not be restarted.

The fix was to make a file copy of a names.nsf from another server.

What seem to happen
The REPLICA option is a rather new Compact option on Domino. It compacts by creating a new database with suffix .ORIG and when done switches with the original database. This means the database is accessible while compacting, and only while switching/renaming the old with the one it is not accessible.

If the server can not rename a database (like names.nsf, log.nsf..) because it is in use by the server, it will instead compact with the options -REPLICA - RESTART. This means when the server is restarted it will do the compacting on the databases before they are locked by the system again.

Unfortunately this leads to a race condition where the Server and the Compact task tries to get a hold of the same databases. That may be fine for log.nsf but when that happens for names.nsf your in trouble.

Compact task will start compacting names.nsf and server tries to open names.nsf can not find server documents and other documents and crashes. Since the compact task never finished before the crash you may have a "none working" names.nsf.

IBM/HCL knows of the problem, so hopefully we will have fix soon.

Until then...be careful and stay away from the -REPLICA option

From log:

[03352:00002-00007F742382F720] IBM Domino (r) Server (64 Bit), Release 9.0.1FP10HF382 , November 19, 2018
[03352:00002-00007F742382F720] Copyright (c) IBM Corporation 1987, 2013. All Rights Reserved.
[03352:00002-00007F742382F720] Restart Analysis (0 MB): 100%
[03352:00002-00007F742382F720] 04/03/2019 09:13:26 AM Compacting /local/notesdata/log.nsf (), restart completing, -REPLICA -RESTART
[03352:00002-00007F742382F720] 04/03/2019 09:13:26 AM Compacting /local/notesdata/log.nsf (), restart completing, -REPLICA -RESTART
[03352:00002-00007F742382F720] Clearing DBIID CC5E5ECA for DB /local/notesdata/log.ORIG
[03352:00002-00007F742382F720] 04/03/2019 09:13:32 AM Compacted /local/notesdata/log.nsf (), restart completed, -REPLICA -RESTART
[03352:00002-00007F742382F720] 04/03/2019 09:13:32 AM Compacted /local/notesdata/log.nsf (), restart completed, -REPLICA -RESTART
[03352:00002-00007F742382F720] 04/03/2019 09:13:32 AM Compacted /local/notesdata/log.nsf (), 512K bytes recovered (<1%), -REPLICA -RESTART
[03352:00002-00007F742382F720] 04/03/2019 09:13:32 AM Compacted /local/notesdata/log.nsf (), 512K bytes recovered (<1%), -REPLICA -RESTART
[03352:00002-00007F742382F720] 04/03/2019 09:13:34 AM Recovery Manager: Restart Recovery complete. (0/0 databases needed full/partial recovery)
[03352:00002-00007F742382F720] 04/03/2019 09:13:35 AM Recovery Manager: Assigning new DBIID for /local/notesdata/names.nsf (need new backup for media recovery).
[03352:00002-00007F742382F720] 04/03/2019 09:13:35 AM Recovery Manager: Assigning new DBIID for /local/notesdata/names.ndx (need new backup for media recovery).
[03352:00002-00007F742382F720] 04/03/2019 09:13:35 AM Compacting /local/notesdata/names.nsf (), restart completing, -REPLICA -RESTART
[03352:00002-00007F742382F720] Clearing DBIID BB86E638 for DB /local/notesdata/names.ORIG
[03352:00002-00007F742382F720] 04/03/2019 09:13:37 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($AccountsByType)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:37 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Accounts)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:37 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Adminp)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($CertificateAuthorities)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Certifiers)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Clusters)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Connections)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($CrossCertByName)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($CrossCertByRoot)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Domains)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($ExternalDomainNetworkAddresse')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($FileIdentifications)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Groups)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:38 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Holidays)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:39 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($InternetSites)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:39 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($LDAPAlias)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:39 AM Informational, rebuild view needed - invalid collection header (reading /local/notesdata/names.nsf view note Title:'($LDAPCN)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:39 AM Invalid collection data was detected.
[03352:00002-00007F742382F720] 04/03/2019 09:13:39 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($LDAPG)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:39 AM Informational, rebuild view needed - invalid collection header (reading /local/notesdata/names.nsf view note Title:'($LDAPRDNHier)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:39 AM Invalid collection data was detected.
[03352:00002-00007F742382F720] 04/03/2019 09:13:39 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($LDAPS)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:40 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Locations)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:40 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($MailGroups)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:40 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Networks)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:40 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($PeopleGroupsFlat)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:40 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($PoliciesByGroup)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:40 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($PoliciesExt)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:40 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Policies)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:40 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Profiles)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:40 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Programs)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:40 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($RegisterGroups)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:41 AM Informational, rebuild view needed - invalid collection header (reading /local/notesdata/names.nsf view note Title:'($ServerAccess)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:41 AM Invalid collection data was detected.
[03352:00002-00007F742382F720] 04/03/2019 09:13:41 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($ServerConfig)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:41 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($ServerGroups)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:41 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($ServersLookup)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:41 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($Servers)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:42 AM Informational, rebuild view needed - invalid collection header (reading /local/notesdata/names.nsf view note Title:'($Users)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:42 AM Invalid collection data was detected.
[03352:00002-00007F742382F720] 04/03/2019 09:13:42 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($vwServersByMajVer) $vwServers')
[03352:00002-00007F742382F720] 04/03/2019 09:13:42 AM Informational, rebuilding view - no container or index (reading /local/notesdata/names.nsf view note Title:'($WebSSOConfigs)')
[03352:00002-00007F742382F720] 04/03/2019 09:13:43 AM Compacted /local/notesdata/names.nsf (), restart completed, -REPLICA -RESTART
[03352:00002-00007F742382F720] 04/03/2019 09:13:43 AM Compacted /local/notesdata/names.nsf (), 9728K bytes recovered (39%), -REPLICA -RESTART
[03352:00002-00007F742382F720] 04/03/2019 09:13:43 AM Informational - The database /local/notesdata/names.nsf has caused the DAOS catalog to become out of sync. Prune operations may be postponed. Please run 'tell daosmgr resync' at the next convenient opportunity to re-synchronize.
[03352:00002-00007F742382F720] 04/03/2019 09:13:43 AM Configuration Error: Server record not found. The Server Name in ID file [DominoDev/xxxxx], does not match any Server found in the Domino Directory in the ($Locations) view. Check that the Server name field in the Server record is a hierarchical name.
[03352:00002-00007F742382F720] Server exiting: The Domino Directory does not contain a server document for this server

Watch out...Load Compact -REPLICA may kill your server

Blog Search

Blog Categories

About this Blog