361 Lotus blogs updated hourly. Who will post next? Home | Downloads | Events | Pods | Blogs | Search | myPL | About 
 
Latest 7 Posts
Token bucket!
Sun, Jun 8th 2014 67
"Employee of the Quarter" (not)
Fri, Dec 27th 2013 48
Geek-o-Terica 17: Accuracy vs. Precision, The Time Drift Chronicles
Mon, Sep 9th 2013 49
Some Interesting Tech I’ve Been Involved With Lately
Sat, Apr 20th 2013 41
The Interesting Difference Between "Exception" and "RuntimeException" (Java)
Sat, Feb 2nd 2013 33
Geek-o-Terica 14: How Using "LimitEntries" Can Mess Up Db Replication (Unintended Side-Effects)
Wed, Jan 16th 2013 52
"Reflections on Reflecting Light" - Java program
Wed, May 9th 2012 34
Top 10
How to get ALL of the documents: db.search() vs. db.AllDocuments vs. AllDocuments view
Mon, Oct 25th 2010 120
Geek-O-Terica 15: Easy conversion of Notes documents to MIME format (Part 1)
Mon, Mar 21st 2011 68
Token bucket!
Sun, Jun 8th 2014 67
Using Fiddler to monitor HTTP traffic in Domino Java agents
Mon, Mar 14th 2011 62
Geek-o-Terica 14: How Using "LimitEntries" Can Mess Up Db Replication (Unintended Side-Effects)
Wed, Jan 16th 2013 52
Geek-o-Terica 13: Notes Response Hierarchies - Without Views
Mon, Dec 6th 2010 50
Geek-O-Terica 16: Easy conversion of Notes documents to MIME format (Part 2)
Mon, Apr 4th 2011 49
Geek-o-Terica 17: Accuracy vs. Precision, The Time Drift Chronicles
Mon, Sep 9th 2013 49
"Employee of the Quarter" (not)
Fri, Dec 27th 2013 48
Some Interesting Tech I’ve Been Involved With Lately
Sat, Apr 20th 2013 41


How to get ALL of the documents: db.search() vs. db.AllDocuments vs. AllDocuments view
Bob Balaban    

Greetings, Geeks!

Have you ever had to write an agent that looks at ALL of the documents in a database? As is usual with LotusScript/Java and the back-end classes in Notes, there's more than one way to do that.

Actually, there are (at least) 3 common techniques:
1) Database.AllDocuments
2) Get the "All Documents" view and iterate
3) Database.search("@all")

I used to always use (and recommend) Database.AllDocuments, which returns a DocumentCollection instance. It's very efficient, because it maps to an optimized C API call that retrieves a list of all NOTEIDs (as an IDTable) in the database. This translates to a LotusScript DocumentCollection instance very easily, and none of the document objects is actually "opened" until you access it from the DocumentCollection,

Then, somewhere along the way, I discovered that a change had been made to the behavior of this property. Originally (Notes 4.x), the code in the back-end classes implementation filtered out any deletion stubs in the IDTable resulting from the C API call. Again, this was an efficient operation, made possible by the fact that the NOTEIDs of deletion stubs all have a high bit set (0x80000000L). So you could tell the API to remove all values in the IDTable greater than  0x80000000L, and what was left would be all valid NOTEIDs.

At some point (not sure when, or why), this final filtering operation was removed. So now, if you use Database.AllDocuments, you might find that as you iterate through the document objects in the result set, some of those documents will be "invalid" or deleted. It can be hard to work around this in your code, as the DocumentCollection.getNextDocument() call does not usually work if the "current" document object is invalid. See here and here for some of the details.

I recently worked on a project where the speed of acquiring all the document objects in the NSF and iterating through them was really important. The AllDocuments/GetNth technique was too slow. So I thought I'd try iterating through the "All Documents" view instead (this particular project was processing dbs derived from the standard mail template). I thought it would be pretty fast because all the indexing would have already been done, and it was fast. The problem, however, was that it didn't really retrieve ALL of the documents in the database. Why? Here's the mail template "AllDocuments" view selection formula (the "($All)" view):

     SELECT @IsNotMember("A"; ExcludeFromView) & IsMailStationery != 1 & Form != "Group" & Form != "Person"

So any document that has a field "ExcludeFromView" containing an "A", or any document with a Form field containing "Group" or "Person", or any mail stationery document would never show up in the "All" view, and your agent would never see it. For some purposes this would be fine, but not for the project I was doing at the time, I really needed ALL of the documents.

So I fell back on Plan C: do a Database.search using "@all" as the selection criterion. I thought it would be slow, because the search() function visits every document in the database to apply the @function selection formula. But I knew that it would also explicitly exclude any deletion stubs from the result set, and that was a major concern. It turned out that the search was not particularly slow, certainly it's faster than using DocumentCollection.GetNthDocument() on a very large (100K) result set. The performance overall was actually pretty reasonable, and since I know that the resulting DocumentCollection will not contain any deleted documents, I can avoid GetNthDocument and just use GetFirst/NextDocument.

So, there you have it. All 3 techniques "work" (with variations in behavior), you need to think carefully about which one you should use.

Geek ya later!

(Need expert application development architecture/coding help?  Want me to help you invent directory services based on RDBMS?? Contact me at: bbalaban, gmail.com)
Follow me on Twitter @LooseleafLLC
This article ©Copyright 2010 by Looseleaf Software LLC, all rights reserved. You may link to this page, but may not copy without prior approval.



---------------------
http://www.bobzblog.com/tuxedoguy.nsf/dx/how-to-get-all-of-the-documents-db.search-vs.-db.alldocuments-vs.-alldocuments-view
Oct 25, 2010
121 hits



Recent Blog Posts
67


Token bucket!
Sun, Jun 8th 2014 11:10a   Bob Balaban
Greetings, geeks! I came across this problem on a project I'm doing: given a RESTful API on a web application server, how do you "throttle" calls so that a given authenticated user can only make a certain number of calls (N) per time interval (W), where both N and W are configurable. The solution I came up with is basically a “Token Bucket” with a sliding time window. What surprised me when I got something working was how easy it really was to implement. The context in whi [read] Keywords: application applications java server twitter
48


"Employee of the Quarter" (not)
Fri, Dec 27th 2013 2:20p   Bob Balaban
Greetings, geeks! This one is not a technical topic, so feel free to move on. In my work email today, I received a shotgun message announcing the current "Employee of the Quarter". The lucky designee receives preferential parking (near the building entrance) for 3 months. There was a long-ish description of how great the EotQ is, and why they were selected for this honor. Not to take away from someone who is obviously (from the email description) a very hard worker, get-it-done typ [read] Keywords: email twitter
49


Geek-o-Terica 17: Accuracy vs. Precision, The Time Drift Chronicles
Mon, Sep 9th 2013 5:20a   Bob Balaban
Greetings, geeks! Ever need a software timer to see how long it took to do something? This is a common task, right? And the basic technique is simple: capture the current time, do your thing, preferably for a while so that you average out a large number of operations to eliminate random perturbations, capture the current time again, and report on the difference between the two times. Sounds easy, right? But there are complications. Such as: What is the relative granularity of the time span you a [read] Keywords: applications java laptop network server twitter vm
41


Some Interesting Tech I’ve Been Involved With Lately
Sat, Apr 20th 2013 8:00p   Bob Balaban
Greetings, geeks! Some of you have been wondering what I've been up to lately. No, not a stealth-mode startup, not a new API for Notes/Domino, nothing like that. In fact, I'm at about the 6-month mark in a whole new (to me) tech area: storage and storage management. It's one of those things that's a whole lot more complicated than you'd ever think, until you start to dig into it. Me, I thought a "disk" started and ended with that box on the end of a USB cable that you plug into a la [read] Keywords: domino notes laptop twitter
33


The Interesting Difference Between "Exception" and "RuntimeException" (Java)
Sat, Feb 2nd 2013 6:30p   Bob Balaban
Greetings Geeks! I hope everyone who attended Lotusphere IBMConnect '13 had a good time and has returned home safely. I didn't attend (again, I stopped going after the first 17), but I've been hearing that it was pretty good. I learned something new about certain types of exceptions in Java programming the other day. Here's the context: I had an Enum class with a few items in it I had a String that came from outside the program that I wanted to convert into an enum'ed constant Thi [read] Keywords: lotusphere application java twitter
52


Geek-o-Terica 14: How Using "LimitEntries" Can Mess Up Db Replication (Unintended Side-Effects)
Wed, Jan 16th 2013 12:12p   Bob Balaban
Greetings, Geeks! This one is going to get uber-geeky real fast, so hang onto your gaming consoles. We need a little background, but I'll keep it brief. Most of you probably know that whenever a document is saved (written to disk in the NSF), 2 list items automatically get updated: $UpdatedBy is a list of the names of the people who modified (and, initially, created) the document. The $Revisions item is a list of date/time values indicating the times at which the updates occurred. So, t [read] Keywords: lotus notes ods application database development properties twitter




34


"Reflections on Reflecting Light" - Java program
Wed, May 9th 2012 6:10p   Bob Balaban
Greetings Geeks! Today it is my very great pleasure to introduce to you a new guest blogger: my son David. He has graciously allowed me to prevail upon him to write about his recent science fair project, titled "Reflections on Reflecteing Light". That may sound innocuous, but to quote one of my favorite movies, "That's no ordinary rabbit!". David took this project through our local high school's science fair, was invited to the regional fair, and then to the Massachusetts state science [read] Keywords: blogger email java
34


Happy 5th? Yeah, 5th
Tue, Apr 10th 2012 8:40p   Bob Balaban
Greetings Geeks! So. March 30, 2007 was my first blog entry. Evidently I missed my own 5th Blanniversary! :-( Better late than never, I suppose. In theory, anyway :-) I know I don't post here often, but I do hope that when I do post, it's worth reading. For some people, anyway. I'll quote from a nice person who commented on that first post: "energetic,funny,cute--that's what I feel about you from your special explanation about "In Theory".I think you must be very content while wr [read] Keywords: lotus notes xpages application development twitter
30


"Programming Domino With Java" - Final Flush
Mon, Jan 2nd 2012 3:10a   Bob Balaban
Happy New Year, Geeks! Here's the 7th (and final) installment of the book. Thanks again for all the positive feedback. The first installment can be found here The 2nd installment is here The 3rd is here The 4th is here The 5th is here and the 6th is here All of the book content (as is all of the content on this blog) is Copyright 1998 and 2011 by Looseleaf Software, Inc. You may not reproduce or distribute the book's content without permission from me. Some Caveats and ex [read] Keywords: domino ibm lotus notes R5 application development java twitter
34


"Programming Domino With Java" - Sixth Sip
Tue, Dec 6th 2011 10:30a   Bob Balaban
Greetings, Geeks! Here's the 6th installment of the book. Thanks again for all the positive feedback. The first installment can be found here The 2nd installment is here The 3rd is here The 4th is here The 5th is here All of the book content (as is all of the content on this blog) is Copyright 1998 and 2011 by Looseleaf Software, Inc. You may not reproduce or distribute the book's content without permission from me. Some Caveats and explanations: - This book is now 12 ye [read] Keywords: domino ibm lotus lsx notes R5 application development interface java microsoft twitter




Created and Maintained by Yancy Lent - About - Blog Submission - Suggestions - Change Log - Blog Widget - Advertising - Mobile Edition