203 Lotus blogs updated hourly. Who will post next? Home | Blogs | Search | About 
Latest 7 Posts
First Steps to Code Coverage Analysis in Domino Plugins
Thu, Nov 9th 2017 7
New Small Project: p2site-maven-plugin
Thu, Oct 26th 2017 4
Side-Project Monday Evening
Tue, Jun 27th 2017 3
Including a Headless DDE Build in a Maven Tree
Tue, Mar 14th 2017 3
That Java Thing, Part 17: My Current XPages Plug-in Dev Environment
Sun, Feb 26th 2017 7
Slides From My Connect 2017 Presentations
Fri, Feb 24th 2017 6
The State of Domino App Dev Post-Connect-2017
Fri, Feb 24th 2017 6
Top 10
Using a ViewHandler to Serve a Different XPage
Sat, May 3rd 2014 24
How I Maven-ized My Framework
Mon, Dec 8th 2014 11
Setting up nginx in Front of a Domino Server
Thu, Sep 18th 2014 10
Change Is In The Air
Fri, Aug 26th 2016 10
Cramming Rails Into A Maven Tree
Mon, Sep 26th 2016 9
Reforming the Blog in Darwino, Part 2
Thu, Feb 16th 2017 9
I Posted My WrapBootstrap Ace Renderkit
Tue, Sep 30th 2014 8
Things I Rarely Use: sessionScope
Thu, Jul 3rd 2014 7
The Trouble With Developing on Domino
Tue, Jul 8th 2014 7
Be a Better Programmer, Part 4
Thu, Aug 14th 2014 7

Delving Into NSF Raw Item Data
Twitter Google+ Facebook LinkedIn Addthis Email Gmail Flipboard Reddit Tumblr WhatsApp StumbleUpon Yammer Evernote Delicious
Jesse Gallagher    

For the Design package of the OpenNTF Domino API, one of my goals was to add the ability to access the supporting structures around XPages: file resources, images, CSS, and, crucially, Java classes and the XPages themselves. My weapon of choice for the whole API is DXL, and this covered the job nicely for most of those aspects. DXL has clean representations for the first three in the list: specialized XML formats containing the pertinent metadata and a single BASE64-encoded block of the actual file data. That's not the case for Java classes and XPages, however. For those, DXL falls back to the "note" format (inaptly referred to as "Binary DXL" in the Designer prefs). What sets this format apart from the element-specific DXL is that it doesn't attempt to present the data in a particularly friendly way - no specialized file- or image-resource support, no converting of agent code to a readable format, and no converting of rich text to the HTML-like structure you've seen if you've worked with DXL. Not everything is stored as binary data in this format - text, numbers, (some) formulas, and (most) date/time values are still converted to a human-friendly form. But the point remains: this format is closer to the data you would get by using the C API. In fact, for the "rawitemdata" entries, it is the data you would get by using the C API. This came up originally in the Design API and again the last couple days in the "nsfdata" package I'm working on to replace the design innards. The APIs I'd like to step back a moment for that point to go over the various APIs of Domino. The normal Java API we work with - the lotus.domino.local one, not the OpenNTF one - is a third level of abstraction (or maybe fourth - I'm not sure if there's an extra layer of C++ stuff in here). The Java API is a fairly-thin wrapper around the same back-end classes used in LotusScript (hence "LSXBE"). Other than presenting some data types as Java native types like String and int and providing a few convenience methods like DateTime#toJavaDate, it doesn't bring too much to the table. The "LSXBE" objects have a more-complicated job (assuming there's not another layer): they bring object-oriented order to the lowest level, the C API. The C API is basically a big soup of data structures, constants, type aliases (e.g. DWORD = 32-bit integer), and functions to operate on these. As is usually the case with a C API, dealing with it is onerous, and so the higher-level APIs wrap these capabilities into the objects you know and love. This wrapping has a number of significant limitations in functionality and speed, but overall the capabilities you see in the objects have direct analogues at the C level. Those limitations, incidentally, are laid bare by the XPage environment's cheating-like-crazy "napi" classes. But the point is that using the higher-level APIs generally shields you from the harsh realities of the C structures, but they leak through in the "rawitemdata" elements in DXL. As it turns out, data in these elements, when BASE64-decoded, can be accessed according to the specifications in the C API struct documentation. Composite Data Most of the item data structures are both pretty straightforward and not actually presented in DXL anyway (numbers-list items, for example, are basically a count of entries and then a series of 64-bit floating-point values (oddly, the API supports the concept of a number range like a date range, but NSF does not)). There are a couple types that take some more understanding, though, and foremost among them is Composite Data. Composite Data is primarily the format that NSF uses to store rich text data, though it's full of components to store non-RT data like file resources. A Composite Data item is essentially an array of "records", which consists of a few things: A number indicating the type of the record (potentially when combined with the next part). A record type is something like "TEXT", "TABLEBEGIN", or "LINK": one of the core components that make up Composite Data. You can think of these sort of like XML start and end tags. A number indicating either the stated size of the record or, if it's 0 or 255, the size of the number following this that indicates the stated size of the record. C APIs are full of crap like that. If the size number was 0 or 255, a 16- or 32-bit integer indicating the stated size of the record. Any fixed-size structure data stored in the record. For example, CDFILEHEADER, one of the records used in file resources, contains a number of fields indicating things like the length of the file extension (I don't know why), the number of CDFILESEGMENTs that make up the file, and so forth. Any variable-length data stored in this record. Not all record types contain any special data beyond the basic struct, but many do. This usually consists of any associated strings (such as the aforementioned file extension) or raw binary data like that of a file segment. There may also be padding bytes attached to strings when the number of bytes used to store them is non-even. This structure is why RichTextNavigator acts the awkward way it does. The nice thing about understanding this is that, once you do, it opens a window into all sorts of low-level operations. You're no longer restricted to working with just the elements that "friendly" DXL presents: while you still have to do the work of understanding the pertinent records, the work is straightforward. I originally used this as a way to read and write file-resource-type notes, but my new classes are meant to be more generic, opening the door to arbitrary Composite Data manipulation. So... okay... why? As I mentioned at the start, the impetus to my delving this past week was to clean up the backend of the Design API to work more generically, rather than consisting of one-offs to deal with the items just in file resources. The side effects are intriguing in a "mad scentist" sort of way, though. In effect, what I've been building so far is a mechanism for accessing and manipulating native NSF data structures without the presence of an actual NSF (or, in fact, any Notes/Domino dependency at all), in the process completing the utility of DXL. The implications of a fully-function non-NSF store of NSF data are fascinating. It's also important foundational work if I deal with the C API more directly in the future. Years back, I dabbled in a Ruby wrapper around the API, and the concept has never quite left my brain. As the XPage NAPI demonstrates, there's a tremendous amount of speed and functionality benefit to be had in bypassing the legacy Java API completely, and I may travel more fully down that path one day. But the final reason why is "because it's there". Learning more about these underlying structures provides tremendous insight into why Domino does the things it does. It's also just good for my brain to deal with an API with conventions other than the legacy Java API or Java-standard semantics. So I'm interested to see where this will go in the long term. It's possible its life will be primarily as a slightly-better back end for the existing Design API functionality, but it opens up numerous potential roads for future capabilities.

Jul 29, 2014
6 hits

Recent Blog Posts
First Steps to Code Coverage Analysis in Domino Plugins
Thu, Nov 9th 2017 2:53p   Jesse Gallagher
I'm always interested in getting the computer to tell me how to tell it what to do more successfully, and, to further that pursuit, I've started taking an interest in code coverage. If you're not familiar with the term, "code coverage" refers to reporting on which lines of code were actually executed during runtime, most commonly in association with unit tests. Eclipse (and presumably other IDEs) has support for this, and I've decided to give it a shot. Since I'm starting this out
New Small Project: p2site-maven-plugin
Thu, Oct 26th 2017 5:17p   Jesse Gallagher
It's no secret that I have a love/hate relationship with developing for OSGi platforms with Maven. The giant divide between "all-in" Tycho projects (which limit your options with normal Maven features) and trying to bolt on OSGi support in an otherwise-normal project creates an array of problems big and small. Some of those hurdles would be difficult to bridge, such as any automated tests that want to test the proper functioning of OSGi services. However, not all projects need that - i
Side-Project Monday Evening
Tue, Jun 27th 2017 1:49p   Jesse Gallagher
Yesterday, in one of my various Slack chats, the topic of JShell - the Java 9 REPL - came up in the context of how useful it would be for XPages development. Being able to open up a "shell" into a running XPages application could be really useful in a lot of ways - and I think that the XPages Debug Toolbar has an SSJS-evaluate feature that would do something like this. Still, it got me looking around a bit, and I ran across Groovysh Server, which is a project that combines Apache's SSH
Including a Headless DDE Build in a Maven Tree
Tue, Mar 14th 2017 4:45p   Jesse Gallagher
Most of my Domino projects nowadays have two components: a suite of OSGi plugins/features and at least one NSF. Historically, I've kept the NSF part separate from the OSGi plugin projects - I'll keep the ODP in the repo, but then usually also keep a recent "build" made by copying the database from my dev server, and then include that built version in the result using the Maven Assembly plugin. This works, but it's not quite ideal: part of the benefit of having a Maven project being au
That Java Thing, Part 17: My Current XPages Plug-in Dev Environment
Sun, Feb 26th 2017 4:23p   Jesse Gallagher
It's been a while since I started this series on Java development, but I've been meaning for a bit now to crack it back open to discuss my current development setup for plug-ins, since it's changed a bit. The biggest change is that, thanks to Serdar's work on the latest XPages SDK release, I now have Domino running plug-ins from my OS X Eclipse workspace. Previously, I switched between either running on the Mac and doing manual builds or slumming it in Eclipse in Windows. Having just t
Slides From My Connect 2017 Presentations
Fri, Feb 24th 2017 9:29p   Jesse Gallagher
At this year's Connect, Philippe Riand and I co-presented two sessions: one on ways to integrate your apps into the Connections UI and one on Darwino's role for Domino developers. I've uploaded the slides to SlideShare: DEV-1430 - IBM Connections Integration: Exploring the Long List of Options DEV-1467 - Give a New Life to Your Notes/Domino Applications and Leverage IBM Bluemix, Watson, & Connections (effectively, "the Darwino session")
The State of Domino App Dev Post-Connect-2017
Fri, Feb 24th 2017 9:28p   Jesse Gallagher
I'm en route back from this year's IBM Connect in San Francisco, and this plane ride is giving me a good chance to chew over the implications for Domino developers. First off, I'll put my bias in this matter right up front: Darwino, which I've been working on and discussing quite a bit, is one of the three "chosen" vendors for app enhancement/modernization/what-have-you. So, while this post isn't going to be about Darwino specifically, it's certainly pertinent for me. In any case,
Reforming the Blog in Darwino, Part 2
Thu, Feb 16th 2017 8:41p   Jesse Gallagher
During the run-up to Connect next week, I turned my gaze back to my indefinite-term project of reforming this blog in Darwino. When last I left it publicly, I had set up replication between a copy of the database and a Darwino app. After that post, I did a bit of tinkering in the direction of building a (J)Ruby on Rails front-end for it, next to the "j2ee" project. That side effort may bear fruit in time (as I recall, I got the embedded web app serving default pages, but didn't implemen
Connect 2017 Final Stretch
Wed, Feb 15th 2017 12:16p   Jesse Gallagher
IBM Connect 2017 is less than a week away, and I've been furiously prepping for a couple parts of what is promising to be a busy conference. On Monday, before the official kickoff of the conference, OpenNTF is co-hosting a Hackathon, where attendees will work on one of several potential projects. The goal is to learn about new development methods, work with new people, and hopefully kick off some useful open-source projects to boot. During the conference proper, I'll be presenting two se
December Is Self-Aggrandizement Month, Apparently
Sat, Dec 17th 2016 3:21p   Jesse Gallagher
It's been a busy month (couple of years, really), but the last few weeks in particular have involved a couple minor announcements that I'm quite appreciative for. On the 14th, IBM announced the 2017 class of IBM Champions for ICS, and they included me on the list. It's been a joy to be considered a Champion for the last few years, and 2017 promises to be an interesting year to continue that in our slice of the development world. Mere days later, IBM sent out notifications about Connect

Created and Maintained by Yancy Lent - About - Planet Lotus Blog - Advertising - Mobile Edition