|Latest 7 Posts
| First Steps to Code Coverage Analysis in Domino Plugins|
Thu, Nov 9th 2017 7
| New Small Project: p2site-maven-plugin|
Thu, Oct 26th 2017 4
| Side-Project Monday Evening|
Tue, Jun 27th 2017 3
| Including a Headless DDE Build in a Maven Tree|
Tue, Mar 14th 2017 3
| That Java Thing, Part 17: My Current XPages Plug-in Dev Environment|
Sun, Feb 26th 2017 7
| Slides From My Connect 2017 Presentations|
Fri, Feb 24th 2017 6
| The State of Domino App Dev Post-Connect-2017|
Fri, Feb 24th 2017 6
| Using a ViewHandler to Serve a Different XPage|
Sat, May 3rd 2014 24
| How I Maven-ized My Framework|
Mon, Dec 8th 2014 11
| Setting up nginx in Front of a Domino Server|
Thu, Sep 18th 2014 10
| Change Is In The Air|
Fri, Aug 26th 2016 10
| Cramming Rails Into A Maven Tree|
Mon, Sep 26th 2016 9
| Reforming the Blog in Darwino, Part 2|
Thu, Feb 16th 2017 9
| I Posted My WrapBootstrap Ace Renderkit|
Tue, Sep 30th 2014 8
| Things I Rarely Use: sessionScope|
Thu, Jul 3rd 2014 7
| The Trouble With Developing on Domino|
Tue, Jul 8th 2014 7
| Be a Better Programmer, Part 4|
Thu, Aug 14th 2014 7
||Delving Into NSF Raw Item Data
For the Design package of the OpenNTF Domino API, one of my goals was to add
the ability to access the supporting structures around XPages: file resources,
images, CSS, and, crucially, Java classes and the XPages themselves. My weapon
of choice for the whole API is DXL, and this covered the job nicely for most of
those aspects. DXL has clean representations for the first three in the list:
specialized XML formats containing the pertinent metadata and a single
BASE64-encoded block of the actual file data.
That's not the case for Java classes and XPages, however. For those, DXL falls
back to the "note" format (inaptly referred to as "Binary DXL" in the Designer
prefs). What sets this format apart from the element-specific DXL is that it
doesn't attempt to present the data in a particularly friendly way - no
specialized file- or image-resource support, no converting of agent code to a
readable format, and no converting of rich text to the HTML-like structure
you've seen if you've worked with DXL. Not everything is stored as binary data
in this format - text, numbers, (some) formulas, and (most) date/time values
are still converted to a human-friendly form. But the point remains: this
format is closer to the data you would get by using the C API. In fact, for the
"rawitemdata" entries, it is the data you would get by using the C API. This
came up originally in the Design API and again the last couple days in the
"nsfdata" package I'm working on to replace the design innards.
I'd like to step back a moment for that point to go over the various APIs of
Domino. The normal Java API we work with - the lotus.domino.local one, not the
OpenNTF one - is a third level of abstraction (or maybe fourth - I'm not sure
if there's an extra layer of C++ stuff in here). The Java API is a fairly-thin
wrapper around the same back-end classes used in LotusScript (hence "LSXBE").
Other than presenting some data types as Java native types like String and int
and providing a few convenience methods like DateTime#toJavaDate, it doesn't
bring too much to the table.
The "LSXBE" objects have a more-complicated job (assuming there's not another
layer): they bring object-oriented order to the lowest level, the C API. The C
API is basically a big soup of data structures, constants, type aliases (e.g.
DWORD = 32-bit integer), and functions to operate on these. As is usually the
case with a C API, dealing with it is onerous, and so the higher-level APIs
wrap these capabilities into the objects you know and love. This wrapping has a
number of significant limitations in functionality and speed, but overall the
capabilities you see in the objects have direct analogues at the C level.
Those limitations, incidentally, are laid bare by the XPage environment's
cheating-like-crazy "napi" classes.
But the point is that using the higher-level APIs generally shields you from
the harsh realities of the C structures, but they leak through in the
"rawitemdata" elements in DXL. As it turns out, data in these elements, when
BASE64-decoded, can be accessed according to the specifications in the C API
Most of the item data structures are both pretty straightforward and not
actually presented in DXL anyway (numbers-list items, for example, are
basically a count of entries and then a series of 64-bit floating-point values
(oddly, the API supports the concept of a number range like a date range, but
NSF does not)). There are a couple types that take some more understanding,
though, and foremost among them is Composite Data.
Composite Data is primarily the format that NSF uses to store rich text data,
though it's full of components to store non-RT data like file resources. A
Composite Data item is essentially an array of "records", which consists of a
A number indicating the type of the record (potentially when combined with the
next part). A record type is something like "TEXT", "TABLEBEGIN", or "LINK":
one of the core components that make up Composite Data. You can think of these
sort of like XML start and end tags.
A number indicating either the stated size of the record or, if it's 0 or 255,
the size of the number following this that indicates the stated size of the
record. C APIs are full of crap like that.
If the size number was 0 or 255, a 16- or 32-bit integer indicating the stated
size of the record.
Any fixed-size structure data stored in the record. For example, CDFILEHEADER,
one of the records used in file resources, contains a number of fields
indicating things like the length of the file extension (I don't know why), the
number of CDFILESEGMENTs that make up the file, and so forth.
Any variable-length data stored in this record. Not all record types contain
any special data beyond the basic struct, but many do. This usually consists of
any associated strings (such as the aforementioned file extension) or raw
binary data like that of a file segment. There may also be padding bytes
attached to strings when the number of bytes used to store them is non-even.
This structure is why RichTextNavigator acts the awkward way it does.
The nice thing about understanding this is that, once you do, it opens a window
into all sorts of low-level operations. You're no longer restricted to working
with just the elements that "friendly" DXL presents: while you still have to do
the work of understanding the pertinent records, the work is straightforward. I
originally used this as a way to read and write file-resource-type notes, but
my new classes are meant to be more generic, opening the door to arbitrary
Composite Data manipulation.
So... okay... why?
As I mentioned at the start, the impetus to my delving this past week was to
clean up the backend of the Design API to work more generically, rather than
consisting of one-offs to deal with the items just in file resources. The side
effects are intriguing in a "mad scentist" sort of way, though. In effect, what
I've been building so far is a mechanism for accessing and manipulating native
NSF data structures without the presence of an actual NSF (or, in fact, any
Notes/Domino dependency at all), in the process completing the utility of DXL.
The implications of a fully-function non-NSF store of NSF data are fascinating.
It's also important foundational work if I deal with the C API more directly in
the future. Years back, I dabbled in a Ruby wrapper around the API, and the
concept has never quite left my brain. As the XPage NAPI demonstrates, there's
a tremendous amount of speed and functionality benefit to be had in bypassing
the legacy Java API completely, and I may travel more fully down that path one
But the final reason why is "because it's there". Learning more about these
underlying structures provides tremendous insight into why Domino does the
things it does. It's also just good for my brain to deal with an API with
conventions other than the legacy Java API or Java-standard semantics.
So I'm interested to see where this will go in the long term. It's possible its
life will be primarily as a slightly-better back end for the existing Design
API functionality, but it opens up numerous potential roads for future
Jul 29, 2014
| Recent Blog Posts
First Steps to Code Coverage Analysis in Domino Plugins|
Thu, Nov 9th 2017 2:53p Jesse Gallagher
I'm always interested in getting the computer to tell me how to tell it what to
do more successfully, and, to further that pursuit, I've started taking an
interest in code coverage.
If you're not familiar with the term, "code coverage" refers to reporting on
which lines of code were actually executed during runtime, most commonly in
association with unit tests. Eclipse (and presumably other IDEs) has support
for this, and I've decided to give it a shot.
Since I'm starting this out
New Small Project: p2site-maven-plugin|
Thu, Oct 26th 2017 5:17p Jesse Gallagher
It's no secret that I have a love/hate relationship with developing for OSGi
platforms with Maven. The giant divide between "all-in" Tycho projects (which
limit your options with normal Maven features) and trying to bolt on OSGi
support in an otherwise-normal project creates an array of problems big and
Some of those hurdles would be difficult to bridge, such as any automated tests
that want to test the proper functioning of OSGi services. However, not all
projects need that - i
Side-Project Monday Evening|
Tue, Jun 27th 2017 1:49p Jesse Gallagher
Yesterday, in one of my various Slack chats, the topic of JShell - the Java 9
REPL - came up in the context of how useful it would be for XPages development.
Being able to open up a "shell" into a running XPages application could be
really useful in a lot of ways - and I think that the XPages Debug Toolbar has
an SSJS-evaluate feature that would do something like this.
Still, it got me looking around a bit, and I ran across Groovysh Server, which
is a project that combines Apache's SSH
Including a Headless DDE Build in a Maven Tree|
Tue, Mar 14th 2017 4:45p Jesse Gallagher
Most of my Domino projects nowadays have two components: a suite of OSGi
plugins/features and at least one NSF. Historically, I've kept the NSF part
separate from the OSGi plugin projects - I'll keep the ODP in the repo, but
then usually also keep a recent "build" made by copying the database from my
dev server, and then include that built version in the result using the Maven
Assembly plugin. This works, but it's not quite ideal: part of the benefit of
having a Maven project being au
That Java Thing, Part 17: My Current XPages Plug-in Dev Environment|
Sun, Feb 26th 2017 4:23p Jesse Gallagher
It's been a while since I started this series on Java development, but I've
been meaning for a bit now to crack it back open to discuss my current
development setup for plug-ins, since it's changed a bit.
The biggest change is that, thanks to Serdar's work on the latest XPages SDK
release, I now have Domino running plug-ins from my OS X Eclipse workspace.
Previously, I switched between either running on the Mac and doing manual
builds or slumming it in Eclipse in Windows. Having just t
Slides From My Connect 2017 Presentations|
Fri, Feb 24th 2017 9:29p Jesse Gallagher
At this year's Connect, Philippe Riand and I co-presented two sessions: one on
ways to integrate your apps into the Connections UI and one on Darwino's role
for Domino developers. I've uploaded the slides to SlideShare:
DEV-1430 - IBM Connections Integration: Exploring the Long List of Options
DEV-1467 - Give a New Life to Your Notes/Domino Applications and Leverage IBM
Bluemix, Watson, & Connections (effectively, "the Darwino session")
The State of Domino App Dev Post-Connect-2017|
Fri, Feb 24th 2017 9:28p Jesse Gallagher
I'm en route back from this year's IBM Connect in San Francisco, and this plane
ride is giving me a good chance to chew over the implications for Domino
First off, I'll put my bias in this matter right up front: Darwino, which I've
been working on and discussing quite a bit, is one of the three "chosen"
vendors for app enhancement/modernization/what-have-you. So, while this post
isn't going to be about Darwino specifically, it's certainly pertinent for me.
In any case,
Reforming the Blog in Darwino, Part 2|
Thu, Feb 16th 2017 8:41p Jesse Gallagher
During the run-up to Connect next week, I turned my gaze back to my
indefinite-term project of reforming this blog in Darwino.
When last I left it publicly, I had set up replication between a copy of the
database and a Darwino app. After that post, I did a bit of tinkering in the
direction of building a (J)Ruby on Rails front-end for it, next to the "j2ee"
project. That side effort may bear fruit in time (as I recall, I got the
embedded web app serving default pages, but didn't implemen
Connect 2017 Final Stretch|
Wed, Feb 15th 2017 12:16p Jesse Gallagher
IBM Connect 2017 is less than a week away, and I've been furiously prepping for
a couple parts of what is promising to be a busy conference.
On Monday, before the official kickoff of the conference, OpenNTF is co-hosting
a Hackathon, where attendees will work on one of several potential projects.
The goal is to learn about new development methods, work with new people, and
hopefully kick off some useful open-source projects to boot.
During the conference proper, I'll be presenting two se
December Is Self-Aggrandizement Month, Apparently|
Sat, Dec 17th 2016 3:21p Jesse Gallagher
It's been a busy month (couple of years, really), but the last few weeks in
particular have involved a couple minor announcements that I'm quite
On the 14th, IBM announced the 2017 class of IBM Champions for ICS, and they
included me on the list. It's been a joy to be considered a Champion for the
last few years, and 2017 promises to be an interesting year to continue that in
our slice of the development world.
Mere days later, IBM sent out notifications about Connect