|Latest 7 Posts
| Side-Project Monday Evening|
Tue, Jun 27th 2017 6
| Including a Headless DDE Build in a Maven Tree|
Tue, Mar 14th 2017 10
| That Java Thing, Part 17: My Current XPages Plug-in Dev Environment|
Sun, Feb 26th 2017 9
| Slides From My Connect 2017 Presentations|
Fri, Feb 24th 2017 9
| The State of Domino App Dev Post-Connect-2017|
Fri, Feb 24th 2017 8
| Reforming the Blog in Darwino, Part 2|
Thu, Feb 16th 2017 7
| Connect 2017 Final Stretch|
Wed, Feb 15th 2017 7
| Setting up nginx in Front of a Domino Server|
Thu, Sep 18th 2014 18
| How I Maven-ized My Framework|
Mon, Dec 8th 2014 15
| Things I Rarely Use: sessionScope|
Thu, Jul 3rd 2014 13
| Working with Rich Text's MIME Structure|
Wed, Jul 8th 2015 13
| Including a Headless DDE Build in a Maven Tree|
Tue, Mar 14th 2017 10
| Building an App with the frostillic.us Framework, Part 4|
Thu, Jul 17th 2014 9
| Arbitrary Authentication with an nginx Reverse Proxy|
Mon, Sep 22nd 2014 9
| Release Weekend: ODA and Darwino|
Tue, Aug 2nd 2016 9
| Quick XPages Utility: Keep Alive and Alert|
Tue, Aug 30th 2016 9
| Slides From My Connect 2017 Presentations|
Fri, Feb 24th 2017 9
||Delving Into NSF Raw Item Data
For the Design package of the OpenNTF Domino API, one of my goals was to add
the ability to access the supporting structures around XPages: file resources,
images, CSS, and, crucially, Java classes and the XPages themselves. My weapon
of choice for the whole API is DXL, and this covered the job nicely for most of
those aspects. DXL has clean representations for the first three in the list:
specialized XML formats containing the pertinent metadata and a single
BASE64-encoded block of the actual file data.
That's not the case for Java classes and XPages, however. For those, DXL falls
back to the "note" format (inaptly referred to as "Binary DXL" in the Designer
prefs). What sets this format apart from the element-specific DXL is that it
doesn't attempt to present the data in a particularly friendly way - no
specialized file- or image-resource support, no converting of agent code to a
readable format, and no converting of rich text to the HTML-like structure
you've seen if you've worked with DXL. Not everything is stored as binary data
in this format - text, numbers, (some) formulas, and (most) date/time values
are still converted to a human-friendly form. But the point remains: this
format is closer to the data you would get by using the C API. In fact, for the
"rawitemdata" entries, it is the data you would get by using the C API. This
came up originally in the Design API and again the last couple days in the
"nsfdata" package I'm working on to replace the design innards.
I'd like to step back a moment for that point to go over the various APIs of
Domino. The normal Java API we work with - the lotus.domino.local one, not the
OpenNTF one - is a third level of abstraction (or maybe fourth - I'm not sure
if there's an extra layer of C++ stuff in here). The Java API is a fairly-thin
wrapper around the same back-end classes used in LotusScript (hence "LSXBE").
Other than presenting some data types as Java native types like String and int
and providing a few convenience methods like DateTime#toJavaDate, it doesn't
bring too much to the table.
The "LSXBE" objects have a more-complicated job (assuming there's not another
layer): they bring object-oriented order to the lowest level, the C API. The C
API is basically a big soup of data structures, constants, type aliases (e.g.
DWORD = 32-bit integer), and functions to operate on these. As is usually the
case with a C API, dealing with it is onerous, and so the higher-level APIs
wrap these capabilities into the objects you know and love. This wrapping has a
number of significant limitations in functionality and speed, but overall the
capabilities you see in the objects have direct analogues at the C level.
Those limitations, incidentally, are laid bare by the XPage environment's
cheating-like-crazy "napi" classes.
But the point is that using the higher-level APIs generally shields you from
the harsh realities of the C structures, but they leak through in the
"rawitemdata" elements in DXL. As it turns out, data in these elements, when
BASE64-decoded, can be accessed according to the specifications in the C API
Most of the item data structures are both pretty straightforward and not
actually presented in DXL anyway (numbers-list items, for example, are
basically a count of entries and then a series of 64-bit floating-point values
(oddly, the API supports the concept of a number range like a date range, but
NSF does not)). There are a couple types that take some more understanding,
though, and foremost among them is Composite Data.
Composite Data is primarily the format that NSF uses to store rich text data,
though it's full of components to store non-RT data like file resources. A
Composite Data item is essentially an array of "records", which consists of a
A number indicating the type of the record (potentially when combined with the
next part). A record type is something like "TEXT", "TABLEBEGIN", or "LINK":
one of the core components that make up Composite Data. You can think of these
sort of like XML start and end tags.
A number indicating either the stated size of the record or, if it's 0 or 255,
the size of the number following this that indicates the stated size of the
record. C APIs are full of crap like that.
If the size number was 0 or 255, a 16- or 32-bit integer indicating the stated
size of the record.
Any fixed-size structure data stored in the record. For example, CDFILEHEADER,
one of the records used in file resources, contains a number of fields
indicating things like the length of the file extension (I don't know why), the
number of CDFILESEGMENTs that make up the file, and so forth.
Any variable-length data stored in this record. Not all record types contain
any special data beyond the basic struct, but many do. This usually consists of
any associated strings (such as the aforementioned file extension) or raw
binary data like that of a file segment. There may also be padding bytes
attached to strings when the number of bytes used to store them is non-even.
This structure is why RichTextNavigator acts the awkward way it does.
The nice thing about understanding this is that, once you do, it opens a window
into all sorts of low-level operations. You're no longer restricted to working
with just the elements that "friendly" DXL presents: while you still have to do
the work of understanding the pertinent records, the work is straightforward. I
originally used this as a way to read and write file-resource-type notes, but
my new classes are meant to be more generic, opening the door to arbitrary
Composite Data manipulation.
So... okay... why?
As I mentioned at the start, the impetus to my delving this past week was to
clean up the backend of the Design API to work more generically, rather than
consisting of one-offs to deal with the items just in file resources. The side
effects are intriguing in a "mad scentist" sort of way, though. In effect, what
I've been building so far is a mechanism for accessing and manipulating native
NSF data structures without the presence of an actual NSF (or, in fact, any
Notes/Domino dependency at all), in the process completing the utility of DXL.
The implications of a fully-function non-NSF store of NSF data are fascinating.
It's also important foundational work if I deal with the C API more directly in
the future. Years back, I dabbled in a Ruby wrapper around the API, and the
concept has never quite left my brain. As the XPage NAPI demonstrates, there's
a tremendous amount of speed and functionality benefit to be had in bypassing
the legacy Java API completely, and I may travel more fully down that path one
But the final reason why is "because it's there". Learning more about these
underlying structures provides tremendous insight into why Domino does the
things it does. It's also just good for my brain to deal with an API with
conventions other than the legacy Java API or Java-standard semantics.
So I'm interested to see where this will go in the long term. It's possible its
life will be primarily as a slightly-better back end for the existing Design
API functionality, but it opens up numerous potential roads for future
Jul 29, 2014
| Recent Blog Posts
Side-Project Monday Evening|
Tue, Jun 27th 2017 1:49p Jesse Gallagher
Yesterday, in one of my various Slack chats, the topic of JShell - the Java 9
REPL - came up in the context of how useful it would be for XPages development.
Being able to open up a "shell" into a running XPages application could be
really useful in a lot of ways - and I think that the XPages Debug Toolbar has
an SSJS-evaluate feature that would do something like this.
Still, it got me looking around a bit, and I ran across Groovysh Server, which
is a project that combines Apache's SSH
Including a Headless DDE Build in a Maven Tree|
Tue, Mar 14th 2017 4:45p Jesse Gallagher
Most of my Domino projects nowadays have two components: a suite of OSGi
plugins/features and at least one NSF. Historically, I've kept the NSF part
separate from the OSGi plugin projects - I'll keep the ODP in the repo, but
then usually also keep a recent "build" made by copying the database from my
dev server, and then include that built version in the result using the Maven
Assembly plugin. This works, but it's not quite ideal: part of the benefit of
having a Maven project being au
That Java Thing, Part 17: My Current XPages Plug-in Dev Environment|
Sun, Feb 26th 2017 4:23p Jesse Gallagher
It's been a while since I started this series on Java development, but I've
been meaning for a bit now to crack it back open to discuss my current
development setup for plug-ins, since it's changed a bit.
The biggest change is that, thanks to Serdar's work on the latest XPages SDK
release, I now have Domino running plug-ins from my OS X Eclipse workspace.
Previously, I switched between either running on the Mac and doing manual
builds or slumming it in Eclipse in Windows. Having just t
Slides From My Connect 2017 Presentations|
Fri, Feb 24th 2017 9:29p Jesse Gallagher
At this year's Connect, Philippe Riand and I co-presented two sessions: one on
ways to integrate your apps into the Connections UI and one on Darwino's role
for Domino developers. I've uploaded the slides to SlideShare:
DEV-1430 - IBM Connections Integration: Exploring the Long List of Options
DEV-1467 - Give a New Life to Your Notes/Domino Applications and Leverage IBM
Bluemix, Watson, & Connections (effectively, "the Darwino session")
The State of Domino App Dev Post-Connect-2017|
Fri, Feb 24th 2017 9:28p Jesse Gallagher
I'm en route back from this year's IBM Connect in San Francisco, and this plane
ride is giving me a good chance to chew over the implications for Domino
First off, I'll put my bias in this matter right up front: Darwino, which I've
been working on and discussing quite a bit, is one of the three "chosen"
vendors for app enhancement/modernization/what-have-you. So, while this post
isn't going to be about Darwino specifically, it's certainly pertinent for me.
In any case,
Reforming the Blog in Darwino, Part 2|
Thu, Feb 16th 2017 8:41p Jesse Gallagher
During the run-up to Connect next week, I turned my gaze back to my
indefinite-term project of reforming this blog in Darwino.
When last I left it publicly, I had set up replication between a copy of the
database and a Darwino app. After that post, I did a bit of tinkering in the
direction of building a (J)Ruby on Rails front-end for it, next to the "j2ee"
project. That side effort may bear fruit in time (as I recall, I got the
embedded web app serving default pages, but didn't implemen
Connect 2017 Final Stretch|
Wed, Feb 15th 2017 12:16p Jesse Gallagher
IBM Connect 2017 is less than a week away, and I've been furiously prepping for
a couple parts of what is promising to be a busy conference.
On Monday, before the official kickoff of the conference, OpenNTF is co-hosting
a Hackathon, where attendees will work on one of several potential projects.
The goal is to learn about new development methods, work with new people, and
hopefully kick off some useful open-source projects to boot.
During the conference proper, I'll be presenting two se
December Is Self-Aggrandizement Month, Apparently|
Sat, Dec 17th 2016 3:21p Jesse Gallagher
It's been a busy month (couple of years, really), but the last few weeks in
particular have involved a couple minor announcements that I'm quite
On the 14th, IBM announced the 2017 class of IBM Champions for ICS, and they
included me on the list. It's been a joy to be considered a Champion for the
last few years, and 2017 promises to be an interesting year to continue that in
our slice of the development world.
Mere days later, IBM sent out notifications about Connect
The New Podcast is a Real Thing: WTF Tech Episode 1|
Mon, Oct 31st 2016 10:31a Jesse Gallagher
As intimated at the end of the last This Week in Lotus, Stuart, Darren, and I
have launched a new podcast in a similar vein: WTF Tech. Since we're all in the
IBM sphere, that'll be the natural starting point for the topics we cover, but
it's not going to be IBM-focused as such. For this first episode, we lucked out
and had a couple-weeks period chock full of announcements, so we had plenty of
material. Give it a listen!
Cramming Rails Into A Maven Tree|
Mon, Sep 26th 2016 1:25p Jesse Gallagher
Because I'm me, one of the paths I'm investigating for my long-term
blog-reformation project is seeing if I can get Ruby on Rails in there. I've
been carrying a torch for the language and framework for forever, and so it'd
be good to actually write a real thing in it for once.
This has been proving to be a very interesting thing to try to do well.
Fortunately, the basics of "run Rails in a Java server" have been well worked
out: the JRuby variant of the language is top-notch and the