Next Project: ODP Compiler
Mon Mar 05 17:32:06 EST 2018
- Next Project: ODP Compiler
- NSF ODP Tooling 1.0
- NSF ODP Tooling Example Project
- NSF ODP Tooling 1.2
- How the ODP Compiler Works, Part 1
- How the ODP Compiler Works, Part 2
- How the ODP Compiler Works, Part 3
- How the ODP Compiler Works, Part 4
- How the ODP Compiler Works, Part 5
- How the ODP Compiler Works, Part 6
- How the ODP Compiler Works, Part 7
One of the larger thorns in my side with my Domino development lately has been trying to automate builds of on-disk projects into NSFs via Jenkins. In theory, the process is pretty straightforward. It even works sometimes! However, particularly once you add in the necessity to deploy OSGi plugins to Designer first and want to run it from Jenkins, things get extraordinarily flaky: Designer may not launch properly from a behind-the-scenes Jenkins runner, the plugin installation may mysteriously fail, and so forth - and the error reporting is difficult at best.
So it's been on my mind for a good while to find a way to get from an ODP to an NSF without involving Designer, and I decided over the last couple days to really take a swing at it. It's not a small task, though; the process involves a number of difficult steps:
- Install and activate provided OSGi plugins
- Create an XPages registry that knows about the XPages libraries installed on the server, including those just contributed
- Translate XPages and Custom Controls into Java source, with intra-file knowledge of the just-added CCs
- Create a Java classpath that matches the plug-in dependencies that Designer derives from the dependant XPages Libraries
- Compile the resultant Java source and any Java classes in the NSF into bytecode
- Recompose the composite data form of the file data for these elements and many file resources into their DXL ".metadata" files for import
- Create an NSF and import all of this
- Compile any LotusScript in source-based libraries from the ODP
- Uninstall any hot-loaded OSGi plugins
Those steps even leave out some fiddly details, like components defined via .xsp-config files in the NSF or XSP-associated .properties files, not to mention any steps I haven't encountered yet. It's a lot of work!
My first hope was to be able to hook into the process that Designer uses, perhaps grabbing a couple pertinent OSGi plugins and going from there. However, from what I can tell, all the involved plugins are intricately tied into many layers of Designer-the-IDE and so are no small matter to use on their own without also including the entire stack. So that left me to cobble together an equivalent process out of parts.
Fortunately, a couple projects have already provided a solid foundation for this. First and foremost is the XPages Bazaar. This is a project that Philippe Riand created a number of years ago, meant to be a workshop for really experimental components in a form less contrained than the ExtLib became. Since he left IBM, it's sat unmaintained, but I figured it'd be a perfect incubator for this project, so I tossed it up on GitHub, recomposed its Maven structure, and cleaned it up a bit for FP10 use. The reason why it makes such a perfect shell is a pair of its features: an XSP interpreter and an on-the-fly Java compiler. The former hooks into the mysterious guts of the XSP runtime to allow for translation of XSP to Java and the latter wraps the official Java compiler API with some OSGi knowledge to compile that along into bytecode.
Even starting with this, the first couple steps still required a lot of digging around. I learned how to install and activate OSGi bundles, how XPages Registries work internally, and made some tweaks to work around problems I encountered. I also encountered the joy of a bizarre javac bug to do with annotations in enum constructors, which my target project used.
Once I had the XPages-side components compiled, the next step was to start composing the NSF. The ODP format for XPages elements and other "file resource"-type entities is to put the code in its "normal" form in a file and then a subset of the DXL in a ".metadata" file next to it. The trouble here is that, even for entities where the file data is stored in the note unprocessed, the storage format isn't a strict binary blob of the file data: it's a composite data stream of file header and segment structures. I thought of two main ways I could go about getting these file resources into the NSF: via IBM's NAPI and by building the structures into the DXL files before import. The NAPI has a convenient FileAccess class for this purpose (presumably used by Designer), but my attempts to use it met primarily with server crashes. I'm sure it's possible to go this route, but I'd already "solved" the DXL problem years ago, for ODA's Design API. So, at least for now, I took the tack of writing out the binary structure manually, pouring it into the DXL as Base64, and importing that. It's a little inefficient, but it works.
Overall, I've made a lot of progress so far, but there's still a lot to be done: not all file types have their data put into the right places, LotusScript isn't properly compiled, Agents don't do anything at all yet, and XPages+CCs aren't actually imported into the NSF. Still, it's in a spot where I'm confident that it can one day work, whichis more than I could have said a week ago. If you'd like, browse around the code and pitch in if it's an itch you'd like to scratch as well.
Cameron Gregor - Mon Mar 05 19:43:12 EST 2018
A mammoth effort, I hope you get there! If only this was something that was supported by IBM / HCL ...
Just to offer an alternative, there is also another way to launch Designer with plugins from a 'dropins' style directory without having to install them first. It was covered in one of Niklas Hiedloff's blog posts a while back but is very hard to find nowadays. I'm still on FP7 so haven't tried it with FP8 and greater.
You place a .link file in the <notes program dir>/framework/rcp/eclipse/links directory
The link file contains a single line 'path=<path to your dropins folder'
Then you set up that dropins folder with a subfolder called eclipse and then that subfolder has plugins and features subfolders
Lastly you need to make changes to the following file (make a backup!)
${notesDataDir}/workspace/.config/org.eclipse.update/platform.xml
This way you can prepare a directory with the dependencies, (a bit like a target platform) and just turn them off an on by using the link files.
I also use this method when developing locally as well mainly for XPages Controls, whenever I add new properties to a control, I re build the update site to the dropins folder, and then restart Designer. Saves having to go through the whole install plugins process. It's also easy to clean out old plugins by deleting the plugins (but Designer must not be running to do that)
On a jenkins server your build can prepare these update site folders before running the NSF builds
I made some ant tasks that do these steps in my BuildXPages project http://camac.github.io/BuildXPages/#TasksDesigner
they are called updateDesignerLink initdesignersite checkplatformxml configuredynamicplugins
WK - Tue Mar 06 04:05:53 EST 2018
Wow, that is amazing, what You did there. The road ahead is long, but still, that is really nice way to start.
I was also tired with instability that designer brings to Jenkins builds and tried to decompile plugins responsible for xpages translation (intellij has great decompiler build in) but complicated internal structure and strong ties to virtual file system was to much for me to handle. The only thing I was able to achieve was to swap module class loader so that we can now work on java part of our projects using outside IDE (we do not use plugins and keep java inside nsf).
No wonder that when we asked IBM (through support, long time ago) if they could provide a way to translate xpages outside IBM designer they responded that this will never happen. I suppose after Philippe R. left there is no one who cares about this technology there. And judging by last webinar HCL will not be different.
Patrick Kwinten - Wed Mar 28 09:37:55 EDT 2018
CI/CD should be important for Domino too. HCL should take a look at it.
Andrew - Thu Oct 25 22:46:39 EDT 2018
Thank you for publishing such a great project!
Just curious whether it is possible to run the ODPExporter in the stand-alone Java app (using only Notes Local Session)?
I tried to the code (ODBExporter) into sample stand-alone app, which looks like the following:
...
NotesThread.sinitThread();
s = NotesFactory.createSession();
NotesSession nSession = new NotesSession();
NotesDatabase db = nSession.getDatabase("/local/_apiTest/crmtest.nsf");
db.open();
System.out.println(">> DB: " + db.getDatabasePath());
System.out.println(">> DB ReplicaID: " + db.getReplicaID());
ODPExporter odpExporter = new ODPExporter(db);
Path exportedPath = odpExporter.export();
System.out.println(">> Export done! Location: " + exportedPath.toFile().getAbsolutePath());
...
But running the code getting the following exception:
>> DB: /local/_apiTest/crmtest.nsf
>> DB ReplicaID: 48258331:003970BF
Exception in thread "main" java.lang.UnsatisfiedLinkError: com.ibm.domino.napi.c.callback.NotesProcInit.initNative(ILjava/lang/Class;)V
at com.ibm.domino.napi.c.callback.NotesProcInit.initNative(Native Method)
at com.ibm.domino.napi.c.callback.IDENUMERATEPROC.<clinit>(IDENUMERATEPROC.java:26)
at org.openntf.nsfodp.exporter.ODPExporter.export(ODPExporter.java:182)
at net.prominic.Exporter.exporterTest1(Exporter.java:50)
at net.prominic.Exporter.main(Exporter.java:23)
Is it feasible at all, or it should be running only in Domino OSGi context?
Thank you!
Andrew - Mon Nov 05 00:54:02 EST 2018
As Jesse suggested me - all what I need is to add the following method call:
com.ibm.domino.napi.c.C.initLibrary(null);
After adding that - stand-alone ODP Exporter successfully exported the NSF to ODP!
So the code should look like:
...
NotesThread.sinitThread();
s = NotesFactory.createSession();
NotesSession nSession = new NotesSession();
com.ibm.domino.napi.c.C.initLibrary(null);
NotesDatabase db = nSession.getDatabase("/local/_apiTest/crmtest.nsf");
db.open();
System.out.println(">> DB: " + db.getDatabasePath());
System.out.println(">> DB ReplicaID: " + db.getReplicaID());
ODPExporter odpExporter = new ODPExporter(db);
Path exportedPath = odpExporter.export();
System.out.println(">> Export done! Location: " + exportedPath.toFile().getAbsolutePath());
...