XSLTProc in the Buff

Intro

The DORA project (Domino On-disk Repository Assistant) from Cameron Gregor has gone through a couple incarnations, originally as DORA, then again as a plugin to Domino Designer (DDE), called Swiper. These are both great projects, benefitting the community in that our (git for dora) scm repositories (for swiper) are much tidier and there are less issues with the overhead of the metadata/etc. files (particularly with swiper).

What It Does

DORA (and Swiper) both accomplish the same filtering/stripping of metadata content which is unnecessary for a Domino On Disk Project (ODP), at least when it comes to source control integration (git or mercurial, etc.) and the importing of an ODP to create an NSF.

Stripping out the overhead of the unnecessary metadata means that our scm commits (generally git) can be far more "atomic" in nature, making them more meaningful and to-the-point. If you've had to sift through commit history and keep reading through lines of <last-updated... and the like, you know the pain all too well.

My Use Case

In large Domino applications, specifically the largest one I maintain at the day job, the biggest hurdle to my being able to complete the automation of the "headless designer build" of an NSF has been in relation to the fact that the app has an incredibly large number of design elements and, in most builds (virtually every instance), the build will fail to fully import the project to the NSF during the initialization, making it fail to initialize properly, and results a useless NSF with none of the design elements. The most damning thing was that performing the same steps of importing the ODP and creating a new NSF from the ODP were entirely successful, making my first few goes at headless DDE builds seem fruitless, for my efforts.

After running some tests with ingesting a copy of my app's git repo into an isolated environment, I was able to get nearly every design element processed by Swiper using a combination of Build Automatically (which I do not use for my day job, on account of not wanting to wait for a minor eternity for Built Automatically to do it's thing, whenever I open the correspondingly gargantuan NSF) and manually applying the filter. Running the headless build task after my git repo's ODP contents had been processed meant my builds started succeeding, every single one.

Swiper

Swiper is great, but it's biggest limitation is the fact that it hooks into the Build Automatically task for DDE (from my perspective). Since I don't use this (currently, until I find a better way) at the day job, it's my limiting factor. What makes swiper great is that it limits this output, before it even hits the ODP, making the scm/git integration seamless; this is what I believe should be a part of DDE's integrated process, in my opinion, as part of the use of source control implies that any imported ODP is trusted by the signer/creator/you and that things like last updated properties on design elements are irrelevant.

Enter: Automation

Wanting to automate my builds (nightly, per release to the master branch, on-demand, however I configure my Jenkins CI instance), this meant I wanted the ability to run the task of filtering the XML (which DXL is, with the binary option disabled) as an on-demand (build pipeline invoked), "one-off" process. This would ideally be invoked from the CLI, to be just another task in the shell/PowerShell script for my Jenkins CI instance.

Read and Understand

Something Cameron did a great job with, looking back at DORA, was to embrace a documentation of the moving parts. The dora.pl script was where I started, which is where I should have ended, in retrospect. In the Read Me for the project, there's a section called "Manual Installation of Dora", which outlines the specifics of what the install script does during the setup for a project.

What Setup Does for a Git Repo

As best as I can tell, it achieves the following few things:

  • globally installs (if it isn't there already) scripts in the user's home directory (~/bin) and the XSL files that will filter the assets (~/dora) and adds the script to the user's PATH (for cli access)
  • installs into the local git repo (.git/config) a definition of a filter, called "dxlmetadata" (with tasks of clean and smudge) and sets it as required
  • adds the XSL files to xsl/
  • sets the file associations in the .gitattributes file to enforce the files to be processed to be handled as text, filtered by "dxlmetadata", and use a consistent LF for the end of line (and consistency during the file's subsequent tracking in git)
  • also tells .gitignore to not track the files or changes in the xsl path, or some of the other files associated with some settings and whatnot

To Replicate The Results

I need to filter the correct files using the appropriate XSL (the DXLClean.xsl being the most relevant here, since I don't need to do anything other than run it each time before a headless build, no commiting back to the git repo form a build pipeline standpoint).

Proof of Concept

DORA installs the xsltproc binary, which is necessary on Windows; on *nix (based) OSes, most come with a copy of it. I installed dora then, in a freshly cloned copy of my app's git repository, I ran the following command (adapted from Cameron's DORA ReadMe):

xsltproc ~/dora/DXLClean.xsl ODP/XPages/SomePage.xsp.metadata

This prints the output of the processed file to the console. The command flag for outputting to a specific file is -o <outputFileName>. Now that I know I can set my build environment for consistency, all I need to do is apply the filter to the appropriate files, performing an in-place save.

Expanding

I should probably just create a shell script to perform the bootstraping needed here; but since I'm lazy and have already added my scripts to my personal dotfiles repository, I'm unlikely to do so at this point, as it won't benefit me (and is a somewhat trivial script to write at this point).

In reality, I've gone with what I know and created a task runner based solution, using grunt. The benefit of writing a bootstrap.sh at this point should be minimal, as I ought to incorporate some prompt mechanism as a part of my default grunt task to perform the work of updating a package.json file for the user (if it exists), otherwise clone one down and perform its setup of dependencies. This relies on my packaging my tasks into a proper npm package. The true advantage of a bootstrap.sh would be that I could eliminate the requirement of running some npm install ... requirements, but this assumes installation of node + npm already, as it's a build automation server (which hopefully has things like nvm/nvm-windows installed).

Gruntfile.js Config

I wrote a Gruntfile.js which performs the tasks of:

  • checking for the DXLClean.xsl filter in the current project path (alternatively, one could not do this, and work out of the DORA installed copy in ~/dora/xsl/DXLClean.xsl)

  • processes the specified files (*.metadata, etc.) into a temp folder

    • then back to the origin location (processing in-place caused some empty file issues that would fail out; this corrects that behavior)
    • cleans the temp folder back out

Here's a copy of the Gruntfile.js. Note the file array, which I duplicated from the dora project, also note the variable definition of the ODP path; since I can't assume every project has the same ODP name, setting it up front means that my Jenkins task can update the Gruntfile.js for correct path, depending on anything from a per-project definition to an environment variable. Here it is:

// Gruntfile.js
/*
* This is a single-purpose Gruntfile which establishes a one-off mechanism for running the
* xsltproc task involved in DORA, but for more on-demand use in a build automation environment.
*
*/
// our wrapper function (required by grunt and its plugins)
module.exports = function(grunt) {
// sets the ODP path, which defaults to './ODP/'; change to your ODP path relative to your Gruntfile.
var odp = './ODP/';
// array of source file matching (globbing) for files to process
// ref: https://github.com/camac/dora#installing-filters
var fileAr = [
odp+'**/*.aa',
odp+'**/*.column',
odp+'**/*.dcr',
odp+'**/*.fa',
odp+'**/*.field',
odp+'**/*.folder',
odp+'**/*.form',
odp+'**/*.frameset',
odp+'**/*.ija',
odp+'**/*.ja',
odp+'**/*.javalib',
odp+'**/*.lsa',
odp+'**/*.lsdb',
odp+'**/*.metadata',
odp+'**/*.navigator',
odp+'**/*.outline',
odp+'**/*.page',
odp+'**/*.subform',
odp+'**/*.view',
odp+'Resources/AboutDocument',
odp+'AppProperties/database.properties',
odp+'Resources/IconNote',
odp+'Code/actions/Shared?Actions',
odp+'Resources/UsingDocument'
];
// time-grunt injecting to grunt instance
require('time-grunt')(grunt);
// CONFIGURE GRUNT
grunt.initConfig({
// all of our configuration will go here
xsltproc: {
options: {
stylesheet: './DXLClean.xsl', // either copy the DXLClean.xsl to the project folder for portabilty, or use the ~/dora/xsl/ path
novalid: true,
html: false
},
destTmp: {
files: [{
expand: true,
src: fileAr,
dest: 'tmp'
}]
},
destInPlace: {
files: [{
expand: true,
src: fileAr
}]
}
},
clean: {
tmp: ['./tmp']
},
curl: {
filter: {
src: 'https://raw.githubusercontent.com/camac/dora/master/xsl/DXLClean.xsl',
dest: './DXLClean.xsl'
},
gitignore: {
src: 'https://raw.githubusercontent.com/github/gitignore/master/Node.gitignore',
dest: './.gitignore'
}
},
copy: {
main: {
files: [{
expand: true,
cwd: './tmp/'+odp,
src: ['**'],
dest: odp
}]
}
}
});
// LOAD GRUNT PLUGINS
grunt.loadNpmTasks('grunt-xsltproc');
grunt.loadNpmTasks('grunt-contrib-clean');
grunt.loadNpmTasks('grunt-curl');
grunt.loadNpmTasks('grunt-contrib-copy');
// CREATE TASKS
grunt.registerTask('filterExistsCheck', function(){ // can refactor to use home dir relative, like above
if( !grunt.file.exists('./DXLClean.xsl') ){
console.log('no filter file found');
grunt.task.run('curl:filter');
}else{
console.log('filter file found');
}
});
grunt.registerTask('warn', function(){
var msg = "\nYou have invoked grunt with no options.\n";
msg += "======================================\n";
msg += "Run grunt with either the inPlace or tmp task specified, for the corresponding action.\n";
grunt.fail.fatal(msg);
});
grunt.registerTask('inPlace', ['filterExistsCheck','xsltproc:destInPlace']);
grunt.registerTask('tmp', ['filterExistsCheck','xsltproc:destTmp','copy','clean']);
grunt.registerTask('default', ['tmp']);
};
view raw Gruntfile.js hosted with ❤ by GitHub

An interesting thing I had to overcome was, in a larger ODP, I had to write out the modified versions of the files to a tmp/ path, as re-writing them in-place was causing an error of "no contents", or some weirdness. Copying to a temporary location, then copying back in solved this, apparently. Also, you can see that aside from the Gruntfile.js and package.json, if it doesn't find a copy of the DXLClean.xsl, it pulls a copy from the dora repository.

Here's a copy of the corresponding package.json, which I copy in if my Jenkins task script doesn't detect the Gruntfile.js and package.json. It's there exclusively for the dependency installation via npm install prior to the build.

{
"name": "some-big-xpages-project",
"version": "1.0.0",
"description": "a big XPages project that needs some filtering during build automation, prior to headless DDE build task",
"main": "Gruntfile.js",
"dependencies": {},
"devDependencies": {
"grunt": "^0.4.5",
"grunt-cli": "^0.1.13",
"grunt-contrib-clean": "^1.0.0",
"grunt-contrib-copy": "^0.8.2",
"grunt-curl": "^2.2.0",
"grunt-httpcopy": "^0.3.0",
"grunt-xsltproc": "^0.6.1",
"time-grunt": "^1.3.0"
},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"clean": "grunt"
},
"repository": {
"type": "git",
"url": "git@github.com:edm00se/xsltproc4domino.git"
},
"keywords": [],
"author": "Eric McCormick, @edm00se",
"license": "MIT"
}
view raw package.json hosted with ❤ by GitHub

Executing Grunt

Provided that your automatic build environment has access to an installed version of node and npm, then your build script/task should be able to execute the grunt tasks with the default task association, grunt. By convention, it's best to ensure that all dependencies are installed first with npm install from within the project. Alternatively, this could be specified in the package.json of your project to be an npm script definition, such as npm run clean (see the script "clean" in the package.json). The benefit of this is that the npm "clean" script can be expanded to perform any other npm or command line tasks needed for the project, with the same, single command governing all execution.

Onward and Upward

There's a bit more to this story, and next time, I'll cover the scripts I run and general Jenkins CI setup. I'm really thankful I've had some good help from a number of sources, including being able to pick Cameron Gregor's brain on occasion, which has led me to be able to set this all up. I have a couple hurdles yet before my normal application release cycle is the way I want it, but I'm past some hefty hurdles already. Stay tuned, and as always, thanks for reading.