Draining the happy soup - Part 1

Unleashing unlocked packages promises to reduce risk, improve agility and drive home the full benefits of SFDX

Some planning required

I'm following the approach "Throw and see what sticks to the wall". The rough idea: retrieve all meta data, convert it into SFDX format, distribute it over a number of packages and put it back together.

To make it more fun I picked an heavily ~~abused~~ customized and used org with more than 20,000 meta data artifacts (and a few surprises). Follow along.

Learning

Trailhead has a module on unlocked packages on its trail Get Started with Salesforce DX.

While you are there, check out the (at time of writing the 15) modules on Application Lifecycle Management.

Downloading

The limits for retrieving packages (10,000 elements, 39MB zip or about 400 MB raw) posed an issue for my XL org. So I used, growing fond of it, PackageBuilder to download all sources. It automatically creates multiple package.xml files when you exceed the limits.

Of course, no action without headache. I ran out of memory on my first attempts, so I had to tweak the jvm parameters with -Xms512M -Xmx1G. Furthermore I limited the number of items to retrieve to 4000 per package and excluded the meta data type Document. Documents, not visible in Lightning are user creatable, why they are meta data is beyond me. For reference my build.properties file:

# Package builder
sf.apiversion=45.0
sf.username=user@salesforce.io
sf.password=Sup3rS3cr3t
sf.serverurl=https://test.salesforce.com
sf.maxPoll=200
includechangedata=false
metadatatargetdir=src
destination=packages
download=true
unpack=false
gitcommit=false
skipItems=Document:*
maxitems=4000

You might need to play with your own parameters. A little experimenting goes a long way.

Conversion

You create a SFDX project sfdx force:project:create and you have moved the zip files to the root of the project. Now unzip one archive to mdapisrc and execute
sfdx force:mdapi:convert -r mdapisrc -d force-app. Rinse and repeat.

So far so good. Most likely, until I get it right, I have to repeat these steps more than once, so I created a small shell script (time to get a Mac or the Windows Subsystem for Linux) to repeat the whole exercise:

#!/bin/bash
TITLE="Happy Soup to SFDX retrieval"
SECONDS=0

process_package() {
 rm -rf mdapisrc/*
 unzip -d mdapisrc $1.zip
 sfdx force:mdapi:convert -r mdapisrc -d force-app
}

#Create packages
java -Xms512M -Xmx1G -jar ~/bin/PackageBuilder.jar -o build.properties -download

# Retrive the packages and convert
# First the one without a number
process_package package

# Then the numbered ones
for fname in packages/package.*.xml; do 
 fn="${fname%.xml}"
 f=${fn##*/}
 process_package $f
done

#Time information
LAPSED="$(($SECONDS / 3600))hrs $((($SECONDS / 60) % 60))min $(($SECONDS % 60))sec"
MESSAGE="SFDX Operations completed in: $LAPSED"

#Let the Mac know
osascript -e "display notification \"${MESSAGE}\" with title \"${TITLE}\""
echo "Done with $TITLE $MESSAGE"

On Linux you want to replace the osascript call with growl or notify-send or skip it completely.

Now we got local files to play with, they need to be distributed to their new projects and kept current. But that's the story for next time.

As usual YMMV