Using Cluster for Image Processing with ImageJ-headless mode

Using Cluster for Image Processing with ImageJ-headless mode

The simplest case: processing with no use of plugins

We take a sample case of adding noise to an image as an example. You need following files in the current directory (in the following, we assume that the current directory is /home/miura/test. In your case, it could be /home/<yourname>/test.

Four files should be in the directory:

headless.jar
- This .jar file manages to work with ImageJ headlessly.
- accessible by /g/almf/software/ij/headless.jar from within EMBL network.
  - Otherwise, download headless.jar file from http://sourceforge.net/projects/imageja. Place it somewhere you could access.
blobs.tif
- You could download this by Normal ImageJ (with-head, or GUI). [File → Open Samples → Blob]. Save it in the current directory /home/<your username>/test
headlesstest.ijm
- You must write an ImageJ macro file and save it in the current directory. See below.
testpbs.sh
- You must write a shell script and save it in the current directory. See below.

Write an ImageJ macro that does the actual processing.

Example macro is named “headlesstest.ijm”
This macro does not use ij.gui.* classes, so it should basically be running even without headless.jar (not tested).

srcname = getArgument();
open(srcname);
destpath = "test/";
run("Add Noise");
saveAs("Tiff", destpath + "blobprocessed.tif");

Tip: for passing arguments to the macro, use getArgument() command. See also this page, section "Running Macros from the Command Line".

Write a short shell script , that runs the ImageJ macro.

Example shell script “testpbs.sh” is shown below.

#!/bin/sh
#PBS -N TestPBS
#PBS -l walltime=1:00:00
/usr/struct/bin/java -classpath /g/almf/software/ij/headless.jar:/g/almf/software/ij/ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch /home/<your user name>/test/headlesstest.ijm /home/<your user name>/test/blobs.tif

Some more explanation:

Java command for the headless mode is based on instruction in this page
see more information at the batch farm @ EMBL
Above script assumes that you will run it at /home/miura/test
- you cannot use the default java, since it is GNU java, not Sun Java.
- full path to the right Java runtime environment should be assigned as above.
- ij.jar, the main body of ImageJ is in SCB unit software central repository.
- Argument for the IJ macro is a file with full path information. This is just to be explicit, as base directory is /home/miura

Login to sub-master from SSH secure shell, and then type below:

cd /home/<yourname>/test
sh testpbs.sh

If you see that the processed file appearing in /home/<yourname>/test, then you are successfully using ImageJ from command line via shell script.

Throwing the job to cluster:

/usr/pbs/bin/qsub -q clusterng ./testpbskm.sh

if you see the processed file in /home/<yourname>/test, then you are successfully using the EMBL cluster.

using ImageJ Plugins

Using plugin is a bit tricky. In case of desktop ImageJ, what you need to install plugin is simply drug and drop .class or .jar files to plugins folder within imageJ folder. With headless mode, you need to do the following:

explicitly state where the plugins folder resides by setting -ijpath.
~~all plugins must not be in jar. They all have to be class files.~~
- ~~if you only have jar file, then you should open it by zip archiver and extract all class files. File hierarchy should be maintained within the plugin folder.~~

… so the java command would look like this: Macro file “headlesstest2.ijm” is supposed to be at $HOME and images to be processed are under ~/test, and headlesstest2.ijm takes this path as an argument.

cd ~

/usr/struct/bin/java -cp /g/almf/software/ij/headless.jar:/g/almf/software/ij/ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -ijpath /g/almf/software/ij -batch headlesstest2.ijm /home/miura/test

and the plugins folder resides within /g/almf/software/ij/ in the above case. If you need to add a plugin, please ask Kota.

Spreading Jobs with qsub

For faster computation of many files, a job array consisting of many job scripts would speed up the processing.

For example, if there are 100 files to be image-processed, job script for each file could be prepared with numerated job file name, and submitted to the cluster. Jobs are then spread to vacant resources (uses vacant CPU).

Preparation of scripts could be automated. I will paste an example ImageJ macro for doing this. The macro generates job scripts for all files in a directory (e.g. for 100 files, then 100 scripts + job array script is generated).

There are 7 arguments required to run this script.

arg[0]: path to the folder cotaining images to be processed
arg[1]: path to the output folder for processed images
arg[2]: path to the folder where scripts will be saved
arg[3]: prefix of job shell scrips. should be longer than 2 chars
arg[4]: path to ij jars
arg[5]: path to IJ macro
arg[6]: IJ macro name

After generating files, change the permission to all the script files so that they could be executed. Should be something like (if your job script prefix is job_):


chmod +x job_*

Example work flow

In this example, job scripts generator (written in ImageJ macro) is executed. This generator also generates job array script. Each job script then executes java command on corresponding image/stack, using single image processing ImageJ macro file (this file name is currently static, written in the beginning part of generator macro).

File hierarchy environment is as follows:

current directory: /home/miura/test
images are in: /home/miura/test/testsrc
ij.jar and headless.jar are in /home/miura/test
imageJ macro for generating shell scripts is at: /home/miura/test
imageJ macro for image processing is at: /home/miura/test

% /usr/struct/bin/java -cp headless.jar:ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch JCreate2.ijm /home/miura/test/testsrc:/home/miura/test/testdest:jobbb
% chmod +x testdest/jobbb*
% qsub testdest/jobarray.sh

1st line creates job scripts and a job array script.
2nd line changed the permission of job scripts
3rd line throws the job to the cluster

Above steps could be written in a meta-shell script as follows. Then all you need to do is just execute one line, sh <name of script>.sh.

#!/bin/sh
#script for imageJ cluster calculation

# path to IJ jar file
IJJARS="/home/miura/test"

# path to images and stacks
SRCPATH="/home/miura/test/testsrc"

# path to job array generator macro
JOBGENPATH="/home/miura/test"

# name of job array generator
JOBGENNAME="JCreate2.ijm"

# path to save job scripts and job array script
JOBPATH="/home/miura/test/testdest"

# base name (prefix) of job script generated for each images/stacks
JOBPREF="jobbb"

echo "IJ full-path ${IJJARS}/headless.jar"

#timer
jobstart=$(date +%s)
#jobstartN=$(date +%N)

/usr/struct/bin/java -cp ${IJJARS}/headless.jar:${IJJARS}/ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch ${JOBGENPATH}/${JOBGENNAME} ${SRCPATH}:${JOBPATH}:${JOBPREF}

chmod +x ${JOBPATH}/${JOBPREF}*

qsub ${JOBPATH}/jobarray.sh

# timer
jobend=$(date +%s)
#jobendN=$(date +%N)
echo "Time: $((jobend-jobstart)) secs."
#echo "Time: $((jobendN-jobstartN)) nano-sec."

Automated Script (Only from within EMBL network)

Automated Script is available in the ALMF server (clust5fullp.sh). With this script, you only need to invoke the script with two arguments.

arg[0]: full path to the directory containing image/stack files
arg[1]: name of the imageJ macro you want to apply.

sh /g/almf/software/ij/clust5fullp.sh <path>/<to>/<ImageContainingFolder> <Name of ImageJ Macro>

argument <path>/<to>/<ImageContainingFolder> is full path to the folder where images / stacks are stored.

argument <Name of ImageJ Macro> is the file name of ImageJ macro, and this file should be placed under /g/almf/software/ij.

For example, your command could be

sh /g/almf/software/ij/clust5fullp.sh /g/almf/miura/testsmalls headlesstest3.ijm

This script will create two new folders in the directory same as where the image directory is (*_job and *_proc). Processed images or stacks will be saved under *_proc.
Besides, so-called job reports (text files containing output messages) will appear in your current directory where you executed the command.

The script could be viewed in the link below: clust5fulllpath.sh

Example Scripts

Cluster ImageJ processing script

Sample Shell Script that does all.

#!/bin/sh
#script for imageJ cluster calculation
#clust5fullp.sh argument must be the full path to the folder containing source images/stacks. 
#

imgdir=$1
if test -d ${imgdir}
then
	echo ${imgdir}" ...input directory"
else
	echo ${imgdir}" ...no such directory."
	exit 1
fi
outdir=${imgdir}"_proc"
if test -d ${outdir}
then
	echo ${outdir}" ...output directory"
else
	mkdir ${outdir}
	echo ${outdir}" ...output directory created"
fi
jobdir=${imgdir}"_jobs"
if test -d ${jobdir}
then
	echo ${jobdir}" ...job scripts dir"
else
	mkdir ${jobdir}
	echo ${jobdir}" ...job scripts directory created"
fi

#curdir=`pwd`

# path to IJ jar file
IJJARS="/g/almf/miura/pub"

# path to images and stacks
#SRCPATH="/home/miura/test/testsrc"
#SRCPATH="/g/almf/miura/testsmalls"
#SRCPATH=${curdir}"/"${imgdir}
SRCPATH=${imgdir}

#path to output directory
#OUTPATH=${curdir}"/"${outdir}
OUTPATH=${outdir}

# path to job array generator macro
#JOBGENPATH="/home/miura/test/ij"
JOBGENPATH="/g/almf/miura/pub"

# name of job array generator
JOBGENNAME="JCreate3.ijm"

# path to save job scripts and job array script
#JOBPATH="/home/miura/test/job3"
#JOBPATH=${curdir}"/"${jobdir}
JOBPATH=${jobdir}

# base name (prefix) of job script generated for each images/stacks
#JOBPREF="job_"${imgdir}
JOBPREF="job_"

# path to image processing IJ macro
#IJMACROPATH="/home/miura/test/ij"
IJMACROPATH="/g/almf/miura/pub"

# image processing IJ macro name
IJMACRONAME="headlesstest3.ijm"

echo "IJ full-path ${IJJARS}/headless.jar"

#timer
jobstart=$(date +%s)
#jobstartN=$(date +%N)

macroarg=${SRCPATH}:${OUTPATH}:${JOBPATH}:${JOBPREF}:${IJJARS}:${IJMACROPATH}:${IJMACRONAME}
echo ${macroarg}

/usr/struct/bin/java -cp ${IJJARS}/headless.jar:${IJJARS}/ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch ${JOBGENPATH}/${JOBGENNAME} ${macroarg}

chmod +x ${JOBPATH}/${JOBPREF}*

qsub ${JOBPATH}/jobarray.sh

# timer
jobend=$(date +%s)
#jobendN=$(date +%N)
echo "Time: $((jobend-jobstart)) secs."
#echo "Time: $((jobendN-jobstartN)) nano-sec."

Sample ImageJ macro for Generating Job Array

The ImageJ macro below is the actual content of the script above (called in line 75). it generates one script for one image/stack, and also meta-script called jobarray.sh for the submission to cluster.

/* ImageJ macro for generating shell scripts to be used by qsub array
	Kota miura (miura@embl.de) 2010

	...better be written as a shell script in future. 

 to use this macro, adjust path variable SUN_JAVA if required
 you might probably need to change the name of imageJ jar file of var_ijjar

 requres three arguments separated by colon. 
 arg[0]: path to the folder cotaining images to be processed
 arg[1]: path to the output folder for processed images
 arg[2]: path to the folder where scripts will be saved
 arg[3]: prefix of job shell scrips. should be longer than 2 chars 
  arg[4]: path to ij jars
  arg[5]: path to IJ macro
  arg[6]: IJ macro name

 Example command for execution
 /usr/struct/bin/java -cp headless.jar:ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch JCreate.ijm /home/miura/test/testsrc:/home/miura/test/testdest:jobbb
*/

/* path where batch file (IJ macro) is placed,*/
var BASEP = "\/home\/miura\/test\/ij";

/* file name of IJ macro*/
var IJMACROFILE = "headlesstest3.ijm";

/* path to folder where ij.jar and plugins folder is*/
var IJP = "\/home\/miura\/test\/ij";

/* path to SUN java*/
var SUN_JAVA = "\/usr\/struct\/bin\/java";

var var_headless;
var var_ijjar;
var var_ijpath;
var var_batchfile;
var var_batcharg;
var vars;
var jobfilePrefix = "job_";
	
arg = getArgument();
print(arg);
argA = split(arg, ":");
//for (i=0; i<argA.length; i++) print(argA[i]);
if (argA.length != 7){
	print("Number of Arguments:" + argA.length);
	print("Abort Generator: There should be exactly 7 arguments for running script generator.");
	exit();
}
srcdir = argA[0];
outdir = argA[1];
destdir = argA[2];
argjobprefix = argA[3];
if (lengthOf(argjobprefix)>2){
	jobfilePrefix = argjobprefix;
}
IJP = argA[4];
BASEP = argA[5];
IJMACROFILE = argA[6];

print("variables taken from arguments");

setVariables();

vars += srcdir + File.separator;

print("input: " + srcdir);
print("output: " + outdir);
print("jobs: " + destdir);
print("ImageJ Path: " + IJP);
print("ImageJ macro Path: " + BASEP);
print("ImageJ macro Name: " + IJMACROFILE);


if (!File.isDirectory(srcdir)) {
	print("Abort: source directory does not exist!");
	exit();
}
if (!File.isDirectory(outdir)) {
	print("Abort: output directory does not exist!");
	exit();
}
if (!File.isDirectory(destdir)) {
	File.makeDirectory(destdir);
}
if (!File.isDirectory(IJP)) {
	print("Abort: ImageJ directory does not exist!");
	exit();	
}
if (!File.isDirectory(BASEP)) {
	print("Abort: ImageJ Image Processing Macro directory does not exist!");
	exit();	
}
if (!File.exists(BASEP + File.separator + IJMACROFILE)) {
	print("Abort: Image Processing Macro File does not exist!");
	exit();	
}

filesA = getFileList(srcdir);
jobfilename = "job_";
for (i = 0; i< filesA.length; i++){
	generateJobScript(filesA[i], i, destdir);
}

jobarraystring = "#!/bin/sh\n"
		+ "\n"
		+ "#PBS -J 1-" + filesA.length + "\n"
		+ "#PBS -q clusterng\n"
		+ destdir+  File.separator 
		+ jobfilePrefix + "$PBS_ARRAY_INDEX.sh";
		
jobarrayfile_fullpath = destdir + File.separator + "jobarray.sh";

File.saveString(jobarraystring, jobarrayfile_fullpath);

function setVariables(){
	var_headless =  "HEADLESS=" + IJP + "\/headless.jar\n";
	var_ijjar =     "IJ_JAR=" + IJP + "\/ij-1.44h.jar\n";
	var_ijpath =    "IJ_PATH=" + IJP + "\n";
	var_batchfile = "BATCH_FILE=" + BASEP + "\/" + IJMACROFILE + "\n";
	var_batcharg =  "BATCH_ARG=";

	vars = var_headless
		+ var_ijjar
		+ var_ijpath
		+ var_batchfile
		+ var_batcharg;
}


function generateJobScript(filename, number, destdir){
	header = "#!/bin/sh\n"
	+ "#PBS -N TestPBS\n"
	+ "#PBS -l walltime=1:00:00\n";
	varslocal = vars +  filename;

	command = SUN_JAVA + " "
		+ "-cp $HEADLESS:$IJ_JAR -Djava.awt.headless=true "
		+ "ij.ImageJ -ijpath $IJ_PATH -batch $BATCH_FILE $BATCH_ARG";
	job = header + "\n\n" + varslocal + "\n" + command + "\n";
	print(job);
	jobname = jobfilePrefix+ (number+1) + ".sh";
	fullpath = destdir + File.separator + jobname;
	File.saveString(job, fullpath);
}

A bit of details on errors

java.awt.headless should be explicitly set to true. If you do not use the flag “-Djava.awt.headless=true”, error could be returned that looks like this:

	at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
	at sun.awt.X11GraphicsEnvironment.access$100(Unknown Source)
	at sun.awt.X11GraphicsEnvironment$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.awt.X11GraphicsEnvironment.<clinit>(Unknown Source)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Unknown Source)
	at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(Unknown Source)
	at sun.awt.X11.XToolkit.<clinit>(Unknown Source)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Unknown Source)
	at java.awt.Toolkit$2.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.awt.Toolkit.getDefaultToolkit(Unknown Source)
	at ij.process.FloatProcessor.createImage(FloatProcessor.java:157)
	at ij.process.TypeConverter.convertFloatToByte(TypeConverter.java:78)
	at ij.process.TypeConverter.convertToByte(TypeConverter.java:40)
	at ij.process.ImageProcessor.convertToByte(ImageProcessor.java:1956)
	at emblcmci.FFTFilter_NoGenDia.filter(FFTFilter_NoGenDia.java:135)
	at emblcmci.FFTFilter_NoGenDia.core(FFTFilter_NoGenDia.java:55)
	at emblcmci.PreprocessChromosomeDots.run(PreprocessChromosomeDots.java:54)
	at Preprocess_ChromosomeDots.run(Preprocess_ChromosomeDots.java:11)
	at ij.IJ.runUserPlugIn(IJ.java:193)
	at ij.IJ.runPlugIn(IJ.java:154)
	at ij.Executer.runCommand(Executer.java:147)
	at ij.Executer.run(Executer.java:78)
	at ij.IJ.run(IJ.java:269)
	at ij.IJ.run(IJ.java:247)
	at ij.macro.Functions.doRun(Functions.java:561)
	at ij.macro.Functions.doFunction(Functions.java:79)
	at ij.macro.Interpreter.doStatement(Interpreter.java:201)
	at ij.macro.Interpreter.doStatements(Interpreter.java:189)
	at ij.macro.Interpreter.run(Interpreter.java:100)
	at ij.macro.Interpreter.run(Interpreter.java:72)
	at ij.macro.Interpreter.run(Interpreter.java:82)
	at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:94)
	at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:79)
	at ij.IJ.runMacroFile(IJ.java:121)
	at ij.ImageJ.main(ImageJ.java:620)

Acknowledgements

Thanks to Frank Thommen(SCB), Andres Lindau (IT support) and Christian Tischer (ALMF) for their great help and suggestions.

Table of Contents