Differences

This shows you the differences between two versions of the page.

--- documents:100922imagej_cluster [2010/10/01 11:07] – kota
+++ documents:100922imagej_cluster [2016/05/24 12:46] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ====== Using Cluster for Image Processing with ImageJ-headless mode ======
+===== The simplest case: processing with no use of plugins =====
-Roughly, work flow is as follows
+We take a sample case of adding noise to an image as an example. You need following files in the current directory (in the following, we assume that the current directory is /home/miura/test. In your case, it could be /home/<yourname>/test.
-  * download headless.jar file from [[http://sourceforge.net/projects/imageja|http://sourceforge.net/projects/imageja]]. Place it somewhere under your home such as /home/yourname/test
+Four files should be in the directory:
-  *  write a ImageJ macro that does what you want.
+  * headless.jar
-    * for passing arguments to the macro, [[http://rsbweb.nih.gov/ij/developer/macro/macros.html#cli |see this page, section "Running Macros from the Command Line"]]. In this example, name of the macro file is headlesstest.ijm
+    * This .jar file manages to work with ImageJ headlessly.
-    * note that the base directory will be /home/yourname (in my case, /home/miura)
+    * accessible by /g/almf/software/ij/headless.jar from within EMBL network.
-    * for this reason, when you need to open or save files, be careful with the path assignment.
+      * Otherwise, download headless.jar file from [[http://sourceforge.net/projects/imageja|http://sourceforge.net/projects/imageja]]. Place it somewhere you could access.
-  * write a short shell script, that runs the ImageJ macro. Example is below.
+  * blobs.tif
+    * You could download this by Normal ImageJ (with-head, or GUI). [File -> Open Samples -> Blob]. Save it in the current directory /home/<your username>/test
+  * headlesstest.ijm
+    * You must write an ImageJ macro file and save it in the current directory. See below.
+  * testpbs.sh
+    * You must write a shell script and save it in the current directory. See below.
+**Write an ImageJ macro** that does the actual processing.
+    * Example macro is named "headlesstest.ijm"
+    * This macro does not use ij.gui.* classes, so it should basically be running even without headless.jar (not tested).
+<code>
+srcname = getArgument();
+open(srcname);
+destpath = "test/";
+run("Add Noise");
+saveAs("Tiff", destpath + "blobprocessed.tif");
+</code>
+Tip: for passing arguments to the macro, use getArgument() command. [[http://rsbweb.nih.gov/ij/developer/macro/macros.html#cli |See also this page, section "Running Macros from the Command Line"]].
+**Write a short shell script** , that runs the ImageJ macro.
+    * Example shell script "testpbs.sh" is shown below.
 <code>
 #!/bin/sh
 #PBS -N TestPBS
 #PBS -l walltime=1:00:00
-/usr/struct/bin/java -classpath /home/miura/test/headless.jar:/struct/software/linux/imagej-1.42/ij.jarr -Djava.awt.headless=true ij.ImageJ -batch /home/miura/test/headlesstest.ijm /home/miura/test/blobs.tif
+/usr/struct/bin/java -classpath /g/almf/software/ij/headless.jar:/g/almf/software/ij/ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch /home/<your user name>/test/headlesstest.ijm /home/<your user name>/test/blobs.tif
 </code>
-  * java command for the headless mode is based on [[http://imagejdocu.tudor.lu/doku.php?id=faq:technical:how_do_i_run_imagej_without_a_graphics_environment_headless&do=show|instruction in this page]]
+Some more explanation:
-  * save this script as "testpbs.sh" or so.
+  * Java command for the headless mode is based on [[http://imagejdocu.tudor.lu/doku.php?id=faq:technical:how_do_i_run_imagej_without_a_graphics_environment_headless&do=show|instruction in this page]]
   * see more information at [[http://intranet.embl.de/it_services/services/pbs/|the batch farm @ EMBL]]
-  * Above script assumes that you are running it at /home/yourname/test
+  * Above script assumes that you will run it at /home/miura/test
     * you cannot use the default java, since it is GNU java, not Sun Java.
     * full path to the right Java runtime environment should be assigned as above.
     * ij.jar, the main body of ImageJ is in SCB unit software central repository.
-  * in /home/<yourname>/test, there should be four files.
+    * Argument for the IJ macro is a file with full path information. This is just to be explicit, as base directory is /home/miura
-    * headless.jar
-    * jeadlesstest.ijm
+**Login to sub-master** from SSH secure shell, and then type below:
-    * testpbs.sh
-    * blobs.tif
-  * below is the test macro. It does not use ij.gui.* classes, so it should basically be running even without headless.jar (not tested).
-<code>
-srcname = getArgument();
-open(srcname);
-destpath = "test/";
-run("Add Noise");
-saveAs("Tiff", destpath + "blobprocessed.tif");
-</code>
-  * Login to sub-master from SSH secure shell, and then type as the following:
 <code>
 cd /home/<yourname>/test
 sh testpbs.sh
 </code>
-    * if you see that the processed file appearing in /home/<yourname>/test, then you are successfully using ImageJ from commandline via shell script.
+If you see that the processed file appearing in /home/<yourname>/test, then you are successfully using ImageJ from command line via shell script.
   * Throwing the job to cluster:
-    *  /usr/pbs/bin/qsub -q clusterng ./testpbskm.sh
+<code>
-  * if you see the processed file in /home/<yourname>/test, then you are successfully using the EMBL cluster.
+/usr/pbs/bin/qsub -q clusterng ./testpbskm.sh
+</code>
+if you see the processed file in /home/<yourname>/test, then you are successfully using the EMBL cluster.
 ===== using ImageJ Plugins =====
@@ Line 51: / Line 61: @@
 Using plugin is a bit tricky. In case of desktop ImageJ, what you need to install plugin is simply drug and drop .class or .jar files to plugins folder within imageJ folder. With headless mode, you need to do the following:
   * explicitly state where the plugins folder resides by setting -ijpath.
-  * all plugins must not be in jar. They all have to be class files.
+  * <del>all plugins must not be in jar. They all have to be class files.</del>
-    * if you only have jar file, then you should open it by zip archiver and extract all class files. File hierarchy should be maintained within the plugin folder.
+    * <del>if you only have jar file, then you should open it by zip archiver and extract all class files. File hierarchy should be maintained within the plugin folder.</del>
-... so the java command would look like this:
+... so the java command would look like this: Macro file "headlesstest2.ijm" is supposed to be at $HOME and images to be processed are under ~/test, and headlesstest2.ijm takes this path as an argument.
 <code>
-/usr/struct/bin/java -cp /home/miura/test/headless.jar:/home/miura/test/ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -ijpath /home/miura/test -batch headlesstest2.ijm /home/miura/test
+cd ~
+/usr/struct/bin/java -cp /g/almf/software/ij/headless.jar:/g/almf/software/ij/ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -ijpath /g/almf/software/ij -batch headlesstest2.ijm /home/miura/test
 </code>
-and the plugins folder resides within /home/miura/test.
+and the plugins folder resides within /g/almf/software/ij/ in the above case. If you need to add a plugin, please ask Kota.
 ===== Spreading Jobs with qsub =====
@@ Line 64: / Line 78: @@
 For example, if there are 100 files to be image-processed, job script for each file could be prepared with numerated job file name, and submitted to the  cluster. Jobs are then spread to vacant resources (uses vacant CPU).
-Here is a script for generating job scripts for all files in a directory (e.g. for 100 files, then 100 scripts + job array script is generated).
+Preparation of scripts could be automated. [[http://cmci.embl.de/documents/100922imagej_cluster#sample_imagej_macro_for_generating_job_array|I will paste an example ImageJ macro for doing this.]] The macro generates job scripts for all files in a directory (e.g. for 100 files, then 100 scripts + job array script is generated).
+There are 7 arguments required to run this script.
+  * arg[0]: path to the folder cotaining images to be processed
+  * arg[1]: path to the output folder for processed images
+  * arg[2]: path to the folder where scripts will be saved
+  * arg[3]: prefix of job shell scrips. should be longer than 2 chars
+  * arg[4]: path to ij jars
+  * arg[5]: path to IJ macro
+  * arg[6]: IJ macro name
+After generating files, change the permission to all the script files so that they could be executed. Should be something like (if your job script prefix is job_):
+<code>
+chmod +x job_*
+</code>
+==== Example work flow ====
+In this example, job scripts generator (written in ImageJ macro) is executed. This generator also generates job array script. Each job script then executes java command on corresponding image/stack, using single image processing ImageJ macro file (this file name is currently static, written in the beginning part of generator macro).
+File hierarchy environment is as follows:
+  * current directory: /home/miura/test
+  * images are in: /home/miura/test/testsrc
+  * ij.jar and headless.jar are in /home/miura/test
+  * imageJ macro for generating shell scripts is at: /home/miura/test
+  * imageJ macro for image processing is at: /home/miura/test
+<code>
+% /usr/struct/bin/java -cp headless.jar:ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch JCreate2.ijm /home/miura/test/testsrc:/home/miura/test/testdest:jobbb
+% chmod +x testdest/jobbb*
+% qsub testdest/jobarray.sh
+</code>
+  * 1st line creates job scripts and a job array script.
+  * 2nd line changed the permission of job scripts
+  * 3rd line throws the job to the cluster
+Above steps could be written in a meta-shell script as follows. Then all you need to do is just execute one line, sh <name of script>.sh.
+<sxh shell>
+#!/bin/sh
+#script for imageJ cluster calculation
+# path to IJ jar file
+IJJARS="/home/miura/test"
+# path to images and stacks
+SRCPATH="/home/miura/test/testsrc"
+# path to job array generator macro
+JOBGENPATH="/home/miura/test"
+# name of job array generator
+JOBGENNAME="JCreate2.ijm"
+# path to save job scripts and job array script
+JOBPATH="/home/miura/test/testdest"
+# base name (prefix) of job script generated for each images/stacks
+JOBPREF="jobbb"
+echo "IJ full-path ${IJJARS}/headless.jar"
+#timer
+jobstart=$(date +%s)
+#jobstartN=$(date +%N)
+/usr/struct/bin/java -cp ${IJJARS}/headless.jar:${IJJARS}/ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch ${JOBGENPATH}/${JOBGENNAME} ${SRCPATH}:${JOBPATH}:${JOBPREF}
+chmod +x ${JOBPATH}/${JOBPREF}*
+qsub ${JOBPATH}/jobarray.sh
+# timer
+jobend=$(date +%s)
+#jobendN=$(date +%N)
+echo "Time: $((jobend-jobstart)) secs."
+#echo "Time: $((jobendN-jobstartN)) nano-sec."
+</sxh>
+===== Automated Script (Only from within EMBL network) =====
+Automated Script is available in the ALMF server (clust5fullp.sh). With this script, you only need to invoke the script with two arguments.
+  * arg[0]: full path to the directory containing image/stack files
+  * arg[1]: name of the imageJ macro you want to apply.
+<code>
+sh /g/almf/software/ij/clust5fullp.sh <path>/<to>/<ImageContainingFolder> <Name of ImageJ Macro>
+</code>
+argument <path>/<to>/<ImageContainingFolder> is full path to the folder where images / stacks are stored.
+argument <Name of ImageJ Macro> is the file name of ImageJ macro, and **this file should be placed under /g/almf/software/ij**.
+For example, your command could be
+<code>
+sh /g/almf/software/ij/clust5fullp.sh /g/almf/miura/testsmalls headlesstest3.ijm
+</code>
+This script will create two new folders in the directory same as where the image directory is (*_job and *_proc). Processed images or stacks will be saved under *_proc. \\
+Besides, so-called job reports (text files containing output messages) will appear in your current directory where you executed the command.
+The script could be viewed in the link below: [[http://cmci.embl.de/documents/100922imagej_cluster#jobarray_generator|clust5fulllpath.sh]]
+===== Example Scripts =====
+==== Cluster ImageJ processing script ====
+Sample Shell Script that does all.
+<sxh shell>
+#!/bin/sh
+#script for imageJ cluster calculation
+#clust5fullp.sh argument must be the full path to the folder containing source images/stacks.
+#
+imgdir=$1
+if test -d ${imgdir}
+then
+	echo ${imgdir}" ...input directory"
+else
+	echo ${imgdir}" ...no such directory."
+	exit 1
+fi
+outdir=${imgdir}"_proc"
+if test -d ${outdir}
+then
+	echo ${outdir}" ...output directory"
+else
+	mkdir ${outdir}
+	echo ${outdir}" ...output directory created"
+fi
+jobdir=${imgdir}"_jobs"
+if test -d ${jobdir}
+then
+	echo ${jobdir}" ...job scripts dir"
+else
+	mkdir ${jobdir}
+	echo ${jobdir}" ...job scripts directory created"
+fi
+#curdir=`pwd`
+# path to IJ jar file
+IJJARS="/g/almf/miura/pub"
+# path to images and stacks
+#SRCPATH="/home/miura/test/testsrc"
+#SRCPATH="/g/almf/miura/testsmalls"
+#SRCPATH=${curdir}"/"${imgdir}
+SRCPATH=${imgdir}
+#path to output directory
+#OUTPATH=${curdir}"/"${outdir}
+OUTPATH=${outdir}
+# path to job array generator macro
+#JOBGENPATH="/home/miura/test/ij"
+JOBGENPATH="/g/almf/miura/pub"
+# name of job array generator
+JOBGENNAME="JCreate3.ijm"
+# path to save job scripts and job array script
+#JOBPATH="/home/miura/test/job3"
+#JOBPATH=${curdir}"/"${jobdir}
+JOBPATH=${jobdir}
+# base name (prefix) of job script generated for each images/stacks
+#JOBPREF="job_"${imgdir}
+JOBPREF="job_"
+# path to image processing IJ macro
+#IJMACROPATH="/home/miura/test/ij"
+IJMACROPATH="/g/almf/miura/pub"
+# image processing IJ macro name
+IJMACRONAME="headlesstest3.ijm"
+echo "IJ full-path ${IJJARS}/headless.jar"
+#timer
+jobstart=$(date +%s)
+#jobstartN=$(date +%N)
+macroarg=${SRCPATH}:${OUTPATH}:${JOBPATH}:${JOBPREF}:${IJJARS}:${IJMACROPATH}:${IJMACRONAME}
+echo ${macroarg}
+/usr/struct/bin/java -cp ${IJJARS}/headless.jar:${IJJARS}/ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch ${JOBGENPATH}/${JOBGENNAME} ${macroarg}
+chmod +x ${JOBPATH}/${JOBPREF}*
+qsub ${JOBPATH}/jobarray.sh
+# timer
+jobend=$(date +%s)
+#jobendN=$(date +%N)
+echo "Time: $((jobend-jobstart)) secs."
+#echo "Time: $((jobendN-jobstartN)) nano-sec."
+</sxh>
+==== Sample ImageJ macro for Generating Job Array ====
+The ImageJ macro below is the actual content of the script above (called in line 75). it generates one script for one image/stack, and also meta-script called jobarray.sh for the submission to cluster.
 <code>
 /* ImageJ macro for generating shell scripts to be used by qsub array
@@ Line 72: / Line 289: @@
 	...better be written as a shell script in future.
- to use this macro, adjust path variables BASEP, IJMACROFILE, IJP, SUN_JAVA
+ to use this macro, adjust path variable SUN_JAVA if required
  you might probably need to change the name of imageJ jar file of var_ijjar
  requres three arguments separated by colon.
  arg[0]: path to the folder cotaining images to be processed
- arg[1]: path to the folder where scripts will be saved
+ arg[1]: path to the output folder for processed images
- arg[2]: prefix of job shell scrips. should be longer than 2 chars
+ arg[2]: path to the folder where scripts will be saved
+ arg[3]: prefix of job shell scrips. should be longer than 2 chars
+  arg[4]: path to ij jars
+  arg[5]: path to IJ macro
+  arg[6]: IJ macro name
  Example command for execution
@@ Line 85: / Line 306: @@
 /* path where batch file (IJ macro) is placed,*/
-var BASEP = "\/home\/miura\/test";
+var BASEP = "\/home\/miura\/test\/ij";
 /* file name of IJ macro*/
@@ Line 91: / Line 312: @@
 /* path to folder where ij.jar and plugins folder is*/
-var IJP = "\/home\/miura\/test";
+var IJP = "\/home\/miura\/test\/ij";
 /* path to SUN java*/
 var SUN_JAVA = "\/usr\/struct\/bin\/java";
-var var_headless =  "HEADLESS=" + IJP + "\/headless.jar\n";
+var var_headless;
-var var_ijjar =     "IJ_JAR=" + IJP + "\/ij-1.44h.jar\n";
+var var_ijjar;
-var var_ijpath =    "IJ_PATH=" + IJP + "\n";
+var var_ijpath;
-var var_batchfile = "BATCH_FILE=" + BASEP + "\/" + IJMACROFILE + "\n";
+var var_batchfile;
-var var_batcharg =  "BATCH_ARG=";
+var var_batcharg;
+var vars;
-var vars = var_headless
-	+ var_ijjar
-	+ var_ijpath
-	+ var_batchfile
-	+ var_batcharg;
 var jobfilePrefix = "job_";
 arg = getArgument();
+print(arg);
 argA = split(arg, ":");
+//for (i=0; i<argA.length; i++) print(argA[i]);
+if (argA.length != 7){
+	print("Number of Arguments:" + argA.length);
+	print("Abort Generator: There should be exactly 7 arguments for running script generator.");
+	exit();
+}
 srcdir = argA[0];
-destdir = argA[1];
+outdir = argA[1];
-argjobprefix = argA[2];
+destdir = argA[2];
+argjobprefix = argA[3];
 if (lengthOf(argjobprefix)>2){
 	jobfilePrefix = argjobprefix;
 }
+IJP = argA[4];
+BASEP = argA[5];
+IJMACROFILE = argA[6];
+print("variables taken from arguments");
+setVariables();
 vars += srcdir + File.separator;
 print("input: " + srcdir);
-print("output: " + destdir);
+print("output: " + outdir);
+print("jobs: " + destdir);
+print("ImageJ Path: " + IJP);
+print("ImageJ macro Path: " + BASEP);
+print("ImageJ macro Name: " + IJMACROFILE);
 if (!File.isDirectory(srcdir)) {
 	print("Abort: source directory does not exist!");
+	exit();
+}
+if (!File.isDirectory(outdir)) {
+	print("Abort: output directory does not exist!");
 	exit();
 }
@@ Line 129: / Line 370: @@
 	File.makeDirectory(destdir);
 }
+if (!File.isDirectory(IJP)) {
+	print("Abort: ImageJ directory does not exist!");
+	exit();
+}
+if (!File.isDirectory(BASEP)) {
+	print("Abort: ImageJ Image Processing Macro directory does not exist!");
+	exit();
+}
+if (!File.exists(BASEP + File.separator + IJMACROFILE)) {
+	print("Abort: Image Processing Macro File does not exist!");
+	exit();
+}
 filesA = getFileList(srcdir);
 jobfilename = "job_";
@@ Line 145: / Line 399: @@
 File.saveString(jobarraystring, jobarrayfile_fullpath);
+function setVariables(){
+	var_headless =  "HEADLESS=" + IJP + "\/headless.jar\n";
+	var_ijjar =     "IJ_JAR=" + IJP + "\/ij-1.44h.jar\n";
+	var_ijpath =    "IJ_PATH=" + IJP + "\n";
+	var_batchfile = "BATCH_FILE=" + BASEP + "\/" + IJMACROFILE + "\n";
+	var_batcharg =  "BATCH_ARG=";
+	vars = var_headless
+		+ var_ijjar
+		+ var_ijpath
+		+ var_batchfile
+		+ var_batcharg;
+}
 function generateJobScript(filename, number, destdir){
@@ Line 162: / Line 431: @@
 }
-</code>
-Create a text file as above and save it to a directory where you could access (in the command below, the file is named JCreate.ijm). Then type the following command.
-**/usr/struct/bin/java -cp headless.jar:ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch JCreate.ijm /home/miura/test/testsrc:/home/miura/test/testdest
-**\\
-In above case, all the scripts will be generated under
-/home/miura/test/testdest
-for all the files in
-/home/miura/test/testsrc
-after generating files, change the permission to all the script files so that they could be executed. Should be something like
-<code>
-chmod +x job_*
 </code>
-==== Example work flow ====
-In this example, job scripts generator (written in ImageJ macro) is executed. This generator also generates job array script. Each job script then executes java command on corresponding image/stack, using single image processing ImageJ macro file (this file name is currently static, written in the beginning part of generator macro).
-File hierarchy environment is as follows:
-  * current directory: /home/miura/test
-  * images are in: /home/miura/test/testsrc
-  * ij.jar and headless.jar are in /home/miura/test
-  * imageJ macro for generating shell scripts is at: /home/miura/test
-  * imageJ macro for image processing is at: /home/miura/test
-<code>
-% /usr/struct/bin/java -cp headless.jar:ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch JCreate2.ijm /home/miura/test/testsrc:/home/miura/test/testdest:jobbb
-% chmod +x testdest/jobbb*
-% qsub testdest/jobarray.sh
-</code>
-  * 1st line creates job scripts and a job array script.
-  * 2nd line changed the permission of job scripts
-  * 3rd line throws the job to the cluster
-Above steps could be written in a meta-shell script as follows. Then all you need to do is just execute one line, sh <name of script>.sh.
-<code>
-#!/bin/sh
-#script for imageJ cluster calculation
-# path to IJ jar file
-IJJARS="/home/miura/test"
-# path to images and stacks
-SRCPATH="/home/miura/test/testsrc"
-# path to job array generator macro
-JOBGENPATH="/home/miura/test"
-# name of job array generator
-JOBGENNAME="JCreate2.ijm"
-# path to save job scripts and job array script
-JOBPATH="/home/miura/test/testdest"
-# base name (prefix) of job script generated for each images/stacks
-JOBPREF="jobbb"
-echo "IJ full-path ${IJJARS}/headless.jar"
-#timer
-jobstart=$(date +%s)
-#jobstartN=$(date +%N)
-/usr/struct/bin/java -cp ${IJJARS}/headless.jar:${IJJARS}/ij-1.44h.jar -Djava.awt.headless=true ij.ImageJ -batch ${JOBGENPATH}/${JOBGENNAME} ${SRCPATH}:${JOBPATH}:${JOBPREF}
-chmod +x ${JOBPATH}/${JOBPREF}*
-qsub ${JOBPATH}/jobarray.sh
-# timer
-jobend=$(date +%s)
-#jobendN=$(date +%N)
-echo "Time: $((jobend-jobstart)) secs."
-#echo "Time: $((jobendN-jobstartN)) nano-sec."
-</code>
@@ Line 302: / Line 484: @@
 ===== Acknowledgements =====
-Thanks to Frank Thommen(SCB) and Andres Lindau (IT support) for their great help.
+Thanks to Frank Thommen(SCB), Andres Lindau (IT support) and Christian Tischer (ALMF) for their great help and suggestions.