documents:100922imagej_cluster
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
documents:100922imagej_cluster [2010/10/01 12:19] – synthax highlighter kota | documents:100922imagej_cluster [2010/11/16 09:00] – kota | ||
---|---|---|---|
Line 7: | Line 7: | ||
* headless.jar | * headless.jar | ||
* This .jar file manages to work with ImageJ headlessly. | * This .jar file manages to work with ImageJ headlessly. | ||
- | * download headless.jar file from [[http:// | + | * accessible by / |
+ | * Otherwise, | ||
* blobs.tif | * blobs.tif | ||
- | * You could download this by Normal ImageJ (with-head, or GUI). [File -> Open Samples -> Blob]. Save it in the current directory /home/miura/test | + | * You could download this by Normal ImageJ (with-head, or GUI). [File -> Open Samples -> Blob]. Save it in the current directory /home/<your username> |
* headlesstest.ijm | * headlesstest.ijm | ||
- | * You must created this ImageJ macro file and save it in the current directory. See below. | + | * You must write an ImageJ macro file and save it in the current directory. See below. |
* testpbs.sh | * testpbs.sh | ||
- | * You must created this ImageJ macro file and save it in the current directory. See below. | + | * You must write a shell script |
**Write an ImageJ macro** that does the actual processing. | **Write an ImageJ macro** that does the actual processing. | ||
Line 33: | Line 34: | ||
#PBS -N TestPBS | #PBS -N TestPBS | ||
#PBS -l walltime=1: | #PBS -l walltime=1: | ||
- | / | + | / |
</ | </ | ||
Some more explanation: | Some more explanation: | ||
Line 73: | Line 74: | ||
For example, if there are 100 files to be image-processed, | For example, if there are 100 files to be image-processed, | ||
- | Here is a script | + | Preparation of scripts could be automated. [[http:// |
- | < | + | There are 7 arguments required |
- | /* ImageJ macro for generating shell scripts | + | |
- | Kota miura (miura@embl.de) 2010 | + | |
- | ...better | + | * arg[0]: path to the folder cotaining images to be processed |
+ | * arg[1]: path to the output folder for processed images | ||
+ | * arg[2]: path to the folder where scripts will be saved | ||
+ | * arg[3]: prefix of job shell scrips. should be longer than 2 chars | ||
+ | * arg[4]: path to ij jars | ||
+ | * arg[5]: path to IJ macro | ||
+ | * arg[6]: IJ macro name | ||
- | to use this macro, adjust path variables BASEP, IJMACROFILE, | ||
- | you might probably need to change the name of imageJ jar file of var_ijjar | ||
- | | + | After generating files, change |
- | | + | < |
- | | + | |
- | | + | |
- | | + | chmod +x job_* |
- | / | + | |
- | */ | + | |
- | + | ||
- | /* path where batch file (IJ macro) is placed,*/ | + | |
- | var BASEP = " | + | |
- | + | ||
- | /* file name of IJ macro*/ | + | |
- | var IJMACROFILE = " | + | |
- | + | ||
- | /* path to folder where ij.jar and plugins folder is*/ | + | |
- | var IJP = " | + | |
- | + | ||
- | /* path to SUN java*/ | + | |
- | var SUN_JAVA = " | + | |
- | + | ||
- | var var_headless = " | + | |
- | var var_ijjar = " | + | |
- | var var_ijpath = " | + | |
- | var var_batchfile = " | + | |
- | var var_batcharg = " | + | |
- | + | ||
- | var vars = var_headless | + | |
- | + var_ijjar | + | |
- | + var_ijpath | + | |
- | + var_batchfile | + | |
- | + var_batcharg; | + | |
- | var jobfilePrefix = "job_"; | + | |
- | + | ||
- | arg = getArgument(); | + | |
- | argA = split(arg, ":" | + | |
- | srcdir = argA[0]; | + | |
- | destdir = argA[1]; | + | |
- | argjobprefix = argA[2]; | + | |
- | if (lengthOf(argjobprefix)> | + | |
- | jobfilePrefix = argjobprefix; | + | |
- | } | + | |
- | + | ||
- | vars += srcdir + File.separator; | + | |
- | print(" | + | |
- | print(" | + | |
- | + | ||
- | if (!File.isDirectory(srcdir)) { | + | |
- | print(" | + | |
- | exit(); | + | |
- | } | + | |
- | if (!File.isDirectory(destdir)) { | + | |
- | File.makeDirectory(destdir); | + | |
- | } | + | |
- | filesA = getFileList(srcdir); | + | |
- | jobfilename = " | + | |
- | for (i = 0; i< filesA.length; | + | |
- | generateJobScript(filesA[i], | + | |
- | } | + | |
- | + | ||
- | jobarraystring = "# | + | |
- | + " | + | |
- | + "#PBS -J 1-" + filesA.length + " | + | |
- | + "#PBS -q clusterng\n" | + | |
- | + destdir+ | + | |
- | + jobfilePrefix + " | + | |
- | + | ||
- | jobarrayfile_fullpath = destdir + File.separator + " | + | |
- | + | ||
- | File.saveString(jobarraystring, | + | |
- | + | ||
- | function generateJobScript(filename, | + | |
- | header = "# | + | |
- | + "#PBS -N TestPBS\n" | + | |
- | + "#PBS -l walltime=1: | + | |
- | varslocal = vars + filename; | + | |
- | + | ||
- | command = SUN_JAVA + " " | + | |
- | + "-cp $HEADLESS: | + | |
- | + " | + | |
- | job = header + " | + | |
- | print(job); | + | |
- | jobname = jobfilePrefix+ (number+1) + " | + | |
- | fullpath = destdir + File.separator + jobname; | + | |
- | File.saveString(job, | + | |
- | } | + | |
</ | </ | ||
- | Create a text file as above and save it to a directory where you could access (in the command below, the file is named JCreate.ijm). Then type the following command. | ||
- | |||
- | **/ | ||
- | **\\ | ||
- | |||
- | |||
- | In above case, all the scripts will be generated under | ||
- | |||
- | / | ||
- | |||
- | for all the files in | ||
- | |||
- | / | ||
- | |||
- | after generating files, change the permission to all the script files so that they could be executed. Should be something like | ||
- | < | ||
- | chmod +x job_* | ||
- | </ | ||
==== Example work flow ==== | ==== Example work flow ==== | ||
Line 256: | Line 159: | ||
</ | </ | ||
+ | ===== Automated Script (Only from within EMBL network) ===== | ||
+ | Automated Script is available in the ALMF server (clust5fullp.sh). With this script, you only need to invoke the script with two arguments. | ||
+ | * arg[0]: full path to the directory containing image/stack files | ||
+ | * arg[1]: name of the imageJ macro you want to apply. | ||
+ | < | ||
+ | sh / | ||
+ | </ | ||
+ | argument < | ||
- | ===== Automated Script (Only from within EMBL network) ===== | + | argument <Name of ImageJ Macro> is the file name of ImageJ macro, and **this file should be placed under / |
- | Automated Script is available, with which you need to change only two lines defining the ImageJ macro that you want to apply to images in a folder. | + | For example, your command could be |
- | + | ||
- | [[clust5fulllpath.sh]] | + | |
- | + | ||
- | lines to be changed are | + | |
< | < | ||
- | 64 IJMACROPATH=" | + | sh /g/almf/software/ij/ |
- | ... | + | |
- | 67 IJMACRONAME=" | + | |
</ | </ | ||
- | Path to the macro and name of the macro file should be replaced according to your situation. | ||
- | rename | + | This script will create two new folders in the directory same as where the image directory is (*_job |
+ | Besides, so-called job reports (text files containing output messages) will appear in your current directory where you executed the command. | ||
+ | |||
+ | The script could be viewed in the link below: [[http:// | ||
+ | |||
+ | ===== Example Scripts ===== | ||
+ | ==== Cluster ImageJ processing script ==== | ||
+ | |||
+ | Sample Shell Script that does all. | ||
+ | <sxh shell> | ||
+ | #!/bin/sh | ||
+ | #script for imageJ cluster calculation | ||
+ | # | ||
+ | # | ||
+ | |||
+ | imgdir=$1 | ||
+ | if test -d ${imgdir} | ||
+ | then | ||
+ | echo ${imgdir}" | ||
+ | else | ||
+ | echo ${imgdir}" | ||
+ | exit 1 | ||
+ | fi | ||
+ | outdir=${imgdir}" | ||
+ | if test -d ${outdir} | ||
+ | then | ||
+ | echo ${outdir}" | ||
+ | else | ||
+ | mkdir ${outdir} | ||
+ | echo ${outdir}" | ||
+ | fi | ||
+ | jobdir=${imgdir}" | ||
+ | if test -d ${jobdir} | ||
+ | then | ||
+ | echo ${jobdir}" | ||
+ | else | ||
+ | mkdir ${jobdir} | ||
+ | echo ${jobdir}" | ||
+ | fi | ||
+ | |||
+ | # | ||
+ | |||
+ | # path to IJ jar file | ||
+ | IJJARS="/ | ||
+ | |||
+ | # path to images and stacks | ||
+ | # | ||
+ | # | ||
+ | # | ||
+ | SRCPATH=${imgdir} | ||
+ | |||
+ | #path to output directory | ||
+ | # | ||
+ | OUTPATH=${outdir} | ||
+ | |||
+ | # path to job array generator macro | ||
+ | # | ||
+ | JOBGENPATH="/ | ||
+ | |||
+ | # name of job array generator | ||
+ | JOBGENNAME=" | ||
+ | |||
+ | # path to save job scripts and job array script | ||
+ | # | ||
+ | # | ||
+ | JOBPATH=${jobdir} | ||
+ | |||
+ | # base name (prefix) of job script generated for each images/ | ||
+ | # | ||
+ | JOBPREF=" | ||
+ | |||
+ | # path to image processing IJ macro | ||
+ | # | ||
+ | IJMACROPATH="/ | ||
+ | |||
+ | # image processing IJ macro name | ||
+ | IJMACRONAME=" | ||
+ | |||
+ | echo "IJ full-path ${IJJARS}/ | ||
+ | |||
+ | #timer | ||
+ | jobstart=$(date +%s) | ||
+ | # | ||
+ | |||
+ | macroarg=${SRCPATH}: | ||
+ | echo ${macroarg} | ||
+ | |||
+ | / | ||
+ | |||
+ | chmod +x ${JOBPATH}/ | ||
+ | |||
+ | qsub ${JOBPATH}/ | ||
+ | |||
+ | # timer | ||
+ | jobend=$(date +%s) | ||
+ | # | ||
+ | echo "Time: $((jobend-jobstart)) secs." | ||
+ | #echo "Time: $((jobendN-jobstartN)) nano-sec." | ||
+ | </ | ||
+ | ==== Sample ImageJ macro for Generating Job Array ==== | ||
+ | The ImageJ macro below is the actual content of the script | ||
< | < | ||
- | sh < | + | /* ImageJ macro for generating shell scripts |
- | </ | + | Kota miura (miura@embl.de) 2010 |
- | argument <path>/<to>/< | + | ...better be written as a shell script in future. |
+ | |||
+ | to use this macro, adjust | ||
+ | you might probably need to change the name of imageJ jar file of var_ijjar | ||
+ | |||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | arg[4]: path to ij jars | ||
+ | arg[5]: path to IJ macro | ||
+ | arg[6]: IJ macro name | ||
+ | |||
+ | | ||
+ | /usr/ | ||
+ | */ | ||
+ | |||
+ | /* path where batch file (IJ macro) | ||
+ | var BASEP = " | ||
+ | |||
+ | /* file name of IJ macro*/ | ||
+ | var IJMACROFILE = " | ||
+ | |||
+ | /* path to folder where ij.jar and plugins folder is*/ | ||
+ | var IJP = " | ||
+ | |||
+ | /* path to SUN java*/ | ||
+ | var SUN_JAVA = " | ||
+ | |||
+ | var var_headless; | ||
+ | var var_ijjar; | ||
+ | var var_ijpath; | ||
+ | var var_batchfile; | ||
+ | var var_batcharg; | ||
+ | var vars; | ||
+ | var jobfilePrefix = " | ||
+ | |||
+ | arg = getArgument(); | ||
+ | print(arg); | ||
+ | argA = split(arg, ":" | ||
+ | //for (i=0; i<argA.length; i++) print(argA[i]); | ||
+ | if (argA.length != 7){ | ||
+ | print(" | ||
+ | print(" | ||
+ | exit(); | ||
+ | } | ||
+ | srcdir = argA[0]; | ||
+ | outdir = argA[1]; | ||
+ | destdir = argA[2]; | ||
+ | argjobprefix = argA[3]; | ||
+ | if (lengthOf(argjobprefix)> | ||
+ | jobfilePrefix = argjobprefix; | ||
+ | } | ||
+ | IJP = argA[4]; | ||
+ | BASEP = argA[5]; | ||
+ | IJMACROFILE = argA[6]; | ||
+ | |||
+ | print(" | ||
+ | |||
+ | setVariables(); | ||
+ | |||
+ | vars += srcdir + File.separator; | ||
+ | |||
+ | print(" | ||
+ | print(" | ||
+ | print(" | ||
+ | print(" | ||
+ | print(" | ||
+ | print(" | ||
+ | |||
+ | |||
+ | if (!File.isDirectory(srcdir)) { | ||
+ | print(" | ||
+ | exit(); | ||
+ | } | ||
+ | if (!File.isDirectory(outdir)) { | ||
+ | print(" | ||
+ | exit(); | ||
+ | } | ||
+ | if (!File.isDirectory(destdir)) { | ||
+ | File.makeDirectory(destdir); | ||
+ | } | ||
+ | if (!File.isDirectory(IJP)) { | ||
+ | print(" | ||
+ | exit(); | ||
+ | } | ||
+ | if (!File.isDirectory(BASEP)) { | ||
+ | print(" | ||
+ | exit(); | ||
+ | } | ||
+ | if (!File.exists(BASEP + File.separator + IJMACROFILE)) { | ||
+ | print(" | ||
+ | exit(); | ||
+ | } | ||
+ | |||
+ | filesA = getFileList(srcdir); | ||
+ | jobfilename = " | ||
+ | for (i = 0; i< filesA.length; | ||
+ | generateJobScript(filesA[i], | ||
+ | } | ||
+ | |||
+ | jobarraystring = "# | ||
+ | + " | ||
+ | + "#PBS -J 1-" + filesA.length + " | ||
+ | + "#PBS -q clusterng\n" | ||
+ | + destdir+ | ||
+ | + jobfilePrefix + " | ||
+ | |||
+ | jobarrayfile_fullpath = destdir + File.separator + " | ||
+ | |||
+ | File.saveString(jobarraystring, | ||
+ | |||
+ | function setVariables(){ | ||
+ | var_headless = " | ||
+ | var_ijjar = " | ||
+ | var_ijpath = " | ||
+ | var_batchfile = " | ||
+ | var_batcharg = " | ||
+ | |||
+ | vars = var_headless | ||
+ | + var_ijjar | ||
+ | + var_ijpath | ||
+ | + var_batchfile | ||
+ | + var_batcharg; | ||
+ | } | ||
+ | |||
+ | |||
+ | function generateJobScript(filename, | ||
+ | header = "# | ||
+ | + "#PBS -N TestPBS\n" | ||
+ | + "#PBS -l walltime=1: | ||
+ | varslocal = vars + filename; | ||
+ | |||
+ | command = SUN_JAVA + " " | ||
+ | + "-cp $HEADLESS: | ||
+ | + " | ||
+ | job = header + " | ||
+ | print(job); | ||
+ | jobname = jobfilePrefix+ (number+1) + " | ||
+ | fullpath = destdir + File.separator + jobname; | ||
+ | File.saveString(job, | ||
+ | } | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | </ | ||
Line 329: | Line 480: | ||
===== Acknowledgements ===== | ===== Acknowledgements ===== | ||
- | Thanks to Frank Thommen(SCB) | + | Thanks to Frank Thommen(SCB), Andres Lindau (IT support) and Christian Tischer (ALMF) for their great help and suggestions. |
documents/100922imagej_cluster.txt · Last modified: 2016/05/24 12:46 by 127.0.0.1