documents:100922imagej_cluster
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
documents:100922imagej_cluster [2010/10/01 14:02] – kota | documents:100922imagej_cluster [2016/05/24 12:46] (current) – external edit 127.0.0.1 | ||
---|---|---|---|
Line 7: | Line 7: | ||
* headless.jar | * headless.jar | ||
* This .jar file manages to work with ImageJ headlessly. | * This .jar file manages to work with ImageJ headlessly. | ||
- | * download headless.jar file from [[http:// | + | * accessible by / |
+ | * Otherwise, | ||
* blobs.tif | * blobs.tif | ||
- | * You could download this by Normal ImageJ (with-head, or GUI). [File -> Open Samples -> Blob]. Save it in the current directory /home/miura/test | + | * You could download this by Normal ImageJ (with-head, or GUI). [File -> Open Samples -> Blob]. Save it in the current directory /home/<your username> |
* headlesstest.ijm | * headlesstest.ijm | ||
- | * You must created this ImageJ macro file and save it in the current directory. See below. | + | * You must write an ImageJ macro file and save it in the current directory. See below. |
* testpbs.sh | * testpbs.sh | ||
- | * You must created this ImageJ macro file and save it in the current directory. See below. | + | * You must write a shell script |
**Write an ImageJ macro** that does the actual processing. | **Write an ImageJ macro** that does the actual processing. | ||
Line 33: | Line 34: | ||
#PBS -N TestPBS | #PBS -N TestPBS | ||
#PBS -l walltime=1: | #PBS -l walltime=1: | ||
- | / | + | / |
</ | </ | ||
Some more explanation: | Some more explanation: | ||
Line 60: | Line 61: | ||
Using plugin is a bit tricky. In case of desktop ImageJ, what you need to install plugin is simply drug and drop .class or .jar files to plugins folder within imageJ folder. With headless mode, you need to do the following: | Using plugin is a bit tricky. In case of desktop ImageJ, what you need to install plugin is simply drug and drop .class or .jar files to plugins folder within imageJ folder. With headless mode, you need to do the following: | ||
* explicitly state where the plugins folder resides by setting -ijpath. | * explicitly state where the plugins folder resides by setting -ijpath. | ||
- | * all plugins must not be in jar. They all have to be class files. | + | * <del>all plugins must not be in jar. They all have to be class files.</ |
- | * if you only have jar file, then you should open it by zip archiver and extract all class files. File hierarchy should be maintained within the plugin folder. | + | * <del>if you only have jar file, then you should open it by zip archiver and extract all class files. File hierarchy should be maintained within the plugin folder.</ |
- | ... so the java command would look like this: | + | ... so the java command would look like this: Macro file " |
< | < | ||
- | / | + | cd ~ |
+ | |||
+ | / | ||
</ | </ | ||
- | and the plugins folder resides within /home/miura/test. | + | and the plugins folder resides within /g/almf/software/ |
===== Spreading Jobs with qsub ===== | ===== Spreading Jobs with qsub ===== | ||
Line 73: | Line 78: | ||
For example, if there are 100 files to be image-processed, | For example, if there are 100 files to be image-processed, | ||
- | Here is a script | + | Preparation of scripts could be automated. [[http:// |
- | < | + | There are 7 arguments required |
- | /* ImageJ macro for generating shell scripts | + | |
- | Kota miura (miura@embl.de) 2010 | + | |
- | ...better | + | * arg[0]: path to the folder cotaining images to be processed |
+ | * arg[1]: path to the output folder for processed images | ||
+ | * arg[2]: path to the folder where scripts will be saved | ||
+ | * arg[3]: prefix of job shell scrips. should be longer than 2 chars | ||
+ | * arg[4]: path to ij jars | ||
+ | * arg[5]: path to IJ macro | ||
+ | * arg[6]: IJ macro name | ||
- | to use this macro, adjust path variables BASEP, IJMACROFILE, | ||
- | you might probably need to change the name of imageJ jar file of var_ijjar | ||
- | | + | After generating files, change |
- | | + | < |
- | | + | |
- | | + | |
- | | + | chmod +x job_* |
- | / | + | |
- | */ | + | |
- | + | ||
- | /* path where batch file (IJ macro) is placed,*/ | + | |
- | var BASEP = " | + | |
- | + | ||
- | /* file name of IJ macro*/ | + | |
- | var IJMACROFILE = " | + | |
- | + | ||
- | /* path to folder where ij.jar and plugins folder is*/ | + | |
- | var IJP = " | + | |
- | + | ||
- | /* path to SUN java*/ | + | |
- | var SUN_JAVA = " | + | |
- | + | ||
- | var var_headless = " | + | |
- | var var_ijjar = " | + | |
- | var var_ijpath = " | + | |
- | var var_batchfile = " | + | |
- | var var_batcharg = " | + | |
- | + | ||
- | var vars = var_headless | + | |
- | + var_ijjar | + | |
- | + var_ijpath | + | |
- | + var_batchfile | + | |
- | + var_batcharg; | + | |
- | var jobfilePrefix = "job_"; | + | |
- | + | ||
- | arg = getArgument(); | + | |
- | argA = split(arg, ":" | + | |
- | srcdir = argA[0]; | + | |
- | destdir = argA[1]; | + | |
- | argjobprefix = argA[2]; | + | |
- | if (lengthOf(argjobprefix)> | + | |
- | jobfilePrefix = argjobprefix; | + | |
- | } | + | |
- | + | ||
- | vars += srcdir + File.separator; | + | |
- | print(" | + | |
- | print(" | + | |
- | + | ||
- | if (!File.isDirectory(srcdir)) { | + | |
- | print(" | + | |
- | exit(); | + | |
- | } | + | |
- | if (!File.isDirectory(destdir)) { | + | |
- | File.makeDirectory(destdir); | + | |
- | } | + | |
- | filesA = getFileList(srcdir); | + | |
- | jobfilename = " | + | |
- | for (i = 0; i< filesA.length; | + | |
- | generateJobScript(filesA[i], | + | |
- | } | + | |
- | + | ||
- | jobarraystring = "# | + | |
- | + " | + | |
- | + "#PBS -J 1-" + filesA.length + " | + | |
- | + "#PBS -q clusterng\n" | + | |
- | + destdir+ | + | |
- | + jobfilePrefix + " | + | |
- | + | ||
- | jobarrayfile_fullpath = destdir + File.separator + " | + | |
- | + | ||
- | File.saveString(jobarraystring, | + | |
- | + | ||
- | function generateJobScript(filename, | + | |
- | header = "# | + | |
- | + "#PBS -N TestPBS\n" | + | |
- | + "#PBS -l walltime=1: | + | |
- | varslocal = vars + filename; | + | |
- | + | ||
- | command = SUN_JAVA + " " | + | |
- | + "-cp $HEADLESS: | + | |
- | + " | + | |
- | job = header + " | + | |
- | print(job); | + | |
- | jobname = jobfilePrefix+ (number+1) + " | + | |
- | fullpath = destdir + File.separator + jobname; | + | |
- | File.saveString(job, | + | |
- | } | + | |
</ | </ | ||
- | Create a text file as above and save it to a directory where you could access (in the command below, the file is named JCreate.ijm). Then type the following command. | ||
- | |||
- | **/ | ||
- | **\\ | ||
- | |||
- | |||
- | In above case, all the scripts will be generated under | ||
- | |||
- | / | ||
- | |||
- | for all the files in | ||
- | |||
- | / | ||
- | |||
- | after generating files, change the permission to all the script files so that they could be executed. Should be something like | ||
- | < | ||
- | chmod +x job_* | ||
- | </ | ||
==== Example work flow ==== | ==== Example work flow ==== | ||
Line 256: | Line 163: | ||
</ | </ | ||
- | |||
- | |||
- | |||
- | |||
===== Automated Script (Only from within EMBL network) ===== | ===== Automated Script (Only from within EMBL network) ===== | ||
- | Automated Script is available, | + | Automated Script is available |
+ | * arg[0]: full path to the directory containing image/stack files | ||
+ | * arg[1]: name of the imageJ | ||
- | [[http:// | ||
- | |||
- | lines to be changed are | ||
< | < | ||
- | 64 IJMACROPATH=" | + | sh /g/almf/software/ij/ |
- | ... | + | |
- | 67 IJMACRONAME=" | + | |
</ | </ | ||
- | Path to the macro and name of the macro file should be replaced according to your situation. | ||
- | rename and save the script, and do | + | argument < |
+ | |||
+ | argument <Name of ImageJ Macro> is the file name of ImageJ macro, and **this file should be placed under / | ||
+ | |||
+ | For example, your command could be | ||
< | < | ||
- | sh < | + | sh / |
</ | </ | ||
- | argument < | + | This script will create two new folders in the directory same as where the image directory is (*_job and *_proc). Processed |
+ | Besides, so-called job reports (text files containing output messages) will appear in your current directory where you executed the command. | ||
+ | |||
+ | The script could be viewed in the link below: [[http:// | ||
===== Example Scripts ===== | ===== Example Scripts ===== | ||
Line 375: | Line 281: | ||
#echo "Time: $((jobendN-jobstartN)) nano-sec." | #echo "Time: $((jobendN-jobstartN)) nano-sec." | ||
</ | </ | ||
- | + | ==== Sample ImageJ macro for Generating Job Array ==== | |
- | ==== Sample ImageJ macro for Generating Job Arrays | + | The ImageJ macro below is the actual content of the script above (called in line 75). it generates one script for one image/ |
- | The ImageJ macro below is the actual content of the script above (called in line 75). | + | |
< | < | ||
/* ImageJ macro for generating shell scripts to be used by qsub array | /* ImageJ macro for generating shell scripts to be used by qsub array | ||
Line 579: | Line 484: | ||
===== Acknowledgements ===== | ===== Acknowledgements ===== | ||
- | Thanks to Frank Thommen(SCB) | + | Thanks to Frank Thommen(SCB), Andres Lindau (IT support) and Christian Tischer (ALMF) for their great help and suggestions. |
documents/100922imagej_cluster.txt · Last modified: 2016/05/24 12:46 by 127.0.0.1