Parasol how to
parasol is software for running batches of jobs on the cluster of computing machines.
links
This is the up-to-date parasol documentation: http://genecats.soe.ucsc.edu/eng/parasol.html
Please read it! I am not going to cover all that information here.
Google returns this OLD link which is now out of date and should NOT be used:
DO NOT USE! http:/users.soe.ucsc.edu/~donnak/eng/parasol.htm DO NOT USE!
A nice page written by Hiram on Cluster_Jobs:
http://genomewiki.ucsc.edu/index.php/Cluster_Jobs
history
parasol is the cluster job-running software written by Jim Kent around 2002 when Condor and other compute cluster software proved to be too slow.
parasol is used in many established pipelines and genome-browser build procedures.
Described as "embarrassingly parallelizable", genomics data is often the perfect target for parallelization since you can just chop up the input data into pieces and then run a cluster job against each piece (or pair of pieces), and then combine the final results. There is software developed here for dividing genomes and FASTA and other kinds of inputs.
Sometimes there are also routines for lifting the input or output from one coordinate system (e.g. single-chunk) to another (e.g. its place on the full chromosome). Data broken into blocks may simply divided into each pieces, but sometimes the pieces are designed to overlap and then the results are chained back together at the end.
Because of variability in input data, e.g. the amount or repeats, or some other feature, some jobs have chunks of data that will end up running quite a bit longer than others. Jim Kent recommends dividing the chunks up smaller to reduce the amount of variation in job runtime. Ideally the average runtime for a job would be just a few minutes. parasol can handle quite a large input queue, with a large number of jobs in a batch, so do not be afraid to divide your work into smaller pieces if needed. It also helps the cluster run more efficiently, and reach equilibrium more quickly, with shorter-running jobs.
gotchas
CRITICAL for performance and good cluster citizenship: If you have jobs that do much input/output, copy the data needed for them to the cluster nodes in /scratch/ so that they will be doing mostly local IO. Your job can copy the final result files back to your batch directory when done. Not doing this can result in slowness or crashing of the cluster, the storage system, or even users home directories.
Another gotcha to avoid with parasol and cluster jobs, do not run your batch and data from your home directory, as this may cause the NFS server that supports the homedirs to crash or become unusably slow. Instead, your parasol batches and their results should be located on the /hive file system. Typically for personal work or research, use /hive/users/$USER. For production browser work such as creating data tracks, use /hive/data/genomes/$assembly/. The makedocs in kent/src/hg/makeDb/doc/ have many examples.
clusters
The current cluster as of 2015-09-18 is "ku" and it has 32 machines or nodes, each with 32 CPUs and 256GB ram which is 1024 CPUs total.
The original KiloKluster had 1024 single CPU machines. There have been many clusters since then, and a general trend to having fewer machines with more CPUs and RAM on a machine with each generation. Now dead and gone heroic clusters of the past:
kk kk2 kk9 kki encodek pk swarm
access
Before starting as a new engineer, make sure that your umask is 2, and that you can automatically login via SSH with no passphrase to ku:
ssh ku
If the server cert has changed, you will get a warning that is trying to protect you from man-in-the-middle attacks. Just delete the old entry from $HOME/.ssh/knownhosts if needed.
Usually the parasol-system related files are installed in /parasol on both the hub and nodes. However only the admins will have access to changing the cluster, adding or removing machines, or other config changes.
parasol list batches is used so often that you may wish to make an alias in your shell config: alias plb "/parasol/bin/parasol list batches"
It may be handy to have /parasol/bin/ on your path.
To use the cluster you must also be a member of the unix group "kluster_access". Before you are granted membership to this group, you must learn a bit about parasol and the clusters from this very wiki page (or from Galt Barber). Then ask for approval from Jim, followed by a cluster-admin email request.
Because the cluster hub, e.g. "ku", and the machines or "nodes" in the cluster mount both your home dirs and the /hive file system, it is easy to switch back and from hgwdev to various machines and work.
parasol nodes run your jobs as you, if the parasol cluster has been correctly started as root. Each job in the batch is run from the shell, but the shell may have different paths than your home shell. If your job's script cannot find a program, you might need to specify the full path.
Temporary files created on the nodes will be deleted after they reach a certain age, however, if you can have your own job cleanup unneeded files, that is helpful.
examples
Here are some simple hands-on examples:
Example 1: Simple test jobs, every thing runs OK
ssh ku cd /hive/users/$USER mkdir testParasol1 cd testParasol1
See what is already running:
parasol list batches
paraTestJob is a simple test program that comes with parasol. It uses up CPU for 1/10 of a second for each count given. So 99 will increment an integer for 9.9 seconds and then exit(0) normally. We will make a jobList that repeats that simple test command many times.
echo "paraTestJob 99" | awk '{for(i=0;i<13320;i++)print}' > jobList
See the resulting jobList
head jobList
paraTestJob 99 paraTestJob 99 paraTestJob 99 paraTestJob 99 paraTestJob 99 paraTestJob 99 paraTestJob 99 paraTestJob 99 paraTestJob 99 paraTestJob 99
parasol is batch-oriented, at least the para client which is used for most work here. (jobTree written by Benedict Paten in python is unable to use the usual "para" client because his dynamic job requests are not known ahead of time, and are created dynamically based on the output of previous jobs. jobTree submits jobs to the hub directly.)
Now create your parasol batch FROM the jobList. parasol and the "para" client will take care of submitting all of the jobs in your batch until they have all been run successfully. But jobs are not added or removed from the batch.
para create jobList
This will create a file called "batch" which contains one line for each line in jobList, plus it stores other meta information like jobIds, results of run, results of checking output, etc.
[ku.sdsc.edu:testParasol1> ll total 448 -rw-rw-r-- 1 galt genecats 652680 Sep 18 11:49 batch -rw-rw-r-- 1 galt genecats 652680 Sep 18 11:49 batch.bak drwxrwxr-x 2 galt genecats 512 Sep 18 11:49 err -rw-rw-r-- 1 galt genecats 199800 Sep 18 11:48 jobList -rw-rw-r-- 1 galt genecats 2 Sep 18 11:49 para.bookmark The batch.bak is a backup file of batch. err/ is a directory where para problems runs gather the stderr of failed jobs. jobList is the original list of jobs to put in the batch. para.bookmark is a file that remembers how much of the results file we have seen so far to speed up processing a little.
Push the first 10 jobs to make sure the jobs are working and do not have a bug.
para try
plb # my alias for parasol list batches #user run wait done crash pri max cpu ram plan min batch galt 4 6 0 0 10 -1 1 8.0g 341 0 /hive/users/galt/testParasol1/ This shows that galt has run 4 jobs out of 10, 6 are still queued, 0 have crashed so far, and priority is default 10. (1 is the "highest" priority and should not be used with permission.) max = -1 means that there is no maximum number of nodes to use for this batch. cpu = 1 means that the batch is using the default number of CPUs, which is 1. ram = 8.0g means that it is using the default amount of RAM, 8GB which is 256GB/32CPUs. plan = 341 means that the parasol planner will give me 341 default-size job slots, if it runs long enough to reach equilibrium. min = 0 shows the average number of minutes jobs in the batch are taking. batch path is given so you can find your directory. Note: plb shows all batches running by all users, but we did not show it here for brevity.
para check [ku.sdsc.edu:testParasol1> para check 13320 jobs in batch 44182 jobs (including everybody's) in Parasol queue or running. Checking finished jobs unsubmitted jobs: 13310 ranOk: 10 total jobs in batch: 13320
This shows that my jobs are not crashing, the 10 from para try ran ok.
At this point, we can just push the rest of the jobs manually. Or, we can use para shove which will automatically repush failed jobs up to the default of 4 retries.
para push plb [ku.sdsc.edu:testParasol1> plb #user run wait done crash pri max cpu ram plan min batch galt 316 2716 10288 0 10 -1 1 8.0g 341 0 /hive/users/galt/testParasol1/
Over a few minutes my running, my running count has increased and almost reached the plan count of 341, at which point it will reach "equilibrium", where everybody has their fair share and the cluster is being used efficiently, nothing is waiting on other jobs/batches to finish to free up resources on nodes.
All jobs finished now:
[ku.sdsc.edu:testParasol1> para check 13320 jobs in batch 40812 jobs (including everybody's) in Parasol queue or running. Checking finished jobs ranOk: 13320 total jobs in batch: 13320 [ku.sdsc.edu:testParasol1> para time 13320 jobs in batch 40860 jobs (including everybody's) in Parasol queue or running. Checking finished jobs Completed: 13320 of 13320 jobs CPU time in finished jobs: 23817s 396.96m 6.62h 0.28d 0.001 y IO & Wait Time: 33602s 560.03m 9.33h 0.39d 0.001 y Average job time: 4s 0.07m 0.00h 0.00d Longest finished job: 9s 0.15m 0.00h 0.00d Submission to last job: 806s 13.43m 0.22h 0.01d Estimated complete: 0s 0.00m 0.00h 0.00d
You can run para time both to get an estimate of when your batch will be finished, and also add to documents in the makeDb/doc/ how much time the run took, which can be useful when people can compare to previous or future runs.
parasol can detect a problem or bug that makes all of your jobs fail in a row, and thus automatically chill your batch, flushing your batch's queue on the hub. This means that you will not waste cluster resources on a "sick" batch that will fail. Why run 100,000 failing jobs?
parasol can also detect sick nodes in which all jobs from multiple batches are failing.
Example 2: more testing, with crashes
cd /hive/users/$USER
mkdir testParasol2 cd testParasol2
Edit to create a new job list like this: vi jobList
paraTestJob 00 paraTestJob 01 cat /this/file/name/probably/does/not/exist paraTestJob 02 paraTestJob 03 paraTestJob 55 -input=spec paraTestJob 04 paraTestJob 66 -crash paraTestJob 77 -output=77.out
This time we see that when we run paraTestJob with the -crash parameter, it will cause the job to crash. Also, the "cat" statement will fail because the file does not exist. So we can test what happens when things do NOT finish normally.
para create jobList para try plb para check
[ku.sdsc.edu:testParasol2> para check 9 jobs in batch 33118 jobs (including everybody's) in Parasol queue or running. Checking finished jobs crashed: 3 ranOk: 6 total jobs in batch: 9
para push
para problems
[ku.sdsc.edu:testParasol2> para problems 9 jobs in batch 32915 jobs (including everybody's) in Parasol queue or running. Checking finished jobs job: cat /this/file/name/probably/does/not/exist id: 377753997 failure type: crash host: ku-1-04.local start time: Fri Sep 18 12:33:02 2015 return: 1 stderr: cat: /this/file/name/probably/does/not/exist: No such file or directory job: paraTestJob 55 -input=spec id: 377754000 failure type: crash host: ku-1-04.local start time: Fri Sep 18 12:33:03 2015 return: 255 stderr: Couldn't open spec , No such file or directory job: paraTestJob 66 -crash id: 377754002 failure type: crash host: ku-1-11.local start time: Fri Sep 18 12:33:03 2015 return: signal 11 stderr: 3 problems total [ku.sdsc.edu:testParasol2> para problems 9 jobs in batch 32915 jobs (including everybody's) in Parasol queue or running. Checking finished jobs job: cat /this/file/name/probably/does/not/exist id: 377753997 failure type: crash host: ku-1-04.local start time: Fri Sep 18 12:33:02 2015 return: 1 stderr: cat: /this/file/name/probably/does/not/exist: No such file or directory job: paraTestJob 55 -input=spec id: 377754000 failure type: crash host: ku-1-04.local start time: Fri Sep 18 12:33:03 2015 return: 255 stderr: Couldn't open spec , No such file or directory job: paraTestJob 66 -crash id: 377754002 failure type: crash host: ku-1-11.local start time: Fri Sep 18 12:33:03 2015 return: signal 11 stderr: 3 problems total
para.results is created by the hub running as root. It contains one line for each job that ran, and contains result of execution or error, the node it ran on, the runtime, etc. para.bookmark keeps track of which results have already been updated into your batch file and do not need checking again. If you delete para.bookmark, it will simply re-process the entire para.results again and create a new bookmark.
[ku.sdsc.edu:testParasol2> head para.results 256 ku-1-04.local 377753997 cat 0 0 1442604781 1442604782 1442604783 galt /tmp/para377753997.err 'cat /this/file/name/probably/does/not/exist' 0 ku-1-19.local 377753999 paraTestJob 5 0 1442604781 1442604782 1442604783 galt /tmp/para377753999.err 'paraTestJob 03' 0 ku-1-16.local 377753998 paraTestJob 3 0 1442604781 1442604782 1442604784 galt /tmp/para377753998.err 'paraTestJob 02' 0 ku-1-11.local 377753996 paraTestJob 1 0 1442604781 1442604782 1442604784 galt /tmp/para377753996.err 'paraTestJob 01' 0 ku-1-25.local 377754001 paraTestJob 7 0 1442604781 1442604783 1442604784 galt /tmp/para377754001.err 'paraTestJob 04' 0 ku-1-02.local 377753995 paraTestJob 0 0 1442604781 1442604782 1442604787 galt /tmp/para377753995.err 'paraTestJob 00' 11 ku-1-11.local 377754002 paraTestJob 117 0 1442604781 1442604783 1442604787 galt /tmp/para377754002.err 'paraTestJob 66 -crash' 65280 ku-1-04.local 377754000 paraTestJob 0 0 1442604781 1442604783 1442604787 galt /tmp/para377754000.err 'paraTestJob 55 -input=spec' 0 ku-1-14.local 377754003 paraTestJob 140 0 1442604781 1442604783 1442604788 galt /tmp/para377754003.err 'paraTestJob 77 -output=77.out'
paraTestJob 77 -output=77.out
This causes it to create a small output file called 77.out
paraTestJob 55 -input=spec
This causes it to read file "spec" which did not actually exist here, so it caused an error.
Example 3: more testing, with gensub2
gensub2 is a handy utility for creating a parasol jobList with many lines by applying a template to one or two lists. Refer to the parasol document listed at the top of this page for details.
cd /hive/users/$USER
mkdir testParasol3 cd testParasol3
Make an output dir out/
mkdir out
Make a template
vi gsub-template
#LOOP
paraTestJob $(path1) -output={check out line+ out/job$(num1).out}
#ENDLOOP
Clearly $(path1) will refer to the each line in input.list. $(num1) refers to the line # of the input.list line being read.
vi input-list
99 24 65 76 34 23 11 1 2 3 77 34 56 9
Usually this is a list of file-paths, not numbers.
Now that we have a template and a data list, we can create the jobList
gensub2 input.list single gsub-template jobList
This takes each line of input.list and applies it to the template, substituting whatever special substitution words you have used.
[ku.sdsc.edu:testParasol3> head jobList
paraTestJob 99 -output={check out line+ out/job0.out}
paraTestJob 24 -output={check out line+ out/job1.out}
paraTestJob 65 -output={check out line+ out/job2.out}
paraTestJob 76 -output={check out line+ out/job3.out}
paraTestJob 34 -output={check out line+ out/job4.out}
paraTestJob 23 -output={check out line+ out/job5.out}
paraTestJob 11 -output={check out line+ out/job6.out}
paraTestJob 1 -output={check out line+ out/job7.out}
paraTestJob 2 -output={check out line+ out/job8.out}
paraTestJob 3 -output={check out line+ out/job9.out}
If you want to use a second file input list, simply place it where the word "single" appears above, it would run the template against every combinination of lines from the first and second lists. For example, all chunks of human against all chunks of mouse.
We are showing off a parasol "para" feature,
{check out line+ out/job9.out}
that it can automatically check that the output exists and contains at least one line.
para create jobList para try para check para shove
Wait for it to finish. Or hit Ctrl-C to return to commandline for further para client commands, but automatic shoving will stop. You can either shove or push again as needed.
para check para time
Note that if we had stuck a line with "0" in input.list above,
paraTestJob 0 -output=out/N.out
would create an empty output file, and then para client would complain and show it as a failure because of the check output clause in the template, jobList, and batch.
[ku.sdsc.edu:testParasol3> para check 14 jobs in batch 23005 jobs (including everybody's) in Parasol queue or running. Checking finished jobs ranOk: 14 total jobs in batch: 14
[ku.sdsc.edu:testParasol3> ls -l out total 704 -rw-rw-r-- 1 galt genecats 5436 Sep 18 12:59 job0.out -rw-rw-r-- 1 galt genecats 1311 Sep 18 12:59 job1.out -rw-rw-r-- 1 galt genecats 4226 Sep 18 13:01 job10.out -rw-rw-r-- 1 galt genecats 1861 Sep 18 13:01 job11.out -rw-rw-r-- 1 galt genecats 3071 Sep 18 13:01 job12.out -rw-rw-r-- 1 galt genecats 477 Sep 18 13:01 job13.out -rw-rw-r-- 1 galt genecats 3566 Sep 18 12:59 job2.out -rw-rw-r-- 1 galt genecats 4171 Sep 18 12:59 job3.out -rw-rw-r-- 1 galt genecats 1861 Sep 18 12:59 job4.out -rw-rw-r-- 1 galt genecats 1256 Sep 18 12:59 job5.out -rw-rw-r-- 1 galt genecats 596 Sep 18 12:59 job6.out -rw-rw-r-- 1 galt genecats 53 Sep 18 12:59 job7.out -rw-rw-r-- 1 galt genecats 106 Sep 18 12:59 job8.out -rw-rw-r-- 1 galt genecats 159 Sep 18 12:59 job9.out
