The code in this package can be run for SMALL datasets on a single machine. However, it is most effective
to use many processors in parallel. The code in parallel.R and big.R is designed for this task. However,
it must be combined with other materials and external engines such as Condor (www.cs.wisc.edu/condor).
While we have running versions, it is not currently in "user friendly" format. This will be coming in
later releases of R/qtlhot. Brian Yandell 17 may 2012

===========================================================================================
The runparallel.sh shell script is a mockup of what could be done on a parallel system.
We use SOAR from CHTC, which automates use of HTConder.
See http://submit.chtc.wisc.edu/SOAR/SOAR.html .

Phase0.R	Initialization of dataset (done off line)
Phase1.R	Initialization of run. Use cross.RData and params.txt.
		Create Phase1.RData and groups.txt. Run nruns times.
Phase2.R	Permutation runs. Parallelize. Use Phase1.RData and argument based on groups.txt
		Create perm.R_N.RData files. Run nruns*n.split times with n.perm permutations each.
Phase3.R	Wrapup. Wait for all the Phase2 jobs to complete. Use Phase1.RData and perm*.RData.
		Create Phase3.RData. Run nruns times.
		
===========================================================================================
There are actually two ways this is used, depending on the hidden parameter "big".
The first approach (big=FALSE) works for smaller data sets, such as yeast.
For larger data sets (say mouse, with 40K traits), use big=TRUE.

The main difference is that with big=TRUE, the sets of traits are divided up into batches,
say of size.set=250, run with the random permutation, then stitched back together.
This is not explained in the vignette.
You can do is run in parallel n.perm permutations on num.sets = [n.traits/size.set] batches.
That is, there are n.perm * num.sets parallel sandbox runs, rather than just n.perm.
I first create the n.perm random seeds and save them, so that each permutation is done with
the same rearrangement of labels. Try the setting in your params.txt file of:

big = TRUE;
size.set = 250;
n.perm = 1000;

The number 250 was based on a reasonable sized job using our parallel setup
on a 64-bit Linux. This keeps the size of R below 2GB.
R and HTCondor get bogged down if the running R object gets too big,
which happens with large numbers of traits.

===========================================================================================
big = FALSE (use routines in parallel.R)
    Used for studying properties of qtlhot. Can do double resampling: nruns * (n.split * n.perm)
    each run R = 1:nruns is executed as a separate job
    run = 1 uses cross object
    run = 2:nruns uses map from cross object, resimulates genos, phenos

    for each run, Phase2 starts index N = 1:n.split invocations
    each invocation has n.perm random permutations of data
    thus each run has n.split * n.perm permutated data sets
    the n.split invocations are combined together in Phase3

===========================================================================================
big = TRUE (use routines in big.R)
    Used for analysis of large datasets
    run R = 1 (not recommended for nruns > 1)
    
    index = 1 is original data
    index = 2:n.perm are randomly permuted data

    For convenience, n.perm = 1 and n.split = number of permutations.
    That is, each permutation is invoked separately.
    (code has n.perm=n.split for arcane reasons)

    Data are split in big.phase0 into convenient number of traits (~250) to keep cross object small.
    Each of the split data (Trait.*.RData) is run for the same permutation in big.phase2

    Permutations are combined together in big.phase3 to produce one Phase31.RData.

There is pre-processing using big.phase0(), and post-processing Phase31.RData for SOAR. 
These are currently specific to the SOAR/CHTC implementation.
See SOAR.phase0.R and SOAR.post.R for scripts that currently depend on local files.

