QA scripts

This is a list of the most frequently-used programs and scripts that QAers use. It does not include track-specific programs, such as chainNetTrio.csh, or these. Some are devised by QA and live in the source tree at kent/src/utils/qa (these mostly end in ".csh"). Some are programs that the engineers or system administrators have written.

Running Scripts on Multiple Tables at Once

Here's a little bash script that you can use to run any of these scripts on multiple tables. Save the list of tables in a file (in your hive directory) called 'tableList_temp', and then type (note: you can press return after each line):

for i in $(cat tableList_temp)
   do
   [script_name] [database/assembly] $i
   echo $i
done

Must know about!

bigPush.csh

(also see: mypush)

Pushes tables in list to mysqlbeta and records size.
  Requires sudo access to mypush to run.
  
  Do not redirect output or run in the background,
  as it will require you to type your password in.
  Program will ask you for your password again after
  large tables. If you take too long to re-type in
  the table the script stalled on might not get
  pushed. Double-check that all tables have been
  pushed!
  
  Will report total size of push and write two files:
  db.tables.push -> output for all tables from mypush
  db.tables.pushSize -> size of push

commTrio.csh

 Sorts and compares two files.  
 Counts unique and common records.

     usage:  leftFileName rightFileName [rm]
             optional [rm]: remove the three output files when finished

compareWholeColumn.csh

  gets a column from a table on dev and beta and checks diffs.
  reports numbers of rows unique to each and common.
  can compare to older database.
  writes files of everything.

    usage:  database table column [db2] 

compareWholeTable.csh

  gets an entire table from two machines and checks diffs.
  reports numbers of rows unique to each and common.
  writes files of everything.
  not real-time on RR -- uses genome-mysql.

    usage:  database table [machine1] [machine2]
      (defaults to dev and beta)

countPerChrom.csh

  check to see if there are annotations on all chroms.
  will check to see if chrom field is named tName or genoName.

    usage:  database1 table [database2] [RR] [histogram]

      checks database1 on dev
      database2 will be checked on beta by default
        if RR is specified, will use genome-mysql
      histogram option prints bar graph, not values

featureBits

(also see: getYield.csh)

 featureBits - Correlate tables via bitmap projections. 
 usage:
   featureBits database table(s)

(truncated for brevity)

findLevel

  searches trackDb hierarchy for your table and corresponding .html file
  also returns the value of the priority and visibility entries
  and the .ra file location for each

    usage:  database tableName

findOrg.csh

 Finds the organism name given the assembly name
  usage: assemblyName [date]
         will accept name with or without digit
         use 'date' to also retrieve assembly date
         (e.g. 'ornAna2' or 'ornAna')

getAssemblies.csh

  gets the names of all databases that contain a given table.
  will accept the MySQL wildcard, %, but not on RR machines
  note: not real-time on RR.  uses nightly TABLE STATUS dump.

    usage:  tablename [machine] [verbose] - defaults to beta
              "verbose" prints list of assemblies checked

getTrackName.csh

 Returns the short label and group of the track for this table.
 In the case of a composite track, it returns the short label
 for both the sub track and the parent track.

    usage:  database tableName

joinerCheck

(also see: runJoiner.csh)

 joinerCheck - Parse and check joiner file
 usage:
   joinerCheck file.joiner
 options:
   -fields - Check fields in joiner file exist, faster with -fieldListIn
      -fieldListOut=file - List all fields in all databases to file.
      -fieldListIn=file - Get list of fields from file rather than mysql.
   -keys - Validate (foreign) keys.  Takes about an hour.
   -tableCoverage - Check that all tables are mentioned in joiner file
   -dbCoverage - Check that all databases are mentioned in joiner file
   -times - Check update times of tables are after tables they depend on
   -all - Do all tests: -fields -keys -tableCoverage -dbCoverage -times
   -identifier=name - Just validate given identifier.
                    Note only applies to keys and fields checks.
   -database=name - Just validate given database.
                    Note only applies to keys and times checks.
   -verbose=N - use verbose to diagnose difficulties. N = 2, 3 or 4 to
              - show increasing level of detail for some functions.

mypush

used with sudo, generally like so, from hgwdev:

• sudo mypush $db $table mysqlbeta

Usage: mypush database table-pattern [hostlist]
 NOTE: use single quotes around table-pattern  
       if it contains shell special chars like * or ? 

realTime.csh

(also see: updateTimes.csh)

  gets update times from all machines in real time for tables in list.

    usage:  database tablelist (will accept single table)

runBits.csh

  runs featureBits and checks for overlap with gaps.

    usage:  database trackname [checkUnbridged]
              where overlap with unbridged gaps can be turned on

updateTimes.csh

(also see: realTime.csh)

  gets update times for three machines for tables in list.
  if table is trackDb, trackDb_public will also be checked.
  warning:  not in real time for RR.  uses overnight dump.

    usage:  database tablelist 

            reports on dev, beta and RR
            tablelist will accept single table

Might also like!

checkBOT.csh

This is useful for answering mailing list questions. Also see the "bottleneck" command.

  wrapper around bottleneck check.
  gives delay stats for IP address(es).

      usage:  ipAddress [terse]
              (terse gives only data)

      (use ipAddress = "all" to get all IPs having delays)

compareTrackDbAll.csh

  checks all fields in trackDb

    usage: database [machine1] [machine2] [mode] 

      (defaults to hgw1 and hgwbeta)
      mode = (fast | verbose | fastVerbose) 
       - verbose is for html field - defaults to terse
       - fast = (genome-mysql) - defaults to realTime (WGET)

checkPushedFiles.csh

 checks to see if files are in place, after a push

 usage: website files(s)

 website should include the path of the directory where
 the files reside, such as:
   http://hgdownload.cse.ucsc.edu/goldenPath/hg19/liftOver/ 

 file(s) is either a single name or a list of names, and can
 include items with additional directory structure, like so:
   filename
   dir/filename
   dir/dir/dir/filename

 any output other than '200 OK' indicates an error.

compareTableToFile.csh

 Ensures that a table correlates with its associated file.
 Only prints results if there is a diff between table and file.
 Works for these file types: narrowPeak, broadPeak, gappedPeak,
                             bedGraph, NRE, BiP, gcf
 For wiggle files, you must specify [wig] parameter.

  usage:  database tableName fileName [wig] [verbose]
   fileName includes path of download file 
   e.g. /goldenPath/<db>/fileName.gz

   use verbose for more details

copyExtSeqRows.csh

 Automatically copies appropriate rows from the extFile and seq tables
 from hgwdev to hgwbeta.

(truncated for brevity)

countRows.csh

  gets the rowcount for a list of tables from dev, beta and RR.

    usage:  database tablelist [genome-mysql]
      tablelist can be just name of single table

    RR results not in real time, but from dumps
    genome-mysql option adds results from public mysql server

findBlatServer.csh

 gets info about which blat server hosts which genome(s)

 usage:  db|host|all  [db|host]  [machine]
   first parameter required: one specific db or host or all dbs
   second parameter optional: order by db or by host (blatServer)
     defaults to order by db
   third parameter optional: specify machine
     defaults to RR

findColumn.csh

  searches database for all tables containing a specified column name.

    usage:  database, field, [machine = hgwdev|hgwbeta] 
      (defaults to beta)

findPushQLocks.csh

  find all locks in the pushQ on hgwbeta

    usage: go|real
     run with 'go' to see a list of locks
     run with 'real' to unlock all the locks

getChainLines.csh

(also see: getMatrixLines.csh)

  Searches the README.txt files to find the correct parameters for the 
  $chainMinScore and $chainLinearGap variables. 

    usage:  fromDb toDb (these can be in either order)

getChromlist.csh

  prints the chrom names for an assembly.

    usage:  database [norandom]

getMatrixLines.csh

(aslo see: getChainLines.csh)

  Searches the README.txt files to find the correct parameters for the 
  $matrix variable.  This is the q-parameter from the blastz run. 

    usage:  fromDb toDb (these can be in either order)

getYield.csh

(also see: featureBits)

  uses featureBits to get yield and enrichment.

    usage:  database trackname [reference track]
              refTrack defaults to refGene

runJoiner.csh

  runs joinerCheck -keys, finding all identifiers for a table.
  runs joinerCheck -times (use "noTimes" to disable).
  set database to "all" for global.
  for chains/nets, use tablename format: chainDb.

    usage:  database table [all.joiner file to use] [noTimes]