CGI Build Process: Difference between revisions

From genomewiki
Jump to navigationJump to search
m (removed testSearch robot)
(Replacing page with 'This page is no longer maintained.')
 
(119 intermediate revisions by 8 users not shown)
Line 1: Line 1:
This page explains the process we use for building and releasing our CGIs.  This is done on a two-week [[CGI_Build_Schedule | schedule]].
This page is no longer maintained.
 
The builds are built into sandboxes which are located here:
hgwbeta:/data/tmp/releaseBuild
titan:/scratch/releaseBuild
 
==Setting Up the Environment for the Build==
NOTE: This section is a one-time only set up that you must do before starting to do the builds.
 
Before starting as buildmeister, you will need to set up the following in your log in file:
* go to your home directory
hgwdev> cd
 
* edit your .tcshrc file
hgwdev> vi .tcshrc file
 
* add the following lines to set permissions on files and to set the environment variables ($BOX32, $WEEKLYBLD, $BUILDDIR and $BRANCHNN are set in one of the build scripts - buildEnv.csh):
umask 002
source /cluster/bin/build/scripts/buildEnv.csh
alias wb 'cd $WEEKLYBLD'
  if (( "$HOST" == "$BOX32" ) || ( "$HOST" == "hgwbeta" )) then
    setenv hg $BUILDDIR/v${BRANCHNN}_branch/kent/src/hg
endif
 
* To be able to run the Java robot programs, add the following to the top of your path setting:
set path = ( /usr/java/jdk1.5.0_10/bin \
  $path \
      .
      .
      .
  )
 
* optional helper functions:
alias wb 'cd $WEEKLYBLD'
 
if (( "$HOST" == "$BOX32" ) || ( "$HOST" == "hgwbeta" )) then
    setenv hg $BUILDDIR/v${BRANCHNN}_branch/kent/src/hg
endif
 
 
wb gets me quickly to the scripts dir.
 
cd $hg gets me quickly to the sandbox
on beta when I have to recompile stuff
after branch tag moves, etc.
 
 
 
* source your .tcshrc to check the variables:
hgwdev> tcsh
 
* Set up autologin among the general cluster machines
On your local cse box (i.e. screech, pfft, whatever)
>ssh-keygen -t dsa  (use enter for all defaults)
>cd ~/.ssh,
add yourself to the authorized keys
>cp id_dsa.pub authorized_keys
 
Also put these in hgwdev:~/.ssh
>scp -r .ssh/ hgwdev:
 
Permissions on .ssh should be 700.
 
Permissions on files in .ssh/ should be 600 or 640.
 
 
* Set up autologin to hgdownload by copying your public key to the list of authorized keys on those machines:
hgwdev> vi ~/.ssh/id_dsa.pub
copy the public key into the clipboard and then log into hgdownload as user qateam
hgwdev> ssh qateam@hgwdownload
hgwdownload> cd ~/.ssh
paste the key to the authorized_keys file
hgdownload> vi authorized_keys
 
===Build Symlinks===
These are critical for building 32 bit and 64 bit utilities
* Go to ~/bin directory
hgwdev> cd ~/bin
 
* Make sure you have $MACHTYPE directories
hgwdev> mkdir i386
hgwdev> mkdir x86_64
 
* Create a symlink for each $MACHTYPE
hgwdev> ln -s /cluster/bin/i386 i386.cluster
hgwdev> ln -s /cluster/bin/x86_64 x86_64.cluster
 
* The symtrick.csh uses these automatically.
* If a script crashes and leaves the symlinks in an incorrect state, use unsymtrick.csh to restore. Build scripts check to see if unsymtrick.csh should be executed.
 
==Preview Day Build==
'''This is day 1 in the [[CGI_Build_Schedule | schedule]].'''
 
===Run CVS Reports===
* go to /cluster/bin/build/scripts/oldsettings
hgwbeta> cd $WEEKLYBLD
* edit buildEnv.csh: change the 5th line then the 4th line
hgwbeta> vi buildEnv.csh
 
  line 5 (OLD): setenv LASTREVIEWDAY 2006-03-20 # preview of v110
  line 5 (NEW): // change it to look like the date and comment from line 4
 
  line 4 (OLD): setenv REVIEWDAY 2006-04-03  # preview of v111
  line 4 (NEW): setenv REVIEWDAY 2006-today  # preview of v112
 
* copy this edited buildEnv.csh file to the oldsettings directory to a new file called buildEnv.csh.vXXX[+ 1].preview
hgwbeta> cp buildEnv.csh oldsettings/buildEnv.csh.v112.preview
 
* source your .tcshrc
hgwbeta> tcsh
 
* check that the vars are correct
hgwbeta> env
 
* run doNewReview.csh as a test to see what it will do (review the variablies)
hgwbeta> ./doNewReview.csh
 
* run for real (this takes about 20 minutes - it does the run by ssh'ing to hgwdev) and direct output to a log file
hgwbeta> ./doNewReview.csh real >& doNewRev.log
 
* review the output file as it is written to
hgwbeta> tail -f doNewRev.log
 
===Check the reports===
* The reports are automatically built by the script into this [http://hgwdev.cse.ucsc.edu/cvs-reports location].
* Briefly review the reports quickly as a sanity check.
 
===Generate summary and review pairings=== (Donna takes care of this)
* Summarize the code changes that were committed during the past week.
* Update this [http://hgwdev.cse.ucsc.edu/builds/versions.html page] with the summary.
* Assign code-review partners.
* Send an email to browser-staff with the summary and code-review pairings.
 
===Test on hgwbeta===
* Wait to hear from QA about how their CGIs look on hgwbeta. Each member of the QA team has [http://hgwdev.cse.ucsc.edu/qa/cgiTesting.html testing responsiblities].
 
==Make changes to code base as necessary==
'''This happens on days 9, 10, and 11 in the [[CGI_Build_Schedule | schedule]].'''
* If there are problems with the build a developer will fix the code.  This fix needs to be patched into the build on hgwbeta.  This [[ Moving_a_Branch_Tag | page]] explains how to move a Branch Tag on hgwbeta.
 
==Final Build==
'''This is day 8 in the [[CGI_Build_Schedule | schedule]].'''
 
===Do the Build===
 
* go to /cluster/bin/build/scripts/oldsettings
hgwbeta> cd $WEEKLYBLD
 
* edit the buildEnv.csh file
hgwbeta> vi buildEnv.csh
setenv BRANCHNN 131
setenv TODAY 2006-04-10    # v131 final
setenv LASTWEEK 2006-03-27  # v130 final
 
* edit file: add one to the BRANCHNN
setenv BRANCHNN 132
 
* edit file: change the 3rd line then the 2nd line
 
line 3 (OLD): setenv LASTWEEK 2006-03-27  # v130 final
line 3 (NEW): // change it to look like the date and comment from line 2
 
line 2 (OLD): setenv TODAY 2006-04-10    # v131 final
line 2 (NEW): setenv TODAY (today)        # v132 final
 
* copy this edited buildEnv.csh file to the oldsettings directory to a new file called buildEnv.csh.vXXX.final
hgwbeta> cp buildEnv.csh oldsettings/buildEnv.csh.v112.final
 
* source your .tcshrc
hgwbeta> tcsh
 
* check that the vars are correct
hgwbeta> env
 
* run doNewBranch.csh  as a test to see what it will do (review the variables)
hgwbeta> ./doNewBranch.csh
 
* run for real and send the output to a file
hgwbeta> ./doNewBranch.csh real >& doNew.log
 
* review the output file as it is written to
hgwbeta> tail -f doNew.log
 
* look for files that tell you it was successful
hgwbeta> /cluster/bin/build/scripts/32bitUtils.ok 
hgwbeta> /cluster/bin/build/scripts/CvsReports.ok
 
 
* What the doNewBranch.csh script does:
 
# edits the versionInfo.h file
# makes tags (takes several minutes)
# builds CVS reports (takes 5-10 minutes)
# does build (takes 5-10 minutes)
## builds CGIs (most important)
## builds utils (of secondary importance)
# runs robots one at a time
## hgNear (20 min)
## hgTables (1 hour)
## TrackCheck (several hours)
## LiftOverTest (quick)
 
* '''Review the error logs for the robots:'''
/build/scripts/logs/ (error logs)
hgNear -- sends email with results
hgTables -- send email with results
hgTrackCheck -- must check by hand+
LiftOverTest -- must check by hand+
(+ located here: hgwbeta:/cluster/bin/build/scripts/logs)
 
==Push the CGIs==
'''This is day 12 in the [[CGI_Build_Schedule | schedule]].'''
 
===Push to hgwN only===
* hgwN is one of the RR machines, hgw1-8. Each build, rotate to the next machine in numeric order i.e. hgw1 then hgw2 etc. so that one machine is not being worked more than the others.
* QA will ask for push of CGIs from hgwbeta to hgwN only.  Make sure you add a note to the cluster-admins to make sure they know which RR machine to push to first. If there is a NEW CGI going out this week, be sure to make a prominent note of it in your push request.  The admins push from a script, and they will need to add your new CGI to the script.
 
As of '''August 2007''', here's a list of the CGIs and data files we push.  Note that CGIs and data files may have been added since this list was created -- this is meant to be a starting point.
 
cartDump cartReset das hgc
hgBlat hgConvert hgCustom hgGateway hgGene hgGenome
hgLiftOver hgNear hgPcr hgSession hgTables hgText hgTracks hgTrackUi
hgVisiGene mkEncodeFrameset hgEncodeDataVersions
pbGateway pbGlobal pbTracks phyloGif
/usr/local/apache/cgi-bin/all.joiner
/usr/local/apache/cgi-bin/galaAvail.tab
/usr/local/apache/cgi-bin/hgNearData/*
/usr/local/apache/cgi-bin/hgGeneData/*
/usr/local/apache/cgi-bin/hgcData/*
/usr/local/apache/cgi-bin/hgCgiData/*
/usr/local/apache/cgi-bin/visiGeneData/*
/usr/local/apache/cgi-bin/loader/*
 
* Run TrackCheck.java on hgwN. This is the responsibility of the QA person who tests hgTracks.
 
# start here: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck>
# make a props file which specifies the machine/db to check.  If you are in a hurry, set zoomCount=0 and it will only check the default position for each assembly (much quicker!).
# run it: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck> nohup TrackCheck hgw1.props > &  $WEEKLYBLD/logs/hgw1.07-13-2006
# grep the output for 'error' (and 'Error').
 
* Monitor Apache Error Log (QA does this):
hgwN:/usr/local/apache/logs/error_log
 
* Wait to hear from QA about how their CGIs look on hgwN. Each member of the QA team has [http://hgwdev.cse.ucsc.edu/qa/cgiTesting.html testing responsiblities]. Check also that TrackCheck ran successfully.
 
===Push to the rest of the RR===
* QA will ask for push from hgwbeta to the rest of the hgwN machines & MGC.
 
==Final Build Wrap-up==
'''This is day 12 in the [[CGI_Build_Schedule | schedule]].'''
 
* zip the source code
hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> tcsh
hgwbeta> env
hgwbeta> doZip.csh  (this is automatically pushed to hgdownload)
 
* build and push hgcentral
hgwdev> ./buildHgCentralSql.csh
hgwdev> ./buildHgCentralSql.csh real
** request push of hgcentral to hgdownload
  Please push from hgwdev --> hgdownload
  /usr/local/apache/htdocs/admin/hgcentral.sql
  This includes settings for the mirror sites.
 
* build 32bit build and push to hgdownload
hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> doBuildCgi32.csh
** the push to hgdownload happens automatically to here:
hgdownload:/mirrordata/apache/cgi-bin-i386/
 
 
* zip liftOver utils on hgwbeta and scp them to hgdownload
hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> doHgDownloadUtils.csh
 
* send email to genome-mirror.  Include link to latest source: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip.
 
[[Category:Browser QA]]

Latest revision as of 18:39, 10 March 2011

This page is no longer maintained.