|
|
| (112 intermediate revisions by 7 users not shown) |
| Line 1: |
Line 1: |
| This page explains the process we use for building and releasing our CGIs. This is done on a two-week [[CGI_Build_Schedule | schedule]]. | | This page is no longer maintained. |
| | |
| The builds are built into sandboxes which are located here:
| |
| hgwbeta:/data/tmp/releaseBuild
| |
| titan:/scratch/releaseBuild
| |
| | |
| ==Setting Up the Environment for the Build==
| |
| NOTE: This section is a one-time only set up that you must do before starting to do the builds.
| |
| | |
| Before starting as buildmeister, you will need to set up the following in your log in file:
| |
| * go to your home directory
| |
| hgwdev> cd
| |
| | |
| * edit your .tcshrc file
| |
| hgwdev> vi .tcshrc file
| |
| | |
| * add the following lines to set permissions on files and to set the environment variables ($BOX32, $WEEKLYBLD, $BUILDDIR and $BRANCHNN are set in one of the build scripts - buildEnv.csh):
| |
| umask 002
| |
| source /cluster/bin/build/scripts/buildEnv.csh
| |
| alias wb 'cd $WEEKLYBLD'
| |
| if (( "$HOST" == "$BOX32" ) || ( "$HOST" == "hgwbeta" )) then
| |
| setenv hg $BUILDDIR/v${BRANCHNN}_branch/kent/src/hg
| |
| endif
| |
| | |
| * To be able to run the Java robot programs, add the following to the top of your path setting:
| |
| set path = ( /usr/java/jdk1.5.0_10/bin \
| |
| $path \
| |
| .
| |
| .
| |
| .
| |
| )
| |
| | |
| * optional helper functions:
| |
| alias wb 'cd $WEEKLYBLD'
| |
| | |
| if (( "$HOST" == "$BOX32" ) || ( "$HOST" == "hgwbeta" )) then
| |
| setenv hg $BUILDDIR/v${BRANCHNN}_branch/kent/src/hg
| |
| endif
| |
| | |
| | |
| wb gets me quickly to the scripts dir.
| |
| | |
| cd $hg gets me quickly to the sandbox
| |
| on beta when I have to recompile stuff
| |
| after branch tag moves, etc.
| |
| | |
| | |
| | |
| * source your .tcshrc to check the variables:
| |
| hgwdev> tcsh
| |
| | |
| * Set up autologin among the general cluster machines
| |
| On your local cse box (i.e. screech, pfft, whatever)
| |
| >ssh-keygen -t dsa (use enter for all defaults)
| |
| >cd ~/.ssh,
| |
| add yourself to the authorized keys
| |
| >cp id_dsa.pub authorized_keys
| |
| | |
| Also put these in hgwdev:~/.ssh
| |
| >scp -r .ssh/ hgwdev:
| |
| | |
| Permissions on .ssh should be 700.
| |
| | |
| Permissions on files in .ssh/ should be 600 or 640.
| |
| | |
| | |
| * Set up autologin to hgdownload by copying your public key to the list of authorized keys on those machines:
| |
| hgwdev> vi ~/.ssh/id_dsa.pub
| |
| copy the public key into the clipboard and then log into hgdownload as user qateam
| |
| hgwdev> ssh qateam@hgwdownload
| |
| hgwdownload> cd ~/.ssh
| |
| paste the key to the authorized_keys file
| |
| hgdownload> vi authorized_keys
| |
| | |
| ===Build Symlinks===
| |
| These are critical for building 32 bit and 64 bit utilities
| |
| * Go to ~/bin directory
| |
| hgwdev> cd ~/bin
| |
| | |
| * Make sure you have $MACHTYPE directories
| |
| hgwdev> mkdir i386
| |
| hgwdev> mkdir x86_64
| |
| | |
| * Create a symlink for each $MACHTYPE
| |
| hgwdev> ln -s /cluster/bin/i386 i386.cluster
| |
| hgwdev> ln -s /cluster/bin/x86_64 x86_64.cluster
| |
| | |
| * The symtrick.csh uses these automatically.
| |
| * If a script crashes and leaves the symlinks in an incorrect state, use unsymtrick.csh to restore. Build scripts check to see if unsymtrick.csh should be executed.
| |
| | |
| ==Preview Day Build==
| |
| '''This is day 1 in the [[CGI_Build_Schedule | schedule]].'''
| |
| | |
| ===Run CVS Reports===
| |
| * go to /cluster/bin/build/scripts/oldsettings
| |
| hgwbeta> cd $WEEKLYBLD
| |
|
| |
| * edit buildEnv.csh: change the 5th line then the 4th line
| |
| hgwbeta> vi buildEnv.csh
| |
| | |
| line 5 (OLD): setenv LASTREVIEWDAY 2006-03-20 # preview of v110
| |
| line 5 (NEW): // change it to look like the date and comment from line 4
| |
| | |
| line 4 (OLD): setenv REVIEWDAY 2006-04-03 # preview of v111
| |
| line 4 (NEW): setenv REVIEWDAY 2006-today # preview of v112
| |
| | |
| * copy this edited buildEnv.csh file to the oldsettings directory to a new file called buildEnv.csh.vXXX[+ 1].preview
| |
| hgwbeta> cp buildEnv.csh oldsettings/buildEnv.csh.v112.preview
| |
| | |
| * source your .tcshrc
| |
| hgwbeta> tcsh
| |
| | |
| * check that the vars are correct
| |
| hgwbeta> env
| |
| | |
| * run doNewReview.csh as a test to see what it will do (review the variablies)
| |
| hgwbeta> ./doNewReview.csh
| |
| | |
| * run for real (this takes about 20 minutes - it does the run by ssh'ing to hgwdev) and direct output to a log file
| |
| hgwbeta> ./doNewReview.csh real >& doNewRev.log
| |
| | |
| * review the output file as it is written to
| |
| hgwbeta> tail -f doNewRev.log
| |
| | |
| ===Check the reports===
| |
| * The reports are automatically built by the script into this [http://hgwdev.cse.ucsc.edu/cvs-reports location].
| |
| * Briefly review the reports quickly as a sanity check.
| |
| | |
| ===Generate summary and review pairings=== (Donna takes care of this)
| |
| * Summarize the code changes that were committed during the past week.
| |
| * Update this [http://hgwdev.cse.ucsc.edu/builds/versions.html page] with the summary.
| |
| * Assign code-review partners.
| |
| * Send an email to browser-staff with the summary and code-review pairings.
| |
| | |
| ===Test on hgwbeta===
| |
| * Wait to hear from QA about how their CGIs look on hgwbeta. Each member of the QA team has [http://hgwdev.cse.ucsc.edu/qa/cgiTesting.html testing responsiblities].
| |
| | |
| ==Make changes to code base as necessary==
| |
| '''This happens on days 9, 10, and 11 in the [[CGI_Build_Schedule | schedule]].'''
| |
| * If there are problems with the build a developer will fix the code. This fix needs to be patched into the build on hgwbeta. This [[ Moving_a_Branch_Tag_And_Patching_a_Branch | page]] explains how to move a Branch Tag and how to Patch a Branch on hgwbeta.
| |
| | |
| ==Final Build==
| |
| '''This is day 8 in the [[CGI_Build_Schedule | schedule]].'''
| |
| | |
| ===Do the Build===
| |
| | |
| * go to /cluster/bin/build/scripts/oldsettings
| |
| hgwbeta> cd $WEEKLYBLD
| |
| | |
| * edit the buildEnv.csh file
| |
| hgwbeta> vi buildEnv.csh
| |
| setenv BRANCHNN 131
| |
| setenv TODAY 2006-04-10 # v131 final
| |
| setenv LASTWEEK 2006-03-27 # v130 final
| |
| | |
| * edit file: add one to the BRANCHNN
| |
| setenv BRANCHNN 132
| |
| | |
| * edit file: change the 3rd line then the 2nd line
| |
| | |
| line 3 (OLD): setenv LASTWEEK 2006-03-27 # v130 final
| |
| line 3 (NEW): // change it to look like the date and comment from line 2
| |
| | |
| line 2 (OLD): setenv TODAY 2006-04-10 # v131 final
| |
| line 2 (NEW): setenv TODAY (today) # v132 final
| |
| | |
| * copy this edited buildEnv.csh file to the oldsettings directory to a new file called buildEnv.csh.vXXX.final
| |
| hgwbeta> cp buildEnv.csh oldsettings/buildEnv.csh.v112.final
| |
| | |
| * re-source buildEnv.csh
| |
| hgwbeta> source buildEnv.csh
| |
| | |
| * check that the vars are correct
| |
| hgwbeta> env
| |
| | |
| * run doNewBranch.csh as a test to see what it will do (review the variables)
| |
| hgwbeta> ./doNewBranch.csh
| |
| | |
| * run for real and send the output to a file
| |
| hgwbeta> ./doNewBranch.csh real >& doNew.log
| |
| | |
| * review the output file as it is written to
| |
| hgwbeta> tail -f doNew.log
| |
| | |
| * look for files that tell you it was successful (script will report whether these files were created):
| |
| hgwbeta> /cluster/bin/build/scripts/32bitUtils.ok
| |
| hgwbeta> /cluster/bin/build/scripts/CvsReports.ok
| |
| | |
| * What the doNewBranch.csh script does:
| |
| | |
| # edits the versionInfo.h file
| |
| # makes tags (takes several minutes)
| |
| # builds CVS reports (takes 5-10 minutes)
| |
| # does build (takes 5-10 minutes)
| |
| ## builds CGIs (most important)
| |
| ## builds utils (of secondary importance)
| |
| | |
| * run doRobots.csh
| |
| ./doRobots.csh >& NNN.robots.log
| |
| | |
| * What the doRobots.csh script does:
| |
| # runs robots one at a time
| |
| ## hgNear (20 min)
| |
| ## hgTables (1 hour)
| |
| ## TrackCheck (several hours)
| |
| ## LiftOverTest (quick)
| |
| | |
| * '''Review the error logs for the robots:'''
| |
| (error logs located here: hgwbeta:/cluster/bin/build/scripts/logs)
| |
| hgNear -- sends email with results
| |
| hgTables -- send email with results
| |
| hgTrackCheck -- must check by hand+
| |
| LiftOverTest -- must check by hand+
| |
| | |
| ==Push the CGIs==
| |
| '''This is day 12 in the [[CGI_Build_Schedule | schedule]].'''
| |
| | |
| ===Push to hgwN only===
| |
| * hgwN is one of the RR machines, hgw1-8. Each build, rotate to the next machine in numeric order i.e. hgw1 then hgw2 etc. so that one machine is not being worked more than the others.
| |
| * QA will send an email to push-request the morning of the push letting the pushers know that today is a CGI push day (this is their notice to be vigilant about pushing quickly).
| |
| * QA will ask for push of CGIs from hgwbeta to hgwN only. Make sure you add a note to the cluster-admins to make sure they know which RR machine to push to first. If there is a NEW CGI going out this week, be sure to make a prominent note of it in your push request. The admins push from a script, and they will need to add your new CGI to the script.
| |
| | |
| As of '''January 2008''', here's a list of the CGIs and data files we push. Note: CGIs and data files may have been added since this list was created -- this is meant to be a starting point.
| |
| | |
| cartDump cartReset das hgc
| |
| hgBlat hgConvert hgCustom hgGateway hgGene hgGenome
| |
| hgLiftOver hgNear hgPcr hgSession hgTables hgTracks hgTrackUi
| |
| hgVisiGene mkEncodeFrameset hgEncodeDataVersions
| |
| pbGateway pbGlobal pbTracks phyloGif
| |
|
| |
| /usr/local/apache/cgi-bin/all.joiner
| |
| /usr/local/apache/cgi-bin/galaAvail.tab
| |
| /usr/local/apache/cgi-bin/hgNearData/*
| |
| /usr/local/apache/cgi-bin/hgGeneData/*
| |
| /usr/local/apache/cgi-bin/hgcData/*
| |
| /usr/local/apache/cgi-bin/hgCgiData/*
| |
| /usr/local/apache/cgi-bin/visiGeneData/*
| |
| /usr/local/apache/cgi-bin/loader/*
| |
| | |
| * Run TrackCheck.java on hgwN. This is the responsibility of the QA person who tests hgTracks.
| |
| | |
| # start here: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck>
| |
| # make a props file which specifies the machine/db to check. If you are in a hurry, set zoomCount=0 and it will only check the default position for each assembly (much quicker!). Example props file for hgw4:
| |
| machine hgwbeta.cse.ucsc.edu
| |
| server hgw4.cse.ucsc.edu
| |
| quick false
| |
| dbSpec all
| |
| table all
| |
| zoomCount 3
| |
| | |
| # run it: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck> nohup TrackCheck hgw1.props > & $WEEKLYBLD/logs/TrackCheck-hgw1.07-13-2006
| |
| # grep the output for 'error' (and 'Error').
| |
| | |
| * Monitor Apache Error Log (QA does this):
| |
| hgwN:/usr/local/apache/logs/error_log
| |
| | |
| * Wait to hear from QA about how their CGIs look on hgwN. Each member of the QA team has [http://hgwdev.cse.ucsc.edu/qa/cgiTesting.html testing responsiblities]. Check also that TrackCheck ran successfully.
| |
| | |
| ===Push to the rest of the RR===
| |
| * QA will ask for push from hgwbeta to the rest of the hgwN machines & MGC.
| |
| * QA will send email to the build-meister to let him/her know that the CGIs are on the RR.
| |
| | |
| ==Final Build Wrap-up==
| |
| '''This is day 12 in the [[CGI_Build_Schedule | schedule]].'''
| |
| | |
| * zip the source code
| |
| hgwbeta> cd /cluster/bin/build/scripts
| |
| hgwbeta> tcsh
| |
| hgwbeta> env
| |
| hgwbeta> doZip.csh (this is automatically pushed to hgdownload)
| |
| | |
| * build and push hgcentral
| |
| hgwdev> ./buildHgCentralSql.csh
| |
| hgwdev> ./buildHgCentralSql.csh real
| |
| ** request push of hgcentral to hgdownload
| |
| Please push from hgwdev --> hgdownload
| |
| /usr/local/apache/htdocs/admin/hgcentral.sql
| |
| This includes settings for the mirror sites.
| |
| | |
| * build 32bit build and push to hgdownload
| |
| hgwbeta> cd /cluster/bin/build/scripts
| |
| hgwbeta> doBuildCgi32.csh
| |
| ** the push to hgdownload happens automatically to here:
| |
| hgdownload:/mirrordata/apache/cgi-bin-i386/
| |
| | |
| | |
| * zip liftOver utils on hgwbeta and scp them to hgdownload
| |
| hgwbeta> cd /cluster/bin/build/scripts
| |
| hgwbeta> doHgDownloadUtils.csh
| |
| | |
| * send email to genome-mirror. Include link to latest source: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip.
| |
| | |
| [[Category:Browser QA]]
| |
