|
|
| Line 1: |
Line 1: |
| This page explains the process we use for building and releasing our CGIs. This is done on a three-week [[CGI_Build_Schedule | schedule]]. | | This page is no longer maintained. |
| | |
| The builds are built into sandboxes which are located here:
| |
| hgwbeta:/data/releaseBuild
| |
| titan:/scratch/releaseBuild
| |
| | |
| Older hgwbeta builds are periodically relocated here
| |
| to save space on hgwbeta which has a small SSD disk:
| |
| /hive/groups/browser/build
| |
| | |
| ==Setting Up the Environment for the Build==
| |
| NOTE: This section is a one-time only set up that you must do before starting to do the builds.
| |
| | |
| Before starting as buildmeister, you will need to set up the following in your log in file:
| |
| * go to your home directory
| |
| hgwdev> cd
| |
| | |
| * edit your .tcshrc file
| |
| hgwdev> vi .tcshrc file
| |
| | |
| * add the following lines to set permissions on files and to set the environment variables ($BOX32, $WEEKLYBLD, $BUILDDIR and $BRANCHNN are set in one of the build scripts - buildEnv.csh):
| |
| umask 002
| |
| source /cluster/bin/build/scripts/buildEnv.csh
| |
| | |
| * To be able to run the Java robot programs, add the following to the top of your path setting:
| |
| set path = ( /usr/java/default/bin \
| |
| $path \
| |
| .
| |
| .
| |
| .
| |
| )
| |
| | |
| * optional helper functions:
| |
| alias wb 'cd $WEEKLYBLD'
| |
| | |
| if (( "$HOST" == "$BOX32" ) || ( "$HOST" == "hgwbeta" )) then
| |
| setenv hg $BUILDDIR/v${BRANCHNN}_branch/kent/src/hg
| |
| endif
| |
| | |
| wb gets you to the scripts dir.
| |
| | |
| cd $hg gets you to the sandbox
| |
| on beta when you have to recompile stuff
| |
| after branch tag moves, etc.
| |
| | |
| * source your .tcshrc to check the variables:
| |
| hgwdev> tcsh
| |
| | |
| * Set up autologin among the general cluster machines
| |
| On your local cse box (i.e. screech, pfft, whatever)
| |
| >ssh-keygen -t dsa (use enter for all defaults)
| |
| >cd ~/.ssh,
| |
| add yourself to the authorized keys
| |
| >cp id_dsa.pub authorized_keys
| |
| | |
| Also put these in hgwdev:~/.ssh
| |
| >scp -r .ssh/ hgwdev:
| |
| | |
| Permissions on .ssh should be 700.
| |
| | |
| Permissions on files in .ssh/ should be 600 or 640.
| |
| | |
| | |
| * Set up autologin to hgdownload by copying your public key to the list of authorized keys on those machines:
| |
| hgwdev> vi ~/.ssh/id_dsa.pub
| |
| copy the public key into the clipboard and then log into hgdownload as user qateam
| |
| hgwdev> ssh qateam@hgdownload
| |
| hgdownload> cd ~/.ssh
| |
| paste the key to the authorized_keys file
| |
| hgdownload> vi authorized_keys
| |
| | |
| ===Build Symlinks===
| |
| These are critical for building 32 bit and 64 bit utilities
| |
| * Go to ~/bin directory
| |
| hgwdev> cd ~/bin
| |
| | |
| * Make sure you have $MACHTYPE directories
| |
| hgwdev> mkdir i386
| |
| hgwdev> mkdir x86_64
| |
| | |
| * Create a symlink for each $MACHTYPE
| |
| hgwdev> ln -s /cluster/bin/i386 i386.cluster
| |
| hgwdev> ln -s /cluster/bin/x86_64 x86_64.cluster
| |
| | |
| * The symtrick.csh uses these automatically.
| |
| * If a script crashes and leaves the symlinks in an incorrect state, use unsymtrick.csh to restore. Build scripts check to see if unsymtrick.csh should be executed.
| |
| | |
| == Preview Day Build : Day 2 ==
| |
| '''This is day 2 in the [[CGI_Build_Schedule | schedule]].'''
| |
| | |
| ===Run Git Reports===
| |
| * go to /cluster/bin/build/scripts/
| |
| hgwbeta> cd $WEEKLYBLD
| |
|
| |
| * edit buildEnv.csh: change the 5th line then the 4th line
| |
| hgwbeta> vi buildEnv.csh
| |
| | |
| line 5 (OLD): setenv LASTREVIEWDAY 2006-03-20 # preview of v110
| |
| line 5 (NEW): // change it to look like the date and comment from line 4
| |
| | |
| line 4 (OLD): setenv REVIEWDAY 2006-04-03 # preview of v111
| |
| line 4 (NEW): setenv REVIEWDAY 2006-today # preview of v112
| |
| | |
| * re-source buildEnv.csh
| |
| hgwbeta> source buildEnv.csh # or just restart your shell windows
| |
| | |
| * check that the vars are correct
| |
| hgwbeta> env
| |
| | |
| * make sure you are on master branch
| |
| git checkout master
| |
| | |
| * commit the changes to this file to Git:
| |
| hgwbeta> @ NEXTNN = ( $BRANCHNN + 1 ) ; git commit -m "v$NEXTNN preview" buildEnv.csh; git pull; git push
| |
| | |
| * use screen to run
| |
| screen
| |
| * run doNewReview.csh as a test to see what it will do (review the variablies)
| |
| hgwbeta> ./doNewReview.csh
| |
| * run for real (this takes about 2 minutes - it runs git reports by ssh'ing to hgwdev) and direct output to a log file
| |
| hgwbeta> ./doNewReview.csh real >& doNewRev.log
| |
| ctrl-a, d to detach
| |
| hgwbeta> tail -f doNewRev.log
| |
| | |
| ===Check the reports===
| |
| * The reports are automatically built by the script into this [http://genecats.cse.ucsc.edu/git-reports location].
| |
| * Briefly review the reports quickly as a sanity check.
| |
| | |
| ===Generate summary and review pairings=== (Ann takes care of this)
| |
| * Summarize the code changes that were committed during the past week.
| |
| * Update this [http://hgwdev.cse.ucsc.edu/builds/versions.html page] with the summary.
| |
| * Assign code-review partners.
| |
| * Send an email to browser-staff with the summary and code-review pairings.
| |
| | |
| ===Test on hgwbeta===
| |
| * Wait to hear from QA about how their CGIs look on hgwbeta. Each member of the QA team has [http://hgwdev.cse.ucsc.edu/qa/cgiTesting.html testing responsiblities].
| |
| | |
| ==Make changes to code base as necessary==
| |
| '''This happens on days 9, 10, and 11 in the [[CGI_Build_Schedule | schedule]].'''
| |
| * If there are problems with the build a developer will fix the code. This fix needs to be patched into the build on hgwbeta. This [[ Moving_a_Branch_Tag_And_Patching_a_Branch | page]] explains how to move a Branch Tag and how to Patch a Branch on hgwbeta.
| |
| | |
| ==Final Build : Day 9 ==
| |
| '''This is day 9 in the [[CGI_Build_Schedule | schedule]].'''
| |
| | |
| ===Do the Build===
| |
| | |
| * go to weekly build dir on beta
| |
| hgwbeta> cd $WEEKLYBLD
| |
| | |
| * edit the buildEnv.csh file
| |
| hgwbeta> vi buildEnv.csh
| |
| setenv BRANCHNN 131
| |
| setenv TODAY 2006-04-10 # v131 final
| |
| setenv LASTWEEK 2006-03-27 # v130 final
| |
| | |
| * edit file: add one to the BRANCHNN
| |
| setenv BRANCHNN 132
| |
| | |
| * edit file: change the 3rd line then the 2nd line
| |
| | |
| line 3 (OLD): setenv LASTWEEK 2006-03-27 # v130 final
| |
| line 3 (NEW): // change it to look like the date and comment from line 2
| |
| | |
| line 2 (OLD): setenv TODAY 2006-04-10 # v131 final
| |
| line 2 (NEW): setenv TODAY (today) # v132 final
| |
| | |
| * re-source buildEnv.csh
| |
| hgwbeta> source buildEnv.csh # probably better to just close and restart your shell windows
| |
| | |
| * check that the vars are correct
| |
| hgwbeta> env
| |
| | |
| * check that you are on master branch
| |
| git checkout master
| |
| | |
| * commit the changes to this file to Git:
| |
| hgwbeta> git commit -m "v$BRANCHNN final build" buildEnv.csh;git pull;git push
| |
| | |
| * do a dummy-login to the 32-bit box so that the script will be more likely not to hang later on building 32-bit cgis.
| |
| ssh $BOX32 "ls -l"
| |
| | |
| * run doNewBranch.csh
| |
| hgwbeta> screen
| |
| * test to see what it will do (review the variables)
| |
| hgwbeta> ./doNewBranch.csh
| |
| * run for real, send the output to a file and review while it is written (takes ~1 hour)
| |
| hgwbeta> ./doNewBranch.csh real >& doNew.log
| |
| ctrl-a, d # to detach from the screen
| |
| hgwbeta> tail -f doNew.log
| |
| | |
| * look for files that tell you it was successful (script will report whether these files were created):
| |
| hgwbeta> /cluster/bin/build/scripts/32bitUtils.ok
| |
| hgwbeta> /cluster/bin/build/scripts/GitReports.ok
| |
| | |
| * If you get errors it might be because the script is wrong, rather than there actually be an error. For example, to check for errors, the 'make' log file is grepped for 'error|warn' so any new 'C' file with error or warn in its name will show up as an error whether or not it compiled cleanly. You might need to change the script to remove references to files like this, eg edit buildBeta.csh to ignore references to files like gbWarn.c and gbWarn.o in the log:
| |
| hgwbeta> vi buildBeta.csh
| |
| ...
| |
| make alpha >& make.alpha.log
| |
| # These flags and programs will trip the error detection
| |
| sed -i -e "s/-DJK_WARN//g" make.alpha.log
| |
| sed -i -e "s/-Werror//g" make.alpha.log
| |
| #-- report any compiler warnings, fix any errors (shouldn't be any)
| |
| #-- to check for errors:
| |
| set res = `/bin/egrep -i "error|warn" make.alpha.log | /bin/grep -v "gbWarn.o -c gbWarn.c" | /bin/grep -v "gbExtFile.o gbWarn.o gbMiscDiff.o"`
| |
| | |
| * What the doNewBranch.csh script does:
| |
| | |
| # edits the versionInfo.h file
| |
| # makes tags (takes 1 minute)
| |
| # builds Git reports (takes 1 minute)
| |
| # does build (takes 5-10 minutes)
| |
| ## builds utils (of secondary importance)
| |
| ## builds CGIs (most important)
| |
| | |
| ===Check the reports===
| |
| * The reports are automatically built by the script into this [http://genecats.cse.ucsc.edu/git-reports location].
| |
| * Briefly review the reports quickly as a sanity check.
| |
| | |
| ===Run the Robots===
| |
| | |
| * ['''build-meister'''] run doRobots.csh, and watch the log if you are interested (most log messages go to the logs/ dir mentioned below)
| |
| hgwbeta> screen # startup a new screen
| |
| hgwbeta> ./doRobots.csh >& $BRANCHNN.robots.log
| |
| # ctrl-a, d to detach from screen
| |
| hgwbeta> tail -f $BRANCHNN.robots.log
| |
| | |
| * What the doRobots.csh script does:
| |
| # runs robots one at a time
| |
| ## hgNear (20 min)
| |
| ## hgTables (several hours)
| |
| ## TrackCheck (several hours)
| |
| ## LiftOverTest (quick)
| |
| | |
| * ['''build shepherd'''] Review the error logs for the robots:
| |
| | |
| error logs located here: hgwbeta:/cluster/bin/build/scripts/logs
| |
| | |
| * hgNear -- sends email with results
| |
| * hgTables -- send email with results
| |
| * TrackCheck -- must check by hand: grep -i "error" logs/TrackCheck-v$BRANCHNN.log (''TrackCheck person does this'')
| |
| * LiftOverTest -- must check by hand: cat logs/LiftOverTest-v$BRANCHNN.log
| |
| | |
| ==Fixing problems in the Build ==
| |
| '''This usually happens between days 9 and 12.'''
| |
| | |
| ===QA advises buildmeister to patch or move the branch tags===
| |
| | |
| * see [[Moving_a_Branch_Tag_And_Patching_a_Branch|these instructions]].
| |
| | |
| ==Push the CGIs==
| |
| '''This is day 16 in the [[CGI_Build_Schedule | schedule]].'''
| |
| | |
| ===Push to hgwN only===
| |
| * hgwN is one of the RR machines, hgw1-8. Each build, rotate to the next machine in numeric order i.e. hgw1 then hgw2 etc. so that one machine is not being worked more than the others.
| |
| * QA will send an email to push-request the morning of the push letting the pushers know that today is a CGI push day (this is their notice to be vigilant about pushing quickly).
| |
| * QA will ask for push of CGIs from hgwbeta to hgwN only. Make sure you add a note to the cluster-admins to make sure they know which RR machine to push to first. If there is a NEW CGI going out this week, be sure to make a prominent note of it in your push request. The admins push from a script, and they will need to add your new CGI to the script. (the build-meister should not be cc'd on this email.)
| |
| | |
| As of '''February 2011''', here's a list of the CGIs and data files we push. Note: CGIs and data files may have been added since this list was created -- this is meant to be a starting point.
| |
| | |
| cartDump cartReset das hgc hgEncodeVocab
| |
| hgBlat hgConvert hgCustom hgGateway hgGene hgGenome
| |
| hgLiftOver hgNear hgPcr hgSession hgTables hgTracks hgTrackUi
| |
| hgVisiGene mkEncodeFrameset hgEncodeDataVersions
| |
| pbGateway pbGlobal pbTracks phyloGif hgPal hgSuggest hgApi
| |
| hgFileUi
| |
|
| |
| /usr/local/apache/cgi-bin/all.joiner
| |
| /usr/local/apache/cgi-bin/galaAvail.tab
| |
| /usr/local/apache/cgi-bin/hgNearData/*
| |
| /usr/local/apache/cgi-bin/hgGeneData/*
| |
| /usr/local/apache/cgi-bin/hgcData/*
| |
| /usr/local/apache/cgi-bin/hgCgiData/*
| |
| /usr/local/apache/cgi-bin/visiGeneData/*
| |
| /usr/local/apache/cgi-bin/loader/*
| |
| /usr/local/apache/cgi-bin/lsSnpPdbChimera.py
| |
| /usr/local/apache/cgi-bin/greatData/*
| |
| | |
| For these directories we request an rsync --delete (from hgwbeta to the RR)
| |
| /usr/local/apache/htdocs/js/*
| |
| /usr/local/apache/htdocs/style/*
| |
| | |
| * Run TrackCheck.java on hgwN. This is the responsibility of the QA person who tests hgTracks.
| |
| | |
| # make a props file which specifies the machine/db to check. If you are in a hurry, set zoomCount=1 and it will only check the default position for each assembly (much quicker!). Example props file for hgw4:
| |
| machine mysqlbeta.cse.ucsc.edu
| |
| server hgw4.cse.ucsc.edu
| |
| quick false
| |
| dbSpec all
| |
| table all
| |
| zoomCount 3
| |
| | |
| # run it: nohup TrackCheck hgw1.props > & $WEEKLYBLD/logs/TrackCheck-hgw1.07-13-2006
| |
| # examine the file for errors.
| |
| | |
| * Monitor Apache Error Log (QA does this):
| |
| hgwN:/usr/local/apache/logs/error_log
| |
| To watch the log without line wraps, use [http://en.wikipedia.org/wiki/Less_(Unix) less -S]. Use "F" to follow incoming errors.
| |
| | |
| * Wait to hear from QA about how their CGIs look on hgwN. Each member of the QA team has [http://hgwdev.cse.ucsc.edu/qa/cgiTesting.html testing responsiblities]. Check also that TrackCheck ran successfully.
| |
| | |
| ===Push to the rest of the RR and hgwbeta-public===
| |
| * QA will ask for push from hgwbeta to the rest of the hgwN machines, as well as hgwbeta-public. The js and style directory files should also go to /usr/local/apache/htdocs/js-public/* (or style-public/*) on hgwbeta in order to keep the javascript the same on the RR and hgwbeta-public. So, in addition to asking for the rsync --delete of the directories from hgwbeta to the RR machines, we also need to ask for an rsync --delete:
| |
| from
| |
| /usr/local/apache/htdocs/js/*
| |
| /usr/local/apache/htdocs/style/* (on hgwbeta)
| |
| to
| |
| /usr/local/apache/htdocs/js-public/*
| |
| /usr/local/apache/htdocs/style-public/* (on hgwbeta)
| |
| | |
| * QA will send email to the build-meister to let him/her know that the CGIs are on the RR.
| |
| | |
| ===Remember to keep track of new features===
| |
| Anyone can add to this list at any time, but if no notes for this release have been made on the [[Genome_Browser_Software_Features | new features page]], now is a good time to add some.
| |
| | |
| ==Final Build Wrap-up==
| |
| '''This is day 16 in the [[CGI_Build_Schedule | schedule]].'''
| |
| | |
| The buildmeister should do these steps once QA has notified you that all
| |
| RR machines have been updated.
| |
| | |
| * build and push hgcentral IF there are any changes
| |
| hgwbeta> ssh hgwdev
| |
| hgwdev> tcsh
| |
| hgwdev> cd $WEEKLYBLD
| |
| hgwdev> ./buildHgCentralSql.csh
| |
| hgwdev> ./buildHgCentralSql.csh real
| |
| | |
| * check that the hgcentral.sql has been updated:
| |
| http://hgdownload.cse.ucsc.edu/admin/
| |
| | |
| * build 'userApps' target (various utilities) on hgwbeta and scp them to hgdownload
| |
| ssh hgwbeta
| |
| hgwbeta> cd $WEEKLYBLD
| |
| hgwbeta> ./doHgDownloadUtils.csh
| |
| patch as needed
| |
| | |
| * check that the utils version is right:
| |
| http://hgdownload.cse.ucsc.edu/admin/exe
| |
| | |
| * build 32bit build and push to hgdownload
| |
| hgwbeta> tcsh
| |
| hgwbeta> cd $WEEKLYBLD
| |
| hgwbeta> ./doBuildCgi32.csh
| |
| patch as needed
| |
| | |
| * check that the source file version is right:
| |
| ssh qateam@hgdownload "ls -ltr /mirrordata/apache/cgi-bin-i386"
| |
| | |
| * update the beta tag to match the release:
| |
| hgwbeta> tcsh
| |
| hgwbeta> cd $WEEKLYBLD
| |
| hgwbeta> env (just to make sure it looks right)
| |
| hgwbeta> ./tagBeta.csh
| |
| hgwbeta> ./tagBeta.csh real
| |
| | |
| * tag the official release
| |
| ssh hgwbeta
| |
| cd $WEEKLYBLD
| |
| git fetch
| |
| git tag | grep "v${BRANCHNN}_branch"
| |
| Note: use .1 or .2 or whatever is the next unused subversion number
| |
| git push origin origin/v${BRANCHNN}_branch:refs/tags/v${BRANCHNN}_branch.1
| |
| git fetch
| |
| | |
| * zip the source code
| |
| hgwbeta> tcsh
| |
| hgwbeta> cd $WEEKLYBLD
| |
| hgwbeta> ./doZip.csh (this is automatically pushed to hgdownload)
| |
| | |
| * check that the source file version is right:
| |
| http://hgdownload.cse.ucsc.edu/admin/
| |
| | |
| * send email to genome-mirror@soe.ucsc.edu.
| |
| | |
| Include this link to latest source: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip.
| |
| Use the last email as a template (see https://www.soe.ucsc.edu/pipermail/genome-mirror).
| |
| If you push the hgcentral.sql, make sure to mention this has also changed in the email.
| |
| | |
| [[Category:Browser QA]]
| |
