CGI Build Process: Difference between revisions

From genomewiki
Jump to navigationJump to search
(Replacing page with 'This page is no longer maintained.')
 
(146 intermediate revisions by 8 users not shown)
Line 1: Line 1:
This page explains the process we use for building and releasing our CGIs.  This is done on a two-week [[CGI_Build_Schedule | schedule]].
This page is no longer maintained.
 
The builds are built into sandboxes which are located here:
hgwbeta:/data/tmp/releaseBuild
titan:/scratch/releaseBuild
 
==Setting Up the Environment for the Build==
Before starting as buildmeister, you will need to set up the following in your log in file:
* go to your home directory
hgwdev> cd
 
* edit your .tcshrc file
hgwdev> vi .tcshrc file
 
* add the following lines to set environment variables ($BOX32, $WEEKLYBLD, $BUILDDIR and $BRANCHNN are set in one of the build scripts - buildEnv.csh):
alias wb 'cd $WEEKLYBLD'
if (( "$HOST" == "$BOX32" ) || ( "$HOST" == "hgwbeta" )) then
    setenv hg $BUILDDIR/v${BRANCHNN}_branch/kent/src/hg
endif
 
* To be able to run the Java robot programs, add the following to the top of your path setting:
set path = ( /usr/java/jdk1.5.0_10/bin \
  $path \
      .
      .
      .
  )
 
* source your .tcshrc to check the variables:
hgwdev> tcsh
 
* Set up autologin to hgwdev and titan:
 
==Preview Day Build==
'''This is day 1 in the [[CGI_Build_Schedule | schedule]].'''
 
===Run CVS Reports===
* go to /cluster/bin/build/scripts/oldsettings
hgwbeta> cd $WEEKLYBLD
* edit buildEnv.csh: change the 5th line then the 4th line
hgwbeta> vi buildEnv.csh
 
  line 5 (OLD): setenv LASTREVIEWDAY 2006-03-20 # preview of v110
  line 5 (NEW): // change it to look like the date and comment from line 4
 
  line 4 (OLD): setenv REVIEWDAY 2006-04-03  # preview of v111
  line 4 (NEW): setenv REVIEWDAY 2006-today  # preview of v112
 
* copy this edited buildEnv.csh file to the oldsettings directory to a new file called buildEnv.csh.vXXX[+ 1].preview
hgwbeta> cp buildEnv.csh oldsettings/buildEnv.csh.v112.preview
 
* source your .tcshrc
hgwbeta> tcsh
 
* check that the vars are correct
hgwbeta> env
 
* run doNewReview.csh as a test to see what it will do (review the variablies)
hgwbeta> ./doNewReview.csh
 
* run for real (this takes about 20 minutes - it does the run by ssh'ing to hgwdev) and direct output to a log file
hgwbeta> ./doNewReview.csh real >& doNewRev.log
 
* review the output file as it is written to
hgwbeta> tail -f doNewRev.log
 
===Check the reports===
* The reports are automatically built by the script into this [http://hgwdev.cse.ucsc.edu/cvs-reports location].
* Briefly review the reports quickly as a sanity check.
 
===Generate summary and review pairings=== (Donna takes care of this)
* Summarize the code changes that were committed during the past week.
* Update this [http://hgwdev.cse.ucsc.edu/builds/versions.html page] with the summary.
* Assign code-review partners.
* Send an email to browser-staff with the summary and code-review pairings.
 
===Test on hgwbeta===
* Wait to hear from QA about how their CGIs look on hgwbeta. Each member of the QA team has [http://hgwdev.cse.ucsc.edu/qa/cgiTesting.html testing responsiblities].
 
==Make changes to code base as necessary==
'''This happens on days 9, 10, and 11 in the [[CGI_Build_Schedule | schedule]].'''
* If there are problems with the build a developer will fix the code.  This fix needs to be patched into the build on hgwbeta.  This [http://genome-test.cse.ucsc.edu/qa/test-protocols/movingBranchTags.html document] explains how to move a Branch Tag on hgwbeta.
 
==Final Build==
'''This is day 8 in the [[CGI_Build_Schedule | schedule]].'''
 
===Do the Build===
 
* go to /cluster/bin/build/scripts/oldsettings
hgwbeta> cd $WEEKLYBLD
 
* edit the buildEnv.csh file
hgwbeta> vi buildEnv.csh
setenv BRANCHNN 131
setenv TODAY 2006-04-10    # v131 final
setenv LASTWEEK 2006-03-27  # v130 final
 
* edit file: add one to the BRANCHNN
setenv BRANCHNN 132
 
* edit file: change the 3rd line then the 2nd line
 
line 3 (OLD): setenv LASTWEEK 2006-03-27  # v130 final
line 3 (NEW): // change it to look like the date and comment from line 2
 
line 2 (OLD): setenv TODAY 2006-04-10    # v131 final
line 2 (NEW): setenv TODAY (today)        # v132 final
 
* copy this edited buildEnv.csh file to the oldsettings directory to a new file called buildEnv.csh.vXXX.final
hgwbeta> cp buildEnv.csh oldsettings/buildEnv.csh.v112.final
 
* source your .tcshrc
hgwbeta> tcsh
 
* check that the vars are correct
hgwbeta> env
 
* run doNewBranch.csh  as a test to see what it will do (review the variables)
hgwbeta> ./doNewBranch.csh
 
* run for real and send the output to a file
hgwbeta> ./doNewBranch.csh real >& doNew.log
 
* review the output file as it is written to
hgwbeta> tail -f doNew.log
 
* look for files that tell you it was successful
hgwbeta> /cluster/bin/build/scripts/32bitUtils.ok 
hgwbeta> /cluster/bin/build/scripts/CvsReports.ok
 
 
* What the doNewBranch.csh script does:
 
# edits the versionInfo.h file
# makes tags (takes several minutes)
# builds CVS reports (takes 5-10 minutes)
# does build (takes 5-10 minutes)
## builds CGIs (most important)
## builds utils (of secondary importance)
# runs robots one at a time
## hgNear (20 min)
## TB (1 hour)
## TrackCheck (several hours)
## LiftOverTest (quick)
 
* Review the error logs for the three robots:
/build/scripts/logs/ (error logs)
hgNear -- sends email with results
hgTables -- send email with results
hgTrackCheck -- must check by hand+
LiftOverTest -- must check by hand+
+ located here: hgwbeta:/cluster/bin/build/scripts/logs
 
==Push the CGIs==
'''This is day 12 in the [[CGI_Build_Schedule | schedule]].'''
 
===Push to hgwN only===
* hgwN is one of the RR machines, hgw1-8. Each build, rotate to the next machine in numeric order i.e. hgw1 then hgw2 etc. so that one machine is not being worked more than the others.
* Wait to hear that all pair-reviews are complete.
* Ask for push of CGIs from hgwbeta to hgwN only.  Make sure you add a note to the cluster-admins to make sure they know which RR machine to push to first. If there is a NEW CGI going out this week, be sure to make a prominent note of it in your push request.  The admins push from a script, and they will need to add your new CGI to the script.
* Run TrackCheck.java on hgwN. This is the responsibility of the QA person who tests hgTracks.
 
# start here: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck>
# make a props file which specifies the machine/db to check.  If you are in a hurry, set zoomCount=0 and it will only check the default position for each assembly (much quicker!).
# run it: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck> nohup TrackCheck hgw1.props > &  $WEEKLYBLD/logs/hgw1.07-13-2006
# grep the output for 'error'.
 
* Monitor Apache Error Log
hgwN:/usr/local/apache/logs/error_log
 
* Wait to hear from QA about how their CGIs look on hgwN. Each member of the QA team has [http://hgwdev.cse.ucsc.edu/qa/cgiTesting.html testing responsiblities]. Check also that TrackCheck ran successfully.
 
===Push to the rest of the RR===
* Ask for push from hgwbeta to hgw2-8 & MGC.
 
==Final Build Wrap-up==
'''This is day 12 in the [[CGI_Build_Schedule | schedule]].'''
 
* zip the source code
hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> tcsh
hgwbeta> env
hgwbeta> doZip.csh  (this is automatically pushed to hgdownload)
 
* build and push hgcentral
hgwdev> ./buildHgCentralSql.csh
hgwdev> ./buildHgCentralSql.csh real
** request push of hgcentral to hgdownload
  Please push from hgwdev --> hgdownload
  /usr/local/apache/htdocs/admin/hgcentral.sql
  This includes settings for the mirror sites.
 
* build 32bit build and push to hgdownload
hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> doBuildCgi32.csh
** the push to hgdownload happens automatically to here:
hgdownload:/mirrordata/apache/cgi-bin-i386/
 
 
* zip liftOver utils on hgwbeta and scp them to hgdownload
hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> doHgDownloadUtils.csh
 
* send email to genome-mirror.  Include link to latest source: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip.
 
[[Category:Browser QA]]

Latest revision as of 18:39, 10 March 2011

This page is no longer maintained.