CGI Build Process

This page explains the process we use for building and releasing our CGIs. This is done on a two-week schedule.

The builds are built into sandboxes which are located here:

hgwbeta:/data/tmp/releaseBuild
hgwdevold:/scratch/releaseBuild

Preview Day Build

This is day 1 in the schedule.

Run CVS Reports

• go to /cluster/bin/build/scripts/oldsettings

hgwbeta> cd $WEEKLYBLD
hgwbeta> cd oldsettings

• copy the last buildEnv.csh file to a new file called buildEnv.csh.vXXX[+ 1].preview

hgwbeta> cp buildEnv.csh.v111.final buildEnv.csh.v112.preview

• edit this newly-created file: change the 5th line then the 4th line

hgwbeta> vi buildEnv.csh.v112.preview

 line 5 (OLD): setenv LASTREVIEWDAY 2006-03-20 # preview of v110
 line 5 (NEW): // change it to look like the date and comment from line 4

 line 4 (OLD): setenv REVIEWDAY 2006-04-03  # preview of v111
 line 4 (NEW): setenv REVIEWDAY 2006-today  # preview of v112

• copy the newly created file over the file located up one directory

hgwbeta> cd ../
hgwbeta> cp oldsettings/buildEnv.csh.v112.preview buildEnv.csh

• source your .tcshrc

hgwbeta> tcsh

• check that the vars are correct

hgwbeta> env

• run doNewReview.csh as a test to see what it will do (review the variablies)

hgwbeta> ./doNewReview.csh

• run for real (this takes about 20 minutes - it does the run by ssh'ing to hgwdev)

hgwbeta> ./doNewReview.csh real

Check the reports

• The reports are automatically built by the script into this location.
• Briefly review the reports quickly as a sanity check.

Generate summary and review pairings

• Summarize the code changes that were committed during the past week.
• Update this page with the summary.
• Assign code-review partners.
• Send an email to browser-staff with the summary and code-review pairings.

Test on hgwbeta

• Wait to hear from QA about how their CGIs look on hgwbeta. Each member of the QA team has testing responsiblities.

Make changes to code base as necessary

This happens on days 9, 10, and 11 in the schedule.

• If there are problems with the build a developer will fix the code. This fix needs to be patched into the build on hgwbeta. This document explains how to move a Branch Tag on hgwbeta.

Final Build

This is day 8 in the schedule.

Do the Build

• go to /cluster/bin/build/scripts/oldsettings

hgwbeta> cd $WEEKLYBLD
hgwbeta> cd oldsettings

• copy the last buildEnv.csh file to a new file called buildEnv.csh.vXXX.final

hgwbeta> cp buildEnv.csh.v112.preview buildEnv.csh.v112.final

• edit this file

hgwbeta> vi buildEnv.csh.v112.final
setenv BRANCHNN 131
setenv TODAY 2006-04-10     # v131 final
setenv LASTWEEK 2006-03-27  # v130 final

• edit file: add one to the BRANCHNN

setenv BRANCHNN 132

• edit file: change the 3rd line then the 2nd line

line 3 (OLD): setenv LASTWEEK 2006-03-27  # v130 final
line 3 (NEW): // change it to look like the date and comment from line 2

line 2 (OLD): setenv TODAY 2006-04-10     # v131 final
line 2 (NEW): setenv TODAY (today)        # v132 final

• copy the newly created file over the file located up one directory

hgwbeta> cd ../
hgwbeta> cp oldsettings/buildEnv.csh.v112.final buildEnv.csh

• source your .tcshrc

hgwbeta> tcsh

• check that the vars are correct

hgwbeta> env

• run doNewBranch.csh as a test to see what it will do (review the variables)

hgwbeta> ./doNewBranch.csh

• run for real and send the output to a file

hgwbeta> ./doNewBranch.cvs real >& doNew.log

• review the output file as it is written to

hgwbeta> tail -f doNew.log

• look for files that tell you it was successful

hgwbeta> /cluster/bin/build/scripts/32bitUtils.ok  
hgwbeta> /cluster/bin/build/scripts/CvsReports.ok

• What the doNewBranch.cvs script does:

1. edits the versionInfo.h file
2. makes tags (takes several minutes)
3. builds CVS reports (takes 5-10 minutes)
4. does build (takes 5-10 minutes)
   1. builds CGIs (most important)
   2. builds utils (of secondary importance)
5. runs robots one at a time
   1. hgNear (20 min)
   2. TB (1 hour)
   3. TrackCheck (several hours)
   4. LiftOverTest (quick)

• Review the error logs for the three robots:

/build/scripts/logs/ (error logs)

hgNear -- sends email with results
hgTables -- send email with results
hgTrackCheck -- must check by hand+
LiftOverTest -- must check by hand+
+ located here: hgwbeta:/cluster/bin/build/scripts/logs 

Push the CGIs

This is day 12 in the schedule.

Push to hgw1 only

• Wait to hear that all pair-reviews are complete.
• Ask for push of CGIs from hgwbeta to hgw1 only. If there is a NEW CGI going out this week, be sure to make a prominent note of it in your push request. The admins push from a script, and they will need to add your new CGI to the script.
• Run TrackCheck.java on hgw1. This is the responsibility of the QA person who tests hgTracks.

1. start here: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck>
2. make a props file which specifies the machine/db to check. If you are in a hurry, set zoomCount=0 and it will only check the default position for each assembly (much quicker!).
3. run it: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck> nohup TrackCheck hgw1.props > & hgw1.07-13-2006
4. grep the output for 'error'.

• Check Apache Error Log

hgw1:/usr/local/apache/logs/error_log

• Wait to hear from QA about how their CGIs look on hgw1. Each member of the QA team has testing responsiblities. Check also that TrackCheck ran successfully.

Push to the rest of the RR

• Ask for push from hgwbeta to hgw2-8 & MGC.

Final Build Wrap-up

This is day 12 in the schedule.

• zip the source code

hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> tcsh
hgwbeta> env
hgwbeta> doZip.csh  (this is automatically pushed to hgdownload)

• build and push hgcentral

hgwdev> ./buildHgCentralSql.csh
hgwdev> ./buildHgCentralSql.csh real

• • request push of hgcentral to hgdownload

 Please push from hgwdev --> hgdownload
 /usr/local/apache/htdocs/admin/hgcentral.sql
 This includes settings for the mirror sites. 

• build 32bit build and push to hgdownload

hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> doBuildCgi32.csh

• • after this completes, check the directory for this file: 32bitUtils.ok
  • the push to hgdownload happens automatically to here:

hgdownload:/mirrordata/apache/cgi-bin-i386/

• zip liftOver utils on hgwbeta and scp them to hgdownload

hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> doHgDownloadUtils.csh

• send email to genome-mirror. Include link to latest source: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip.