Introduction to using the UCSC Genome browser Assembly
Introduction to using the UCSC Genome browser, Assembly Hubs and Table Browser intersections. IBM 029 keypunch BME 230 Winter 2016 12 January 2016 Hiram Clawson http: //genomewiki. ucsc. edu/index. php/User: Hiram Check for a most recent copy of this presentation at: http: //genomewiki. ucsc. edu/index. php/File: BME 230_Winter_2016. ppt
today’s topics: • • UCSC Genome Browser Track/Assembly Hubs Table Browser – example intersection not covered: C language programming with the ‘kent’ source tree
thank you Angie Hinrichs Matt Schwartz Terry Furey Chuck Sugnet Heather Trumbower Kayla Smith Brooke Rhead Ann Zweig Daryl Thomas Robert Baertsch Jakob Peterson Katherine Pollard Adam Siepel Rachel Harte Mark Diekhans Fan Hsu Bob Kuhn Andy Pohl Kate Rosenbloom Galt Barber Jorge Garcia Archana Thakkapallayil Ali Sultan-Qurraie Jennifer Jackson Krishna Roskin Andrew Kern Brian Raney Craig Lowe Yontao Lu Erich Weiler Ryan Weber Gary Moro Todd Lowe Josh Stewart Gill Bejerano Donna Karolchik Jim Kent David Haussler Jim Kent
1. genome browser • • • http: //genome. ucsc. edu/ (89 genomes) 14 primates 33 other mammals 18 other vertebrates 13 flies, mosquito, bee 6 nematodes 5 others, yeast, etc. see also: http: //genome-test. cse. ucsc. edu/ over 100 genomes at last count Just for fun: (search for “gateway” at genomewiki. ucsc. edu) http: //genomewiki. ucsc. edu/index. php/Genome_browser_photo_gateway
see also: • Human Genome Project History: http: //plato. stanford. edu/entries/human-genome/ • The race for the assembly http: //news. ucsc. edu/2015/06/genome-anniversary. html http: //www. ncbi. nlm. nih. gov/pmc/articles/PMC 311095/ • Genome browser You. Tube channel: http: //www. youtube. com/channel/UCQn. UJepy. NOw 0 p 8 s 2 ot. X 4 RYQ/ • Genome browser twitter: http: //twitter. com/genomebrowser/ • 49, 000 genome browsers: (PROTOTYPE !): http: //genome-preview. cse. ucsc. edu/gbdb/hubs/refseq/ • genome-source. cse. ucsc. edu/gitweb/? p=kent. git; a=blob; f=src/user. Apps/README
see also: • associated lectures: http: //genomewiki. ucsc. edu/index. php/Presentations • Open Helix training: genome. ucsc. edu/training. html • Galaxy analysis workflow management system: galaxy. psu. edu • “Powers of Ten” video by Charles & Ray Eames, 1977 http: //www. youtube. com/watch? v=0 f. KBhv. Djuy 0 • Genome browser documentation: genome. ucsc. edu/golden. Path/help/ • email help: genome at soe. ucsc. edu
UCSC Genome Browser • • A set of tools, more than just the browser: Genome Browser -> viewing annotations Blat Server -> find sequence Table Browser -> intersections, correlations Gene Sorter -> gene centric viewpoint In Silico PCR -> determine primer products Genome Graphs, Galaxy, Visi. Gene, etc … ~15, 000 users/day ~500, 000 pages/day ~1. 3 Tb/day browser statistics
http: //genome. ucsc. edu/ or bravely: http: //genome-test. cse. ucsc. edu/ default tracks display:
hide all, base position RTM ! chr location
hide all, base position How does one operate the genome browser ?
track options Title your slides
configure adjust sizes useful navigation options
1024 width, 12 pt font readjust to base due to font size change space enough to show bases again
UCSC gene track next gene Ensembl gene track next exon track controls
repeated click 10 X or search for ‘chr 21’ or click drag select Track visibilities set to ‘squish’ for this display navigation
zoom in on target position click drag to select region on ideogram left click in left label column to change track visibility
item click
a gene item details page best to use blue navigation bar
position/search
2. Assembly/Track hubs http: //genome. ucsc. edu/golden. Path/help/hg. Track. Hub. Help. html
My Hubs URL entry
Gateway page hub selection
External genome sequence display
Hub file relationships, e. g. hub. txt -> genomes. txt
3. Table Browser DEC PDP 8/S desktop computer ASR 33 teletype, paper tape punch reader/writer Lear-Siegler ADM 3 A terminal
design your experiment Task: find highly conserved non-coding regions 1. Select highly conserved regions from conservation track filter phast. Cons 100 way > 0. 1 2. Intersect with NOT exons 3. result
table browser
ensembl exons
gene tracks output options: exons output (the ‘name=‘ string should be unique from other custom tracks)
same settings, get introns
select output introns output
measure results
visualize results
conservation track
conservation track settings press ‘Submit’ to return to the browser 100 pixels Vertebrates only
larger graph, full visibilities use blue navigation menus to TB:
table browser filter experiment with small area first select greater than filter > 0. 1 size of answer set
create custom track
create intersections primary table
secondary table base pair AND function NOT exons
output intersection results NOT exons AND phast. Cons 100 way > 0. 1
and the answer is:
4. C language programming http: //genome-source. cse. ucsc. edu/gitweb/? p=kent. git; a=blob; f=src/user. Apps/README IBM System 360 Model 40, late 1960’s, 256 Kb, 10, 000 FLOPS SWARM cluster, UCSC 2008, 1024 CPU cores, Intel Xeon 2. 3 Ghz
Fetch and build kent source • • • GIT: http: //genome. ucsc. edu/admin/git. html - hourly updated ZIP file: http: //hgdownload. cse. ucsc. edu/admin/jksrc. zip - ~biweekly updated build instructions: http: //genome. ucsc. edu/admin/jk-install. html see also, in the source tree kent/src/product/README. * BME 230 Winter 2011 presentation (similar to this one) http: //genomewiki. ucsc. edu/index. php/File: BME 230_Winter_2011. ppt • Angie’s presentation of this material: http: //genomewiki. ucsc. edu/index. php/Image: Bejerano_Lab_2008_03_31. ppt • • Jim Kent source tree design discussions, several at: http: //genomewiki. ucsc. edu/index. php/Presentations BME 230 Winter 2008 presentation: http: //genomewiki. ucsc. edu/index. php/Image: Baertsch-code-talk. ppt The presentations mentioned at the genomewiki are complementary to this presentation. Please take a look at them.
source tree organization kent/src/ - top level - see README here kent/src/inc/*. h - API manual for common functions kent/src/utils/ - file manipulation utilities kent/src/lib/$MACHTYPE/ - built libraries kent/src/lib/ - builds lib jkweb. a kent/src/hg/ - database manipulation commands (mostly) kent/src/hg/lib/ - builds lib jkhgap. a kent/src/jk. Own. Lib/ - blat support – COPYRIGHT © J. Kent see also www. kentinformatics. com
using the database utilities 1. Create public My. SQL user/password access file: $ cd $HOME $ cat << ‘_EOF_’ >. hg. conf db. host=genome-mysql. cse. ucsc. edu db. user=genomep db. password=password ‘_EOF_’ $ chmod 600. hg. conf # will not work without this security 2. That file is used by kent src commands to access My. SQL: $ feature. Bits hg 19 gap 239845127 bases of 2897316137 (8. 278%) in intersection (try this same command with the argument -count. Gaps)
writing a new command Use ‘new. Prog’ to establish the skeleton of a new command: $ new. Prog - make a new C source skeleton. usage: new. Prog prog. Name description words This will make a directory 'prog. Name' and a file in it 'prog. Name. c' with a standard skeleton Options: -jkhgap - include jkhgap. a and mysql libraries as well as jkweb. a archives -cgi - create shell of a CGI script for web -cvs - obsolete option, needs to update to git commands (note: this works only within the source tree hierarchy)
command line arguments #include "common. h" // see also: src/inc/common. h "options. h" // see also: src/inc/options. h static long offset = 0; static char * chrom = NULL; static double min. Val = 0; static double max. Val = BIGNUM; static boolean do. Nothing = FALSE; static struct option. Specs[] = { {"offset", OPTION_LONG}, {"chrom", OPTION_STRING}, {"min. Val", OPTION_DOUBLE}, {"max. Val", OPTION_DOUBLE}, {"do. Nothing", OPTION_BOOLEAN}, {NULL, 0} };
command line argument processing #include "common. h" // see also: src/inc/common. h "options. h" // see also: src/inc/options. h int main( int argc, char *argv[] ) /* program demonstrates fetching options from the command line */ { … offset = option. Long("offset", 0); chrom = option. Val("chrom", NULL); min. Val = option. Double("min. Val", -1 * INFINITY); max. Val = option. Double("max. Val", INFINITY); do. Nothing = option. Exists(”do. Nothing"); … exit(0); }
line oriented file I/O #include "common. h" // see also: src/inc/common. h #include "linefile. h" // see also: src/inc/linefile. h void some. Function(char *file. Name) /* function demonstrates reading a file line by line */ { struct line. File *lf = line. File. Open(file. Name, TRUE); char * line = NULL; int size = 0; while (line. File. Next(lf, &line, &size)) { char *words[128]; int word. Count = chop. By. White(line, words, Array. Size(words)); } line. File. Close(&lf); }
structured file I/O #include "common. h" #include "maf. h" // see also: src/inc/common. h // see also: src/inc/maf. h void scan. Maf(char *maf. File) /* function demonstrates reading a MAF file, record by record */ { struct maf. File *mf = maf. Open(maf. File); struct maf. Ali *ali = NULL; while ((ali = maf. Next(mf)) != NULL) { int c. Count = sl. Count(ali->components); … processing maf alignment record … maf. Ali. Free(&ali); // need to free the maf record structure } // maf. Next() just happens to close the file on the last record } // see also: src/hg/mouse. Stuff/maf. Coverage/
reading DNA sequence files Example is from src/utils/find. Motif. c Include files would be, from src/inc/ common. h dnaseq. h dna. Load. h These functions read fasta files, nib files, or 2 bit files. DNA sequence is any of: ACGTacgt. Nn lower case is ‘masked’ static void find. Motif(char *input) /* find. Motif - find specified motif in sequence file. */ { struct dna. Load *dl = dna. Load. Open(input); struct dna. Seq *seq; while ((seq = dna. Load. Next(dl)) != NULL) { verbose(2, "#tprocessing: %sn", seq->name); DNA *dna = seq->dna; for (i=0; i < seq->size; ++i) { val = nt. Val[(int)dna[i]]; switch (val) { case T_BASE_VAL: case C_BASE_VAL: case A_BASE_VAL: case G_BASE_VAL: … etc … } } }
memory management #include "common. h" // see also: src/inc/common. h #include "memalloc. h" // see also: src/inc/memalloc. h #include "bed. h" // see also: src/hg/inc/bed. h void some. Function(int count) /* demonstrate use of need. Mem(), free. Mem(), Alloc. Var() */ { int *pile. Of. Ints = need. Mem(sizeof(int) * count); … use ‘em or lose ‘em … free. Mem(pile. Of. Ints); // or: freez(&pile. Of. Ints); struct bed *bed. Item; Alloc. Var(bed. Item); // macro defined in src/inc/common. h bed. Item->chrom = clone. String("chr 23"); … etc … }
linked lists #include "common. h" struct sl. List { struct sl. List *next; } // see also: src/inc/common. h int sl. Count(void *list) /* count elements in list. */ { struct sl. List *pt = (struct sl. List *)list; int len = 0; while (pt != NULL) // or more concisely: for ( ; pt = pt->next) { // ++len; len += 1; pt = pt->next; } return len; } HP 2100 A mini computer
sorting a linked list #include "common. h" // see also: src/inc/common. h #include ”bed. h" // see also: src/inc/bed. h int bed. Cmp(const void *va, const void *vb) /* Compare to sort based on chrom, chrom. Start. */ { const struct bed *a = *((struct bed **)va); const struct bed *b = *((struct bed **)vb); int dif; dif = strcmp(a->chrom, b->chrom); if (dif == 0) dif = a->chrom. Start - b->chrom. Start; return dif; } /* somewhere in code later */ struct bed *bed. List; // a linked list of bed items sl. Sort(&bed. List, bed. Cmp) // voilà, sorted IBM 082 card sorter
inheritance via C structures struct axt. Score. Scheme /* A scoring scheme or DNA alignment. */ { struct score. Matrix *next; int matrix[256]; /* Look up with letters. */ int gap. Open; /* Gap open cost. */ int gap. Extend; /* Gap extension. */ char *extra; /* extra parameters */ }; This structure “inherits” the methods of the sl. List structure Because the first element(s) in the structure are the same as in struct sl. List --> ‘*next’. Therefore, sl. Count() will work on this list too.
hashed storage structures #include "common. h" #include "hash. h" // see also: src/inc/common. h // see also: src/inc/hash. h struct hash. El *hash. Add(struct hash *hash, char *name, void *val); /* Add new element to hash table. If an item with name, already exists, a new * item is added in a LIFO manner. The last item added for a given name is * the one returned by the hash. Lookup functions. hash. Lookup. Next must be used * to find the preceding entries for a name. */ The given ‘name’ becomes the key to this item. The item to save in the hash is in the pointer ‘val’, which can be any type of structured item.
creating a hash #include "common. h" // see also: src/inc/common. h #include "hash. h" // see also: src/inc/hash. h #include "hg. Tracks. h" // see also: src/hg/hg. Tracks. h DEC PDP 11/40 struct maf. Item *some. Function(struct track *tg) /* create a hash of maf. Items from tg->items simple list */ { struct maf. Item *mi. List = tg->items, *mi; struct hash *mi. Hash = new. Hash(9); /* Make hash of items keyed by database. */ int i = 0; for (mi = mi. List; mi != NULL; mi = mi->next) { mi->ix = i++; if (mi->db != NULL) hash. Add(mi. Hash, mi->db, mi); } return mi. Hash; }
scanning through a hash #include "common. h" #include "hash. h" // see also: src/inc/common. h // see also: src/inc/hash. h void hash. Verify(struct hash *bed. Name. Hash) /* verify that bed element name is identical to hash name */ { struct hash. Cookie cookie; struct hash. El *el; cookie = hash. First(bed. Name. Hash); while ((el = hash. Next(&cookie)) != NULL) { struct bed *bed. El = (struct bed *)el->val; if (different. String(bed. El->name, el->name)) err. Abort("hash broken '%s' != ‘'%s'", bed. El->name, el->name); } }
dynamic strings #include "common. h" #include "hash. h" #include "dystring. h" // see also: src/inc/common. h // see also: src/inc/hash. h // see also: src/inc/dystring. h struct dy. String *concat. Names(struct hash *bed. Name. Hash) /* return comma delimited string of all bed element names */ { struct hash. Cookie cookie; struct hash. El *el; cookie = hash. First(bed. Name. Hash); struct dy. String *big. String = dy. String. New(256); while ((el = hash. Next(&cookie)) != NULL) { struct bed *bed. El = (struct bed *)el->val; dy. String. Printf(big. String, "%s, ", bed. El->name, ); } return big. String; }
thank you Angie Hinrichs Matt Schwartz Terry Furey Chuck Sugnet Heather Trumbower Kayla Smith Brooke Rhead Ann Zweig Daryl Thomas Robert Baertsch Jakob Peterson Katherine Pollard Adam Siepel Rachel Harte Mark Diekhans Fan Hsu Bob Kuhn Andy Pohl Kate Rosenbloom Galt Barber Jorge Garcia Archana Thakkapallayil Ali Sultan-Qurraie Jennifer Jackson Krishna Roskin Andrew Kern Brian Raney Craig Lowe Yontao Lu Erich Weiler Ryan Weber Gary Moro Todd Lowe Josh Stewart Gill Bejerano Donna Karolchik Jim Kent David Haussler Jim Kent
- Slides: 61