Telegraph A Universal System for Information Telegraph History

  • Slides: 8
Download presentation
Telegraph: A Universal System for Information

Telegraph: A Universal System for Information

Telegraph History & Plans • Initial Vision – Carey, Hellerstein, Stonebraker – “Regres”, “B-1”

Telegraph History & Plans • Initial Vision – Carey, Hellerstein, Stonebraker – “Regres”, “B-1” • Sweat, ideas and further vision – – 4 of my grads committed Brewer + 2 grads committed Franklin will play obvious tie-ins with other projects

Telegraph Architecture Query/Browse/Mine & synergies! Control, Dig. Lib Mariposa, Millenium, Control Global Agoric Federation

Telegraph Architecture Query/Browse/Mine & synergies! Control, Dig. Lib Mariposa, Millenium, Control Global Agoric Federation Continuously Reoptimizing Query Processor Adaptive Data Placement Storage Manager (FS, DB, Web) Ninja, Gi. ST, IStore River, Ninja, Aetherstore, Control, STIX

Storage Manager • Historic chance to start over! – new hardware realities • variable-length

Storage Manager • Historic chance to start over! – new hardware realities • variable-length segments, not blocks • big main memories • extra CPUs at the devices (IStore) – revisit and clean up infrastructure for transactions • clean API supporting both log-based & version-based schemes; version-based runs today! • big SW Eng. challenge – unify DB/FS/Web server! • Clients: Ninja’s persistent hash table, query processing, web server, Linux (NT? ) filesystem. – (Mohan Lakhamraju, Rob von Behren, Steve Gribble )

Query Engine • Shared-nothing (cluster) – all data flow (no blocking ops) • •

Query Engine • Shared-nothing (cluster) – all data flow (no blocking ops) • • auto load-balance to micro/macro changes in environment adaptivity more important than raw performance!! CONTROL! || ripple join, online reordering (Shankar Raman) – continuously reoptimizing query plans • tie-ins with STIX (Christos/Sinclair/Russell/Hellerstein) • (Ron Avnur) – first steps in handling streaming sources

Cluster Data Layout – issues: fragmentation, placement, replication on 10^6 disks. For DB/FS/Web. –

Cluster Data Layout – issues: fragmentation, placement, replication on 10^6 disks. For DB/FS/Web. – goals: availability, efficiency, consistency, manageability. – Adaptivity: cooperative vs. competitive ($$) techniques? – (Mehul Shah)

Global Federation • Global distribution – federated DBMS layer a la Mariposa/Cohera • address

Global Federation • Global distribution – federated DBMS layer a la Mariposa/Cohera • address all the hard stuff they dropped! – Global data placement • as in cluster, but must be competitive. (Mehul Shah) – Global query processing (Amol Deshpande) • Agoric query optimization • distributed query processing – Global metadata • yellow pages both for services & datasets • Millenium/Ninja tie-ins?

Applications • Really finding stuff in all the world’s data? – UI meets AI

Applications • Really finding stuff in all the world’s data? – UI meets AI meets Logic (browse/mine/query) • • CONTROL is key: seamless, non-blocking interaction multi-res output and feedback during browse/query hints, wizards, training (AI mining, user in the loop) build on existing “scalable spreadsheet”/xform tools (Shankar Raman)