The Japanese Virtual Observatory JVO Yuji Shirasaki National
The Japanese Virtual Observatory (JVO) Yuji Shirasaki National Astronomical Observatory of Japan
Goals of the JVO project n Provides a seamless access to the distributed astronomical data (catalogs, archives) NAOJ, JAXA, CDS, ESA, ESO, HESARC, SDSS, STSCI, NED, CADC… n Data discovery, Database federation Data Grid n n Provides analysis environment under which multi-wavelength study can be easily conducted An analysis tool is dedicated for each instrument n A homogeneous interface to the software n Easy access to the software n Server side data analysis Computing Grid n
History of JVO and Grid n FY 2002: Prototype 1 n n FY 2003: Prototype 2 n n Globus TK 3. 0 FY 2004: Prototype 3 n n Globus TK 2. 4 Web Service FY 2005: Operation Model Web Service n MDS (Monitoring and Discovery Service) Grid Computing (Subaru Data reduction, Hyper. Z, …) n
Experience of GT 2. 4 (Proto 1) n All the data and analysis service are implemented based on GT 2. 4 n GRAM, Grid. FTP, GSI, Simple CA UDDI is used to resolve the service URL n Too hard to learn the usage of this TK. n Slow Job submission (20 sec/submission) n n Needed adjustment of Poling time UDDI has no functionality to store various kind of metadata n The most of the component was replaced by WS based component in GT 3… n
User Simple. CA Analysis Servers Auth. ser es u eq t R U GRAM HTTP Grid FTP GSI Web Service Se Di rvic sco e ver y Job Submit UDDI Portal User Storage Data Trans. Data Service
Experience of GT 3. 0 (Proto 2) n All the data service and analysis service are implemented based on GT 3. 0 n Grid Service, RFTP, GSI-SFS (by Osaka univ. ), Simple. CA Relatively easy but still too hard… n Performance is good (because of no authentication is made for using Grid Service ? ) n Merger of Grid (OGSA) and Web Service is promising, but now it is rapidly evolving wait until the toolkit become stable ? n
User Simple. CA Analysis Servers Auth. ser es u eq t R U Grid Service HTTP RFT LDAP Se Di rvic sco e ver y Invoke Registry (XML DB) Portal Data Trans. User Storage Data Sevice
Database Federation with the IVOA standard (Proto 3 ~ Operational Model) First implementation of the IVOA standards n All the data services are implemented based on Web Service or simply an HTTP get method. n MDS for Grid computing n Workflow Language and Workflow Engine. n There are plenty of information about WS implementation over the Internet. development was rather easy and efficient. n Succeeded in the interoperability with the major VO projects. n
User 解析 解析 解析 Analysis Servers LDAP t Auth. In vo Web Service HTTP FTP LDAP P. Reg. Se Di rvic sco e ver y MDS ke S. Reg. ta a d g eta estin M rv Ha er s U es u eq R Invoke Portal User Storage In v Sky. Node 1. 0 ok e SIAP/SSAP Other VOs P. Reg. Sky. Node 0. 7. 4
Grid Computing with MDS n Parallel Data Analysis n Data reduction of Subaru Suprime-Cam n 250, 000 Frames (~5 TB) n n Analysis service for JVO users MDS server is a registry of computing resources CPU type, Memory size, Disk size, … n Load average, number of submitted job, … n Job status listener for asynchronous job n resolves the server location for a job execution considering the load balancing. n
Workflow n Workflow enables to build a higher level service utilizing the existing services. Automate a typical procedure which includes data access, data retrieval, data analysis. n Repeat the same procedure by changing parameters’ values. n Same as a shell script to use the data on local machine. n
JVO Workflow Language is defined based on the BPEL 4 WS. n Currently only a small part of BPEL 4 WS is implemented, and some extension is made for realizing our purposes. n Supported tags: n n <Workflow>, <variables>, <variable>, <sequence>, <flow>, <invoke>, <command>, <script>, <set>, <for>, <parfor>, <while>, <if>, <switch> …
Application to Science Use Case n Flow of the QSO-Galaxy clustering study 1. 2. 3. 4. 5. Get coordinates of QSOs from a catalog service Get images around the QSOs from a image service Measure the coordinate and color of sources on the image Estimate the photometric redshift (distance) of the sources. Cacluate a cross-correlation between the QSO and garaxies.
<sequence> … <!-- QSO Catalog and Image DB XMatch Query --> <command xsi: type="builtin" name="execute. Query"> <input> <var. Ref>jvoql</var. Ref> </input> <output> <var. Ref>votable. For. Image</var. Ref> </output> </command> … <!-- Source Detection --> <for items="list. Of. URL"> <invoke identifier=" ivo: //jvo/tools/sextractor " protocol="" namespace="" url="http: //jvoc. dc. nao. ac. jp: 8080/tools/services/SExtractor? wsdl" operation=" perform. For. URL"> <input> <var. Ref>list. Of. URL[_count]</var. Ref> </input> <output> <var. Ref>catalog. DH</var. Ref> </output> </invoke> </for> … <!-- Hyper. Z --> <parfor items="list. Of. Object. Name"> <command xsi: type="builtin" name="execute. Hyper. Z"> <input> <var. Ref>list. Of. Catalog. Name[_count ]</var. Ref> <var. Ref>list. Of. Object. Name[_count]</var. Ref> </input> </command> </parfor> …
SEQUENCE jvoql = QSOStudy. create. JVOQL(qso. Table, ra, dec, radi, image. Table, image. Size, mag. Limit) votable. For. Image = execute. Query(jvoql) store. VOTable(votable. For. Image[0], result_file 1) list. Of. URL = QSOStudy. get. List. Of. URL(votable. For. Image[0]) list. Of. File. Name = QSOStudy. get. List. Of. File. Name(votable. For. Image[0]) list. Of. Object. Name = QSOStudy. get. List. Of. Object. Name(votable. For. Image[0]) list. Of. Catalog. Name = QSOStudy. get. List. Of. Catalog. Name. For. Object(list. Of. Object. Name) FOR list. Of. URL INVOKE ivo: //jvo/tools/sextractor catalog. DH = perform. For. URL(list. Of. URL[_count]) store. VOTable(catalog. DH, list. Of. File. Name[_count]) END wdir = get. Current. Work. Dir() FOR list. Of. Object. Name num. Bands = QSOStudy. prepare. For. Hyper. Z(wdir, list. Of. Object. Name[_count], list. Of. File. Name, list. Of. Catalog. Name[_count]) END PARFOR list. Of. Object. Name execute. Hyper. Z(list. Of. Catalog. Name[_count], list. Of. Object. Name[_count]) END
Future Plan n Public release at the end of this year n Increase the functionality of the workflow system n Data visualization, data analysis n Virtual file system. n Single-sign on n Globus 4 ?
- Slides: 16