Japanese Virtual Observatory Masatoshi Ohishi National Astronomical Observatory
Japanese Virtual Observatory Masatoshi Ohishi National Astronomical Observatory of Japan and Sokendai masatoshi. ohishi@nao. ac. jp 2021/2/26 高エネルギー物理データグリッド研究会
Collaborators NAOJ • Y. Mizumoto • N. Yasuda • Y. Shirasaki • M. Tanaka Ochanomizu U. • Y. Masunaga 2021/2/26 Fujitsu Ltd. • K. Monzen • K. Kawarai • Y. Ishihara • H. Yanaka • Y. Yamaguchi • M. Ishida • N. Saito 高エネルギー物理データグリッド研究会
SUBARU telescope in Hawaii (optical & IR) 2021/2/26 高エネルギー物理データグリッド研究会
45 m Radio Telescope at Nobeyama 2021/2/26 高エネルギー物理データグリッド研究会
Time and Spectral Dimensions The Multiwavelength Crab Nebulae Crab star 1053 AD X-ray, optical, infrared, and radio views of the nearby Crab Nebula, which is now in a state of chaotic expansion after a supernova explosion first sighted in 1054 A. D. by Chinese Astronomers. 2021/2/26 高エネルギー物理データグリッド研 究会 Slide courtesy of Robert Brunner @ Cal. Tech.
Unknowns of Astronomy • Physics just after the Big Bang ? • How Galaxies were formed ? • Averaged density of the Universe etc. → statistical analyses for homogeneous survey data are crucial 2021/2/26 高エネルギー物理データグリッド研究会
Data Productivity • Nobeyama Radio :~1 TB/yr • Subaru@Hawaii :~20 TB/yr • ALMA(planned) :~PB/yr Flood of excellent data (survey data) Digitized & Archived 2021/2/26 高エネルギー物理データグリッド研究会
HPC + High Speed NETWORK +GRID Technology → it is possible to connect and seamlessly use remote DBs and computing facilities for data analyses → Virtual Observatory 2021/2/26 高エネルギー物理データグリッド研究会
JVO (will) consists of Seamless Access to Federated DBs u Various Data Analysis tools u Advanced Analysis such as Data Mining u Federation with Real Telescopes → crucial to control data quality in VO u 2021/2/26 高エネルギー物理データグリッド研 究会
JVO prototype : Ver. 1 Establish seamless access to federated DBs by using the Grid technology u Define JVO Query Language, and implement its parser u Test environment is in a closed subnet u 2021/2/26 高エネルギー物理データグリッド研 究会
Query request Web Broser サービス JVO Browser ASTRONOMER JVO Query Language 利用者独自 Catalog Query Service Commands callable from the GT 2 Parse JVO QL into each query sequence JVO Portal JVO Viewer skycat JVO Contoroller Catalog DB Data Archive Service Globus Toolokit V 2 Data Management DB UDDI Server (Free. Soft) Globus Toolkit Service Registry Security Service Resource Management Data Management Grid. FTP each query Resolve servers to request queries 2021/2/26 Data 高エネルギー物理データグリッド研 究会 Data Analysis Services
JVO Query Language(1) Catalog Query 2021/2/26 高エネルギー物理データグリッド研 究会
JVO Query Language(2) Image Query 2021/2/26 高エネルギー物理データグリッド研 究会
Redhat Linux JVO Prototype System Architecture GSDL UDDI Maintenance Tool JVO Client Apache Tomcat Netscape Communicator JVO Service (X match) Maintenance Tool UDDI 4 J soapuddi Apache SOAP ① Servlet Down. Load Globus Toolkit (Server) Java 2 SE 1. 4 Hypertext contents Redhat Linux JVO Service Gb Ether Solaris 8 (Possible at Linux) JVO Controller UDDI Registory / Servlet Engine 2 Mass Data. Base JVO Service (select) • register • update • delete JDBC SQL library Java 2 Plugin JVO Client (servlet) ③ Find Service ② Request Observation JVO Controller DB Service Java 2 SE 1. 4 Postgre. SQL (DBMS) UDDI 4 J ⑤ execute Service ⑤’ execute Service Image Data SUBARU Sup. Cam z-band UDDI Data • get & parse GSDL • pre condition check • execute Service • post condition check Internet ③ get GSDL Check status ④’ Get GSDL by Grid. FTP ⑥’ spawn Service Grid. FTP SQL library JVO Service (select) SQL library JVO Service (image) Image Data Free Software Commercial Software 2021/2/26 GSDL ⑥ spawn Service JVO Service (count) In House Software Function JVO Service (image) JVO Service (X match) Globus Toolkit (Server) Globus Toolkit GSDL Globus Toolkit (Server) JVO Service SUBARU Data. Base JVO Service ⑦ issue SQL GSDL (copy) ④ Get GSDL by Grid. FTP AVO 2 Mass DB Redhat Linux Globus Toolkit (Client) NVO Postgre. SQL JVO Server Monitoring Globus Toolkit DB Service Image Data (copy) JVO Service (X match) Image Data (copy) The following free software is included by globus toolkit. • Open. SSL (Secure Socket Library) • Open. LDAP (LDAP Server) • wu-ftpd (ftp server) 高エネルギー物理データグリッド研 究会 JVO Service (select) SQL library JVO Service (count) SQL library ⑦’ issue SQL Copy imate data by Grid. FTP Image Data (copy) Image Data DB Service ORACLE SUBARU Sup. Cam i-band
How it works ? Read JVO QL u Create “Work Flow”, and execute each step u When necessary, access to the registry to expand the step of the work flow u Thus the steps increases dynamically u 2021/2/26 高エネルギー物理データグリッド研 究会
Query Files can be stored in a local PC (WS) 2021/2/26 高エネルギー物理データグリッド研 究会
Monitor Window: Refreshed automatically in every 15 seconds 2021/2/26 高エネルギー物理データグリッド研 究会
FAQs on our Prototype Q) Contents of environment variables are not passed to the executing machines. A) Prepare a shell script to get environment variables and another script to execute the command. Then Grid. FTP these shell scripts , and execute the shells in the remote machine. Q) It is impossible to prepare all users’ account to all relevant machines. A) Map all accounts on a portal to a single representative one, such as globus. 2021/2/26 高エネルギー物理データグリッド研 究会
FAQs Q) GT 2 has too large overhead before starting actual jobs !! They are several seconds. Request) Overheads must be < 0. 1 seconds excluding the network delays. A) Use NMI (NSF Middleware Initiative) where each key generation is quite fast through assembler. 2021/2/26 高エネルギー物理データグリッド研 究会
Future u u Prototype version 2 (in progress) implement data analysis tools Interoperation with VOs in the world We have joined the International Virtual Observatory Alliance, and are contributing standardizing protocols for interoperability. 2021/2/26 高エネルギー物理データグリッド研 究会
VOs in the World • NVO (USA), AVO (EU), Astro. Grid (UK) • Germany, Canada, Russia, Australia • India, China, JAPAN Protocols to interconnect VOs → International Virtual Observatory Alliance (IVOA) 2021/2/26 高エネルギー物理データグリッド研 究会
- Slides: 33