ALMA SW Logs and ELK Which logs Three

ALMA SW Logs and ELK

Which logs? Three categories of log messages are being input into ELK 1. 'Online SW' logs 2. 'Offline SW' logs 3. Apache httpd access logs Each of these categories are stored in elasticsearch in a separate index: 1. Online 2. Offline 3. Apache

Online SW logs are the logs generated by the software that controls ALMA correlator, antennas and detectors and their data acquisition. Log example: "Source. Object": "CONTROL/DV 11/Mount. Controller", "Audience": "Operator", "origin": "APE 1", "Host": "dv 11 -abm", "Process": "CONTROL/DV 11/cpp. Container", "Routine": "virtual void Control: : Mount. Controller. Impl: : wait. Until. On. Source()", "Log. Level": "Debug", "Thread": "ORBTask", "Line": "611", "@timestamp": "2018 -10 -19 T 20: 56: 37. 947 Z", "text": "Antenna DV 11 has finished moving. Current position (Az, El) is (+045. 00. 213, +045. 00. 03. 255)", "File": "Mount. Controller. Impl. cpp”

Online SW logs (cont. ) JLOG Operational GUI container Component ACS Log Service ACS Logger Log to files TAR. GZ Component Log file repo

Online SW logs (cont. ) Log file repo Acs. Log. Producer Rabbit. MQ Log. Stash nodes Elasticsearch Cluster

Offline SW logs are the logs generated by web applications that are used to prepare observations or to handle the lifecycle of observing projects and their associated data. Log example: "logger. Fqcn": "org. apache. logging. log 4 j. jul. Api. Logger", "level": "INFO", "release": "production", "origin": "OFFLINE", "thread": "http-nio-8080 -exec-8", "source": { "file": "State. Engine. Impl. java", "method": "log. With. Detail", "line": 766, "class": "alma. lifecycle. stateengine. State. Engine. Impl” }, "message": "State. Engine. Impl. change. State() for OUSStatus. Entity. T: uid: //A 001/X 1284/Xc 9 bn Called with params [target. State: Processing][target. State. Flag: Pipeline. Cal. And. Img][subsystem: obops][user. Id: devaky][state. Change. Comment: (PT) ][timestamp: 2018 -10 -19 T 21: 07: 32] ", "thread. Priority": 5, "type": "log", "thread. Id": 42, "@timestamp": "2018 -10 -19 T 21: 07: 32. 341 Z", "application": "protrack", "stage": "production", "location": "sco", "logger. Name": "alma. lifecycle. stateengine. State. Engine. Impl”

Offline SW logs (cont. ) OSF SCO Offline Web Application Log file repo File. Beat Log. Stash Rabbit. MQ Log. Stash nodes Elasticsearch Cluster

Apache httpd access logs These logs are the logs generated by the httpd server that stands in front of ALMASW web applications deployed at JAO. Log example: http: //kibana. alma. cl/app/kibana? #/doc/apache-*/apache 2018. 10. 19/logs/? id=AWa. OEZ 1 Dk. WGye. Ib. XDf 3 n Local. IP: 200. 2. 1. 25 Virtual. Host: cycle-6. asa. alma. cl Local. Port: 80 Time. Taken: 20766980 Remote. IP: 165. 124. 145. 149 Remote. Port: 43805 Timestamp: 19/Oct/2018: 20: 43: 01 +0000 First. Line. Of. Request: "POST /Obsprep. Submission. Service/Submission. Service? action=store HTTP/1. 1” Return. Code: 200 Bytes. Received: 588077 Bytes. Sent: 576566 Referrer: "-” User. Agent: "ALMA-OT/OFFLINE-2018 AUG- B” Balancer. Worker. Name: "http: //10. 200. 67. 83: 82"

Apache httpd access logs (cont. ) OSF SCO Apache HTTP Server File. Beat Log. Stash Rabbit. MQ Log. Stash nodes Elasticsearch Cluster

ELK Cluster Elasticsearch Cluster at OSF elk-master 2 kibana elk-node 01 elk-node 02 elastic 1 elastic 2 elastic 3 elastic 4 elk-master

How do we use Kibana?

How do we use Kibana? (cont. )

Important Definitions ● Retention policy (aka logrotation) ● Disaster recovery procedure ● ELK stack update policy/procedure

Future plans ● Online SW logs to be ingested in -almost- real time into ELK ● Docker daemon logs to be ingested into ELK ● Application logs used inside STEs (quicklook, acsdaemons, etc) to be ingested into ELK ● Update ELK to 6. x