Splunk Y 2 K all over again presented

Splunk „Y 2 K“ all over again presented by Mika Borner und Tom Vogt October 3, 2020 1

Splunk “(y 2 k)” Timestamp Issue Description • Beginning on January 1, 2020, un-patched Splunk platform instances will be unable to recognize timestamps from events where the date contains a two-digit year. • Beginning on September 13, 2020 at 12: 26: 39 PM Coordinated Universal Time (UTC), un-patched Splunk platform instances will be unable to recognize timestamps from events with dates that are based on Unix time. • Check the official Documentation! • https: //docs. splunk. com/Documentation/Splunk/latest/Release. No tes/Fix. Datetimexml 2020 October 3, 2020 2

Captain, we have a parsing problem! October 3, 2020 3

Effected Splunk Enterprise Instances • Splunk Cloud • Splunk Light • Splunk Enterprise Indexers, clustered or not • Heavy forwarders • Search heads, clustered or not • Search head deployers • Deployment servers • Cluster masters • License masters October 3, 2020 4

But there is more! October 3, 2020 5

Facepalm with a few thousand people October 3, 2020 6

Effected Splunk Universal Forwarders! • Splunk universal forwarders, under the following conditions: • When they have been configured to process structured data such as CSV files, using the INDEXED_EXTRACTIONS setting in props. conf • When they have been configured to process data locally, using the force_local_processing setting in props. conf • BTW don’t do that! • When they have been configured with a monitor input, and that input subsequently encounters an unknown file type October 3, 2020 7

Find Forwarders doing indexed extractions (may not be 100% correct) index=_internal earliest=-48 h@h group=tcpin_connections sourcetype=splunkd component=Metrics [search index=_internal earliest=-48 h@h sourcetype=splunkd component=Metrics name=structuredparsing (processor=linebreaker OR processor=aggregator OR processor=regexreplacement) | stats count by host | fields - count | rename host AS hostname] | stats avg(tcp_KBps) AS avg_kbps first(version) AS version first(os) AS os first(fwd. Type) AS fwd_type first(guid) AS guid by hostname | eval avg_kbps=round(avg_kbps, 2) | sort - avg_kbps October 3, 2020 8

October 3, 2020 9

Splunk “(y 2 k)” Timestamp Issue • Insufficient regex in datetime. xml leads to: • Incorrect timestamping of incoming data • Incorrect rollover of data buckets due to the incorrect timestamping • Incorrect retention of data overall • Incorrect search results due to data ingested with incorrect timestamps October 3, 2020 10

Splunk “(y 2 k)” Timestamp Issue Details Wrong Regexes • <text><![CDATA[(20dd|19dd|[901]d (? !d))]]></text> Correct Regexes • <text><![CDATA[(? : ^|source: : ). *? (? <!d| d. |)(? : 20)? ([901]d)(0d|1[012])([012]d |3[01])(? !d|-| {2, })]]></text> • <text><![CDATA[(? : ^|source: : ). *? (? <!d|d. |)(? : 20)? ([9012]d)(0d|1[012])([012]d|3[01])(? !d|-| {2, })]]></text> 2]d(? !d))]]></text> • <text><![CDATA[(? : ^|source: : ). *? (? <!d| d. )(0d|1[012])([012]d|3[01])(? : 20)? ([9 01]d)(? !d| {2, })]]></text> • <text><![CDATA[(? : ^|source: : ). *? (? <!d|d. )(0 d|1[012])([012]d|3[01])(? : 20)? ([9012]d)(? !d| {2, })]]></text> • <text><![CDATA[((? <=^|[s#, "=([|{])(? : 1[012345]|9)d{8}|^@[da-f. AF]{16, 24})(? : . ? (d{1, 6}))? (? ![d(])]]></te xt> • <text><![CDATA[((? <=^|[s#, "=([|{])(? : 1[012 3456]|9)d{8}|^@[da-f. AF]{16, 24})(? : . ? (d{1, 6}))? (? ![d(])]]></text> October 3, 2020 • <text><![CDATA[(20dd|19dd|[9012]d( ? !d))]]></text> 11

Splunk “(y 2 k)” Timestamp Issue Details • Why does TIME_PREFIX (=>best practices) not solve all problems? • Because of weird logs! datetime. xml is a fallback for timestamps not matching TIME_PREFIX. October 3, 2020 12

ATTENTION There is no method to correct the timestamps after the Splunk platform has ingested the data when the problem starts. October 3, 2020 13

October 3, 2020 14

Solution 1 • Download the latest datetimestamp xml from splunk. com • http: //download. splunk. com/products/ingest 2020/datetime. zip. • Copy the unarchived updated datetime. xml from the location where you downloaded it to the $SPLUNK_HOME/etc directory on the Splunk platform instance. Ensure that the updated file overwrites the existing file. • Confirm that the new datetime. xml has been written to the $SPLUNK_HOME/etc directory. • Restart the Splunk platform. Your Splunk platform instance is now patched. October 3, 2020 15

Solution 2 • Update to the latest Splunk Version with the fix included as soon as it coming available October 3, 2020 16

Solution for UF with Deployment Server • Download an App https: //drive. google. com/file/d/17 WR 7 m. Bqyh. DLEOw QAipfp-MW 8 a. YWVT 5 w. Z/view • Settings for props. conf in this app: [default] DATETIME_CONFIG = /etc/apps/<appname>/local/datetime. xml • Deploy it to all UF with the deployment server October 3, 2020 17

What about the file integration check? • Splunk uses a manifest file for file integration check • If you replace the datetime. xml, file integration error messages will pop up in the Web UI • You can prevent this with a change in limits. conf: [system_checks] installed_files_integrity = log_only • Works on Windows and Linux October 3, 2020 18

What about the file integration check? • Or you can replace the manifest file (linux only): manifestfile=`ls $SPLUNK_HOME | grep manifest | tail -1` mv ${manifestfile}. backup for oldhash in `grep datetime. xml ${manifestfile}. backup | awk -F " " '{print $6}'` do newhash=`sha 256 sum $SPLUNK_HOME/etc/datetime. xml | awk -F " " '{print $1}'` cat ${manifestfile}. backup | sed s/`echo ${oldhash}`/`echo ${newhash}`/g > ${manifestfile} done October 3, 2020 19

Questions? October 3, 2020 20

October 3, 2020 21