How to match Elasticsearch timestamp to windows format?

Enigmatr · October 9, 2020, 12:32am

I'm currently trying to parse some logs using Elasticsearch and displaying them on Kibana. I have tried to make sure that the timestamp of the filter in the conf.d file matches the format that is displayed in the log file, however, Kibana is displaying the ingest time instead of the log timestamp:

This is how I have configured the timestamp in my conf.d file:

    input {
      file {
             path => "/home/centos/ELK/Sep2020/DMA4a/Logs/SLDMS.txt"

        start_position => "beginning"
        sincedb_path => "/dev/null"
      }
    }
    filter {
        date {
           timezone => "UTC"
           match => ["timestamp", "yyyy\/MM\/dd HH:mm:ss.SSS"]
           target => "@timestamp"
        }
    }
    output {
      elasticsearch {
        hosts => ["localhost:9200"]
        index => "dma4a"
      }
    }

And here is an example of how my logs display the timestamp:

'version:1 message:2020/09/04 06:15:33.050|SLDataMiner.exe'

For some reason Kibana displays the ingest time instead of the timestamp from the logs, any suggestions please? In this case the ingest time is Sep 24, 2020 @ 08:51:08.260 and the timestamp I would ideally want displayed instead of the ingest time would be 2020/09/04 06:18:23.851.

Thanks in advance!

gmmorris · October 9, 2020, 9:40am

Hi Enigmatr,
This might be an ingestion issue (prior to being loaded into Kibana) but it might also be a display issue within Kibana itself.

To help identify where the issue lies, could you share the actual document that you're trying to display in Kibana? I'd like to verify that the timestamp field you're looking for is actually in the document that's ingested into Kibana.
If it is in fact there then it should be a matter of choosing the right field.

Enigmatr · October 11, 2020, 10:33pm

Hi Gidi,

Thanks for your prompt response. Here is the document I'm trying to display in Kibana:

2020/09/04 06:15:21.191|SLDMS.exe|4772|CDMAData::Init|DBG|4|Remote DMA 10.176.144.20 (EMDMA0204014PR) is running 9.6.0.0
2020/09/04 06:15:21.191|SLDMS.exe|4772|CSystem::InitializeDMA|DBG|1|Initialize DMA 10.176.144.20(EMDMA0204014PR)
2020/09/04 06:15:21.191|SLDMS.exe|4772|CSystem::InitializeDMA|INF|0|Addresses to check: 10.176.144.18;10.176.144.20;10.176.144.15;10.178.144.18;10.176.144.21;10.178.144.22;10.178.144.15
2020/09/04 06:15:21.192|SLDMS.exe 9.6.1829.3106|4080|23732|CRequest::Init|DBG|0|** Initializing SLNetCom succeeded
2020/09/04 06:15:21.192|SLDMS.exe 9.6.1829.3106|4080|23732|CRequest::AddLocalModule|DBG|0|-- Added local pointer for SNMP Manager v3
2020/09/04 06:15:21.192|SLDMS.exe 9.6.1829.3106|4080|23732|CRequest::Init|DBG|0|** Initializing SLNetCom
2020/09/04 06:15:21.198|SLDMS.exe 9.6.1829.3106|4080|23732|CRequest::Init|DBG|0|** Initializing SLNetCom succeeded
2020/09/04 06:15:21.200|SLDMS.exe|4772|CDMAData::GetInfo|INF|0|** EMDMA0204014PR - Cleaning old info
2020/09/04 06:15:21.200|SLDMS.exe|4772|CDMAData::GetInfo|INF|0|** EMDMA0204014PR - Getting element/service/redundancy/scheduler info
2020/09/04 06:15:21.209|SLDMS.exe|4772|CDMAData::LoadElementsFromVar|INF|0|** EMDMA0204014PR - Loading 0 elements
2020/09/04 06:15:21.209|SLDMS.exe|4772|CDMAData::LoadRedundancyGroupsFromVar|INF|0|** EMDMA0204014PR - Loading 0 redundancy groups
2020/09/04 06:15:21.210|SLDMS.exe|4772|CDMAData::LoadServicesFromVar|INF|0|** EMDMA0204014PR - Loading 0 services
2020/09/04 06:15:21.210|SLDMS.exe|4772|CDMAData::GetInfo|INF|0|** EMDMA0204014PR - Loading Scheduler info
2020/09/04 06:15:21.211|SLDMS.exe|4772|CSystem::SetDMALoaded|DBG|5|** EMDMA0204014PR (42012) - set as loaded.
2020/09/04 06:15:21.213|SLDMS.exe|23668|CSystem::NotifyAgentThreadFunc|CRU|0|** NotifyThread for 10.176.144.20 (42012) started
2020/09/04 06:15:21.984|SLDMS.exe|29184|CDMS::NotifyFunc|DBG|4|Retrieving status of double check routine: 0
2020/09/04 06:15:24.994|SLDMS.exe|4800|CDMS::NotifyFunc|DBG|4|Retrieving status of double check routine: 0
2020/09/04 06:15:35.478|SLDMS.exe|4800|CDMS::NotifyFunc|DBG|5|Received version for remote DMA 10.176.144.20 => 9.6.0.0
2020/09/04 06:17:31.503|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/97 : Host 42012
2020/09/04 06:17:31.507|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/94 : Host 42012
2020/09/04 06:17:31.511|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/91 : Host 42012
2020/09/04 06:17:31.516|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/85 : Host 42012
2020/09/04 06:17:31.520|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/76 : Host 42012
2020/09/04 06:17:31.523|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/73 : Host 42012
2020/09/04 06:17:31.528|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/70 : Host 42012
2020/09/04 06:17:31.533|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/67 : Host 42012
2020/09/04 06:17:31.537|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/58 : Host 42012
2020/09/04 06:17:31.541|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/55 : Host 42012
2020/09/04 06:17:31.545|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/52 : Host 42012
2020/09/04 06:17:31.549|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/49 : Host 42012
2020/09/04 06:17:31.554|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/46 : Host 42012
2020/09/04 06:17:31.559|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/43 : Host 42012
2020/09/04 06:17:31.562|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/40 : Host 42012
2020/09/04 06:17:31.572|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/37 : Host 42012
2020/09/04 06:17:31.577|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/34 : Host 42012
2020/09/04 06:17:31.581|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/31 : Host 42012
2020/09/04 06:17:31.585|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/28 : Host 42012
2020/09/04 06:17:31.588|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/22 : Host 42012
2020/09/04 06:17:31.592|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/19 : Host 42012
2020/09/04 06:17:31.596|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/16 : Host 42012
2020/09/04 06:17:31.600|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/10 : Host 42012
2020/09/04 06:17:31.604|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/7 : Host 42012
2020/09/04 06:17:31.608|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/4 : Host 42012
2020/09/04 06:17:31.612|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/95 : Host 42012
2020/09/04 06:17:31.616|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/92 : Host 42012
2020/09/04 06:17:31.620|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/68 : Host 42012
2020/09/04 06:17:31.624|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/62 : Host 42012
2020/09/04 06:17:31.628|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/59 : Host 42012
2020/09/04 06:17:31.632|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/56 : Host 42012
2020/09/04 06:17:31.636|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/53 : Host 42012
2020/09/04 06:17:31.640|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/50 : Host 42012
2020/09/04 06:17:31.644|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/47 : Host 42012
2020/09/04 06:17:31.649|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/44 : Host 42012
2020/09/04 06:17:31.652|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/41 : Host 42012
2020/09/04 06:17:31.662|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/38 : Host 42012
2020/09/04 06:17:31.666|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/35 : Host 42012
2020/09/04 06:17:31.670|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/32 : Host 42012
2020/09/04 06:17:31.675|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/29 : Host 42012
2020/09/04 06:17:31.679|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/23 : Host 42012
2020/09/04 06:17:31.683|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/20 : Host 42012
2020/09/04 06:17:31.687|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/17 : Host 42012
2020/09/04 06:17:31.691|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/14 : Host 42012
2020/09/04 06:17:31.695|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/11 : Host 42012
2020/09/04 06:17:31.699|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/8 : Host 42012
2020/09/04 06:17:31.703|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/5 : Host 42012
2020/09/04 06:17:31.707|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/2 : Host 42012
2020/09/04 06:17:31.711|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/96 : Host 42012
2020/09/04 06:17:31.715|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/90 : Host 42012
2020/09/04 06:17:31.719|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/72 : Host 42012
2020/09/04 06:17:31.722|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/69 : Host 42012
2020/09/04 06:17:31.726|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/66 : Host 42012
2020/09/04 06:17:31.731|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/63 : Host 42012
2020/09/04 06:17:31.735|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/60 : Host 42012
2020/09/04 06:17:31.739|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/57 : Host 42012
2020/09/04 06:17:31.742|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/54 : Host 42012
2020/09/04 06:17:31.747|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/51 : Host 42012
2020/09/04 06:17:31.751|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/48 : Host 42012
2020/09/04 06:17:31.755|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/42 : Host 42012
2020/09/04 06:17:31.759|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/39 : Host 42012
2020/09/04 06:17:31.763|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/36 : Host 42012
2020/09/04 06:17:31.767|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/33 : Host 42012
2020/09/04 06:17:31.771|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/30 : Host 42012
2020/09/04 06:17:31.775|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/24 : Host 42012
2020/09/04 06:17:31.779|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/21 : Host 42012
2020/09/04 06:17:31.782|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/18 : Host 42012
2020/09/04 06:17:31.792|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/15 : Host 42012
2020/09/04 06:17:31.796|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/9 : Host 42012
2020/09/04 06:17:31.801|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/6 : Host 42012
2020/09/04 06:17:31.805|SLDMS.exe|10788|CSystem::UpdateHostAgentCache|CRU|-1|Updating LocalAgentHostCache with key 42012/3 : Host 42012
2020/09/04 06:18:23.846|SLDMS.exe 9.6.1829.3106|4080|23668|CRequest::Request|ERR|0|Local Request for -DMS- on -VT_BSTR : 10.176.144.19- failed. Failed. (hr = 0x8004022D)
	Type 0/55/1
	VALUE 1: VT_BSTR : C:\Skyline DataMiner\Security\Ownership\42009_156_Components.xml
2020/09/04 06:18:23.847|SLDMS.exe|23668|CSystem::FileCompare|ERR|-1|Filecompare (C:\Skyline DataMiner\Security\Ownership\42009_156_Components.xml) failed to receive CRC from 10.176.144.19: 0x8004022dh Failed.
2020/09/04 06:18:23.851|SLDMS.exe|23668|CSystem::Notify()|ERR|-1|Error during synchronization of C:\Skyline DataMiner\Security\Ownership\42009_156_Components.xml from 10.176.144.20. Unspecified error (hr = 0x80004005)
2020/09/04 06:18:23.851|SLDMS.exe|23668|CSystem::Notify|ERR|0|Synchronize file failed. - Unspecified error (hr = 0x80004005)
**********

Thanks a lot!

gmmorris · October 12, 2020, 9:35am

Interesting, how are you ingesting these?

I suspect Logstash isn't managing to parse that timestamp (I'll ping someone over there to take a look).

That said, it might still be a Kibana issue - could you show us the index pattern you're using? I'm especially interested in looking at the primary time field.

Andrea_Selva · October 12, 2020, 10:12am

Hi Enigmatr,
your configuration has two issues. The first is that to parse the date, you have to isolate upfront the date parsing, and for this you could use the dissect filter for example. I saw that your log has a pretty static format, 7 columns, you could also use CSV filter with | char separator.
The other problem is the date format, you don't need to escape / character like you would do in Java code.

You could try with this sample pipeline:

input {
  file {
    path => "/home/centos/ELK/Sep2020/DMA4a/Logs/SLDMS.txt"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
    dissect {
      mapping => {
        "message" => "%{log_timestamp}|%{proc_name}|%{pid}|%{args}|%{log_level}|%{i_don_know}|%{message}"
      }
    }
    date {
       timezone => "UTC"
       match => ["log_timestamp", "yyyy/MM/dd HH:mm:ss.SSS"]
    }
}
output {
  stdout {
    codec => rubydebug
  }
}

Enigmatr · October 13, 2020, 9:28am

Hi,

Thanks for your reply. I tried using the sample pipeline, however once I run the logstash, I get this warning that the pattern is not found:

[2020-10-13T04:49:55,013][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-10-13T04:49:57,712][WARN ][org.logstash.dissect.Dissector][10sep20_DMA4aTest] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{log_timestamp}|%{proc_name}|%{pid}|%{args}|%{log_level}|%{i_don_know}|%{message}", "event"=>{"@timestamp"=>2020-10-13T08:49:55.550Z, "path"=>"/home/centos/ELK/Sep2020/DMA4a/Logs/Skyline DataMiner/Logging/SLDMS.txt", "@version"=>"1", "tags"=>["_dissectfailure"], "message"=>"Cleaned Stack !!! \r", "host"=>"ELK.arrissi.local"}}

Thanks in advance!

system · November 10, 2020, 9:28am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.