Date filter not working

Elastic Stack Logstash
1

I have an xml file being ingested by logstash. The xml file has a timestamp element with a unix_ms format (e.g., 1458827492928). My date filter is not properly parsing this and setting the @timestamp variable. I created a field to verify the value that I was using for my match statement and I can see that the added field contains a valid and the expected pre-converted timestamp value. Here is my filter section in my config file.

filter {
xml {
source => "message"
target => "doc"
}

use the embedded Unix timestamp

date {
match => ["%{[doc][timestamp]}", "UNIX_MS"]
}
}

The timestamp is being set with a current date even though the timestamp is for 2016 date.
Is there anything I am missing?

2

Please show the result of a stdout { codec => rubydebug } output so we can see exactly what your events look like.

3

Here is a snippet of the output....

{
"path" => "/var/log/tdrs/scf/TDR.AES_1.SCF_1.ScfAgent_5.20160324.1351-UTC.xml",
"@timestamp" => 2017-03-23T03:41:31.841Z,
"@metadata" => {
"path" => "/var/log/tdrs/scf/TDR.AES_1.SCF_1.ScfAgent_5.20160324.1351-UTC.xml",
"host" => "leda-1"
},
"@version" => "1",
"host" => "leda-1",
"doc" => {
"corrID" => [
[0] "1458827492927_daad49a7-b67a-4dbd-8db4-fa4a1a3c386e"
],
"dn" => [
[0] "AES_1.SCF_1.ScfAgent_5.ServiceComponentNnsPXDC_105"
],
"OrgName" => [
[0] "000-SLP013"
],
"eventType" => [
[0] "beginTrans"
],
"originator" => [
[0] "AES_1.SCF_1.ScfAgent_5.ServiceComponentNnsPXDC_105"
],
"version" => [
[0] "1.0"
],
"spID" => [
[0] "admin"
],
"AppName" => [
[0] "SLP013-APP003"
],
"OrgId" => [
[0] "pv"
],
"flowDirection" => [
[0] "0"
],
"AppId" => [
[0] "a75"
],
"operationType" => [
[0] "PxDCService_GetCapabilities"
],
"seqNumber" => [
[0] "1"
],
"timestamp" => [
[0] "1458827492928"
]
},
"message" => "\n beginTrans\n PxDCService_GetCapabilities</
operationType>\n AES_1.SCF_1.ScfAgent_5.ServiceComponentNnsPXDC_105\n 1.0\n 1458827492928\n 1\n 1458827492927_daad49a7-b67a-4dbd-8db4-fa4a1a3
c386e\n AES_1.SCF_1.ScfAgent_5.ServiceComponentNnsPXDC_105\n 0</
flowDirection>\n pv\n 000-SLP013\n a75\n SLP013-A
PP003\n admin\n",
"type" => "scf_xml",
"tags" => [
[0] "multiline"
]
}

4

[doc][timestamp] is an array so if you want Logstash to parse its first element you need to reference it as [doc][timestamp][0].

Are you running a very old Logstash? IIRC reasonably recent ones don't make arrays out of values unless it's necessary.

5

I have installed logstash-5.2.2-1.noarch. Updated the config as follows:

input {
file {
path => "/var/log/tdrs/scf/*.xml"
sincedb_path => "/var/log/logstash/.sincedb"
type => "scf_xml"
start_position => "beginning"
codec => multiline {
pattern => "(^\s|</TdrType>)"
what => "previous"
}
}
}

filter {

strip the XML prolog and envelope to get the actual TDRs

if [message] =~ "<?xml" or [message] =~ "" or [message] =~ "</TDRS>" {
drop{}
}

xml {
source => "message"
target => "doc"
}

use the embedded Unix timestamp

date {
match => ["%{[doc][timestamp][0]}", "UNIX_MS"]
}
}

output {

elasticsearch {

hosts => ["localhost:9200"]

index => "ngeag-tdrs-%{+YYYY.MM.dd}"

}

stdout { codec => rubydebug { metadata => true } }
}

This config yields the same result.

[root@leda-1 logstash]# bin/logstash -f /etc/logstash/conf.d/*.conf | more
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
09:02:49.967 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>24, "pipeline.batch.size"=>125, "pi
peline.batch.delay"=>5, "pipeline.max_inflight"=>3000}
09:02:50.182 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
09:02:50.242 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9602}
{
"path" => "/var/log/tdrs/scf/TDR.AES_1.SCF_1.ScfAgent_5.20160324.1351-UTC.xml",
"@timestamp" => 2017-03-23T14:08:33.842Z,
"@metadata" => {
"path" => "/var/log/tdrs/scf/TDR.AES_1.SCF_1.ScfAgent_5.20160324.1351-UTC.xml",
"host" => "leda-1"
},
"@version" => "1",
"host" => "leda-1",
"doc" => {
"corrID" => [
[0] "1458827492723_ea5b9d3d-ee7a-4966-850c-0131a3d2a9f5"
],
"subscriberID" => [
[0] "16308111111"
],
"dn" => [
[0] "AES_1.SCF_1.ScfAgent_5.EnablerNnsMMS_81"
],
"OrgName" => [
[0] "000-SLP009"
],
"eventType" => [
[0] "finalTrans"
],
"originator" => [
[0] "AES_1.SCF_1.ScfAgent_5.EnablerNnsMMS_81"
],
"version" => [
[0] "1.0"
],
"AppName" => [
[0] "SLP009-APP003"
],
"OrgId" => [
[0] "pl"
],
"flowDirection" => [
[0] "1"
],
"AppId" => [
[0] "a4x"
],
"operationType" => [
[0] "MMSEnabler_NotifyMessageReception"
],
"seqNumber" => [
[0] "4"
],
"timestamp" => [
[0] "1458827492860"
]
},
"message" => "\n finalTrans\n MMSEnabler_NotifyMessageReception\n AES_
1.SCF_1.ScfAgent_5.EnablerNnsMMS_81\n 1.0\n 1458827492860\n 4\n <corrID

1458827492723_ea5b9d3d-ee7a-4966-850c-0131a3d2a9f5\n AES_1.SCF_1.ScfAgent_5.EnablerNnsMMS_81\n 1<
/flowDirection>\n pl\n 000-SLP009\n a4x\n SLP009-APP003\n 16308111111\n",
"type" => "scf_xml",
"tags" => [
[0] "multiline"
]
}

6

Yes.

Insert the following setting in your xml filter:

force_array => false

(then you can remove the trailing [0] from subsequent field references to the new fields stored under doc.)

and insert the following mutate filter between your xml filter and your date filter:

mutate {
  convert => {
    "doc[timestamp]" => "integer"
  }
}

Why? From the docs:

UNIX_MS - will parse int value

int (integer) value, not a string.

Example output:

"@timestamp" => 2016-03-24T13:51:32.860Z

That done: I don’t like the way the xml filter forces you to store content in a new first-level field (such as doc). Here’s a workaround, to be inserted after your xml filter:

# Copy XML content to first-level fields with all-lowercase names
ruby {
	code => '
		event.get("doc").each do |key, value|
			event.set(key.downcase, value)
		end
	'
}

If you use this workaround, remove the leading doc (or [doc]) qualifier from subsequent field references. And remove the doc field before output. (I set an xml target under @metadata.)

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.