Hi,
I have problem that filebeat is logging incorrect time when parasing agains time of the day.
Example:
{
"pipeline": {
"version": 1,
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{ADHOC_TIME:adhoc.timestamp}%{SPACE}%{LOGLEVEL:adhoc.level}%{SPACE}%{JAVACLASS:adhoc.class}%{SPACE}-%{SPACE},%{SPACE}MTU%{NUMBER:adhoc.mtu},%{SPACE}%{NUMBER:adhoc.report.accountid},%{SPACE}%{NUMBER:adhoc.report.userid},%{SPACE}%{WORD:adhoc.report.format},%{SPACE}%{GREEDYDATA:adhoc.report.times}",
"%{ADHOC_TIME:adhoc.timestamp}%{SPACE}%{LOGLEVEL:adhoc.level}%{SPACE}%{JAVACLASS:adhoc.class}%{SPACE}-%{SPACE}%{GREEDYMULTILINE:jboss.server.trace.full}"
],
"pattern_definitions": {
"ADHOC_TIME": "%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} ([AP]M)",
"GREEDYMULTILINE": "(.|\n)*"
}
}
},
{
"date": {
"field": "adhoc.timestamp",
"target_field": "@timestamp",
"timezone": "{{ beat.timezone }}",
"formats": [
"MMM dd, yyyy HH:mm:ss aa"
]
}
},
{
"remove": {
"field": [
"message"
]
}
}
],
"on_failure": [
{
"set": {
"field": "error.message",
"value": "{{ _ingest.on_failure_message }}"
}
}
],
"description": "Pipeline for parsing report (adhoc) server logs"
},
"docs": [
{
"_source": {
"@timestamp": "2019-01-30T12:58:15.489Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.5.4",
"pipeline": "filebeat-6.5.4-adhoc-report-pipeline"
},
"offset": 485398,
"tags": [
"adhoc"
],
"message": "Jan 30, 2019 01:58:11 PM DEBUG report.internal.impl.ReportImpl - , MTU1232, 1, 18, HTMLPREVIEW, 31, 2, 4, 0, 1, 0, 55 ",
"source": "adhocreport.log",
"fileset": {
"module": "adhoc",
"name": "report"
},
"prospector": {
"type": "log"
},
"input": {
"type": "log"
},
"beat": {
"timezone": "+01:00",
"hostname": "adhoc",
"version": "6.5.4",
"name": "adhoc"
},
"host": {
"name": "adhoc"
}
}
}
]
}
And the output is:
{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_type" : "_type",
"_id" : "_id",
"_source" : {
"offset" : 485398,
"@metadata" : {
"pipeline" : "filebeat-6.5.4-adhoc-report-pipeline",
"beat" : "filebeat",
"type" : "doc",
"version" : "6.5.4"
},
"prospector" : {
"type" : "log"
},
"source" : "adhocreport.log",
"fileset" : {
"name" : "report",
"module" : "adhoc"
},
"tags" : [
"adhoc"
],
"input" : {
"type" : "log"
},
"@timestamp" : "2019-01-30T01:58:11.000+01:00",
"beat" : {
"name" : "adhoc",
"hostname" : "adhoc",
"version" : "6.5.4",
"timezone" : "+01:00"
},
"host" : {
"name" : "adhoc"
},
"adhoc" : {
"report" : {
"format" : "HTMLPREVIEW",
"accountid" : "1",
"times" : "31, 2, 4, 0, 1, 0, 55 ",
"userid" : "18"
},
"level" : "DEBUG",
"class" : "report.internal.impl.ReportImpl",
"timestamp" : "Jan 30, 2019 01:58:11 PM",
"mtu" : "1232"
}
},
"_ingest" : {
"timestamp" : "2019-01-30T13:25:23.508Z"
}
}
}
]
}
In short timestamp is set to: 2019-01-30T01:58:11.000+01:00 when time is: Jan 30, 2019 01:58:11 PM which should end up as: 2019-01-30T13:58:11.000+01:00. Is this bug or am I missing something?
test:
curl -X POST http://localhost:9200/_ingest/pipeline/_simulate?pretty
I am using elasticsearch-oss and filebeat 6.5.4