Hi,
I have logs with a timestamp coming in a format like this.
[9/21/20 10:38:02:681 EEST]
In Logstash, I've used the translate filter plugin to convert EEST to +03:00, I can't find a similar functionality with the ingest node date processor.
Anyone konows a solution ?
Isn't that possible to use this?
I'm using the Date processor, but it won't give the correct result, because the timezone is not recognized in that format.
Example input in test pipeline function:
[
{
"_index": "index",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds."
}
}
]
Error message in result:
{
"ignored_error": {
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "unable to parse date [10/3/20 14:30:12:890]"
}
],
"type": "illegal_argument_exception",
"reason": "unable to parse date [10/3/20 14:30:12:890]",
"caused_by": {
"type": "zone_rules_exception",
"reason": "Unknown time-zone ID: EEST",
"suppressed": [
{
"type": "zone_rules_exception",
"reason": "Unknown time-zone ID: EEST"
},
{
"type": "zone_rules_exception",
"reason": "Unknown time-zone ID: EEST"
},
{
"type": "zone_rules_exception",
"reason": "Unknown time-zone ID: EEST"
}
]
}
}
}
Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.
A full reproduction script is something anyone can copy and paste in Kibana dev console, click on the run button to reproduce your use case. It will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.
I'd suggest using the simulate API for your reproduction script.
Also, please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.
Or use markdown style like:
```
CODE
```
This is the icon to use if you are not using markdown format:
There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.
Here is the updated formatted comment using the formatting.
I'm using the Date processor, but it won't give the correct result, because the timezone is not recognized in that format. I will post the pipeline in the next reply, because the size of a comment is reached otherwise.
Example input in 'Stack Management\Ingest Node Pipelines' 'Test pipeline' function:
[
{
"_index": "index",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds."
}
}
]
Result using verbose output:
"docs": [
{
"processor_results": [
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds."
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2020-10-05T07:27:30.728450168Z"
}
}
},
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"fields": {
"msg": "com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"date": "10/3/20",
"threadID": "000000c2",
"milliseconds": "890",
"hour": "14",
"tz": "EEST",
"shortclass": "TransactionCa",
"loglevel": "W",
"minute": "30",
"second": "12"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2020-10-05T07:27:30.728450168Z"
}
}
},
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"fields": {
"msg": "com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"date": "10/3/20",
"threadID": "000000c2",
"milliseconds": "890",
"hour": "14",
"tz": "EEST",
"shortclass": "TransactionCa",
"loglevel": "W",
"minute": "30",
"second": "12",
"timestamp": "10/3/20 14:30:12:890 EEST"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2020-10-05T07:27:30.728450168Z"
}
}
},
{
"ignored_error": {
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "unable to parse date [10/3/20 14:30:12:890 EEST]"
}
],
"type": "illegal_argument_exception",
"reason": "unable to parse date [10/3/20 14:30:12:890 EEST]",
"caused_by": {
"type": "zone_rules_exception",
"reason": "Unknown time-zone ID: EEST",
"suppressed": [
{
"type": "zone_rules_exception",
"reason": "Unknown time-zone ID: EEST"
},
{
"type": "zone_rules_exception",
"reason": "Unknown time-zone ID: EEST"
},
{
"type": "zone_rules_exception",
"reason": "Unknown time-zone ID: EEST"
}
]
}
}
},
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"fields": {
"msg": "com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"date": "10/3/20",
"threadID": "000000c2",
"milliseconds": "890",
"hour": "14",
"tz": "EEST",
"shortclass": "TransactionCa",
"loglevel": "W",
"minute": "30",
"second": "12",
"timestamp": "10/3/20 14:30:12:890 EEST"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2020-10-05T07:27:30.728450168Z"
}
}
},
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"fields": {
"msg": "com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"threadID": "000000c2",
"milliseconds": "890",
"hour": "14",
"tz": "EEST",
"shortclass": "TransactionCa",
"loglevel": "W",
"minute": "30",
"second": "12",
"timestamp": "10/3/20 14:30:12:890 EEST"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2020-10-05T07:27:30.728450168Z"
}
}
},
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"fields": {
"msg": "com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"threadID": "000000c2",
"milliseconds": "890",
"tz": "EEST",
"shortclass": "TransactionCa",
"loglevel": "W",
"minute": "30",
"second": "12",
"timestamp": "10/3/20 14:30:12:890 EEST"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2020-10-05T07:27:30.728450168Z"
}
}
},
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"fields": {
"msg": "com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"threadID": "000000c2",
"milliseconds": "890",
"tz": "EEST",
"shortclass": "TransactionCa",
"loglevel": "W",
"second": "12",
"timestamp": "10/3/20 14:30:12:890 EEST"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2020-10-05T07:27:30.728450168Z"
}
}
},
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"fields": {
"msg": "com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"threadID": "000000c2",
"milliseconds": "890",
"tz": "EEST",
"shortclass": "TransactionCa",
"loglevel": "W",
"timestamp": "10/3/20 14:30:12:890 EEST"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2020-10-05T07:27:30.728450168Z"
}
}
},
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"fields": {
"msg": "com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"threadID": "000000c2",
"tz": "EEST",
"shortclass": "TransactionCa",
"loglevel": "W",
"timestamp": "10/3/20 14:30:12:890 EEST"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2020-10-05T07:27:30.728450168Z"
}
}
},
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"message": "[10/3/20 14:30:12:890 EEST] 000000c2 TransactionCa W com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"fields": {
"msg": "com.ibm.commerce.datatype.TransactionCache setTransactionKnownToBeActive java.lang.RuntimeException: Stack trace at transaction begin time=1601577583559 milliseconds.",
"threadID": "000000c2",
"tz": "EEST",
"shortclass": "TransactionCa",
"loglevel": "W"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2020-10-05T07:27:30.728450168Z"
}
}
}
]
}
]
}
The relevant lines of the message are this:
"type": "zone_rules_exception",
"reason": "Unknown time-zone ID: EEST"
This is my pipeline definition:
{
"filebeat-was-traditional-pipeline" : {
"description" : "Pipeline for parsing WebSphere Traditional messages.",
"processors" : [
{
"gsub" : {
"field" : "message",
"pattern" : """[
]""",
"replacement" : " ",
"ignore_failure" : true
}
},
{
"grok" : {
"field" : "message",
"patterns" : [
"""\[%{DATE:fields.date} %{HOUR:fields.hour}:%{MINUTE:fields.minute}:%{SECOND:fields.second}(?::%{INT:fields.milliseconds}) %{WORD:fields.tz}\] %{BASE16NUM:fields.threadID} %{DATA:fields.shortclass} %{SPACE}%{WORD:fields.loglevel} %{NOTSPACE:fields.class} %{NOTSPACE:fields.method}\(%{DATA:fields.method_ext}\) %{SPACE}%{GREEDYMULTILINE:fields.msg}""",
"""\[%{DATE:fields.date} %{HOUR:fields.hour}:%{MINUTE:fields.minute}:%{SECOND:fields.second}(?::%{INT:fields.milliseconds}) %{WORD:fields.tz}\] %{BASE16NUM:fields.threadID} %{DATA:fields.shortclass} %{SPACE}%{WORD:fields.loglevel} %{SPACE}%{GREEDYMULTILINE:fields.msg}""",
"""\[%{DATE:fields.date} %{HOUR:fields.hour}:%{MINUTE:fields.minute}:%{SECOND:fields.second}(?::%{INT:fields.milliseconds}) %{WORD:fields.tz}\] %{BASE16NUM:fields.threadID} %{DATA:fields.shortclass} %{SPACE}%{WORD:fields.loglevel} %{NOTSPACE:fields.class} %{NOTSPACE:fields.method} %{SPACE}%{GREEDYMULTILINE:fields.msg}"""
],
"pattern_definitions" : {
"GREEDYMULTILINE" : """(.|
)*"""
},
"ignore_missing" : true
}
},
{
"set" : {
"field" : "fields.timestamp",
"value" : "{{fields.date}} {{fields.hour}}:{{fields.minute}}:{{fields.second}}:{{fields.milliseconds}} {{fields.tz}}"
}
},
{
"date" : {
"field" : "fields.timestamp",
"target_field" : "@timestamp",
"timezone" : "{{fields.tz}}",
"formats" : [
"M/dd/yy HH:mm:ss:SSS ZZZ",
"M/d/yy HH:mm:ss:SSS ZZZ",
"MM/dd/yy HH:mm:ss:SSS ZZZ",
"MM/d/yy HH:mm:ss:SSS ZZZ"
],
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "fields.date"
}
},
{
"remove" : {
"field" : "fields.hour"
}
},
{
"remove" : {
"field" : "fields.minute"
}
},
{
"remove" : {
"field" : "fields.second"
}
},
{
"remove" : {
"field" : "fields.milliseconds"
}
},
{
"remove" : {
"field" : "fields.timestamp"
}
}
],
"on_failure" : [
{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}
]
}
}
Would this work for you?
POST _ingest/pipeline/_simulate
{
"docs": [
{
"_index": "index",
"_id": "id",
"_source": {
"timestamp": "10/3/20 14:30:12:890 EEST"
}
}
],
"pipeline": {
"description": "Pipeline for parsing WebSphere Traditional messages.",
"processors": [
{
"date": {
"field": "timestamp",
"target_field": "@timestamp",
"formats": [
"M/dd/yy HH:mm:ss:SSS z",
"M/d/yy HH:mm:ss:SSS z",
"MM/dd/yy HH:mm:ss:SSS z",
"MM/d/yy HH:mm:ss:SSS z"
]
}
}
]
}
}
This gives:
{
"docs" : [
{
"doc" : {
"_index" : "index",
"_type" : "_doc",
"_id" : "id",
"_source" : {
"@timestamp" : "2020-10-03T11:30:12.890Z",
"timestamp" : "10/3/20 14:30:12:890 EEST"
},
"_ingest" : {
"timestamp" : "2020-10-06T16:11:54.243154749Z"
}
}
}
]
}
This works perfect. Thank you.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.