How can i use the timestamp in the source file rather than the timestamp generate automatically

Elastic Stack Beats
1

i am trying to use FileBeat directly parse the log file and send to elastic search. here is the example of the data
2018-01-06 20:13:22 sending the number 7496 to outchannel
2018-01-06 20:13:22 sending the number 7496 to inchannel
2018-01-06 20:13:23 sending the number 7497 to outchannel
2018-01-06 20:13:23 sending the number 7497 to inchannel
2018-01-06 20:13:24 sending the number 7498 to outchannel
2018-01-06 20:13:24 sending the number 7498 to inchannel
2018-01-06 20:13:25 sending the number 7499 to outchannel
2018-01-06 20:13:25 sending the number 7499 to inchannel
2018-01-06 20:13:26 sending the number 7500 to outchannel
2018-01-06 20:13:26 sending the number 7500 to inchannel

so i use the grok pipeline and the data can load to elasticsearch and below is the output
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 3,
"successful": 3,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 7.06019,
"hits": [
{
"_index": "mlsbeat-6.1.0-2018.01.06",
"_type": "doc",
"_id": "AWDLmoWt8ZoYLzM4Ts54",
"_score": 7.06019,
"_source": {
"@timestamp": "2018-01-06T13:13:08.176Z",
"offset": 235527,
"beat": {
"hostname": "gene-VirtualBox",
"name": "gene-VirtualBox",
"version": "6.1.0"
},
"prospector": {
"type": "log"
},
"source": "/home/devops/mls-log.log",
"message": "2018-01-06 20:05:01 sending the number 7000 to outchannel ",
"rttype": "outchannel ",
"timestamp": "2018-01-06 20:05:01",
"trackingid": "7000"
}
},
{
"_index": "mlsbeat-6.1.0-2018.01.06",
"_type": "doc",
"_id": "AWDLmoWt8ZoYLzM4Ts55",
"_score": 7.00971,
"_source": {
"@timestamp": "2018-01-06T13:13:08.176Z",
"offset": 235585,
"beat": {
"hostname": "gene-VirtualBox",
"name": "gene-VirtualBox",
"version": "6.1.0"
},
"prospector": {
"type": "log"
},
"source": "/home/devops/mls-log.log",
"message": "2018-01-06 20:05:01 sending the number 7000 to inchannel ",
"rttype": "inchannel ",
"timestamp": "2018-01-06 20:05:01",
"trackingid": "7000"
}
}
]
}
}

the @timestamp by default is the system time, but acutally i want to use the timestamp in the timestamp, in Kibana can not choose timestamp when create the index pattern hence not able to show in the dashboard. i understand the reason is that timestamp is string type not a date . is there anyway can easily convert it to date type without using logstash?

2

add the pipeline code

PUT _ingest/pipeline/inspectionmls
{
"description" : "Convert mls rt log to indexed data",
"processors" : [
{
"grok": {
"field": "message",
"patterns": ["(?%{TIMESTAMP_ISO8601}) sending the number %{DATA:trackingid} to %{DATA:rttype}$"]
}
}
],
"on_failure" : [
{
"set" : {
"field" : "error",
"value" : " - Error processing message - "
}
}
]
}

3

Extract the event timestamp into a separate field in the pipeline and the use a date processor to parse it.

4

it can not parse to Date format...

PUT _ingest/pipeline/inspectionmls
{
"description" : "Convert mls rt log to indexed data",
"processors" : [
{
"grok": {
"field": "message",
"patterns": ["(?%{TIMESTAMP_ISO8601}) sending the number %{DATA:trackingid} to %{DATA:rttype}$"]
},
"date": {
"field": "timestamp",
"formats": ["ISO8601"]
}
}
]
}

error:

{
"docs": [
{
"error": {
"root_cause": [
{
"type": "exception",
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"header": {
"processor_type": "date"
}
}
],
"type": "exception",
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": """Invalid format: "2018-01-06 15:51:23" is malformed at " 15:51:23""""
}
}
},
"header": {
"processor_type": "date"
}
}
}
]
}

5

You need to change your grok expression to store the timestamp in a variable, e.g. timestamp, the same way you capture trackingid and rttype. Then you can apply the date processor to that field.

6

thanks..

but actually that already stored in timestamp variable. i feel is the date processor can not convert the string to date format hence throw out the exception

just tried again, the same error.

PUT _ingest/pipeline/inspectionmls
{
"description" : "Convert mls rt log to indexed data",
"processors" : [
{
"grok": {
"field": "message",
"patterns": ["%{TIMESTAMP_ISO8601:processtime} sending the number %{DATA:trackingid} to %{DATA:rttype}$"]
},
"date": {
"field": "processtime",
"formats": ["ISO8601"]
}
}
]
}

{
"docs": [
{
"error": {
"root_cause": [
{
"type": "exception",
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"header": {
"processor_type": "date"
}
}
],
"type": "exception",
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": """Invalid format: "2018-01-06 15:51:23" is malformed at " 15:51:23""""
}
}
},
"header": {
"processor_type": "date"
}
}
}
]
}

7

What does an event you previously got inserted with the incorrect timestamp look like? Can you try specifying the full date format instead of using ISO8601 to see if that helps?

8

Thanks very much , after put below format it works!

  "date": {
    "field": "processtime",
    "formats": ["yyyy-MM-dd HH:mm:ss"]

      "@timestamp": "2018-01-06T15:51:23.000Z",
      "beat": {
        "hostname": "systemx",
        "name": "RestReviews",
        "version": "5.1.1"
      },
      "processtime": "2018-01-06 15:51:23",
      "rttype": "inchannels",
      "trackingid": "999711"
    },
9

This topic was automatically closed after 21 days. New replies are no longer allowed.