How can i use the timestamp in the source file rather than the timestamp generate automatically

yigene · January 6, 2018, 1:48pm

i am trying to use FileBeat directly parse the log file and send to elastic search. here is the example of the data
2018-01-06 20:13:22 sending the number 7496 to outchannel
2018-01-06 20:13:22 sending the number 7496 to inchannel
2018-01-06 20:13:23 sending the number 7497 to outchannel
2018-01-06 20:13:23 sending the number 7497 to inchannel
2018-01-06 20:13:24 sending the number 7498 to outchannel
2018-01-06 20:13:24 sending the number 7498 to inchannel
2018-01-06 20:13:25 sending the number 7499 to outchannel
2018-01-06 20:13:25 sending the number 7499 to inchannel
2018-01-06 20:13:26 sending the number 7500 to outchannel
2018-01-06 20:13:26 sending the number 7500 to inchannel

so i use the grok pipeline and the data can load to elasticsearch and below is the output
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 3,
"successful": 3,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 7.06019,
"hits": [
{
"_index": "mlsbeat-6.1.0-2018.01.06",
"_type": "doc",
"_id": "AWDLmoWt8ZoYLzM4Ts54",
"_score": 7.06019,
"_source": {
"@timestamp": "2018-01-06T13:13:08.176Z",
"offset": 235527,
"beat": {
"hostname": "gene-VirtualBox",
"name": "gene-VirtualBox",
"version": "6.1.0"
},
"prospector": {
"type": "log"
},
"source": "/home/devops/mls-log.log",
"message": "2018-01-06 20:05:01 sending the number 7000 to outchannel ",
"rttype": "outchannel ",
"timestamp": "2018-01-06 20:05:01",
"trackingid": "7000"
}
},
{
"_index": "mlsbeat-6.1.0-2018.01.06",
"_type": "doc",
"_id": "AWDLmoWt8ZoYLzM4Ts55",
"_score": 7.00971,
"_source": {
"@timestamp": "2018-01-06T13:13:08.176Z",
"offset": 235585,
"beat": {
"hostname": "gene-VirtualBox",
"name": "gene-VirtualBox",
"version": "6.1.0"
},
"prospector": {
"type": "log"
},
"source": "/home/devops/mls-log.log",
"message": "2018-01-06 20:05:01 sending the number 7000 to inchannel ",
"rttype": "inchannel ",
"timestamp": "2018-01-06 20:05:01",
"trackingid": "7000"
}
}
]
}
}

the @timestamp by default is the system time, but acutally i want to use the timestamp in the timestamp, in Kibana can not choose timestamp when create the index pattern hence not able to show in the dashboard. i understand the reason is that timestamp is string type not a date . is there anyway can easily convert it to date type without using logstash?

yigene · January 6, 2018, 1:50pm

add the pipeline code

PUT _ingest/pipeline/inspectionmls
{
"description" : "Convert mls rt log to indexed data",
"processors" : [
{
"grok": {
"field": "message",
"patterns": ["(?%{TIMESTAMP_ISO8601}) sending the number %{DATA:trackingid} to %{DATA:rttype}$"]
}
}
],
"on_failure" : [
{
"set" : {
"field" : "error",
"value" : " - Error processing message - "
}
}
]
}

Christian_Dahlqvist · January 6, 2018, 2:05pm

Extract the event timestamp into a separate field in the pipeline and the use a date processor to parse it.

yigene · January 6, 2018, 2:22pm

it can not parse to Date format...

PUT _ingest/pipeline/inspectionmls
{
"description" : "Convert mls rt log to indexed data",
"processors" : [
{
"grok": {
"field": "message",
"patterns": ["(?%{TIMESTAMP_ISO8601}) sending the number %{DATA:trackingid} to %{DATA:rttype}$"]
},
"date": {
"field": "timestamp",
"formats": ["ISO8601"]
}
}
]
}

error:

{
"docs": [
{
"error": {
"root_cause": [
{
"type": "exception",
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"header": {
"processor_type": "date"
}
}
],
"type": "exception",
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": """Invalid format: "2018-01-06 15:51:23" is malformed at " 15:51:23""""
}
}
},
"header": {
"processor_type": "date"
}
}
}
]
}

Christian_Dahlqvist · January 6, 2018, 2:26pm

You need to change your grok expression to store the timestamp in a variable, e.g. timestamp, the same way you capture trackingid and rttype. Then you can apply the date processor to that field.

yigene · January 6, 2018, 2:33pm

thanks..

but actually that already stored in timestamp variable. i feel is the date processor can not convert the string to date format hence throw out the exception

just tried again, the same error.

PUT _ingest/pipeline/inspectionmls
{
"description" : "Convert mls rt log to indexed data",
"processors" : [
{
"grok": {
"field": "message",
"patterns": ["%{TIMESTAMP_ISO8601:processtime} sending the number %{DATA:trackingid} to %{DATA:rttype}$"]
},
"date": {
"field": "processtime",
"formats": ["ISO8601"]
}
}
]
}

{
"docs": [
{
"error": {
"root_cause": [
{
"type": "exception",
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"header": {
"processor_type": "date"
}
}
],
"type": "exception",
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "unable to parse date [2018-01-06 15:51:23]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": """Invalid format: "2018-01-06 15:51:23" is malformed at " 15:51:23""""
}
}
},
"header": {
"processor_type": "date"
}
}
}
]
}

Christian_Dahlqvist · January 6, 2018, 2:38pm

What does an event you previously got inserted with the incorrect timestamp look like? Can you try specifying the full date format instead of using ISO8601 to see if that helps?

yigene · January 6, 2018, 2:44pm

Thanks very much , after put below format it works!

  "date": {
    "field": "processtime",
    "formats": ["yyyy-MM-dd HH:mm:ss"]

      "@timestamp": "2018-01-06T15:51:23.000Z",
      "beat": {
        "hostname": "systemx",
        "name": "RestReviews",
        "version": "5.1.1"
      },
      "processtime": "2018-01-06 15:51:23",
      "rttype": "inchannels",
      "trackingid": "999711"
    },

system · January 27, 2018, 1:48pm

This topic was automatically closed after 21 days. New replies are no longer allowed.

Topic		Replies	Views
Is filebeat parse the timestamp from the log line and use it as the timestamp of the event? Beats filebeat	5	891	January 23, 2019
Get timestamp from log lines Beats filebeat	4	11627	February 28, 2018
Add a timestamp from the log line in filebeat Beats filebeat	3	2620	April 6, 2019
Filebeat sending json date field as text Beats filebeat	12	2985	November 12, 2018
@timestamp picked as string instead of date Elasticsearch	8	10596	December 22, 2017

How can i use the timestamp in the source file rather than the timestamp generate automatically

Related topics