Can date ingest processor process a String to extract a Date from it

Elastic Stack Elasticsearch
1

Hi guys
I find a question about Can date ingest processor process a String to extract a Date from it.

And i follow the guide and use Date Processor, but my log message is complex

2017-12-13 11:10:30.000 [pool-10-thread-7] DEBUG c.h.i.c.m.servicemanager.task.PassiveNodeHbTask - PassiveNodeHbTask started

And in order to use Date Processor i have to use Grok Processor to extract date as a new field such as

PUT _ingest/pipeline/java-pipeline
{
  "description" : "describe pipeline",
  "processors" : [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:logatime}"]
      },
      "date" : {
        "field" : "logatime",
        "target_field": "@timestamp", 
        "formats" : ["yyyy-MM-dd HH:mm:ss.SSS"], 
        "timezone": "Asia/Shanghai"
      },
      "remove": {
        "field": "logatime"
      }
    }
  ]
}

But in this way , i will add an useless field ------ logatime

I don't want to add this field if possibile

So can date ingest processor process a complex String to extract a Date from it.

1 Like
2

logatime is removed by the remove processor. So I don't understand what you mean.

But I doubt your grok pattern is correct. I don't think that %{TIMESTAMP_ISO8601:logatime} matches 2017-12-13 11:10:30.000 [pool-10-thread-7] DEBUG ....

3

In fact i use logatime as a middle field to help me extract date from the complex String, therefore i add logatime at beginning and remove logatime finally.

And it extract date as @timestamp from a complex string.

But i wonder is there have a simpler way to extract date directly without middle field. Such as i could use
regular expression in date processor

4

And my grok pattern is incomplete because i only want to extract date. It still work and i will write it later.

5

But i wonder is there have a simpler way to extract date directly without middle field.

No but what is your concern of adding and removing a field?

6

Thanks a lot!!!
i worried it maybe a little slow.
but if there is no other solution, i will still use a middle field
Thank you!!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.