Date extract in elastic is showing as String Type and NOT Date

Elastic Stack Elasticsearch
1

Extract is working fine , however this is still represented as String rather than Date..
Any help here would be much appreciated

Sample data : 2020-04-06 05:16:36: test again

curl -X PUT "localhost:9200/_ingest/pipeline/fields_extraction?pretty" -H 'Content-Type: application/json' -d'
{
    "description" : "parsing the input log to fields",
    "processors" : [
      {
        "dissect" : {
          "field" : "message",
          "pattern" : "%{event_created}: %{log_output}",
          "on_failure" : [
            {
              "set" : {
                "field" : "field_parse_error",
                "value" : "{{ _ingest.on_failure_message }}"
              }
            }
          ]
        }
      },
      {
        "date" : {
          "field" : "event_created",
          "target_field" : "event_datetime",
          "formats" : [
            "YYYY-MM-dd HH:mm:ss"
          ],
          "timezone" : "Europe/London",
          "on_failure" : [
            {
              "set" : {
                "field" : "date_parse_error",
                "value" : "{{ _ingest.on_failure_message }}"
              }
            }
          ]
        }
      }
    ]
 }'
2

My datatype for field "event_datatime" is date using your ingest pipeline.

Probably because when you first create the index using the pipeline, you've used an incorrect input, which produces a parsing error. After the initial incorrect attempt, the field type "event_created" will be determined as text.

Could you delete the index and try again?

This is the process I use for testing your case:

PUT _ingest/pipeline/fields_extraction?pretty
{
  "description": "parsing the input log to fields",
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{event_created}: %{log_output}",
        "on_failure": [
          {
            "set": {
              "field": "field_parse_error",
              "value": "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    },
    {
      "date": {
        "field": "event_created",
        "target_field": "event_datetime",
        "formats": [
          "YYYY-MM-dd HH:mm:ss"
        ],
        "timezone": "Europe/London",
        "on_failure": [
          {
            "set": {
              "field": "date_parse_error",
              "value": "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    }
  ]
}

DELETE test
POST test/_doc?pipeline=fields_extraction
{
  "message": "2020-04-06 05:16:36: test again"
}

GET test/_search
GET test/_mapping

And this is the attempt which uses an incorrect format as input

DELETE test2
POST test2/_doc?pipeline=fields_extraction
{
  "message": "2020-04-06 05:16:366: test again"
}

GET test2/_search
GET test2/_mapping
1 Like
3

Thanks for the awesome tip , i deleted and reloaded with correct data and is worked fine.

However this seems to be not correct rite.. so for instance now its working fine and if i load incorrect format will it go wrong again or its just first record after index is created ?

Thanks
Praveen

4

It will stay the same after the first index. Dynamic mapping happens in the first time the field is created.

Also, if there is a parsing error, the incorrect document will have a field named "date_parse_error" and the parsing error message will be stored in it.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.