Ingest Pipeline

[spike83]

Hi,

I have am ingest pipeline that set within Elasticsearch ingest node, that takes the message field from a log file and splits it out.

During this I pull out the date which looks fine when it hits the document, but the index pattern is setting it as a string and not a date. is the a way I can get it to set as a date?

[
  {
    "grok": {
      "field": "message",
      "patterns": [
        "%{YUMTIMESTAMP:yum.date} %{NOTSPACE:yum.state} %{NOTSPACE:yum.package}"
      ],
      "pattern_definitions": {
        "YUMTIMESTAMP": "%{MONTH} +%{MONTHDAY} %{TIME}"
      },
      "description": "Yum entry"
    }
  },
  {
    "date": {
      "field": "yum.date",
      "formats": [
        "MMM dd HH:mm:ss"
      ],
      "target_field": "yum.datetime"
    }
  }
]

[dadoonet (David Pilato)]

You need to define the mapping for your index and set the field as a date field.

[spike83]

Thank you for coming back to me.

I haven't carried out a mapping before, where is this setup?

[stephenb (Stephen Brown)]

Explicit mapping | Elasticsearch Guide [7.12] | Elastic

And

Date field type | Elasticsearch Guide [7.12] | Elastic

You can GET your current mapping using

GET /yourindexname

And the adjust from there.

[spike83]

Thanks @stephenb - As im pulling the data in through filebeat, wil i need to update the mapping on every new index created for each release e.g. filebeat-7.11 then filebeat-7.12?

[stephenb (Stephen Brown)]

No you will use an index template that matches the pattern of indices like

my-filebeat-*

Think of a template as a mapping (and settings) that get applied to any index name that matches a pattern

[spike83]

Thanks. Had to wing it a little but got it working with your help