Date format not picked up when creating index

Elastic Stack Elasticsearch
1

I created a template without any problems ( script 1)
Then I create an index by putting data into it ( script 2)
then when I retreve the index the timestamp column is still text ( script 3 )

=============== script 1 =====================

PUT _template/sensordata
{
  "template": "sensordata",
  "order": 1,
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "doc": {
      "_source": {
        "enabled": false
      },
      "properties": {
        "timestamp": {
          "type": "date",
          "format": "yyyy-MM-dd hh:mm:ss"
        }
      }
    }
  }
}

=============== script 2 =================

PUT djan/doc/1
{
  "timestamp" : "2019-02-06 10:11:47"
}

============== script 3 =================

GET djan

=== result

{
  "djan": {
    "aliases": {},
    "mappings": {
      "doc": {
        "properties": {
          "timestamp": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1549527842637",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "pMFsJmxrRLiemRhrhZI3uw",
        "version": {
          "created": "5060999"
        },
        "provided_name": "djan"
      }
    }
  }
}
2

Because you created an index named djan but your template will be applied only on index named sensordata:

  "template": "sensordata",

By the way, you should use a more recent version, like 6.6.0.

3

I looked in the orginal code I had then I found out I was mixing templates with names (
like sensordata* ) ( djan was just an example) and the order they were used was not correct.

Tanks for the quick respons

additional question we get data in a timezone moscow without timezone. Can I add timezone?

4

You can use an ingest pipeline with a date processor to process your date as you wish.

See https://www.elastic.co/guide/en/elasticsearch/reference/6.6/date-processor.html

5

I try to convert a date when the index is build. As I understood I can use pipelines. I got my training stuff, google and tried. But something i going wrong.

delete datetestz
delete _template/datetempz

put _template/datetempz
{
"index_patterns": "datetestz",
"mappings": {
"doc": {
"properties": {
"datefield": { "type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
}
}
}
}
}

put _ingest/pipeline/datconvert
{
"description" : "...",
"processors" : [
{
"date" : {
"field" : "datefield",
"target_field": "tdatefield",
"formats" : ["yyyy-MM-dd HH:mm:ss"],
"timezone" : "Europe/London"
}
}
]
}

-------- amsterdam time ------------
put datetestz/doc/1
{
"datefield": "2019-09-02 15:00:11",
"tdatefield" :""
}

I hoped tdatefield was set with the london time.
GET datetestz/_search

============= result ===========
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : "datetestz",
"_type" : "doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"datefield" : "2019-09-02 15:00:11",
"tdatefield" : "" <<=== this must be 2019-09-02 14:00:11 but it is not
}
}
]
}
}

6

Most likely this is because you are providing it as an empty value.

Here is a script to test it properly. Note that it's well formatted to be easily readable and using uppercase HTTP verbs to make it usable in a Kibana console.
Please next time, make sure you paste similar code. It will reduce the time needed to test your code.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "...",
    "processors": [
      {
        "date": {
          "field": "datefield",
          "target_field": "tdatefield",
          "formats": [
            "yyyy-MM-dd HH:mm:ss"
          ],
          "timezone": "Europe/London"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "datefield": "2019-09-02 15:00:11"
      }
    }
  ]
}

Gives:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_type",
        "_id" : "_id",
        "_source" : {
          "datefield" : "2019-09-02 15:00:11",
          "tdatefield" : "2019-09-02T15:00:11.000+01:00"
        },
        "_ingest" : {
          "timestamp" : "2019-02-09T13:19:29.817848Z"
        }
      }
    }
  ]
}

Where "tdatefield" : "2019-09-02T15:00:11.000+01:00" seems to be OK to me.

7

Thx for the reply I look into it today

8

Thanks for your help, it made me understand this date topic. I played with the pipeline UTC CET etc.

Tomorrow I'll discuss it with the business and see if I understood there problem.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.