Failed to parse timestamp exception, (only) when using a new Elasticsearch index name

Elastic Stack Beats
1

Hi,

I am using Elasticsearch, Kibana and Metricbeat 6.1.1. I would like to use a different index name, for Metricbeat, than the default one. But I have problems as shortly follow:

  1. When I start Metricbeat, with default preferences (index name, pattern etc.), I am able to record the data in Elasticsearch and inspect in Kibana without a problem. (the metricbeat index gets created correctly in Elasticsearch)
  2. Then, in the metricbeat.yml file, I simply changed the Elasticsearch output.elasticsearch.index to something custom, along with the indicated setup.template.name and setup.template.pattern accordingly:

output.elasticsearch.index: "my-index-name-%{[beat.version]}-%{+yyyy.MM.dd}"
setup.template.name: "my-index-name"
setup.template.pattern: "my-index-name-*"

  1. In this case, when Metricbeat starts up, I get the following exception:

failed to execute bulk item (index) BulkShardRequest [[my-index-name][0]] containing [index....] ...
... org.elasticsearch.index.mapper.MapperParsingException: failed to parse [@timestamp] ...
... Caused by: java.lang.IllegalArgumentException: Invalid format: "2018-04-02T07:24:39.162Z"...

  1. If I comment-out the custom output.elasticsearch.index setting, then all is fine again and the default metricbeat index is created without problem.

I do not understand why, without modifying any field mappings or other settings, it does not work for
my custom metricbeat index, yet works if I comment the setting out.

I have searched the reference(s) and forum posts for a possible answer. But I fail to understand what I am missing (or doing wrong).
I apologize in advance if the question already has an answer which I failed to find.

Thank you in advance!
Andrei

2

Hi,

I managed to find the problem, after searching and comparing type mapping specifications in my application. (I am not that experienced with the ELK stack yet.)

Unfortunately, it appears I had (have) an index template in code, that specified a different @timestamp format. The template was specified something like

{ "template": "my-*",....
"mappings": ... "properties": "@timestamp": { "type": "date", "format": "epoch_millis"}
.. }

and this caused the trouble with my-index-name index.

Anyway, hope this answer might help other beginners as myself...

Best regards,
Andrei.