Keyword seem to have disappeared

[Bibas]

Hello

I have logs coming from my application, in those logs, there is a "coType" field that used to be a text that contains a keyword subfield.

For a reason that I ignore (I don't think I changed the conf file in months and I reloaded the index in case of an unapplied auto-mapping), the keyword field kind of disappeared but I still receive those logs, the juste now are only text with no subfield.

here's a screenshot of the mapping before and after :

so I was wondering :

• Do you have any idea of the reason why the keyword field could have disappeared ?
• How can I re-add it on my logs ?

Here is my guess for now :
I'm kind of an ELK newbie so I don't know if it can be related but I have a logstash conf that only have a mutate filter to convert some fields to "string", maybe i should had the "coType" field to this filter ?

Thanks for your help !

[spinscale (Alexander Reelsen)]

Fields never disappear from an existing index.

This looks as if the index has been recreated with a different mapping.

You can add that field back using updating your mapping using multi fields. See fields | Elasticsearch Guide [7.14] | Elastic

However: You need to reindex all the existing documents once so that for all of the already indexed documents this field gets added and indexed. This can be done using the reindex API. See Reindex API | Elasticsearch Guide [7.14] | Elastic

Hope this helps!

[Bibas]

Thanks for your answer !

My field are recreated every month with their lifecycle policies, how can I prevent this re-creation with a different mapping to happen in the future ?

[spinscale (Alexander Reelsen)]

Make sure to use index templates Index templates | Elasticsearch Guide [7.14] | Elastic

[Bibas]

Thank you for your time ! That helps a lot !

[Bibas]

I'm sorry to ask for your help again, but I created a new index like myIndex-77 by reindexing my myIndex-1, the new mapping correctly applied but now my logstash is not able to index data anymore, I suspect that he does not find the index he used to push data onto, do you have any idea on how to correct that ?

I tried to delete myIndex-1 and then reindex myIndex-77 into a new myIndex-1 but it doesn't work

Here is the error I keep seeing on Logstash

logstash_1       | [2021-09-14T09:43:05,090][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://elasticsearch:9200/_bulk"}

There is also an error log on ES