After creating several scripted fields for one of our indexes and using them successfully in several visualizations and dashboards over a period of time, all saved scripted fields suddenly disappeared. In addition, all index fields for which a field format had been specified were reset to the default. What might cause this to happen?
That shouldn't happen as they are stored as part of the docs in the .kibana index.
Can you take a look in there to see if they still exist?
I don't see them there at all. I created a new scripted field and also a new visualization. I see the new visualization in the .kibana index, but I do not see the new scripted field there. I also see visualizations and dashboards that I and others have created, including those that are broken because the scripted fields used to create them are no longer available. It almost seems as if the scripted fields exist only in browser cache and are never saved to the elastic cluster. Is that possible?
It's not no, is there anything in the ES logs that might explain this?
I tried again to create the scripted fields and then looked for them in the .kibana index. This time I found them. I had been expecting to find them as a separate document type; however, they are just listed in the index-pattern document as fields with "scripted" property set to true.
As we have several es clusters and I had added the scripted fields to each one, I checked them all and found that those fields were not listed in the .kibana index-pattern document for the index to which they had been added. I also found that the fieldFormatMap property was empty on all except one cluster (I had specified the Byte format for some of the scripted fields as well as for some of the actual index fields.) On that one cluster, the fieldFormatMap property included three of the scripted fields I had created, even though they did not appear in the field property list.
As far as looking at the es logs is concerned, I don't presently have access to them.
And one more note: I tried creating a new scripted field and setting it's format type to something other than the default. I could see the field in both the fields and fieldFormatMap properties of the .kibana index-pattern document. I then deleted the scripted field and the entry was gone from the fields property, but it still appeared in the fieldFormatMap. In order to have the field removed from the fieldFormatMap property, it was necessary to update the scripted field and change the format type back to default, which removed it from the fieldFormatMap property, before performing the delete to get rid of the scripted field.
2 Likes
I have the same problem. Do you happen to have two kibana instances in your cluster?
Ilija