Deleted mapping is still applied by Elasticsearch


#1

Hi,
after changing a mapping, the old mapping is still applied. I restarted ES, then even restarted Windows, but the problem is still there.

An object type, lets call it ExampleType, is saved in an index, lets call it "example_index".
ExampleType has a property DoubleNumber of type double.

When creating a default ElasticClient, it changes field names to camel case:
DoubleNumber -> doubleNumber

At some point (ES 6.1 or 6.2) we switched to explicit mapping creation. In our custom mapping, property name was changed this way:
DoubleNumber -> double_number

After deleting the index and creating it new the changes were applied as expected.

In the meantime, we switched to ES 6.4.0. Used Kibana is also 6.4.0.

Now I noticed, that field change can be disabled at all, if using
.DefaultFieldNameInferrer(name => name)
on ConnectionSetting of ElasticClient. Decided to make use of it.

Changed the custom mapping creation to not change field names. Expected field name mapping is:
DoubleNumber -> DoubleNumber

Deleted example_index.
In Kibana, 'GET example_index/_mapping' is throwing an exception.
'GET _mapping' is showing all mappings, but no mapping for example_index.

Now I create the index:

elasticClient.CreateIndex("example_index", c => c.
InitializeUsing(new IndexState() { Settings = new IndexSettings() { NumberOfReplicas = 0, NumberOfShards = 1 } })

Then I set my custom mapping:
elasticClient.Map(putMappingRequest);

In Kibana, I can see the new mapping now, the field name is 'DoubleNumber' as expected.

But when I index a new ExampleType with:
elasticClient.IndexDocument(myExample);
and then execute in Kibana:
GET /_search{"query": {"match" : { "_index": "example_index"} } }
I get a result with 'double_number' as property name in the document.

'GET example_index/_mapping'
returnes a mapping, where the property is referenced twice:
"DoubleNumber ": {"type": "double"},
"double_number": {"type": "double"}

I have no idea, where ES gets the old style property name mapping. It is not the default one. And it is not used anywhere.

An interesting fact is, that for a second index (lets call it "second_index"), used in parallel with example_index, the same error occurs, although second_index does not ever used a custom mapping, but always a default mapping. Now second_index also has a mapping with two definitions:
"SomeNumber": {"type": "double"},
"some_number": {"type": "double"}
and the second one used. Although this kind of name mapping was never ever applied to seond_index.
The only thing, that couples the two indices is that they are created/deleted using the same ElasticClient instance.

And for a third index, used in a different solution, where always a default mapping was used, the change of ConnectionSettings just worked and the index showed the expected behaviour after deleting and recreation.

A collegue using ES 6.2.0 is working on the same scenario. He has example_index and second_index. For example_index a custom mapping with our renaming rule is applied (SomeProp -> some_prop), for second_index is default mapping applied. He has not changed anything on that, but taking a look now, we see, that for second_index, our custom renaming was applied on his machine.

How can I force ES to delete the renaming rule, that was applied in some previous mapping, but is no longer applied? Restarting has not worked.
And how can I avoid the corruption of an index with renaming rules, that are defined for some other index only?

Alexander


(Bernt Rostad) #2

The index mapping is saved with the index when the index was created so even if you delete the mapping and create a new one it will only affect future indices (but see Updating existing field mapping).

If you want all your old data to get the new mapping the only way I know of is to use the Reindex API and create a new index with the old data. Because you create a new index it will get the new mapping and by indexing the documents from the old to the new index, they also get the new mapping. I've done this a few times myself and it's a pretty straight forward operation.


#3

The affected index was deleted already. Somehow the renaming rule is saved somewhere and applied to several indices, not only to the one, it was defined for. And even if the index that the rule belonged to is deleted, the rule is still somewhere in ES and is applied.

Today I deleted all affected indices and ran the program with new index names, so new indices with new names were created. The ElasticClient has .DefaultFieldNameInferrer(name => name) on its setting. One index was created with a default mapping. The other index was created with a custom mapping. Both have the no longer existing property name renaming rule applied.


#4

Deleted Elasticsearch. Installed version 6.4.2 (not upgraded, new installation). Created the indices. The old property name renaming is applied for both indices. Although the renaming is no longer defined anywhere in the program and although ES was reinstalled.

Help?


#5

Still fighting with this problem. Deleted ES, deleted C:\ProgramData\Elastic, deleted Kibana.
After this cleanup, no visible ES stuff was on the machine.
Debugged my program to be very sure, that it does not provide some property renaming. The mapping stored to ES uses the unchanged property names. The object, that is indexed, has the unchanged property names.
After reinstalling ES and creating the index, the expected mapping with unchanged property names is created by the program. In Kibana I can see, that the index is there and that it is correct.
When continuing my program and indexing a file, the mapping is extended with changed property names (MyProperty -> my_property). And the added part of the mapping is used to index the document. This custom renaming is no longer present anywhere. But ES finds it somewhere...

Is there any data stored at other places then C:\Program Files\Elastic or C:\ProgramData\Elastic ?


#6

Got the error.
The problem was not on ES side. In the connection settings, we pass a custom JsonSerializer and there we had the following:

protected override void ModifyContractResolver(ConnectionSettingsAwareContractResolver resolver) =>
    resolver.NamingStrategy = new SnakeCaseNamingStrategy();

The SnakeCaseNamingStrategy changes the properties (MyProperty -> my_property) when serializing the object to Json.
Deleting this method from the custom serializer solves the problem.


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.