Deleted mapping is still applied by Elasticsearch

Got the error.
The problem was not on ES side. In the connection settings, we pass a custom JsonSerializer and there we had the following:

protected override void ModifyContractResolver(ConnectionSettingsAwareContractResolver resolver) =>
    resolver.NamingStrategy = new SnakeCaseNamingStrategy();

The SnakeCaseNamingStrategy changes the properties (MyProperty -> my_property) when serializing the object to Json.
Deleting this method from the custom serializer solves the problem.