How do I set an index-level metadata field using the Elasticsearch-DSL Python client?

[wpm (W.P. McNeill)]

I am using the Elasticsearch-DSL Python client to create a new index that stores objects called IndexedDocument that are subclasses of elasticsearch_dsl.Document.

index_name = "my-index"
index = elasticsearch_dsl.Index(index_name)
IndexedDocument.init(index_name)
dict(index.get_settings())
{'my-index': {'settings': {'index': {'routing': {'allocation': {'include': {'_tier_preference': 'data_content'}}},
    'number_of_shards': '1',
    'provided_name': 'my-index',
    'creation_date': '1699980696153',
    'number_of_replicas': '1',
    'uuid': '7dT2dzvjT0elqJ_APSUqHQ',
    'version': {'created': '8500003'}}}}}

I want to add metadata to the index, a key called "my_key" with a value "my value" that I can read back from the index.
Here is what I have tried with Elasticsearch-DSL and the errors I've gotten.

index.put_settings(my_key="my value")
Traceback (most recent call last):
  File "/Users/bill.mcneill/anaconda3/envs/Lingua/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3548, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-10-b032164530ed>", line 1, in <module>
    index.put_settings(my_key="my value")
  File "/Users/bill.mcneill/anaconda3/envs/Lingua/lib/python3.10/site-packages/elasticsearch_dsl/index.py", line 517, in put_settings
    return self._get_connection(using).indices.put_settings(
  File "/Users/bill.mcneill/anaconda3/envs/Lingua/lib/python3.10/site-packages/elasticsearch/_sync/client/utils.py", line 402, in wrapped
    return api(*args, **kwargs)
TypeError: IndicesClient.put_settings() got an unexpected keyword argument 'my_key'

I also tried passing in dictionaries since that seems the most similar to what you do with the REST API.

index.put_settings({"my_key": "my value"})
Traceback (most recent call last):
  File "/Users/bill.mcneill/anaconda3/envs/Lingua/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3548, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-11-ef4da1979ed6>", line 1, in <module>
    index.put_settings({"my_key": "my value"})
  File "/Users/bill.mcneill/anaconda3/envs/Lingua/lib/python3.10/site-packages/elasticsearch_dsl/index.py", line 517, in put_settings
    return self._get_connection(using).indices.put_settings(
AttributeError: 'dict' object has no attribute 'indices'

I get the same error for index.put_settings({"index":{"my_key": "my value"}}) and index.put_settings({"settings":{"index":{"my_key": "my value"}}}).

How do I do this?

[Quentin_Pradet]

Hello! I see how that can be confusing.

TL;DR: the correct syntax is index.put_settings(settings={"my_key": "my value"}).

Let's see how I came up with this answer. The elasticsearch_dsl.Index.put_settings() docs state:

Any additional keyword arguments will be passed to Elasticsearch.indices.put_settings unchanged.

So let's try to make this work using only elasticsearch-py first. Looking at the elasticsearch.indices.put_settings(), you can see that this API does not accept arbitrary index settings, instead they need to be passed as the settings parameter:

from elasticsearch import Elasticsearch

es = Elasticsearch("http://localhost:9200")
es.indices.create(index="test-index")
print(es.indices.get_settings(index="test-index"))  # prints number_of_replicas: 1
es.indices.put_settings(index="test-index", body={"number_of_replicas": 2})
print(es.indices.get_settings(index="test-index"))  # prints number_of_replicas: 2

With that knowledge, we can go back to using elasticsearch-dsl, also using the settings keyword argument:

from elasticsearch_dsl import Document, Index, Text, connections

connections.create_connection(hosts="http://localhost:9200")


class IndexedDocument(Document):
    title = Text()


index_name = "my-index"
index = Index(index_name)
IndexedDocument.init(index_name)

print(index.get_settings())  # prints number_of_replicas: 1
index.put_settings(settings={"number_of_replicas": 2})
print(index.get_settings())  # prints number_of_replicas: 1

I hope that helps!

(By the way, I was able to infer from your traceback that you're using a recent 8.x version of elasticsearch-py and elasticsearch-dsl, but please make sure to mention it next time.)

[wpm (W.P. McNeill)]

That works. The call is being made. Thanks.

However, I get an HTTP 400 error with the message "'unknown setting [index.my_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings')".

Is there a special place in the index settings where I'm allowed to write arbitrary key/value pairs? Some "metadata" section? (I thought this was possible, but now I can't find documentation about it.)

Elasticsearch server and elasticsearch-dsl are both version 8.11.0.

[Quentin_Pradet]

Not sure, to be honest. Maybe you had in mind the _meta mapping field? You can use the put mapping/get mapping APIs like this:

index.put_mapping(meta={"my_key": "my value"})
assert index.get_mapping()[index_name]["mappings"]["_meta"] == {"my_key": "my value"}

Or you can use the DSL mapping support which allows you to update the _meta field too.