Hi
For simplicity I want to store reporting metrics in a single node cluster. I can see the index is created with 1 replicas. Are there any way to set replicas to 0?
Or is my only option to create a cluster with more nodes?
Thanks in advance
Lasse Nedergaard
Having indices yellow in a single node cluster will not affect the ability to store or view data. If you want to change it to 0 you can do so using an index template.
Thanks for fast answer. The yellow index was best guess for why my rally was failing with a failed to index when I use the single node cluster. I can see rally creates index and template but it report 404 when it tries to store metrics with an index not found exception.
I can see the index create is rally-metrics-2021-06
And it’s trying to insert info rally-races-2021-06.
But no idea why only one out of 3 indexes was created. In es log I can see the template for rally-races was created but no index.
Any ideas
It has something to do with the yellow state. If I create rally-races index with replicas 0 it works but for automation purposes I need to figure out why the replicas are set to 1
The template used for the rally-metrics index can be found here and doesn't specify the number of replicas (only number of shards 1). Similar case for the other index templates rally-results and rally-races. So Elasticsearch defaults apply (see index.number_of_replicas) causing the index to have 1 replica.
However, as @Christian_Dahlqvist mentioned above this shouldn't affect your capability to store metric records. I just quickly launched a single node docker based metricsstore and issued a short race using: esrally race --track=geonames --challenge=append-no-conflicts-index-only --runtime-jdk=bundled --car=4gheap --distribution-version=7.13.1 --test-mode --on-error=abort with my ~/.rally.ini [reporting] section pointing to my Docker based single node setup and the race went fine, without any issues or errors, and the results are available on the metricsstore
What is the exact error you are seeing? Seeing ~/.rally/rally.log would help.
2021-06-14 09:26:39,595 ActorAddr-(T|:37697)/PID:22 esrally.metrics INFO [rally-metrics-2021-06] already exists.
2021-06-14 09:26:39,597 ActorAddr-(T|:37697)/PID:22 esrally.metrics INFO Creating ES race store
2021-06-14 09:26:39,597 ActorAddr-(T|:37697)/PID:22 esrally.client INFO Creating ES client connected to [{'host': 'rallyes', 'port': '9200'}] with options [{'use_ssl': False, 'verify_certs': True, 'timeout': 120}]
2021-06-14 09:26:39,598 ActorAddr-(T|:37697)/PID:22 esrally.client INFO SSL support: off
2021-06-14 09:26:39,598 ActorAddr-(T|:37697)/PID:22 esrally.client INFO HTTP basic authentication: off
2021-06-14 09:26:39,598 ActorAddr-(T|:37697)/PID:22 esrally.client INFO HTTP compression: off
2021-06-14 09:26:39,601 ActorAddr-(T|:37697)/PID:22 esrally.actor INFO Asking mechanic to start the engine.
2021-06-14 09:26:39,601 ActorAddr-(T|:37697)/PID:22 esrally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 8, 2, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1623662799194'}] match requirements [{'coordinator': True}].
2021-06-14 09:26:39,701 ActorAddr-(T|:37697)/PID:22 esrally.actor INFO Mechanic has started engine successfully.
2021-06-14 09:26:39,702 ActorAddr-(T|:37697)/PID:22 esrally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 8, 2, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1623662799194'}] match requirements [{'coordinator': True}].
2021-06-14 09:26:39,704 ActorAddr-(T|:37697)/PID:22 esrally.actor INFO Telling driver to prepare for benchmarking.
2021-06-14 09:26:39,719 ActorAddr-(T|:46055)/PID:24 esrally.client INFO Creating ES client connected to [{'host': 'rallyes', 'port': '9200'}] with options [{'use_ssl': False, 'verify_certs': True, 'timeout': 120}]
2021-06-14 09:26:39,719 ActorAddr-(T|:46055)/PID:24 esrally.client INFO SSL support: off
2021-06-14 09:26:39,719 ActorAddr-(T|:46055)/PID:24 esrally.client INFO HTTP basic authentication: off
2021-06-14 09:26:39,720 ActorAddr-(T|:46055)/PID:24 esrally.client INFO HTTP compression: off
2021-06-14 09:26:39,725 ActorAddr-(T|:46055)/PID:24 esrally.metrics INFO Creating Elasticsearch metrics store
2021-06-14 09:26:39,725 ActorAddr-(T|:46055)/PID:24 esrally.metrics INFO Opening metrics store for race timestamp=[20210614T092639Z], track=[eventdata], challenge=[bulk-update], car=[['external']]
2021-06-14 09:26:39,732 ActorAddr-(T|:46055)/PID:24 py.warnings WARNING /usr/local/lib/python3.8/site-packages/elasticsearch/connection/base.py:193: ElasticsearchDeprecationWarning: [types removal] Specifying include_type_name in put index template requests is deprecated. The parameter will be removed in the next major version.
warnings.warn(message, category=ElasticsearchDeprecationWarning)
2021-06-14 09:26:39,734 ActorAddr-(T|:46055)/PID:24 esrally.metrics INFO [rally-metrics-2021-06] already exists.
2021-06-14 09:26:39,736 ActorAddr-(T|:46055)/PID:24 esrally.client INFO Creating ES client connected to [{'host': 'lne.abc.prod', 'port': 443, 'use_ssl': True, 'url_prefix': '/abc_services/es'}] with options [{'use_ssl': True, 'verify_certs': False, 'timeout': 60, 'retry-on-timeout': True}]
2021-06-14 09:26:39,736 ActorAddr-(T|:46055)/PID:24 esrally.client INFO SSL support: on
2021-06-14 09:26:39,743 ActorAddr-(T|:46055)/PID:24 esrally.client INFO SSL certificate verification: off
2021-06-14 09:26:39,743 ActorAddr-(T|:46055)/PID:24 esrally.client WARNING User has enabled SSL but disabled certificate verification. This is dangerous but may be ok for a benchmark. Disabling urllib warnings now to avoid a logging storm. See https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings for details.
2021-06-14 09:26:39,744 ActorAddr-(T|:46055)/PID:24 esrally.client INFO SSL client authentication: off
2021-06-14 09:26:39,744 ActorAddr-(T|:46055)/PID:24 esrally.client INFO HTTP basic authentication: off
2021-06-14 09:26:39,744 ActorAddr-(T|:46055)/PID:24 esrally.client INFO HTTP compression: off
2021-06-14 09:26:39,744 ActorAddr-(T|:46055)/PID:24 esrally.driver.driver INFO Checking if REST API is available.
2021-06-14 09:26:39,755 ActorAddr-(T|:46055)/PID:24 esrally.client INFO REST API is available for >= [1] nodes after [0] attempts.
2021-06-14 09:26:39,756 ActorAddr-(T|:46055)/PID:24 esrally.driver.driver INFO REST API is available.
2021-06-14 09:26:39,759 ActorAddr-(T|:46055)/PID:24 esrally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 8, 2, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1623662799194'}] match requirements [{'coordinator': True}].
2021-06-14 09:26:39,780 ActorAddr-(T|:33557)/PID:25 esrally.track.loader INFO Preparing track [eventdata]
2021-06-14 09:26:39,782 ActorAddr-(T|:33557)/PID:25 esrally.track.loader INFO Reloading track [eventdata] to ensure plugins are up-to-date.
2021-06-14 09:26:39,783 ActorAddr-(T|:33557)/PID:25 esrally.utils.modules INFO Loading component [eventdata] from [/rally/.rally/rally-eventdata-track-master/rally-eventdata-track-master/eventdata]
2021-06-14 09:26:44,785 ActorAddr-(T|:46055)/PID:24 esrally.actor INFO A track preparator has exited.
2021-06-14 09:26:44,804 ActorAddr-(T|:46055)/PID:24 esrally.actor INFO A track preparator has exited.
2021-06-14 09:26:44,825 ActorAddr-(T|:37697)/PID:22 esrally.metrics ERROR An unknown error occurred while running the operation [bulk] against your Elasticsearch metrics store on host [rallyes] at port [9200].
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/esrally/metrics.py", line 119, in guarded
return target(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py", line 390, in bulk
for ok, item in streaming_bulk(client, actions, *args, **kwargs):
File "/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py", line 311, in streaming_bulk
for data, (ok, info) in zip(
File "/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py", line 247, in _process_bulk_chunk
for item in gen:
File "/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py", line 188, in _process_bulk_chunk_success
raise BulkIndexError("%i document(s) failed to index." % len(errors), errors)
elasticsearch.helpers.errors.BulkIndexError: ('1 document(s) failed to index.', [{'index': {'_index': 'rally-races-2021-06', '_type': '_doc', '_id': '38f50d4b-d050-4ad2-93f0-f54cb2832a9f', 'status': 404, 'error': {'type': 'index_not_found_exception', 'reason': 'no such index [rally-races-2021-06] and [action.auto_create_index] is [false]', 'index_uuid': '_na_', 'index': 'rally-races-2021-06'}, 'data': {'rally-version': '2.2.0', 'rally-revision': None, 'environment': 'local', 'race-id': '38f50d4b-d050-4ad2-93f0-f54cb2832a9f', 'race-timestamp': '20210614T092639Z', 'pipeline': 'benchmark-only', 'user-tags': {}, 'track': 'eventdata', 'car': ['external'], 'cluster': {'revision': '3186837139b9c6b6d23c3200870651f10d3343b7', 'distribution-version': '7.12.1', 'distribution-flavor': 'default', 'team-revision': None}, 'challenge': 'bulk-update', 'track-params': {'number_of_shards': 10, 'bulk_indexing_iterations': 10000, 'bulk_indexing_clients': 20}}}}])
2021-06-14 09:26:44,827 ActorAddr-(T|:37697)/PID:22 esrally.actor ERROR Error in race control
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/esrally/metrics.py", line 119, in guarded
return target(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py", line 390, in bulk
for ok, item in streaming_bulk(client, actions, *args, **kwargs):
File "/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py", line 311, in streaming_bulk
for data, (ok, info) in zip(
File "/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py", line 247, in _process_bulk_chunk
for item in gen:
File "/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py", line 188, in _process_bulk_chunk_success
raise BulkIndexError("%i document(s) failed to index." % len(errors), errors)
elasticsearch.helpers.errors.BulkIndexError: ('1 document(s) failed to index.', [{'index': {'_index': 'rally-races-2021-06', '_type': '_doc', '_id': '38f50d4b-d050-4ad2-93f0-f54cb2832a9f', 'status': 404, 'error': {'type': 'index_not_found_exception', 'reason': 'no such index [rally-races-2021-06] and [action.auto_create_index] is [false]', 'index_uuid': '_na_', 'index': 'rally-races-2021-06'}, 'data': {'rally-version': '2.2.0', 'rally-revision': None, 'environment': 'local', 'race-id': '38f50d4b-d050-4ad2-93f0-f54cb2832a9f', 'race-timestamp': '20210614T092639Z', 'pipeline': 'benchmark-only', 'user-tags': {}, 'track': 'eventdata', 'car': ['external'], 'cluster': {'revision': '3186837139b9c6b6d23c3200870651f10d3343b7', 'distribution-version': '7.12.1', 'distribution-flavor': 'default', 'team-revision': None}, 'challenge': 'bulk-update', 'track-params': {'number_of_shards': 10, 'bulk_indexing_iterations': 10000, 'bulk_indexing_clients': 20}}}}])
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/esrally/actor.py", line 85, in guard
return f(self, msg, sender)
File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 119, in receiveMsg_PreparationComplete
self.coordinator.on_preparation_complete(msg.distribution_flavor, msg.distribution_version, msg.revision)
File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 208, in on_preparation_complete
self.race_store.store_race(self.race)
File "/usr/local/lib/python3.8/site-packages/esrally/metrics.py", line 1357, in store_race
self.es_store.store_race(race)
File "/usr/local/lib/python3.8/site-packages/esrally/metrics.py", line 1419, in store_race
self.client.index(index=self.index_name(race), doc_type=EsRaceStore.RACE_DOC_TYPE, item=doc, id=race.race_id)
File "/usr/local/lib/python3.8/site-packages/esrally/metrics.py", line 103, in index
self.bulk_index(index, doc_type, [doc])
File "/usr/local/lib/python3.8/site-packages/esrally/metrics.py", line 93, in bulk_index
self.guarded(elasticsearch.helpers.bulk, self._client, items, index=index, chunk_size=5000)
File "/usr/local/lib/python3.8/site-packages/esrally/metrics.py", line 171, in guarded
raise exceptions.RallyError(msg)
esrally.exceptions.RallyError: An unknown error occurred while running the operation [bulk] against your Elasticsearch metrics store on host [rallyes] at port [9200].
What's the version of Elasticsearch that you are using as a metrics store?
Have you set the cluster setting action.auto_create_index which is by default true to false on your metrics cluster?
7.12.1 for both.
Yes index auto create is set to false. But it’s also set to false on my test es cluster and if I use that for metrics also it works.
I will try to set it to true and see if it has any effect.
The index auto create did the trick. Setting it to true fixed the problem.
Thank you very much for fast and precise help much appreciated.
Lasse
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.