# Elasticsearch **URL:** https://discuss.elastic.co/c/elastic-stack/elasticsearch/6.md?page=481 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 482 --- ## [Reattaching Cold Data](https://discuss.elastic.co/t/reattaching-cold-data/298785)
**Author:** [@Tim\_Mobley](https://discuss.elastic.co/u/Tim_Mobley)\ **Replies:** 7\ **Last updated:** [March 5, 2022, 4:19am UTC](https://discuss.elastic.co/t/reattaching-cold-data/298785 "2022-03-05T04:19:29Z")
If you were to have cold data on removable hard disks (for long-term offline storage), what is the best way to re-attach one of these disks to the cold data node without causing trouble? For example, I was wondering if t… --- ## [Docker - Elasticsearch 8 - unable to create temporary keystore](https://discuss.elastic.co/t/docker-elasticsearch-8-unable-to-create-temporary-keystore/298878)
**Author:** [@Realsnack](https://discuss.elastic.co/u/Realsnack)\ **Replies:** 0\ **Last updated:** [March 4, 2022, 9:50pm UTC](https://discuss.elastic.co/t/docker-elasticsearch-8-unable-to-create-temporary-keystore/298878 "2022-03-04T21:50:00Z")
I am upgrading to Elasticsearch 8 on my home docker cluster and I am getting the error: ERROR: unable to create temporary keystore at \[/usr/share/elasticsearch/config/elasticsearch.keystore.tmp\], write permissions requi… --- ## [ES6.8 vs monitoring retention](https://discuss.elastic.co/t/es6-8-vs-monitoring-retention/298283)
**Author:** [@stefws](https://discuss.elastic.co/u/stefws)\ **Replies:** 3\ **Last updated:** [March 4, 2022, 7:02pm UTC](https://discuss.elastic.co/t/es6-8-vs-monitoring-retention/298283 "2022-03-04T19:02:40Z")
Reading Logging indices retention it seems that ES6.8 might no adhere to xpack.monitoring.history.duration default value of 168h ~7days. And without any default ilm policy I created my own: { "delete\_after\_7days" : {… --- ## [Multi-field mapping/search "highlighting" difference between 7.10.2 and 7.16.3?](https://discuss.elastic.co/t/multi-field-mapping-search-highlighting-difference-between-7-10-2-and-7-16-3/298739)
**Author:** [@mrodent](https://discuss.elastic.co/u/mrodent)\ **Replies:** 1\ **Last updated:** [March 4, 2022, 6:28pm UTC](https://discuss.elastic.co/t/multi-field-mapping-search-highlighting-difference-between-7-10-2-and-7-16-3/298739 "2022-03-04T18:28:15Z")
The following mapping and query DSL dict work OK when the version of ES is 7.10.2, but they never work where the version of ES is 7.16.3: mappings = \\ { "mappings": { "properties": … --- ## [Howto modify num\_of\_shards with docker-compose](https://discuss.elastic.co/t/howto-modify-num-of-shards-with-docker-compose/298868)
**Author:** [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Replies:** 0\ **Last updated:** [March 4, 2022, 5:23pm UTC](https://discuss.elastic.co/t/howto-modify-num-of-shards-with-docker-compose/298868 "2022-03-04T17:23:03Z")
Hey there, is there a way to modify the number\_of\_shards via docker-compose or to provide an index template? --- ## [Design practice to prevent oversharding](https://discuss.elastic.co/t/design-practice-to-prevent-oversharding/298759)
**Author:** [@cyril.r](https://discuss.elastic.co/u/cyril.r)\ **Replies:** 8\ **Last updated:** [March 4, 2022, 4:57pm UTC](https://discuss.elastic.co/t/design-practice-to-prevent-oversharding/298759 "2022-03-04T16:57:03Z")
Hi there, I'm strugling with an application design and the impact that it has on the cluster. Let me explain the use case: My customers can create databases inside my application to store and retrieve data. Every tim… --- ## [6.8 to 7.x - This index has date fields with deprecated formats](https://discuss.elastic.co/t/6-8-to-7-x-this-index-has-date-fields-with-deprecated-formats/298865)
**Author:** [@MatthiasLenhardt](https://discuss.elastic.co/u/MatthiasLenhardt)\ **Replies:** 0\ **Last updated:** [March 4, 2022, 4:42pm UTC](https://discuss.elastic.co/t/6-8-to-7-x-this-index-has-date-fields-with-deprecated-formats/298865 "2022-03-04T16:42:03Z")
Hi, we are planing to upgrade Elasticsearch from 6.8 to 7.x. Kibana " 7.0 Upgrade Assistant" tells us for some indices "Date field format uses patterns which may change meaning in 7.0", e.g.: This index has date field… --- ## [Reading structured streaming data from Elasticsearch into Spark using Python](https://discuss.elastic.co/t/reading-structured-streaming-data-from-elasticsearch-into-spark-using-python/298815)
**Author:** [@saurab](https://discuss.elastic.co/u/saurab)\ **Replies:** 2\ **Last updated:** [March 4, 2022, 4:34pm UTC](https://discuss.elastic.co/t/reading-structured-streaming-data-from-elasticsearch-into-spark-using-python/298815 "2022-03-04T16:34:13Z")
I believe Elasticsearch supports Spark Structured Streaming since version 6. I have seen examples of WRITING structured streaming data from spark to Elasticsearch in append mode. But I have not seen a single example of r… --- ## [Which aggregation should be used for transform operations on multifield?](https://discuss.elastic.co/t/which-aggregation-should-be-used-for-transform-operations-on-multifield/298409)
**Author:** [@Scylanth](https://discuss.elastic.co/u/Scylanth)\ **Replies:** 2\ **Last updated:** [March 4, 2022, 4:33pm UTC](https://discuss.elastic.co/t/which-aggregation-should-be-used-for-transform-operations-on-multifield/298409 "2022-03-04T16:33:24Z")
Hello, I am using 7.16.3 Elasticsearch and I am trying to perform a transform aggregation on an index to create a summary index (basically have the same index but with an aggregated timestamp by day to be smaller). Amo… --- ## [Aggregation on Multisearch responses](https://discuss.elastic.co/t/aggregation-on-multisearch-responses/298862)
**Author:** [@hbot](https://discuss.elastic.co/u/hbot)\ **Replies:** 0\ **Last updated:** [March 4, 2022, 4:29pm UTC](https://discuss.elastic.co/t/aggregation-on-multisearch-responses/298862 "2022-03-04T16:29:02Z")
Hi, Is it possible to run an aggregation on multiple response sets returned by the multisearch API? As an example, say I execute 10 different queries through the multisearch API, and each query returns 10 documents with… --- ## [Read from spark big index and memory usage](https://discuss.elastic.co/t/read-from-spark-big-index-and-memory-usage/298707)
**Author:** [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Replies:** 5\ **Last updated:** [March 4, 2022, 3:37pm UTC](https://discuss.elastic.co/t/read-from-spark-big-index-and-memory-usage/298707 "2022-03-04T15:37:44Z")
We use Spark to read a full index of 80Go then write data as a parquet file. I see memory consumption is going up until 24G per Spark worker. Hence my question: during the scroll search (1 hour), is the data stored in … --- ## [Incorrect indice creation date](https://discuss.elastic.co/t/incorrect-indice-creation-date/298539)
**Author:** [@Helicube](https://discuss.elastic.co/u/Helicube)\ **Replies:** 6\ **Last updated:** [March 4, 2022, 3:15pm UTC](https://discuss.elastic.co/t/incorrect-indice-creation-date/298539 "2022-03-04T15:15:42Z")
Hello, I have an issue with the Index Rotation. My Graylog server automatically create a new Indice every day (Index rotation). But sometimes, the creation date entered in Elasticsearch is not correct. The date entered … --- ## [Unable to form a cluster elasticsearch 8.0.0](https://discuss.elastic.co/t/unable-to-form-a-cluster-elasticsearch-8-0-0/298828)
**Author:** [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Replies:** 0\ **Last updated:** [March 4, 2022, 7:57am UTC](https://discuss.elastic.co/t/unable-to-form-a-cluster-elasticsearch-8-0-0/298828 "2022-03-04T07:57:30Z")
Hi Team, I have downloaded elasticsearch 8.0.0 on windows 2019. I am trying to make a 3 node cluster with master + data eligibility. Below is my Node 1 configuration cluster.name: test node.name: node-1 node.roles: \[… --- ## [SELECT the last three rows in DATA with IN list](https://discuss.elastic.co/t/select-the-last-three-rows-in-data-with-in-list/298771)
**Author:** [@fastxl](https://discuss.elastic.co/u/fastxl)\ **Replies:** 3\ **Last updated:** [March 4, 2022, 2:06pm UTC](https://discuss.elastic.co/t/select-the-last-three-rows-in-data-with-in-list/298771 "2022-03-04T14:06:32Z")
Trying to select the last 3 rows of a group. If I run this query POST /\_sql?format=txt { "query": "SELECT \\"Samples.AccumulatedTime.@@data\\",\\"Samples.AccumulatedTime.name\\",deviceName,timestamp FROM \\"mtconnect-demo\\… --- ## [Xpack.security.authc.api\_key.enabled in ECK with TLS self signed certificate disabled](https://discuss.elastic.co/t/xpack-security-authc-api-key-enabled-in-eck-with-tls-self-signed-certificate-disabled/298767)
**Author:** [@trudesea](https://discuss.elastic.co/u/trudesea)\ **Replies:** 2\ **Last updated:** [March 4, 2022, 1:58pm UTC](https://discuss.elastic.co/t/xpack-security-authc-api-key-enabled-in-eck-with-tls-self-signed-certificate-disabled/298767 "2022-03-04T13:58:59Z")
Hi, I'm using ECK on GKE and terminating SSL with LBs and Google managed certs. I have an issue where I cannot implement Fleet because of TLS being disabled. I remember seeing a post where this requirement would be … --- ## [How to audit search on an Elasticsearch indexes](https://discuss.elastic.co/t/how-to-audit-search-on-an-elasticsearch-indexes/298754)
**Author:** [@chriselk](https://discuss.elastic.co/u/chriselk)\ **Replies:** 6\ **Last updated:** [March 4, 2022, 10:50am UTC](https://discuss.elastic.co/t/how-to-audit-search-on-an-elasticsearch-indexes/298754 "2022-03-04T10:50:52Z")
We would like to have report on what type of search we have on some Elastic index, I cannot find the search in the servers logs, not sure how we could audit the search request done on an index ? kind of 10 most field /… --- ## [Compare fields values in two different documents](https://discuss.elastic.co/t/compare-fields-values-in-two-different-documents/298833)
**Author:** [@elasticity2](https://discuss.elastic.co/u/elasticity2)\ **Replies:** 0\ **Last updated:** [March 4, 2022, 10:18am UTC](https://discuss.elastic.co/t/compare-fields-values-in-two-different-documents/298833 "2022-03-04T10:18:46Z")
Hi All, I am trying to achieve the following search based on the data shown in the screenshot. This is just an example to provide context as to what I am trying to do. For a given PID, in this examples (smss.exe) I nee… --- ## [ElasticSearch 2.3 Purge of deleted documents](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492)
**Author:** [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\ **Replies:** 14\ **Last updated:** [March 4, 2022, 9:37am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492 "2022-03-04T09:37:21Z")
I've a legacy Elasticsearch 2.3 index. It is no longer being updated with new entries, instead the only modifications are deletes. All our new data targets an Elasticsearch 7 cluster. Though we do still read from the ES2… --- ## [Error writing registrar state to statestore: failed in store/get operation on store 'filebeat': rename /var/lib/filebeat/registry/filebeat/checkpoint.new /var/lib/filebeat/registry/filebeat/4102615046.json: no such file or directory](https://discuss.elastic.co/t/error-writing-registrar-state-to-statestore-failed-in-store-get-operation-on-store-filebeat-rename-var-lib-filebeat-registry-filebeat-checkpoint-new-var-lib-filebeat-registry-filebeat-4102615046-json-no-such-file-or-directory/298825)
**Author:** [@lucy0108](https://discuss.elastic.co/u/lucy0108)\ **Replies:** 0\ **Last updated:** [March 4, 2022, 7:36am UTC](https://discuss.elastic.co/t/error-writing-registrar-state-to-statestore-failed-in-store-get-operation-on-store-filebeat-rename-var-lib-filebeat-registry-filebeat-checkpoint-new-var-lib-filebeat-registry-filebeat-4102615046-json-no-such-file-or-directory/298825 "2022-03-04T07:36:22Z")
Hi , I see error as below while running filebeat. 2022-03-03T02:04:20.142Z ERROR \[registrar\] registrar/registrar.go:202 Error writing registrar state to statestore: failed in store/get operation on store… --- ## [How to maintain sql\_last\_value for different source queries](https://discuss.elastic.co/t/how-to-maintain-sql-last-value-for-different-source-queries/298812)
**Author:** [@rbnalluri](https://discuss.elastic.co/u/rbnalluri)\ **Replies:** 0\ **Last updated:** [March 4, 2022, 4:43am UTC](https://discuss.elastic.co/t/how-to-maintain-sql-last-value-for-different-source-queries/298812 "2022-03-04T04:43:31Z")
Hi, I have couple of scripts which reads data from oracle database and creates respective index files on Elasticsearch.I have implemented delta extraction logic by using the inbuilt feature of logstash by using :sql\_las… --- ## [How to change to desc without sorts in es?](https://discuss.elastic.co/t/how-to-change-to-desc-without-sorts-in-es/298801)
**Author:** [@ifree-zm](https://discuss.elastic.co/u/ifree-zm)\ **Replies:** 8\ **Last updated:** [March 4, 2022, 4:40am UTC](https://discuss.elastic.co/t/how-to-change-to-desc-without-sorts-in-es/298801 "2022-03-04T04:40:45Z")
Sort Orderedit The order option can have the following values: asc Sort in ascending order desc Sort in descending order The order defaults to desc when sorting on the \_score , and defaults to asc when sortin… --- ## [ElasticsearchError(error: "Routing Error. The path you have requested is invalid.", status: nil)](https://discuss.elastic.co/t/elasticsearcherror-error-routing-error-the-path-you-have-requested-is-invalid-status-nil/297977)
**Author:** [@tenna.churiki](https://discuss.elastic.co/u/tenna.churiki)\ **Replies:** 14\ **Last updated:** [March 4, 2022, 1:59am UTC](https://discuss.elastic.co/t/elasticsearcherror-error-routing-error-the-path-you-have-requested-is-invalid-status-nil/297977 "2022-03-04T01:59:17Z")
Hi, I am trying to send query with Swift, but it returns error below. Routing Error. The path you have requested is invalid. My code is below (Using ElasticSwift) let host = "myEndPoint" … --- ## [Using Search UI directly with Elasticsearch](https://discuss.elastic.co/t/using-search-ui-directly-with-elasticsearch/298787)
**Author:** [@brad-g](https://discuss.elastic.co/u/brad-g)\ **Replies:** 0\ **Last updated:** [March 3, 2022, 8:03pm UTC](https://discuss.elastic.co/t/using-search-ui-directly-with-elasticsearch/298787 "2022-03-03T20:03:21Z")
I have a React/Django site focused on a search application built with Search UI. Right now I have it set up to query an App Search engine with a small test subset (~1000 documents) of the full dataset (~550,000 documents… --- ## [Elastic Search handle out of order updates](https://discuss.elastic.co/t/elastic-search-handle-out-of-order-updates/298571)
**Author:** [@andy4324](https://discuss.elastic.co/u/andy4324)\ **Replies:** 7\ **Last updated:** [March 3, 2022, 7:44pm UTC](https://discuss.elastic.co/t/elastic-search-handle-out-of-order-updates/298571 "2022-03-03T19:44:01Z")
Hi friends, new to ES and need some help from y'all. How can I handle out of order updates? Some context: We're building a custom monitoring system and planning to send events to ES via Kafka. Producers \> Kafka \> Consume… --- ## [RHEL 8.0, Installing Elasticsearch with dnf](https://discuss.elastic.co/t/rhel-8-0-installing-elasticsearch-with-dnf/298773)
**Author:** [@CoreyCBurton](https://discuss.elastic.co/u/CoreyCBurton)\ **Replies:** 1\ **Last updated:** [March 3, 2022, 7:42pm UTC](https://discuss.elastic.co/t/rhel-8-0-installing-elasticsearch-with-dnf/298773 "2022-03-03T19:42:06Z")
Hello, I am installing Elasticsearch using the command sudo dnf install --enablerepo=Elasticsearch Elasticsearch Is there any way to install this with security disabled? --- ## [ES 7.17 | Exponentially growing query cache](https://discuss.elastic.co/t/es-7-17-exponentially-growing-query-cache/298784)
**Author:** [@shweta.c](https://discuss.elastic.co/u/shweta.c)\ **Replies:** 0\ **Last updated:** [March 3, 2022, 7:29pm UTC](https://discuss.elastic.co/t/es-7-17-exponentially-growing-query-cache/298784 "2022-03-03T19:29:42Z")
We recently migrated our Elastic cluster from version 7.7.1 to 7.17.0, and have been noticing that the query cache keeps growing exponentially if indexing is running on the nodes simultaneously with reasonably high searc… --- ## [Moving Indices from Data Folder](https://discuss.elastic.co/t/moving-indices-from-data-folder/298768)
**Author:** [@Tim\_Mobley](https://discuss.elastic.co/u/Tim_Mobley)\ **Replies:** 3\ **Last updated:** [March 3, 2022, 5:57pm UTC](https://discuss.elastic.co/t/moving-indices-from-data-folder/298768 "2022-03-03T17:57:53Z")
If I wanted to quickly move indices from one node to another, would I break anything if I just cut/pasted the indices files under %ES\_HOME%\\data from one node to another? ...as opposed to using the \_cluster/reroute API, … --- ## [BulkProcessor for new Java API Client](https://discuss.elastic.co/t/bulkprocessor-for-new-java-api-client/298628)
**Author:** [@UglyBarnacle](https://discuss.elastic.co/u/UglyBarnacle)\ **Replies:** 2\ **Last updated:** [March 3, 2022, 3:30pm UTC](https://discuss.elastic.co/t/bulkprocessor-for-new-java-api-client/298628 "2022-03-03T15:30:34Z")
Hi, Currently we are using es7.17.x in our environment and now evaluating what is necessary to upgrade to the new es8. So far we slowly started replacing the RestHighLevelClient with the new JavaAPI Client and then ide… --- ## [What is the \`index.number\_of\_routing\_shards\` setting? How can I calculate it based on the number of shards?](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540)
**Author:** [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\ **Replies:** 7\ **Last updated:** [March 3, 2022, 3:27pm UTC](https://discuss.elastic.co/t/what-is-the-index-number-of-routing-shards-setting-how-can-i-calculate-it-based-on-the-number-of-shards/297540 "2022-03-03T15:27:41Z")
I'm aware of a similar question to this topic. Namely What are "number\_of\_routing\_shards"? - #2 by steffens. But it does not answer my question or help me understand the index.number\_of\_routing\_shards setting. I'm hopin… --- ## [Security issue for indices shrunken by ILM?](https://discuss.elastic.co/t/security-issue-for-indices-shrunken-by-ilm/298616)
**Author:** [@tomhe](https://discuss.elastic.co/u/tomhe)\ **Replies:** 6\ **Last updated:** [March 3, 2022, 2:47pm UTC](https://discuss.elastic.co/t/security-issue-for-indices-shrunken-by-ilm/298616 "2022-03-03T14:47:05Z")
When using an ILM to shrink an index, the shrunken index gets a new name on the format shrink-\-\, e.g. shrink-j6-m-my-index (for index my-index). In order to preserve access to both the … [Previous page](https://discuss.elastic.co/c/elastic-stack/elasticsearch/6.md?page=480) [Next page](https://discuss.elastic.co/c/elastic-stack/elasticsearch/6.md?page=482)