# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=538 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 539 --- ## [Sorting by max value of property of nested object array](https://discuss.elastic.co/t/sorting-by-max-value-of-property-of-nested-object-array/333778)
**Author:** [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\ **Replies:** 1\ **Last updated:** [May 18, 2023, 5:20pm UTC](https://discuss.elastic.co/t/sorting-by-max-value-of-property-of-nested-object-array/333778 "2023-05-18T17:20:01Z")
I have the following object model: { ... "classification": \[ { "label": "aaa", "probability": 0.9923 }, { "label": "bbb", "probability": 0.3452 }, { "label": "ccc", "probability": 0.0012 … --- ## [Cannot remove or resolve metric alerts](https://discuss.elastic.co/t/cannot-remove-or-resolve-metric-alerts/333783)
**Author:** [@cfqnjohn](https://discuss.elastic.co/u/cfqnjohn)\ **Replies:** 0\ **Last updated:** [May 18, 2023, 3:44pm UTC](https://discuss.elastic.co/t/cannot-remove-or-resolve-metric-alerts/333783 "2023-05-18T15:44:22Z")
Hello, I have some alerts that were created for Disk Usage alerting. It seems that they triggered awhile back on April 19th, and have not updated since. The alert threshold was modified to 80% originally it was set to 7… --- ## [Change the index allocation requirement](https://discuss.elastic.co/t/change-the-index-allocation-requirement/333781)
**Author:** [@mpniel](https://discuss.elastic.co/u/mpniel)\ **Replies:** 0\ **Last updated:** [May 18, 2023, 3:39pm UTC](https://discuss.elastic.co/t/change-the-index-allocation-requirement/333781 "2023-05-18T15:39:59Z")
How to change the index and remove the allocation requirement "cold" ? --- ## [I am sending 30 GB data from databricks to elasticsearch through the elasticsearch apache hadoop connector, It is taking around 2 hours for sending it. How to make it fast? How much time it should take ideally?](https://discuss.elastic.co/t/i-am-sending-30-gb-data-from-databricks-to-elasticsearch-through-the-elasticsearch-apache-hadoop-connector-it-is-taking-around-2-hours-for-sending-it-how-to-make-it-fast-how-much-time-it-should-take-ideally/333715)
**Author:** [@Sagnik\_Mandal](https://discuss.elastic.co/u/Sagnik_Mandal)\ **Replies:** 6\ **Last updated:** [May 18, 2023, 2:40pm UTC](https://discuss.elastic.co/t/i-am-sending-30-gb-data-from-databricks-to-elasticsearch-through-the-elasticsearch-apache-hadoop-connector-it-is-taking-around-2-hours-for-sending-it-how-to-make-it-fast-how-much-time-it-should-take-ideally/333715 "2023-05-18T14:40:17Z")
My elasticsearch connector configs are: .option("es.write.operation.parallelism", "4") .option("es.batch.size.bytes", "10mb") .option("es.batch.size.entries", "1000") .option("es.batch.write.retry.coun… --- ## [Referencing canvas variables with outputs of other variables](https://discuss.elastic.co/t/referencing-canvas-variables-with-outputs-of-other-variables/333775)
**Author:** [@dreynolds](https://discuss.elastic.co/u/dreynolds)\ **Replies:** 0\ **Last updated:** [May 18, 2023, 1:51pm UTC](https://discuss.elastic.co/t/referencing-canvas-variables-with-outputs-of-other-variables/333775 "2023-05-18T13:51:24Z")
I have multiple variables defined that have the number of hosts a department should have: hr\_hosts: 35, it\_hosts:45, acct\_hosts:5 I'm running a query to pull host scans, which includes the department names as a column … --- ## [Reindex 1 index to multiple indexes](https://discuss.elastic.co/t/reindex-1-index-to-multiple-indexes/333667)
**Author:** [@elasticvakif](https://discuss.elastic.co/u/elasticvakif)\ **Replies:** 7\ **Last updated:** [May 18, 2023, 12:07pm UTC](https://discuss.elastic.co/t/reindex-1-index-to-multiple-indexes/333667 "2023-05-18T12:07:20Z")
We have an index which is around 120 gb. we want to split it multiple indices. Is there any way to do that ? I guess reindex supports 1 to 1. I need 1 to many. It doesn't matter which document is in which index. We can u… --- ## [Hiding non-indexed/ non-searchable fields in "Available fields" section in Kibana discover](https://discuss.elastic.co/t/hiding-non-indexed-non-searchable-fields-in-available-fields-section-in-kibana-discover/333723)
**Author:** [@Saindra\_K](https://discuss.elastic.co/u/Saindra_K)\ **Replies:** 1\ **Last updated:** [May 18, 2023, 10:50am UTC](https://discuss.elastic.co/t/hiding-non-indexed-non-searchable-fields-in-available-fields-section-in-kibana-discover/333723 "2023-05-18T10:50:45Z")
In Kibana discover is there a way to automatically hide the non indexed fields/ non searchable fields in "Available fields" ? I see in available fields setting, by filtering Searchable to Yes we can hide, but this is no… --- ## [Cross Cluster Replication - Dev Environment](https://discuss.elastic.co/t/cross-cluster-replication-dev-environment/333758)
**Author:** [@to185030](https://discuss.elastic.co/u/to185030)\ **Replies:** 0\ **Last updated:** [May 18, 2023, 10:47am UTC](https://discuss.elastic.co/t/cross-cluster-replication-dev-environment/333758 "2023-05-18T10:47:43Z")
Can someone tell me if it is possible to setup CCR on Elasticsearch for my application in a Dev environment - for a very limited window of time, without having to bear the expenses for the licenses of all the Platinum li… --- ## [Frozen tier - Conenience in multiple zones](https://discuss.elastic.co/t/frozen-tier-conenience-in-multiple-zones/333475)
**Author:** [@Alberallo](https://discuss.elastic.co/u/Alberallo)\ **Replies:** 8\ **Last updated:** [May 18, 2023, 9:48am UTC](https://discuss.elastic.co/t/frozen-tier-conenience-in-multiple-zones/333475 "2023-05-18T09:48:42Z")
HI, since high availability is guaranteed by default for frozen nodes, why should there be any convenience in configuring a cluster with more than one zone for the frozen tier? In particular, during the execution of th… --- ## [Failed to parse date field with format strict\_date\_optional\_time||epoch\_millis](https://discuss.elastic.co/t/failed-to-parse-date-field-with-format-strict-date-optional-time-epoch-millis/333734)
**Author:** [@Sachinda](https://discuss.elastic.co/u/Sachinda)\ **Replies:** 1\ **Last updated:** [May 18, 2023, 9:08am UTC](https://discuss.elastic.co/t/failed-to-parse-date-field-with-format-strict-date-optional-time-epoch-millis/333734 "2023-05-18T09:08:02Z")
Hi All, We are observing the following error in the Logstash serves only for the 2023.05.09 logs. This issue is not occurring in other date indexes. so we can see the 2023.05.08 and 2023.05.10 logs are available on the … --- ## [Refresh API taking too long](https://discuss.elastic.co/t/refresh-api-taking-too-long/333562)
**Author:** [@blacar](https://discuss.elastic.co/u/blacar)\ **Replies:** 4\ **Last updated:** [May 18, 2023, 8:56am UTC](https://discuss.elastic.co/t/refresh-api-taking-too-long/333562 "2023-05-18T08:56:24Z")
Hi folks, I am having some 409 - versioning conflict problems when using deleteByQuery so I decided to run a POST /\_refresh when I receive a 409 and just before trying again. The rate of operations recovery using this … --- ## [Bulk insert in elasticsearch datastream using elasticsearch-java 8.6.2](https://discuss.elastic.co/t/bulk-insert-in-elasticsearch-datastream-using-elasticsearch-java-8-6-2/333744)
**Author:** [@ayanarora0101](https://discuss.elastic.co/u/ayanarora0101)\ **Replies:** 0\ **Last updated:** [May 18, 2023, 8:54am UTC](https://discuss.elastic.co/t/bulk-insert-in-elasticsearch-datastream-using-elasticsearch-java-8-6-2/333744 "2023-05-18T08:54:55Z")
I'm trying to bulk insert documents in datastream in elasticsearch using java with (ElasticsearchClient) (elasticsearch-java) 8.6.2. I checked documentation of elasticsearch-java and found information around bulk indexi… --- ## [Java - not able to load FFI provider: How to start the logstash without the error?](https://discuss.elastic.co/t/java-not-able-to-load-ffi-provider-how-to-start-the-logstash-without-the-error/332788)
**Author:** [@karthic](https://discuss.elastic.co/u/karthic)\ **Replies:** 3\ **Last updated:** [May 18, 2023, 7:33am UTC](https://discuss.elastic.co/t/java-not-able-to-load-ffi-provider-how-to-start-the-logstash-without-the-error/332788 "2023-05-18T07:33:35Z")
Tried to load logstash in a Centos environment \[INFO \]\[logstash.runner \] JVM bootstrap flags: \[-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMem… --- ## [Is there a limit for number of routing values](https://discuss.elastic.co/t/is-there-a-limit-for-number-of-routing-values/333719)
**Author:** [@alper](https://discuss.elastic.co/u/alper)\ **Replies:** 1\ **Last updated:** [May 18, 2023, 7:01am UTC](https://discuss.elastic.co/t/is-there-a-limit-for-number-of-routing-values/333719 "2023-05-18T07:01:14Z")
Hi, The index contains one routing field and has 50 shards. I use the routing field in search requests. I do, however, wonder if there would be a performance problem if I sent 1000 routing values in a single query. Sho… --- ## [Filebeat kubernetes autodiscovery per namespace & kibana missing beats](https://discuss.elastic.co/t/filebeat-kubernetes-autodiscovery-per-namespace-kibana-missing-beats/333722)
**Author:** [@bdols](https://discuss.elastic.co/u/bdols)\ **Replies:** 1\ **Last updated:** [May 18, 2023, 6:26am UTC](https://discuss.elastic.co/t/filebeat-kubernetes-autodiscovery-per-namespace-kibana-missing-beats/333722 "2023-05-18T06:26:44Z")
I've used this as a starting point to get ECK up and running: github/elastic/cloud-on-k8s/2.7/config/recipes/beats/stack\_monitoring.yaml I just want metrics and logs collected for elastic in one namespace, and the file… --- ## [Logstash date parse failure - ruby exception](https://discuss.elastic.co/t/logstash-date-parse-failure-ruby-exception/333710)
**Author:** [@sai\_ravi\_shankar](https://discuss.elastic.co/u/sai_ravi_shankar)\ **Replies:** 2\ **Last updated:** [May 18, 2023, 6:11am UTC](https://discuss.elastic.co/t/logstash-date-parse-failure-ruby-exception/333710 "2023-05-18T06:11:49Z")
Hi, I am trying to use timestamp for each document by the value present in file name but i am getting Ruby exception occurred: wrong argument type DateTime (expected LogStash::Timestamp) when i run ruby code. it is wo… --- ## [Linux client data not visible on ELK server after](https://discuss.elastic.co/t/linux-client-data-not-visible-on-elk-server-after/333521)
**Author:** [@jg23](https://discuss.elastic.co/u/jg23)\ **Replies:** 4\ **Last updated:** [May 18, 2023, 6:07am UTC](https://discuss.elastic.co/t/linux-client-data-not-visible-on-elk-server-after/333521 "2023-05-18T06:07:13Z")
Hi, I recently installed the ELK stack on a Linux server running Ubuntu 22.04 using the following as a guide: After the initial installation and setup, I've also been able to successfully send logs from 4 other linux … --- ## [Elastic shard balancing / allocation](https://discuss.elastic.co/t/elastic-shard-balancing-allocation/333713)
**Author:** [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\ **Replies:** 0\ **Last updated:** [May 18, 2023, 4:54am UTC](https://discuss.elastic.co/t/elastic-shard-balancing-allocation/333713 "2023-05-18T04:54:50Z")
Hi We are on Elastic 8.6 with 38 hot data nodes and ingesting about 140 different indices Top 10 indices have indexing rate about 15-50K events/sec. 20 indices has 1-20 K events/sec. And remaining 100 indices indexin… --- ## [New index es not being created after index 'reset' v7.17.9](https://discuss.elastic.co/t/new-index-es-not-being-created-after-index-reset-v7-17-9/333712)
**Author:** [@Bruceclegg](https://discuss.elastic.co/u/Bruceclegg)\ **Replies:** 0\ **Last updated:** [May 18, 2023, 4:33am UTC](https://discuss.elastic.co/t/new-index-es-not-being-created-after-index-reset-v7-17-9/333712 "2023-05-18T04:33:07Z")
I'm working on fixing our onsite elasticsearch 7.17.9 cluster. I've got a small 3 node cluster capturing our production data. Right now all of the logs get loaded into a single index that I use cron and curator to manu… --- ## [Add ILM to existing index 7.17.9](https://discuss.elastic.co/t/add-ilm-to-existing-index-7-17-9/333003)
**Author:** [@Bruceclegg](https://discuss.elastic.co/u/Bruceclegg)\ **Replies:** 6\ **Last updated:** [May 18, 2023, 4:15am UTC](https://discuss.elastic.co/t/add-ilm-to-existing-index-7-17-9/333003 "2023-05-18T04:15:46Z")
I have an existing index I've applied the 30 day default lifecycle management policy to. But it doesn't seem to be working. I've read through the documentation and I/m obviously missing something. The index I'm attemp… --- ## [Elasticsearch Java Client 8.7 String List to FieldAndFormat List](https://discuss.elastic.co/t/elasticsearch-java-client-8-7-string-list-to-fieldandformat-list/333677)
**Author:** [@tcpeiris](https://discuss.elastic.co/u/tcpeiris)\ **Replies:** 2\ **Last updated:** [May 18, 2023, 3:38am UTC](https://discuss.elastic.co/t/elasticsearch-java-client-8-7-string-list-to-fieldandformat-list/333677 "2023-05-18T03:38:43Z")
How do I pass Java String List to .fields as FieldAndFormat in search(req -\> req.index(index).fields()) --- ## [Filebeat on ELK not sending from configured paths](https://discuss.elastic.co/t/filebeat-on-elk-not-sending-from-configured-paths/333706)
**Author:** [@roman-tasi](https://discuss.elastic.co/u/roman-tasi)\ **Replies:** 0\ **Last updated:** [May 18, 2023, 2:34am UTC](https://discuss.elastic.co/t/filebeat-on-elk-not-sending-from-configured-paths/333706 "2023-05-18T02:34:59Z")
In my /etc/filebeat/filebeat.yml I have this set: paths: - /var/log/audit/audit.log - /var/log/secure But in our Kibana we only seem to be receiving from this log.file.path : /var/log/messages Not sure why --- ## [Help with dissect in filter for logstash.conf to dynamically append filename to index patternNotFound Error](https://discuss.elastic.co/t/help-with-dissect-in-filter-for-logstash-conf-to-dynamically-append-filename-to-index-patternnotfound-error/333692)
**Author:** [@fsaa](https://discuss.elastic.co/u/fsaa)\ **Replies:** 2\ **Last updated:** [May 18, 2023, 1:43am UTC](https://discuss.elastic.co/t/help-with-dissect-in-filter-for-logstash-conf-to-dynamically-append-filename-to-index-patternnotfound-error/333692 "2023-05-18T01:43:40Z")
My folder structure is as follows: main\_directory. | .env | docker-compose.yml | +---elasticsearch | \\---config | elasticsearch.yml | \\---logstash +---config | | logstash.yml | | pipe… --- ## [Automatically balance Shard allocation](https://discuss.elastic.co/t/automatically-balance-shard-allocation/333682)
**Author:** [@NishuGoel](https://discuss.elastic.co/u/NishuGoel)\ **Replies:** 1\ **Last updated:** [May 18, 2023, 1:11am UTC](https://discuss.elastic.co/t/automatically-balance-shard-allocation/333682 "2023-05-18T01:11:36Z")
A maintenance applied by "System" today caused a restart of a node in our production cluster The restarted node started reallocating shards, not receiving traffic after 2 hours and the remaining 2 nodes were under huge … --- ## [Is it possible to migrate data from one cluster to another using Snapshot and Restore](https://discuss.elastic.co/t/is-it-possible-to-migrate-data-from-one-cluster-to-another-using-snapshot-and-restore/333694)
**Author:** [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\ **Replies:** 1\ **Last updated:** [May 18, 2023, 1:06am UTC](https://discuss.elastic.co/t/is-it-possible-to-migrate-data-from-one-cluster-to-another-using-snapshot-and-restore/333694 "2023-05-18T01:06:00Z")
Hi Team, I am planning to migrate data from some indices using the snapshot and restore method. Is it possible to snapshot the indices from Cluster1 to one repository and then restore the same snapshot to Cluster2? --- ## [Logstash - Syslog Output - Custom message](https://discuss.elastic.co/t/logstash-syslog-output-custom-message/333588)
**Author:** [@Nandhini\_Viswanathan](https://discuss.elastic.co/u/Nandhini_Viswanathan)\ **Replies:** 0\ **Last updated:** [May 16, 2023, 4:18pm UTC](https://discuss.elastic.co/t/logstash-syslog-output-custom-message/333588 "2023-05-16T16:18:03Z")
Logstash - Syslog Output - Custom message Hi, I'm I working with Logstash - Syslog Output and I've found out problem with setting custom field message. I'm using Elasticstack 7.8.0. I've installed logstash syslog-outp… --- ## [Normalizing Fields](https://discuss.elastic.co/t/normalizing-fields/333544)
**Author:** [@Felkio](https://discuss.elastic.co/u/Felkio)\ **Replies:** 1\ **Last updated:** [May 18, 2023, 12:02am UTC](https://discuss.elastic.co/t/normalizing-fields/333544 "2023-05-18T00:02:30Z")
Hello, we are currently using wazuh in conjunction with ELK stack 7.17.9, we would like to gradually switch to the full elastic stack, but to do this we would first like to normalize the fields that wazuh sends to elast… --- ## [Elastic Heap size calculation not correct with K8S 1.26](https://discuss.elastic.co/t/elastic-heap-size-calculation-not-correct-with-k8s-1-26/333528)
**Author:** [@lineconnect](https://discuss.elastic.co/u/lineconnect)\ **Replies:** 1\ **Last updated:** [May 17, 2023, 11:58pm UTC](https://discuss.elastic.co/t/elastic-heap-size-calculation-not-correct-with-k8s-1-26/333528 "2023-05-17T23:58:03Z")
Hi, we run several elastic clusters(all with 3 nodes). And until now without any issues until we've upgraded vom k8s version 1.24.8. We don't have any JVM options set and the calculation from elastic was always fine. … --- ## [Get source of queries hitting indexes](https://discuss.elastic.co/t/get-source-of-queries-hitting-indexes/333250)
**Author:** [@callumdowling](https://discuss.elastic.co/u/callumdowling)\ **Replies:** 1\ **Last updated:** [May 17, 2023, 11:52pm UTC](https://discuss.elastic.co/t/get-source-of-queries-hitting-indexes/333250 "2023-05-17T23:52:41Z")
Hi all, just wondering if this is possible. We have several tiers in elastic cloud, we would like to see when the frozen tier is being rules/dashboards/queries for our indexes so we can go through and optimise. Is there… --- ## [Not Getting Kubernetes related filters in Kibana when exporting logs from Logstash](https://discuss.elastic.co/t/not-getting-kubernetes-related-filters-in-kibana-when-exporting-logs-from-logstash/333400)
**Author:** [@Akshay04](https://discuss.elastic.co/u/Akshay04)\ **Replies:** 1\ **Last updated:** [May 17, 2023, 11:44pm UTC](https://discuss.elastic.co/t/not-getting-kubernetes-related-filters-in-kibana-when-exporting-logs-from-logstash/333400 "2023-05-17T23:44:02Z")
Hello, I have configured Fluentbit in my k8s cluster to send logs to S3 and ELK stack to get logs from S3 and Visualise in Kibana. When I create Dataview in kibana for the Index, the kibana dashboard is not giving filte… [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=537) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=539)