# Latest **URL:** https://discuss.elastic.co/latest.md?page=593 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 594 --- ## [Exception while attempting Migration from AWS OSS 1.3 to Elasticsearch 7.17](https://discuss.elastic.co/t/exception-while-attempting-migration-from-aws-oss-1-3-to-elasticsearch-7-17/337797)
**Author:** [@gaurav\_jain](https://discuss.elastic.co/u/gaurav_jain)\ **Replies:** 6\ **Last updated:** [July 25, 2023, 12:29pm UTC](https://discuss.elastic.co/t/exception-while-attempting-migration-from-aws-oss-1-3-to-elasticsearch-7-17/337797 "2023-07-25T12:29:12Z")
Hello. We are planning to migrate existing elastic cluster from AWS opensearch service 1.1 to Elastic Cloud: 7.17 with some indexes of ~50GB in size First, I launched an elastic cloud cluster : 7.17. Data migration e… --- ## [How to read indexed binary field data from doc values (in custom Query plugin)](https://discuss.elastic.co/t/how-to-read-indexed-binary-field-data-from-doc-values-in-custom-query-plugin/339387)
**Author:** [@Pyppe](https://discuss.elastic.co/u/Pyppe)\ **Replies:** 0\ **Last updated:** [July 27, 2023, 6:19am UTC](https://discuss.elastic.co/t/how-to-read-indexed-binary-field-data-from-doc-values-in-custom-query-plugin/339387 "2023-07-27T06:19:40Z")
Hi! We're trying to write a custom query-plugin for Elasticsearch where we would use binary data for calculating scores (disclaimer: I've never written one before). Each document can have multiple vectors, so we cannot … --- ## [Extract fields from a field and add the total count](https://discuss.elastic.co/t/extract-fields-from-a-field-and-add-the-total-count/339386)
**Author:** [@joshuskarki](https://discuss.elastic.co/u/joshuskarki)\ **Replies:** 0\ **Last updated:** [July 27, 2023, 6:09am UTC](https://discuss.elastic.co/t/extract-fields-from-a-field-and-add-the-total-count/339386 "2023-07-27T06:09:12Z")
I have this logs (json format) imported into elastic via logstash. "fields.models": "{'msp': '1', 'tcl': '1'}", with logstash, how do we extract the model name and calculate the total count The desired output should a… --- ## [Hardware Requiremenr](https://discuss.elastic.co/t/hardware-requiremenr/339383)
**Author:** [@umangpatel](https://discuss.elastic.co/u/umangpatel)\ **Replies:** 0\ **Last updated:** [July 27, 2023, 5:51am UTC](https://discuss.elastic.co/t/hardware-requiremenr/339383 "2023-07-27T05:51:06Z")
Hello There!!! I have one question about Elasticsearch hardware requiremnet. So let's say if i want to setup Elasticsearch cluster in on-premises data center and i have daily 20 TB of data is ingress or i would say inge… --- ## [Watcher alert status](https://discuss.elastic.co/t/watcher-alert-status/339374)
**Author:** [@jaja](https://discuss.elastic.co/u/jaja)\ **Replies:** 0\ **Last updated:** [July 27, 2023, 5:16am UTC](https://discuss.elastic.co/t/watcher-alert-status/339374 "2023-07-27T05:16:10Z")
Hi Team, We have tried with the watcher it worked for us but we can't go as in watcher the alert status does not change like as in alert we have active , recover options we have. I tried with Index Threshold alert but … --- ## [Elastic Agent GUI Installation](https://discuss.elastic.co/t/elastic-agent-gui-installation/334780)
**Author:** [@xqaiviwjxzw](https://discuss.elastic.co/u/xqaiviwjxzw)\ **Replies:** 3\ **Last updated:** [May 31, 2023, 10:50pm UTC](https://discuss.elastic.co/t/elastic-agent-gui-installation/334780 "2023-05-31T22:50:56Z")
Elastic Will the elastic agent be installed in a way that supports gui in the future? --- ## [Will the snapshot repository able to capture the changes in mappings?](https://discuss.elastic.co/t/will-the-snapshot-repository-able-to-capture-the-changes-in-mappings/339244)
**Author:** [@ian.chan](https://discuss.elastic.co/u/ian.chan)\ **Replies:** 6\ **Last updated:** [July 27, 2023, 5:19am UTC](https://discuss.elastic.co/t/will-the-snapshot-repository-able-to-capture-the-changes-in-mappings/339244 "2023-07-27T05:19:09Z")
Hi. Let's say I have registered a snapshot repository for an index A, with slm policy taking snapshot every hour. Then I add a new field in the mapping of the index A, and add some new documents with values in this new… --- ## [RefreshPolicy WAIT\_UNTIL does not work when using BulkProcessor in Java Client](https://discuss.elastic.co/t/refreshpolicy-wait-until-does-not-work-when-using-bulkprocessor-in-java-client/337857)
**Author:** [@ChiMu\_Yuan](https://discuss.elastic.co/u/ChiMu_Yuan)\ **Replies:** 5\ **Last updated:** [July 27, 2023, 4:53am UTC](https://discuss.elastic.co/t/refreshpolicy-wait-until-does-not-work-when-using-bulkprocessor-in-java-client/337857 "2023-07-27T04:53:47Z")
Hello everyone, I am a beginner, and my English is not very good. I am using ES 7.10 with the Java programming language, so I am using the Java High-Level REST Client. I want to be able to search for relevant content i… --- ## [The indexing or search request send to down node](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022)
**Author:** [@Chimu](https://discuss.elastic.co/u/Chimu)\ **Replies:** 10\ **Last updated:** [July 27, 2023, 3:35am UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022 "2023-07-27T03:35:03Z")
I have an Elasticsearch (v5.6.10) cluster with 3 nodes. Node A : Master Node B : Master + Data Node C : Master + Data There are 6 shards per data node with replication set as 1. All 6 primary nodes are in Node B and a… --- ## [Rollup job is not working if page size is greater than 65000](https://discuss.elastic.co/t/rollup-job-is-not-working-if-page-size-is-greater-than-65000/339367)
**Author:** [@akhil\_reddy](https://discuss.elastic.co/u/akhil_reddy)\ **Replies:** 0\ **Last updated:** [July 27, 2023, 3:08am UTC](https://discuss.elastic.co/t/rollup-job-is-not-working-if-page-size-is-greater-than-65000/339367 "2023-07-27T03:08:48Z")
Hi, I am facing an issue with rollup jobs. If I give the page size as 100000 it is not working and if I give the size as 65000 it works. I tried with different numbers like 70000, 50000 - when the number is greater than… --- ## [How to test for indexes NOT being created?](https://discuss.elastic.co/t/how-to-test-for-indexes-not-being-created/339346)
**Author:** [@McJava1967](https://discuss.elastic.co/u/McJava1967)\ **Replies:** 1\ **Last updated:** [July 27, 2023, 2:08am UTC](https://discuss.elastic.co/t/how-to-test-for-indexes-not-being-created/339346 "2023-07-27T02:08:27Z")
Hi all. My ELK should be receiving data regularly, and creating a new index daily. Is there any way to automatically test if either of those is NOT happening? --- ## [Physicals host with beats to server with ELK docker containers?](https://discuss.elastic.co/t/physicals-host-with-beats-to-server-with-elk-docker-containers/339352)
**Author:** [@rhyejam](https://discuss.elastic.co/u/rhyejam)\ **Replies:** 1\ **Last updated:** [July 27, 2023, 1:48am UTC](https://discuss.elastic.co/t/physicals-host-with-beats-to-server-with-elk-docker-containers/339352 "2023-07-27T01:48:00Z")
So here’s my conundrum. Currently using a vm with a bunch of docker containers on it. Included in these is the ELK docker compose by deviantony on GitHub Now I have a few laptops that I want forwarding logs to the serve… --- ## [Struggling with '-' into field as value](https://discuss.elastic.co/t/struggling-with-into-field-as-value/339230)
**Author:** [@yquirion](https://discuss.elastic.co/u/yquirion)\ **Replies:** 3\ **Last updated:** [July 26, 2023, 10:51pm UTC](https://discuss.elastic.co/t/struggling-with-into-field-as-value/339230 "2023-07-26T22:51:09Z")
Greetings, For very long time, I'm struggling with those errors into my logstash server: \[2023-07-25T17:13:15,009\]\[WARN \]\[logstash.outputs.elasticsearch\]\[5555\_winlogbeat\]\[413af53fed5d62fe27389e3c6e0cc4781e6d3cffc048c8a… --- ## [Winlogbeat 8.8.2 is not sending events to any pipeline](https://discuss.elastic.co/t/winlogbeat-8-8-2-is-not-sending-events-to-any-pipeline/339349)
**Author:** [@pctrindade](https://discuss.elastic.co/u/pctrindade)\ **Replies:** 1\ **Last updated:** [July 26, 2023, 9:11pm UTC](https://discuss.elastic.co/t/winlogbeat-8-8-2-is-not-sending-events-to-any-pipeline/339349 "2023-07-26T21:11:56Z")
I am currently indexing the Windows Security log, and the events are being sent to Elasticsearch and successfully indexed. However, if I do not specify the pipeline named 'winlogbeat-8.8.2-security' in the output, the ev… --- ## [ELK storage on prem](https://discuss.elastic.co/t/elk-storage-on-prem/339348)
**Author:** [@carl56846453](https://discuss.elastic.co/u/carl56846453)\ **Replies:** 0\ **Last updated:** [July 26, 2023, 9:03pm UTC](https://discuss.elastic.co/t/elk-storage-on-prem/339348 "2023-07-26T21:03:54Z")
ELK is not storing much log data. The log seem to keep turning over. ELK is consuming a lot of syslog data. how do I increase the storge of data. --- ## [ECE fundamentals lab 3 step 7](https://discuss.elastic.co/t/ece-fundamentals-lab-3-step-7/339227)
**Author:** [@mbowman](https://discuss.elastic.co/u/mbowman)\ **Replies:** 3\ **Last updated:** [July 26, 2023, 8:15pm UTC](https://discuss.elastic.co/t/ece-fundamentals-lab-3-step-7/339227 "2023-07-26T20:15:54Z")
Course: ECE Fundamentals Version: E-E04NO1 Question: On lab 3, step 7, it's wanting me to test the accessibility of the API with the "elastic" user. The solution states to use this: https://:9243 I have tried the i… --- ## [Post data to elasticsearch sometimes throws error 429](https://discuss.elastic.co/t/post-data-to-elasticsearch-sometimes-throws-error-429/339345)
**Author:** [@111407](https://discuss.elastic.co/u/111407)\ **Replies:** 1\ **Last updated:** [July 26, 2023, 7:37pm UTC](https://discuss.elastic.co/t/post-data-to-elasticsearch-sometimes-throws-error-429/339345 "2023-07-26T19:37:00Z")
command: curl -u username:pw -X POST 'http://sd-4531-6c55:9200/testindex/\_doc' -H 'Content-type: application/json' -d '{"test":11234}' response: {"error":{"root\_cause":\[{"type":"remote\_transport\_exception","reason":"\[… --- ## [Kibana is not working](https://discuss.elastic.co/t/kibana-is-not-working/338991)
**Author:** [@Miguel2](https://discuss.elastic.co/u/Miguel2)\ **Replies:** 11\ **Last updated:** [July 26, 2023, 7:29pm UTC](https://discuss.elastic.co/t/kibana-is-not-working/338991 "2023-07-26T19:29:34Z")
Hello I'm trying to learn the ELK stack from scratch, I'm currently having problems with kibana not getting active as shown below: myuser@myuser-pc:~$ systemctl status kibana × kibana.service - Kibana Loaded: loade… --- ## [Elasticsearch REST API Authorization](https://discuss.elastic.co/t/elasticsearch-rest-api-authorization/339332)
**Author:** [@ksobon](https://discuss.elastic.co/u/ksobon)\ **Replies:** 2\ **Last updated:** [July 26, 2023, 7:16pm UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-authorization/339332 "2023-07-26T19:16:26Z")
OK, so I tried using the Elastic.Client.Elasticsearch library to get an index template, but it had some JSON serialization issues that was causing an exception. Next up, I tried using just a regular REST call. I have cr… --- ## [Kibana Error Unable to revive connection](https://discuss.elastic.co/t/kibana-error-unable-to-revive-connection/339344)
**Author:** [@fjcd1990](https://discuss.elastic.co/u/fjcd1990)\ **Replies:** 0\ **Last updated:** [July 26, 2023, 7:08pm UTC](https://discuss.elastic.co/t/kibana-error-unable-to-revive-connection/339344 "2023-07-26T19:08:19Z")
hi everyone i need help with this error in my kibana service, because the credencial SSL TLS has expired and i don't know how to use the elastic elasticsearch-certutil. "path" : "/usr/share/elasticsearch/config/http-ce… --- ## [GetIndexTemplate() call throws an exception](https://discuss.elastic.co/t/getindextemplate-call-throws-an-exception/339329)
**Author:** [@ksobon](https://discuss.elastic.co/u/ksobon)\ **Replies:** 2\ **Last updated:** [July 26, 2023, 6:25pm UTC](https://discuss.elastic.co/t/getindextemplate-call-throws-an-exception/339329 "2023-07-26T18:25:47Z")
I am trying to get an index template using the .NET APIs via Elastic.Clients.Elasticsearch and Elastic.Transport libraries. I got my client setup like this: var credentials = new BasicAuthentication(elasticUsername, el… --- ## [Logstash pipeline is getting killed with jnr.enxio.channels.NativeException: Error closing fd 272: Stale file handle"](https://discuss.elastic.co/t/logstash-pipeline-is-getting-killed-with-jnr-enxio-channels-nativeexception-error-closing-fd-272-stale-file-handle/339328)
**Author:** [@jayanthi\_c](https://discuss.elastic.co/u/jayanthi_c)\ **Replies:** 2\ **Last updated:** [July 26, 2023, 6:23pm UTC](https://discuss.elastic.co/t/logstash-pipeline-is-getting-killed-with-jnr-enxio-channels-nativeexception-error-closing-fd-272-stale-file-handle/339328 "2023-07-26T18:23:59Z")
I am using filebeat to transfer log to logstash but we see that pipeline is getting killed with the below error jnr.enxio.channels.NativeException: Error closing fd 272: Stale file handle" Can someone please help --- ## [How to view the backend log of the Python class \`Elasticsearch\`?](https://discuss.elastic.co/t/how-to-view-the-backend-log-of-the-python-class-elasticsearch/339133)
**Author:** [@Mike\_Z](https://discuss.elastic.co/u/Mike_Z)\ **Replies:** 1\ **Last updated:** [July 26, 2023, 6:21pm UTC](https://discuss.elastic.co/t/how-to-view-the-backend-log-of-the-python-class-elasticsearch/339133 "2023-07-26T18:21:22Z")
We are using an instance of the Python class Elasticsearch, e.g., by es = Elasticsearch(hosts="http://test-elastic-host:9200"). For example, when calling the search() method, we need to know what exactly the request is,… --- ## [I want to know why the indices.id\_field\_data.enabled configuration is turned off by default](https://discuss.elastic.co/t/i-want-to-know-why-the-indices-id-field-data-enabled-configuration-is-turned-off-by-default/339338)
**Author:** [@Kurt\_Rudolph](https://discuss.elastic.co/u/Kurt_Rudolph)\ **Replies:** 1\ **Last updated:** [July 26, 2023, 5:50pm UTC](https://discuss.elastic.co/t/i-want-to-know-why-the-indices-id-field-data-enabled-configuration-is-turned-off-by-default/339338 "2023-07-26T17:50:36Z")
This topic got automatically closed without an answer I want to know why the indices.id\_field\_data.enabled configuration is turned off by default I'm evaluating the impacts of upgrading from v7 -\> v8 and found an issue… --- ## [How can i send logs from Elastic to another SIEM?](https://discuss.elastic.co/t/how-can-i-send-logs-from-elastic-to-another-siem/339154)
**Author:** [@Ck1f](https://discuss.elastic.co/u/Ck1f)\ **Replies:** 7\ **Last updated:** [July 26, 2023, 5:39pm UTC](https://discuss.elastic.co/t/how-can-i-send-logs-from-elastic-to-another-siem/339154 "2023-07-26T17:39:54Z")
Good morning, We have installed elastic-agents throughout our environment with everything setup for filebeat and metricbeat monitoring. And now i'm trying to understand how can i send logs to another SIEM? --- ## [Logstash stops processing syslog messages when DNS server not available](https://discuss.elastic.co/t/logstash-stops-processing-syslog-messages-when-dns-server-not-available/339334)
**Author:** [@RJC](https://discuss.elastic.co/u/RJC)\ **Replies:** 0\ **Last updated:** [July 26, 2023, 5:07pm UTC](https://discuss.elastic.co/t/logstash-stops-processing-syslog-messages-when-dns-server-not-available/339334 "2023-07-26T17:07:09Z")
Running Logstash 8.5.2 on RHEL. I implemented DNS filter plugin to resolve IP addresses to hostnames for all syslog nodes reporting to this logstash server. I am using our local DNS server. It all worked perfectly unti… --- ## [Data duplication problem after server migration](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186)
**Author:** [@charlielin](https://discuss.elastic.co/u/charlielin)\ **Replies:** 8\ **Last updated:** [July 26, 2023, 4:41pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186 "2023-07-26T16:41:44Z")
Hi There: Currently, we are using Filebeat (version 7.15.2) to harvest logs of program A and send them to Kafka. Due to some reason, we will do some operations called Server Migration freqently. Server Migration means… --- ## [Vega Sankey Diagram](https://discuss.elastic.co/t/vega-sankey-diagram/338149)
**Author:** [@Ranger\_Rick](https://discuss.elastic.co/u/Ranger_Rick)\ **Replies:** 4\ **Last updated:** [July 26, 2023, 4:09pm UTC](https://discuss.elastic.co/t/vega-sankey-diagram/338149 "2023-07-26T16:09:34Z")
Good afternoon! I created some Sankeys to better visualize certain relationships and they mostly work well. Following the example provided here: Sankey Creation resulting in clean graphs but with a slight issue. The da… --- ## [How to create a Security Rule (SIEM) for Custom Logs Integration](https://discuss.elastic.co/t/how-to-create-a-security-rule-siem-for-custom-logs-integration/339327)
**Author:** [@Nishant\_Chauhan](https://discuss.elastic.co/u/Nishant_Chauhan)\ **Replies:** 0\ **Last updated:** [July 26, 2023, 3:51pm UTC](https://discuss.elastic.co/t/how-to-create-a-security-rule-siem-for-custom-logs-integration/339327 "2023-07-26T15:51:29Z")
Hi Team, I have setup Custom Logs Integration and able to create rules for observability. but rules are not working for Security dashboard. sample log 2023-07-26T08:05:25.661Z ERRO 1 --- \[nio-8080-exec-3\] c.i.c.b.c.H… --- ## [Alerts are not triggering in Kibana 7.17.11](https://discuss.elastic.co/t/alerts-are-not-triggering-in-kibana-7-17-11/339285)
**Author:** [@malak](https://discuss.elastic.co/u/malak)\ **Replies:** 1\ **Last updated:** [July 26, 2023, 3:29pm UTC](https://discuss.elastic.co/t/alerts-are-not-triggering-in-kibana-7-17-11/339285 "2023-07-26T15:29:27Z")
Hello, I have a basic license where I create a Threat matching rule. Previewing rule is showing the expected result however, the rule is not triggering. Any idea? [Previous page](https://discuss.elastic.co/latest.md?page=592) [Next page](https://discuss.elastic.co/latest.md?page=594)