# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1300 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1301 --- ## [Is it possible to monitor elasticsearch using metricbeat and custom index name?](https://discuss.elastic.co/t/is-it-possible-to-monitor-elasticsearch-using-metricbeat-and-custom-index-name/280084)
**Author:** [@Travis](https://discuss.elastic.co/u/Travis)\ **Replies:** 3\ **Last updated:** [July 31, 2021, 4:44pm UTC](https://discuss.elastic.co/t/is-it-possible-to-monitor-elasticsearch-using-metricbeat-and-custom-index-name/280084 "2021-07-31T16:44:40Z")
Hello ! I use metricbeat on the whole stack members (Elasticsearch/Logstash/Kibana) to collect system metrics (enabled by default) I use logstash output and custom index name metricbeat.yml : # ----------------------… --- ## [Reindex by grouping documents by Unique ID](https://discuss.elastic.co/t/reindex-by-grouping-documents-by-unique-id/280129)
**Author:** [@mrunalini](https://discuss.elastic.co/u/mrunalini)\ **Replies:** 0\ **Last updated:** [July 31, 2021, 3:15pm UTC](https://discuss.elastic.co/t/reindex-by-grouping-documents-by-unique-id/280129 "2021-07-31T15:15:35Z")
Hi Everyone, I want to reindex my Index which has different documents with unique id . I want to reindex all the data into single documents against unique id. is it possible to do that. --- ## [MySQL 8: Could not create connection to database server](https://discuss.elastic.co/t/mysql-8-could-not-create-connection-to-database-server/280064)
**Author:** [@Justin\_Morgan](https://discuss.elastic.co/u/Justin_Morgan)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 11:57am UTC](https://discuss.elastic.co/t/mysql-8-could-not-create-connection-to-database-server/280064 "2021-07-30T11:57:29Z")
After upgrading to MySQL 8 I get the following error from my logstash pod: Loading class \`com.mysql.jdbc.Driver'. This is deprecated. The new driver class is \`com.mysql.cj.jdbc.Driver'. The driver is automatically regis… --- ## [Fluentd performance is degraded](https://discuss.elastic.co/t/fluentd-performance-is-degraded/280060)
**Author:** [@Voula\_Mikr](https://discuss.elastic.co/u/Voula_Mikr)\ **Replies:** 3\ **Last updated:** [July 31, 2021, 6:05am UTC](https://discuss.elastic.co/t/fluentd-performance-is-degraded/280060 "2021-07-31T06:05:31Z")
In a deployment with elasticsearch 8.x, fluentd pod performance has been degraded and following log is printed whenever a new message is received and it is about to be delivered in elastic. warning: 299 Elasticsearch-7.… --- ## [Pre-index data in ingest pipeline](https://discuss.elastic.co/t/pre-index-data-in-ingest-pipeline/280104)
**Author:** [@jjammin92](https://discuss.elastic.co/u/jjammin92)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 10:31pm UTC](https://discuss.elastic.co/t/pre-index-data-in-ingest-pipeline/280104 "2021-07-30T22:31:44Z")
Good afternoon, I have some access style logs with a numeric field "player.score." I'm looking to pre-index these fields to leverage patterns for faster retrieval The values can range anywhere from 0 to 5000 being the… --- ## [Ruby hash to fields of an event](https://discuss.elastic.co/t/ruby-hash-to-fields-of-an-event/280102)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 10:03pm UTC](https://discuss.elastic.co/t/ruby-hash-to-fields-of-an-event/280102 "2021-07-30T22:03:53Z")
Hi I got the following ruby hash after parsing a long string with ruby code, { "mfemve"=\>"0,39 ", "vmtoolsd"=\>"0,39 ", "zabbix\_agentd"=\>"0,39 ", "powershell"=\>"0,39 ", "ir\_agent"=\>"0,00 ", "wmiprvse"=\>"0,00 ", "unsecap… --- ## [AWSFargate module for Filebeat not working?](https://discuss.elastic.co/t/awsfargate-module-for-filebeat-not-working/279949)
**Author:** [@stec00](https://discuss.elastic.co/u/stec00)\ **Replies:** 4\ **Last updated:** [July 30, 2021, 9:19pm UTC](https://discuss.elastic.co/t/awsfargate-module-for-filebeat-not-working/279949 "2021-07-30T21:19:15Z")
I am attempting to use the AWSFargate module for Filebeat. Ran the following commands (with placeholder values substituted): ./filebeat.exe modules enable awsfargate ./filebeat -e -E cloud.id=myElasticCloudID -E cloud… --- ## [How to compare exported objects in .ndjson format from two envirornments](https://discuss.elastic.co/t/how-to-compare-exported-objects-in-ndjson-format-from-two-envirornments/279846)
**Author:** [@Kunal\_Sharma](https://discuss.elastic.co/u/Kunal_Sharma)\ **Replies:** 2\ **Last updated:** [July 30, 2021, 8:19pm UTC](https://discuss.elastic.co/t/how-to-compare-exported-objects-in-ndjson-format-from-two-envirornments/279846 "2021-07-30T20:19:11Z")
Dear Team, We have multiple Kibana stacks (dev, test, prod) which are manually managed and have some differences. We are looking for some quick help on comparing the Kibana objects from two environments and finding the… --- ## [Output to json saving only specific fields](https://discuss.elastic.co/t/output-to-json-saving-only-specific-fields/280098)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 3\ **Last updated:** [July 30, 2021, 7:57pm UTC](https://discuss.elastic.co/t/output-to-json-saving-only-specific-fields/280098 "2021-07-30T19:57:42Z")
Hello everyone I am new logstash and I am sorry if the question is so basic. I have been saving my syslog into a csv file, as follow: csv { path =\> "C:\\path\\Desktop\\Adib\\adib-syslogs-%{+yyyy.MM.dd}.csv" csv\_o… --- ## [Logstash mutate field](https://discuss.elastic.co/t/logstash-mutate-field/280097)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 7:56pm UTC](https://discuss.elastic.co/t/logstash-mutate-field/280097 "2021-07-30T19:56:27Z")
Hi everyone, I hope somebody can help me with this issue I am facing with logstash. I am receiving some syslog, and trying to put in place some filtering and merging together 2 fields. the simple structure that I want… --- ## [Elasticsearch not doing local primary indices recovery after node restart](https://discuss.elastic.co/t/elasticsearch-not-doing-local-primary-indices-recovery-after-node-restart/280101)
**Author:** [@pranav\_sah](https://discuss.elastic.co/u/pranav_sah)\ **Replies:** 0\ **Last updated:** [July 30, 2021, 7:45pm UTC](https://discuss.elastic.co/t/elasticsearch-not-doing-local-primary-indices-recovery-after-node-restart/280101 "2021-07-30T19:45:41Z")
We have a cluster of 6 nodes of which one node was restarted due to some patch activity. Before the node was brought down we had set "cluster.routing.allocation.enable" : "primaries" on persistent level so as to have f… --- ## [How to prevent clobbering in translate filter](https://discuss.elastic.co/t/how-to-prevent-clobbering-in-translate-filter/280008)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 4\ **Last updated:** [July 30, 2021, 4:50pm UTC](https://discuss.elastic.co/t/how-to-prevent-clobbering-in-translate-filter/280008 "2021-07-30T16:50:51Z")
Hello , I'm using translate filter with a set of mappings in dictionary . { "@timestamp" =\> 2021-07-23T11:52:08.000Z, "Flag" =\> "16464", "Flag\_Definition" =\> "liveness test timed out… --- ## [Kibana discover tab is lagging](https://discuss.elastic.co/t/kibana-discover-tab-is-lagging/277902)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 17\ **Last updated:** [July 30, 2021, 3:51pm UTC](https://discuss.elastic.co/t/kibana-discover-tab-is-lagging/277902 "2021-07-30T15:51:53Z")
Running elk stack 7.11 Issue is while searching 2 months data in kibana's discover it is taking more than 5 minutes to show the result the index size is 100gb we did top command and we didn't see any serious thing cpu i… --- ## [JDBC last run is updated, but logstash always start from the beggining and duplicate documents](https://discuss.elastic.co/t/jdbc-last-run-is-updated-but-logstash-always-start-from-the-beggining-and-duplicate-documents/280091)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 0\ **Last updated:** [July 30, 2021, 3:11pm UTC](https://discuss.elastic.co/t/jdbc-last-run-is-updated-but-logstash-always-start-from-the-beggining-and-duplicate-documents/280091 "2021-07-30T15:11:53Z")
Hi, I have a pipeline with JDBC input with schedule of 5 min, I ran logstash as a service sudo systemctl start logstash and the JDBC last run is updated, but when logstash start the last run is not used an duplicate do… --- ## [Extract selected properties from a nested JSON with logstash?](https://discuss.elastic.co/t/extract-selected-properties-from-a-nested-json-with-logstash/280090)
**Author:** [@Samurai1](https://discuss.elastic.co/u/Samurai1)\ **Replies:** 0\ **Last updated:** [July 30, 2021, 2:53pm UTC](https://discuss.elastic.co/t/extract-selected-properties-from-a-nested-json-with-logstash/280090 "2021-07-30T14:53:53Z")
{ "billToLocation": { "prop1": "iwant", "prop2": { "prop2prop1": "iwant", "prop2prop2": { }, "prop2prop3:"iDontWant }, … --- ## [Match any character regular expression does not match as expected](https://discuss.elastic.co/t/match-any-character-regular-expression-does-not-match-as-expected/278918)
**Author:** [@alexleen](https://discuss.elastic.co/u/alexleen)\ **Replies:** 6\ **Last updated:** [July 30, 2021, 2:31pm UTC](https://discuss.elastic.co/t/match-any-character-regular-expression-does-not-match-as-expected/278918 "2021-07-30T14:31:45Z")
POST /search\_test/\_doc { "text":"Jack and Jill went up the hill." } GET /search\_test/\_search?q=j%2A GET /search\_test/\_search?q=j... I would expect those two queries to return the same document since both of those regu… --- ## [Want to add multiple log files as a input in filebeat.yml and send to logstash](https://discuss.elastic.co/t/want-to-add-multiple-log-files-as-a-input-in-filebeat-yml-and-send-to-logstash/280087)
**Author:** [@Aboli\_77](https://discuss.elastic.co/u/Aboli_77)\ **Replies:** 0\ **Last updated:** [July 30, 2021, 2:23pm UTC](https://discuss.elastic.co/t/want-to-add-multiple-log-files-as-a-input-in-filebeat-yml-and-send-to-logstash/280087 "2021-07-30T14:23:38Z")
Hi All, My filebeat.yml file input is: filebeat.inputs: type: log enabled: true paths: /home/XXXX/XYZ/abc.send.2\* /home/XXXX/XYZ/abc\_xyz.target\* /home/XXXX/XYZ/abc.sender.6\* /home/XXXX/XYZ/abc.log\* fields: log… --- ## [Grok fails to write events on a file](https://discuss.elastic.co/t/grok-fails-to-write-events-on-a-file/280081)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 2\ **Last updated:** [July 30, 2021, 2:04pm UTC](https://discuss.elastic.co/t/grok-fails-to-write-events-on-a-file/280081 "2021-07-30T14:04:46Z")
I have the following test configuration. Basically I want to write the messages in either the fail or the notfail files depending on whether the grok filter would succeed. I've created the files proactively under a folde… --- ## [ML: difference between partition\_field\_name and by\_field\_name?](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name/280013)
**Author:** [@CameronCenic](https://discuss.elastic.co/u/CameronCenic)\ **Replies:** 3\ **Last updated:** [July 30, 2021, 2:01pm UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name/280013 "2021-07-30T14:01:24Z")
I do not really understand the difference between these two settings. They seem to perform the same function from my perspective. --- ## [Logstash filter take everything after](https://discuss.elastic.co/t/logstash-filter-take-everything-after/280061)
**Author:** [@Betorov](https://discuss.elastic.co/u/Betorov)\ **Replies:** 3\ **Last updated:** [July 30, 2021, 9:56am UTC](https://discuss.elastic.co/t/logstash-filter-take-everything-after/280061 "2021-07-30T09:56:21Z")
Hi avrey one, I wanted to filter this type of message: ip ip text I don't know the format of the text and what is inside of it.I only know that there is 2 ip and the text. How i can geto something that I don't know i… --- ## [Centralization Logstash](https://discuss.elastic.co/t/centralization-logstash/279938)
**Author:** [@Ilya\_21bA](https://discuss.elastic.co/u/Ilya_21bA)\ **Replies:** 3\ **Last updated:** [July 30, 2021, 1:10pm UTC](https://discuss.elastic.co/t/centralization-logstash/279938 "2021-07-30T13:10:47Z")
Hi, tell me, is it necessary to install the entire ELK stack on a new virtual machine, if I want to collect logs from several machines, tell me how it can be done, there are instructions. --- ## [Shard allocation on single node causes cluster overload](https://discuss.elastic.co/t/shard-allocation-on-single-node-causes-cluster-overload/279794)
**Author:** [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\ **Replies:** 3\ **Last updated:** [July 30, 2021, 1:06pm UTC](https://discuss.elastic.co/t/shard-allocation-on-single-node-causes-cluster-overload/279794 "2021-07-30T13:06:12Z")
Cluster creates index with 5 shards on same node. And it picks the node with least disk space. High performance ETL process writes all the data on to single node instead of to 5 multiple nodes. This causes overloads of … --- ## [Working with List Fields in Kibana](https://discuss.elastic.co/t/working-with-list-fields-in-kibana/279860)
**Author:** [@Sandeep\_Raju](https://discuss.elastic.co/u/Sandeep_Raju)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 11:34am UTC](https://discuss.elastic.co/t/working-with-list-fields-in-kibana/279860 "2021-07-30T11:34:13Z")
Hi all, I'm having difficulty in adding list fields in my project. The fields are already defined & cannot be changed. For ex: Groups: QA, audit, domain, company, admin, ip These are values given for a field in a s… --- ## [Concerns with scaling Logstash when using the Aggregate filter](https://discuss.elastic.co/t/concerns-with-scaling-logstash-when-using-the-aggregate-filter/279855)
**Author:** [@nsouth](https://discuss.elastic.co/u/nsouth)\ **Replies:** 5\ **Last updated:** [July 30, 2021, 11:26am UTC](https://discuss.elastic.co/t/concerns-with-scaling-logstash-when-using-the-aggregate-filter/279855 "2021-07-30T11:26:55Z")
The aggregate filter is working great in my testing, but I'm concerned about scale because it is stateful and requires setting filter workers to 1 (-w 1 flag). We have about 150 servers, each of which will generate ~2GB … --- ## [Continuously merging documents from multiple indexes](https://discuss.elastic.co/t/continuously-merging-documents-from-multiple-indexes/279953)
**Author:** [@roll](https://discuss.elastic.co/u/roll)\ **Replies:** 2\ **Last updated:** [July 30, 2021, 10:50am UTC](https://discuss.elastic.co/t/continuously-merging-documents-from-multiple-indexes/279953 "2021-07-30T10:50:38Z")
Hi all, I'm fairly new to elasticsearch and have a question regarding merging of documents and removing duplicates. I got the following data structure: topic1: { "\_id": “a12345", "userId" : "x123”, … --- ## [Get the "request" that a dashboard or a visualization have](https://discuss.elastic.co/t/get-the-request-that-a-dashboard-or-a-visualization-have/279769)
**Author:** [@aza-zil](https://discuss.elastic.co/u/aza-zil)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 10:44am UTC](https://discuss.elastic.co/t/get-the-request-that-a-dashboard-or-a-visualization-have/279769 "2021-07-30T10:44:27Z")
Hey, is there anyway to get the request that a dashboard or a visualization uses to get the data programmatically? Or is there any api that I can use to get the data of a dashboard? thanks --- ## [Kibana monitoring with Metricbeat](https://discuss.elastic.co/t/kibana-monitoring-with-metricbeat/279728)
**Author:** [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 10:16am UTC](https://discuss.elastic.co/t/kibana-monitoring-with-metricbeat/279728 "2021-07-30T10:16:55Z")
I am trying to monitor kibana with Metricbeat kibana-xpack. But i am getting below log messages. Jul 27 05:47:09 xxxxxxxxx metricbeat\[149229\]: 2021-07-27T05:47:09.110-0700#011ERROR#011\[kibana.stats\]#011stats/stats.go:79… --- ## [Limit ElasticSearch repetitive logs after exception](https://discuss.elastic.co/t/limit-elasticsearch-repetitive-logs-after-exception/279695)
**Author:** [@Stefan\_Speranta](https://discuss.elastic.co/u/Stefan_Speranta)\ **Replies:** 0\ **Last updated:** [July 27, 2021, 7:28am UTC](https://discuss.elastic.co/t/limit-elasticsearch-repetitive-logs-after-exception/279695 "2021-07-27T07:28:50Z")
Hi, We have developed a project using Elastic, Logstash and Kibana as log management. After some site attacks one area of application got into endless loops of exceptions and filled Elastic with the same log massage ag… --- ## [How to pass parameter filter from dashboard to Visualize Kibana](https://discuss.elastic.co/t/how-to-pass-parameter-filter-from-dashboard-to-visualize-kibana/279948)
**Author:** [@HuongCV](https://discuss.elastic.co/u/HuongCV)\ **Replies:** 4\ **Last updated:** [July 30, 2021, 9:46am UTC](https://discuss.elastic.co/t/how-to-pass-parameter-filter-from-dashboard-to-visualize-kibana/279948 "2021-07-30T09:46:51Z")
Hi all, In Dashboard, I have range filter: confirmation\_year from A year to B year. In Elastic have field dead\_year. In metric Visualize , I want to count documents that satisfy the condition: 'dead\_year' less than or… --- ## [How to get all buckets of last 24hrs in date histogram](https://discuss.elastic.co/t/how-to-get-all-buckets-of-last-24hrs-in-date-histogram/279616)
**Author:** [@prathamesh7](https://discuss.elastic.co/u/prathamesh7)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 9:45am UTC](https://discuss.elastic.co/t/how-to-get-all-buckets-of-last-24hrs-in-date-histogram/279616 "2021-07-30T09:45:42Z")
Hi All, I am using Date Histogram with Minimum interval as Hourly to get the results of Last 24 Hours and getting below graph. (v is 7.4) The request is , GET access\*/\_search?pretty=true { "aggs": { "2": { … [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1299) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1301)