# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1081 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1082 --- ## [Extract the Key from MQTT JSON message](https://discuss.elastic.co/t/extract-the-key-from-mqtt-json-message/295185)
**Author:** [@folderman](https://discuss.elastic.co/u/folderman)\ **Replies:** 11\ **Last updated:** [January 25, 2022, 9:26am UTC](https://discuss.elastic.co/t/extract-the-key-from-mqtt-json-message/295185 "2022-01-25T09:26:11Z")
Hi All, I have a problem with my testing device support MQTT. Here is the MQTT message: {"device\_info": {"uuid": "XXXXXX","fw\_ver": "V1.5.0" },"measures": \[{"n":"co2", "u":"ppm", "v":1155.995},{"n":"voc", "u":"ppm", "v… --- ## [How to build logstash-input-tcp v6.0.3 plugin?](https://discuss.elastic.co/t/how-to-build-logstash-input-tcp-v6-0-3-plugin/295295)
**Author:** [@SHUBHAM\_JAIN1](https://discuss.elastic.co/u/SHUBHAM_JAIN1)\ **Replies:** 1\ **Last updated:** [January 25, 2022, 9:04am UTC](https://discuss.elastic.co/t/how-to-build-logstash-input-tcp-v6-0-3-plugin/295295 "2022-01-25T09:04:54Z")
please mention step for making build gem for logstash-input-tcp plugin for version 6.0.3 --- ## [Data nodes randomly leaving cluster](https://discuss.elastic.co/t/data-nodes-randomly-leaving-cluster/294838)
**Author:** [@jcasals](https://discuss.elastic.co/u/jcasals)\ **Replies:** 11\ **Last updated:** [January 25, 2022, 8:55am UTC](https://discuss.elastic.co/t/data-nodes-randomly-leaving-cluster/294838 "2022-01-25T08:55:30Z")
Hello, I have a problem that I can't seem to be able to fix nor understand. I have an Elasticsearch cluster with three master nodes and 6 (12) data nodes. Master nodes: virtual, with 2CPU and 4GB of memory. Data node… --- ## [Two commands return different segments info of the same index](https://discuss.elastic.co/t/two-commands-return-different-segments-info-of-the-same-index/295151)
**Author:** [@chembohuang](https://discuss.elastic.co/u/chembohuang)\ **Replies:** 1\ **Last updated:** [January 25, 2022, 7:46am UTC](https://discuss.elastic.co/t/two-commands-return-different-segments-info-of-the-same-index/295151 "2022-01-25T07:46:57Z")
ES 7.7.1, jdk 14. A cluster with 3 dedicated master nodes and 100+ data nodes. During the execution of the forcemerge, myindex/\_stats and \_cat/segments/myindex return different segments info. Here is the forcemerge cm… --- ## [Unable to delete index pattern in Kibana 7.15](https://discuss.elastic.co/t/unable-to-delete-index-pattern-in-kibana-7-15/295179)
**Author:** [@abhinav\_chourasia](https://discuss.elastic.co/u/abhinav_chourasia)\ **Replies:** 3\ **Last updated:** [January 25, 2022, 7:00am UTC](https://discuss.elastic.co/t/unable-to-delete-index-pattern-in-kibana-7-15/295179 "2022-01-25T07:00:29Z")
I have moved from ELK 7.9 to ELK 7.15 in an attempt to solve this problem and it looks like all that effort was of no use. I am still unable to delete the index pattern in Kibana, neither through the console, nor through… --- ## [Roles are not working correctly in kibana](https://discuss.elastic.co/t/roles-are-not-working-correctly-in-kibana/295256)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 1\ **Last updated:** [January 25, 2022, 6:52am UTC](https://discuss.elastic.co/t/roles-are-not-working-correctly-in-kibana/295256 "2022-01-25T06:52:27Z")
Hi, I am trying to create a separate space in kibana based on a read only developer role I have created the roles dedicated for the developers that has read access to the indices. However they only see one pattern of in… --- ## [Warning "Unable to acquire permit to use snapshot files during recovery…" when restarting elasticsearch node](https://discuss.elastic.co/t/warning-unable-to-acquire-permit-to-use-snapshot-files-during-recovery-when-restarting-elasticsearch-node/293543)
**Author:** [@Gregor\_Weckbecker](https://discuss.elastic.co/u/Gregor_Weckbecker)\ **Replies:** 2\ **Last updated:** [January 25, 2022, 6:02am UTC](https://discuss.elastic.co/t/warning-unable-to-acquire-permit-to-use-snapshot-files-during-recovery-when-restarting-elasticsearch-node/293543 "2022-01-25T06:02:12Z")
When upgrading our elasticsearch cluster to version 7.16.1, we performing a rolling upgarde as described in the documentation. Before the restart we trigger a synced flush and increase cluster.routing.allocation.node\_con… --- ## [Metricbeat - Not creating any logfile](https://discuss.elastic.co/t/metricbeat-not-creating-any-logfile/295266)
**Author:** [@sujeetkp](https://discuss.elastic.co/u/sujeetkp)\ **Replies:** 0\ **Last updated:** [January 24, 2022, 5:59pm UTC](https://discuss.elastic.co/t/metricbeat-not-creating-any-logfile/295266 "2022-01-24T17:59:19Z")
I am trying to set up metric beat for my CentOS7 host. I have explictly mentioned the logfile location for the metricbeat and the logging level is debug, but I dont see a log file created. I can see the logs in journalct… --- ## [FileBeats Syslog not being sent to Elastic Cloud](https://discuss.elastic.co/t/filebeats-syslog-not-being-sent-to-elastic-cloud/295300)
**Author:** [@wekimop](https://discuss.elastic.co/u/wekimop)\ **Replies:** 0\ **Last updated:** [January 25, 2022, 4:43am UTC](https://discuss.elastic.co/t/filebeats-syslog-not-being-sent-to-elastic-cloud/295300 "2022-01-25T04:43:11Z")
Hello, i am trying to use the System integration in my Elastic Security agent to capture syslog data from /var/log/syslog file. I am also doing the same on another server but using a FileBeat instance, sending output to … --- ## [DEVELOP JAVA OUTPUT PLUGIN FOR APPLICATION LOG](https://discuss.elastic.co/t/develop-java-output-plugin-for-application-log/295293)
**Author:** [@uday22](https://discuss.elastic.co/u/uday22)\ **Replies:** 0\ **Last updated:** [January 25, 2022, 2:43am UTC](https://discuss.elastic.co/t/develop-java-output-plugin-for-application-log/295293 "2022-01-25T02:43:58Z")
An application log every line of the message is different from another line. In that, need to extract the IP address and username by using the java output plugin. which method and PluginConfigSpec should use in the plugi… --- ## [Ec2 role based access failing for s3-input-plugin if we upgrade ec2 to use metadata v2](https://discuss.elastic.co/t/ec2-role-based-access-failing-for-s3-input-plugin-if-we-upgrade-ec2-to-use-metadata-v2/295271)
**Author:** [@uma\_rengasamy](https://discuss.elastic.co/u/uma_rengasamy)\ **Replies:** 0\ **Last updated:** [January 24, 2022, 6:56pm UTC](https://discuss.elastic.co/t/ec2-role-based-access-failing-for-s3-input-plugin-if-we-upgrade-ec2-to-use-metadata-v2/295271 "2022-01-24T18:56:29Z")
Logstash Version : logstash 7.2.0 aws-cli/1.18.157 Python/3.6.8 Linux/4.18.0-305.12.1.el8\_4.x86\_64 botocore/1.18.16 logstash-input-s3 (3.4.1) aws-cli/1.18.157 Python/3.6.8 Linux/4.18.0-305.12.1.el8\_4.x86\_64 botocore/1… --- ## [Remove \`\_all\` mapping from older indexes](https://discuss.elastic.co/t/remove-all-mapping-from-older-indexes/295263)
**Author:** [@willdennis](https://discuss.elastic.co/u/willdennis)\ **Replies:** 5\ **Last updated:** [January 25, 2022, 1:06am UTC](https://discuss.elastic.co/t/remove-all-mapping-from-older-indexes/295263 "2022-01-25T01:06:50Z")
Upgraded to v6.8 recently; trying to ensure I can now upgrade to v7.x Having an issue when trying to reindex my old logstash indexes created on v5.6.x - getting this message when the system goes to create the new index (… --- ## [Get distinct values from a array in ElasticSearch](https://discuss.elastic.co/t/get-distinct-values-from-a-array-in-elasticsearch/295274)
**Author:** [@smkarnan](https://discuss.elastic.co/u/smkarnan)\ **Replies:** 1\ **Last updated:** [January 25, 2022, 12:27am UTC](https://discuss.elastic.co/t/get-distinct-values-from-a-array-in-elasticsearch/295274 "2022-01-25T00:27:04Z")
Hi, How can I retrieve unique values in the array, i just want to get unique lat from the below collection. "hits" { "function": { "geo\_fence: \[ { "lon": 131.… --- ## [Keyboard Shortcuts for Dev Tools](https://discuss.elastic.co/t/keyboard-shortcuts-for-dev-tools/295267)
**Author:** [@Tim\_Mobley](https://discuss.elastic.co/u/Tim_Mobley)\ **Replies:** 1\ **Last updated:** [January 25, 2022, 12:19am UTC](https://discuss.elastic.co/t/keyboard-shortcuts-for-dev-tools/295267 "2022-01-25T00:19:09Z")
This is a simple one - is there a keyboard shortcut for executing an API call in Dev Tools (rather than clicking the little arrow)? It seems like a small thing, but if it exists, it would save a ton of time. --- ## [Error while connecting logstash to kafka](https://discuss.elastic.co/t/error-while-connecting-logstash-to-kafka/295189)
**Author:** [@Avnish\_Singh](https://discuss.elastic.co/u/Avnish_Singh)\ **Replies:** 0\ **Last updated:** [January 24, 2022, 8:16am UTC](https://discuss.elastic.co/t/error-while-connecting-logstash-to-kafka/295189 "2022-01-24T08:16:31Z")
Hi I'm trying to run logstash with input as Kafka and output to file, here is the pipeline config file: input { kafka { bootstrap\_servers =\> "kafka\_server:9092 topics =\> \["Topic1", "Topic2", "Topic3"\] … --- ## [Logstash throughput ramping up too slowly -- throttle filter?](https://discuss.elastic.co/t/logstash-throughput-ramping-up-too-slowly-throttle-filter/295285)
**Author:** [@rsk0](https://discuss.elastic.co/u/rsk0)\ **Replies:** 0\ **Last updated:** [January 24, 2022, 9:43pm UTC](https://discuss.elastic.co/t/logstash-throughput-ramping-up-too-slowly-throttle-filter/295285 "2022-01-24T21:43:42Z")
We have a (Windows) logs stream pulled out of Kafka and pushed into ES. During site events our rate can spike to many multiples. When this happens, Logstash appears to delay messages getting in to ES. It's not due to … --- ## [Functionbeat GCP pub/sub requiring storage entry point when only pub/sub is enabled](https://discuss.elastic.co/t/functionbeat-gcp-pub-sub-requiring-storage-entry-point-when-only-pub-sub-is-enabled/295254)
**Author:** [@Sebastian\_Borys](https://discuss.elastic.co/u/Sebastian_Borys)\ **Replies:** 4\ **Last updated:** [January 24, 2022, 9:24pm UTC](https://discuss.elastic.co/t/functionbeat-gcp-pub-sub-requiring-storage-entry-point-when-only-pub-sub-is-enabled/295254 "2022-01-24T21:24:44Z")
Deploying functionbeat on GCP with entry point RunPubSub returns an error complaining about storage entry point, which I am not using. I get the following error upon deploy Deployment failure: Build failed: # functions… --- ## [Total shards per node calculation](https://discuss.elastic.co/t/total-shards-per-node-calculation/294829)
**Author:** [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Replies:** 4\ **Last updated:** [January 24, 2022, 8:04pm UTC](https://discuss.elastic.co/t/total-shards-per-node-calculation/294829 "2022-01-24T20:04:57Z")
I've been working on Elastic for a while now but just wanted to do a quick check on learning that may also be useful to others. I've got a customer that would like 4 separate indices for logs. The only difference betwe… --- ## [Iterating over the same grok filter](https://discuss.elastic.co/t/iterating-over-the-same-grok-filter/295089)
**Author:** [@tfinan](https://discuss.elastic.co/u/tfinan)\ **Replies:** 4\ **Last updated:** [January 24, 2022, 7:09pm UTC](https://discuss.elastic.co/t/iterating-over-the-same-grok-filter/295089 "2022-01-24T19:09:23Z")
I am a new user here, and don't know if this is a trivial question or not (it certainly is not for me). What I want to do is create a filter that creates fields with names based on the data itself. I think I can do this… --- ## [Winlog beat set up returning the following error](https://discuss.elastic.co/t/winlog-beat-set-up-returning-the-following-error/294999)
**Author:** [@kgood517](https://discuss.elastic.co/u/kgood517)\ **Replies:** 1\ **Last updated:** [January 24, 2022, 6:46pm UTC](https://discuss.elastic.co/t/winlog-beat-set-up-returning-the-following-error/294999 "2022-01-24T18:46:27Z")
Hi all, I'm getting the following error trying to setup my winlogbeat following along the directions from Winlogbeat quick start: installation and configuration | Winlogbeat Reference \[7.16\] | Elastic. ERROR \[esclientle… --- ## [What happens to the original text field?](https://discuss.elastic.co/t/what-happens-to-the-original-text-field/295262)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 3\ **Last updated:** [January 24, 2022, 6:26pm UTC](https://discuss.elastic.co/t/what-happens-to-the-original-text-field/295262 "2022-01-24T18:26:49Z")
When we use custom or built-in analyzer in our mapping it just convert the stream of characters to stream of tokens. For example when we analyze the text like The QUICK brown foxes jumped over the dog!.Here i am using c… --- ## [Use Date Histogram to group data by day of the week](https://discuss.elastic.co/t/use-date-histogram-to-group-data-by-day-of-the-week/295255)
**Author:** [@TomTom](https://discuss.elastic.co/u/TomTom)\ **Replies:** 1\ **Last updated:** [January 24, 2022, 6:09pm UTC](https://discuss.elastic.co/t/use-date-histogram-to-group-data-by-day-of-the-week/295255 "2022-01-24T18:09:36Z")
My case is as follows, I am conducting a search for sales that occurred between 2021-12-23 to 2022-01-23, and I need to know the total sales by day of the week, regardless of the week and year. So I can theoretically kn… --- ## [How to improve below Logstash grok filters?](https://discuss.elastic.co/t/how-to-improve-below-logstash-grok-filters/295212)
**Author:** [@alytkowski](https://discuss.elastic.co/u/alytkowski)\ **Replies:** 3\ **Last updated:** [January 24, 2022, 5:55pm UTC](https://discuss.elastic.co/t/how-to-improve-below-logstash-grok-filters/295212 "2022-01-24T17:55:59Z")
Hi there. I have a problem with Logstash consuming most of the CPU resources all the time and events not being parsed in real time. Through Stack Monitoring in Kibana, I found that for one set of filters for a certain so… --- ## [Why does ClusterState class need need a data member named stateUUID?](https://discuss.elastic.co/t/why-does-clusterstate-class-need-need-a-data-member-named-stateuuid/295196)
**Author:** [@armsword](https://discuss.elastic.co/u/armsword)\ **Replies:** 1\ **Last updated:** [January 24, 2022, 5:48pm UTC](https://discuss.elastic.co/t/why-does-clusterstate-class-need-need-a-data-member-named-stateuuid/295196 "2022-01-24T17:48:05Z")
https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/cluster/ClusterState.java#L139 This class ClusterState has two data member, named stateUUID and version. The stateUUID is supp… --- ## [Kibana vs. Power BI](https://discuss.elastic.co/t/kibana-vs-power-bi/295240)
**Author:** [@Tim\_Mobley](https://discuss.elastic.co/u/Tim_Mobley)\ **Replies:** 3\ **Last updated:** [January 24, 2022, 5:33pm UTC](https://discuss.elastic.co/t/kibana-vs-power-bi/295240 "2022-01-24T17:33:52Z")
I have a use-case where end users need to be able to quickly and easily view data stored in a MS SQL database for devices, systems and hardware/software assets they own. This would just be a read-only dashboard where the… --- ## [Ruby script for deleting some elements recursively \[Q&A\]](https://discuss.elastic.co/t/ruby-script-for-deleting-some-elements-recursively-q-a/295253)
**Author:** [@SudoNova](https://discuss.elastic.co/u/SudoNova)\ **Replies:** 0\ **Last updated:** [January 24, 2022, 4:31pm UTC](https://discuss.elastic.co/t/ruby-script-for-deleting-some-elements-recursively-q-a/295253 "2022-01-24T16:31:24Z")
Hi Just because I had troubles to make it work,I decided to share it maybe some other folks can use it some time. Requirement: Delete all elements in a parsed json log where they are empty Note: Haproxy sends empty fi… --- ## [Conditionals in pipelines](https://discuss.elastic.co/t/conditionals-in-pipelines/294970)
**Author:** [@Tuckson](https://discuss.elastic.co/u/Tuckson)\ **Replies:** 11\ **Last updated:** [January 24, 2022, 4:17pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970 "2022-01-24T16:17:49Z")
Hi, We are in the process of recreating our logstash pipelines in kibana/elk to use Elasticsearch pipelines instead of logstash. However the lack of documentation on how to do this in Kibana is overwhelming. See the ac… --- ## [OSQuery Manager Pack scheduled queries not executed on ELK 7.16.2 and 7.16.3?](https://discuss.elastic.co/t/osquery-manager-pack-scheduled-queries-not-executed-on-elk-7-16-2-and-7-16-3/295028)
**Author:** [@BSL](https://discuss.elastic.co/u/BSL)\ **Replies:** 1\ **Last updated:** [January 24, 2022, 3:41pm UTC](https://discuss.elastic.co/t/osquery-manager-pack-scheduled-queries-not-executed-on-elk-7-16-2-and-7-16-3/295028 "2022-01-24T15:41:33Z")
Scheduled OSQuery Manager Pack queries don't seem to work on fresh installs of ELK 7.16.2 and 7.16.3 (tried 2 completely separate ELK installations, one hosted on Ubuntu 20.04 with TLS+cert, one on Windows 11 unsecure, b… --- ## [Mongodb jdbc logstash no data output in my console](https://discuss.elastic.co/t/mongodb-jdbc-logstash-no-data-output-in-my-console/295245)
**Author:** [@Kermit\_Liu](https://discuss.elastic.co/u/Kermit_Liu)\ **Replies:** 0\ **Last updated:** [January 24, 2022, 3:23pm UTC](https://discuss.elastic.co/t/mongodb-jdbc-logstash-no-data-output-in-my-console/295245 "2022-01-24T15:23:37Z")
input { jdbc { jdbc\_driver\_library =\> "/usr/share/logstash/pipeline/driver/mongojdbc4.0.jar" jdbc\_driver\_class =\> "com.dbschema.MongoJdbcDriver" jdbc\_connection\_string =\> "jdbc:mongodb://172.17.0.1… --- ## [Reporting option not available while adding Kibana privileges](https://discuss.elastic.co/t/reporting-option-not-available-while-adding-kibana-privileges/295223)
**Author:** [@pranay\_jain](https://discuss.elastic.co/u/pranay_jain)\ **Replies:** 2\ **Last updated:** [January 24, 2022, 2:29pm UTC](https://discuss.elastic.co/t/reporting-option-not-available-while-adding-kibana-privileges/295223 "2022-01-24T14:29:36Z")
Hi, I followed the document (Configure reporting in Kibana | Kibana Guide \[7.16\] | Elastic) which explains how to enable users generate reports and grant users access to the reporting features, and secure the reporting … [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1080) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1082)