# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1444 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1445 --- ## [Using pre-defined grok patterns](https://discuss.elastic.co/t/using-pre-defined-grok-patterns/271216)
**Author:** [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)\ **Replies:** 4\ **Last updated:** [April 27, 2021, 4:48am UTC](https://discuss.elastic.co/t/using-pre-defined-grok-patterns/271216 "2021-04-27T04:48:21Z")
Hi, community. I just discovered that logstash has pre-defined grok patterns, for firewalls, for example (here logstash-patterns-core/patterns at master · logstash-plugins/logstash-patterns-core · GitHub). But I can't u… --- ## [Elasticsearch error license is not available](https://discuss.elastic.co/t/elasticsearch-error-license-is-not-available/271039)
**Author:** [@charles97](https://discuss.elastic.co/u/charles97)\ **Replies:** 2\ **Last updated:** [April 27, 2021, 6:08am UTC](https://discuss.elastic.co/t/elasticsearch-error-license-is-not-available/271039 "2021-04-27T06:08:34Z")
es: 7.12 os:CentOS7 java : java-1.8.0-openjdk.x86\_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.282.b08-1.el7\_9.x86\_64/jre/bin/java) My cluster used to had 2 node, this node as : master another node as : data-1. I deci… --- ## [No monitoring data found](https://discuss.elastic.co/t/no-monitoring-data-found/271008)
**Author:** [@Chris\_Gatihi](https://discuss.elastic.co/u/Chris_Gatihi)\ **Replies:** 8\ **Last updated:** [April 27, 2021, 5:41am UTC](https://discuss.elastic.co/t/no-monitoring-data-found/271008 "2021-04-27T05:41:03Z")
Hi, I'm running on version 7.11.1 and I'm getting a "No monitoring data found" when I navigate to stack monitoring in Kibana. Kibana can access my Elasticsearch cluster just fine when going through the dev tools consol… --- ## [Fetch Agent Status](https://discuss.elastic.co/t/fetch-agent-status/271313)
**Author:** [@Sathish22](https://discuss.elastic.co/u/Sathish22)\ **Replies:** 4\ **Last updated:** [April 27, 2021, 5:04am UTC](https://discuss.elastic.co/t/fetch-agent-status/271313 "2021-04-27T05:04:49Z")
Dear Team, Currently, we have a requirement to fetch the agent status from all servers, is there any way to fetch the agent status from all servers (Windows & Linux) and save it in elasticsearch. Please help me.!!! Th… --- ## [How to create a rollover policy in ILM which automatically moves historical indices into warm node and keeps last 2 quarter indices in hot nodes](https://discuss.elastic.co/t/how-to-create-a-rollover-policy-in-ilm-which-automatically-moves-historical-indices-into-warm-node-and-keeps-last-2-quarter-indices-in-hot-nodes/270957)
**Author:** [@sagar7645](https://discuss.elastic.co/u/sagar7645)\ **Replies:** 1\ **Last updated:** [April 27, 2021, 4:49am UTC](https://discuss.elastic.co/t/how-to-create-a-rollover-policy-in-ilm-which-automatically-moves-historical-indices-into-warm-node-and-keeps-last-2-quarter-indices-in-hot-nodes/270957 "2021-04-27T04:49:53Z")
Hi Team, I am using elasticsearch 7.9 need to create a rollover ILM policy to keep historical indices in warm node and last 2 quarter indices on hot node. Can you please suggest me how to do that in a ILM policy. Is it… --- ## [Workaround to Dynamic Index Names When Ussing ILM Policy](https://discuss.elastic.co/t/workaround-to-dynamic-index-names-when-ussing-ilm-policy/271311)
**Author:** [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Replies:** 1\ **Last updated:** [April 27, 2021, 4:40am UTC](https://discuss.elastic.co/t/workaround-to-dynamic-index-names-when-ussing-ilm-policy/271311 "2021-04-27T04:40:32Z")
I apologize if this has been asked a lot, but I did a lot of reading and I'm still not quite clear. For instance: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#\_writing\_to\_differen… --- ## [How to speed up the segment merging at night?](https://discuss.elastic.co/t/how-to-speed-up-the-segment-merging-at-night/271267)
**Author:** [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Replies:** 2\ **Last updated:** [April 27, 2021, 3:22am UTC](https://discuss.elastic.co/t/how-to-speed-up-the-segment-merging-at-night/271267 "2021-04-27T03:22:16Z")
The ES version is 7.11.1 I use ilm to manage my log indices hot phase: rollover warm phase (12 hours from rollover): forcemerge、read only、data allocation cold phase (5 days from rollover): freeze delete phase(10 day… --- ## [Unable to create a transform that uses "min"](https://discuss.elastic.co/t/unable-to-create-a-transform-that-uses-min/271309)
**Author:** [@Sam\_Critchley](https://discuss.elastic.co/u/Sam_Critchley)\ **Replies:** 1\ **Last updated:** [April 27, 2021, 3:12am UTC](https://discuss.elastic.co/t/unable-to-create-a-transform-that-uses-min/271309 "2021-04-27T03:12:00Z")
I'm trying to create a transform that aggregates on min of a date field. This field is mapped in the index like so: "event\_time": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss.nnnnnn" }, I… --- ## [Last query date for an index](https://discuss.elastic.co/t/last-query-date-for-an-index/270861)
**Author:** [@danielibars](https://discuss.elastic.co/u/danielibars)\ **Replies:** 1\ **Last updated:** [April 27, 2021, 2:56am UTC](https://discuss.elastic.co/t/last-query-date-for-an-index/270861 "2021-04-27T02:56:48Z")
Hi There! I'm trying to create a rule either ILM or programmatically to remove unused indices I'm interested on deleting an index that hasn't been queried in more than 30 days, the index contains static data (html page… --- ## [Kibana performance tuning](https://discuss.elastic.co/t/kibana-performance-tuning/271152)
**Author:** [@ARDiver86](https://discuss.elastic.co/u/ARDiver86)\ **Replies:** 2\ **Last updated:** [April 27, 2021, 2:20am UTC](https://discuss.elastic.co/t/kibana-performance-tuning/271152 "2021-04-27T02:20:03Z")
We are having pretty bad performance in our new environment we setup with one Kibana server and two Elasticsearch nodes. We followed the best practices and adjusted the heap size to 31GB for both values, disabled swappin… --- ## [Logs disappeared after running applications for a few hours - ELK stack](https://discuss.elastic.co/t/logs-disappeared-after-running-applications-for-a-few-hours-elk-stack/271050)
**Author:** [@District](https://discuss.elastic.co/u/District)\ **Replies:** 1\ **Last updated:** [April 27, 2021, 1:50am UTC](https://discuss.elastic.co/t/logs-disappeared-after-running-applications-for-a-few-hours-elk-stack/271050 "2021-04-27T01:50:39Z")
Hi, we are using ELK stack for monitoring our applications. However, it's been quite weird to us that logs kept disappearing after a few hours in kibana. The pattern was like: Deployed applications -\> logs appeared in … --- ## [Issue with elasticsearch backup using GCS](https://discuss.elastic.co/t/issue-with-elasticsearch-backup-using-gcs/271239)
**Author:** [@722826c085cfc824006d](https://discuss.elastic.co/u/722826c085cfc824006d)\ **Replies:** 0\ **Last updated:** [April 26, 2021, 10:01am UTC](https://discuss.elastic.co/t/issue-with-elasticsearch-backup-using-gcs/271239 "2021-04-26T10:01:12Z")
We are running ES cluster as kuberntes pods in google cloud environment. We are planning to trigger the snapshots and need to store the data GCS bucket. As part of snapshot, we are creating the rep0, but the repo creat… --- ## [Replication of shards with more nodes vs parallel cluster](https://discuss.elastic.co/t/replication-of-shards-with-more-nodes-vs-parallel-cluster/271035)
**Author:** [@nchalise](https://discuss.elastic.co/u/nchalise)\ **Replies:** 1\ **Last updated:** [April 27, 2021, 1:35am UTC](https://discuss.elastic.co/t/replication-of-shards-with-more-nodes-vs-parallel-cluster/271035 "2021-04-27T01:35:36Z")
To reduce/mitigate downtime which approach is better and why? doubling the data/client nodes and having replication on the shards OR is it worth considering parallel clusters with a load balancer on the client nodes. Pl… --- ## [Deleted field, result is successfull, but field is still there](https://discuss.elastic.co/t/deleted-field-result-is-successfull-but-field-is-still-there/271287)
**Author:** [@NLSVTN](https://discuss.elastic.co/u/NLSVTN)\ **Replies:** 1\ **Last updated:** [April 27, 2021, 12:53am UTC](https://discuss.elastic.co/t/deleted-field-result-is-successfull-but-field-is-still-there/271287 "2021-04-27T00:53:45Z")
Hi, I am using Kibana for interacting with my indices. I have a field in \_source: '\_nisc\_AC' which I am trying to delete in the following way: POST grch38\_test\_\_wes\_\_grch38/\_update\_by\_query?conflicts=proceed { "scr… --- ## [Unable To Connect To theRemote Cluster](https://discuss.elastic.co/t/unable-to-connect-to-theremote-cluster/271210)
**Author:** [@yahooljy](https://discuss.elastic.co/u/yahooljy)\ **Replies:** 2\ **Last updated:** [April 27, 2021, 12:41am UTC](https://discuss.elastic.co/t/unable-to-connect-to-theremote-cluster/271210 "2021-04-27T00:41:00Z")
Hi there ! I Set up basic security for the Elastic Stack plus secured HTTPS traffic . and then my kibana can connect to the local cluster successfully . But there is something wrong with the connection between the localc… --- ## [Using filebeat, cannot read json log to custom index](https://discuss.elastic.co/t/using-filebeat-cannot-read-json-log-to-custom-index/270914)
**Author:** [@thuynh](https://discuss.elastic.co/u/thuynh)\ **Replies:** 2\ **Last updated:** [April 26, 2021, 9:47pm UTC](https://discuss.elastic.co/t/using-filebeat-cannot-read-json-log-to-custom-index/270914 "2021-04-26T21:47:34Z")
Index template already created. It runs but doesn''t tell me any error and can't see new index which should be call "fb-message-7.12.0-yyyy.mm.dd" I notice other issues i'm missing the param to indicate this is a custo… --- ## [Pie chart w/ data from two diff indexes](https://discuss.elastic.co/t/pie-chart-w-data-from-two-diff-indexes/271199)
**Author:** [@azulgrana](https://discuss.elastic.co/u/azulgrana)\ **Replies:** 4\ **Last updated:** [April 26, 2021, 9:13pm UTC](https://discuss.elastic.co/t/pie-chart-w-data-from-two-diff-indexes/271199 "2021-04-26T21:13:49Z")
Hi team, I have a use case where I need to graph on a single pie chart data from two different indexes. The feel that I need chart exists on both indexes and has the same name. So my question is, is it possible to creat… --- ## [Methods to loadbalance on logstash from Input data stream to Filter section](https://discuss.elastic.co/t/methods-to-loadbalance-on-logstash-from-input-data-stream-to-filter-section/271292)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 2\ **Last updated:** [April 26, 2021, 9:09pm UTC](https://discuss.elastic.co/t/methods-to-loadbalance-on-logstash-from-input-data-stream-to-filter-section/271292 "2021-04-26T21:09:13Z")
I'm looking for methods to loadbalance on logstash or elasticsearch . The requirement is that i have decently large log files with thousands of lines, while logstash is perfectly processing the log file as expected, i wa… --- ## [Replace @timestamp with actual timestamp from log file with two different formats](https://discuss.elastic.co/t/replace-timestamp-with-actual-timestamp-from-log-file-with-two-different-formats/271264)
**Author:** [@kommineni24](https://discuss.elastic.co/u/kommineni24)\ **Replies:** 5\ **Last updated:** [April 26, 2021, 8:25pm UTC](https://discuss.elastic.co/t/replace-timestamp-with-actual-timestamp-from-log-file-with-two-different-formats/271264 "2021-04-26T20:25:38Z")
Hi, I am trying to index my mail-relay log files to Elastic search. All the log entries are being indexed into a field named message . @timestamp field shows the time the entry was indexed and not the timestamp from the… --- ## [Suricata module question](https://discuss.elastic.co/t/suricata-module-question/271286)
**Author:** [@nunex\_17](https://discuss.elastic.co/u/nunex_17)\ **Replies:** 2\ **Last updated:** [April 26, 2021, 6:41pm UTC](https://discuss.elastic.co/t/suricata-module-question/271286 "2021-04-26T18:41:55Z")
Hi! In my setup, the "eve" log files are separated by category. Using the Suricata module, how can I send both eve files to elastic? is it possible to use the example below? - module: suricata eve: enabled: true… --- ## [Metricbeat \> Module PostgreSQL \> wrong mapping of blk\_read\_time and blk\_write\_time](https://discuss.elastic.co/t/metricbeat-module-postgresql-wrong-mapping-of-blk-read-time-and-blk-write-time/270722)
**Author:** [@fabski](https://discuss.elastic.co/u/fabski)\ **Replies:** 2\ **Last updated:** [April 26, 2021, 5:50pm UTC](https://discuss.elastic.co/t/metricbeat-module-postgresql-wrong-mapping-of-blk-read-time-and-blk-write-time/270722 "2021-04-26T17:50:48Z")
Hi, I think there is a bug in the PostgreSQL module of Metricbeat. According to beats/metricbeat/module/postgresql/database/data.go, the following fields are mapped as Integer: "blocks": s.Object{ ... "time": s… --- ## [Timestamp](https://discuss.elastic.co/t/timestamp/271206)
**Author:** [@Akhil2](https://discuss.elastic.co/u/Akhil2)\ **Replies:** 2\ **Last updated:** [April 26, 2021, 5:13pm UTC](https://discuss.elastic.co/t/timestamp/271206 "2021-04-26T17:13:21Z")
Hello everyone, I am trying to use timestamp from my log file instead of the timestamp when my logs were processed. in order to do that, I am using dissect processor and Timestamp processor together as below. processor… --- ## [Improving fingerprint filter performance](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573)
**Author:** [@vasu01](https://discuss.elastic.co/u/vasu01)\ **Replies:** 4\ **Last updated:** [April 26, 2021, 5:12pm UTC](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573 "2021-04-26T17:12:23Z")
We are using the logstash fingerprint filter to avoid duplicate data in elasticsearch. But the data ingestion is taking more time. For Example - A file having ~15000-20000 rows takes approx 2~3 hours to load. Only two f… --- ## [\[Docker\] Using old data folder not working](https://discuss.elastic.co/t/docker-using-old-data-folder-not-working/271280)
**Author:** [@Bo-Wyatt](https://discuss.elastic.co/u/Bo-Wyatt)\ **Replies:** 3\ **Last updated:** [April 26, 2021, 4:59pm UTC](https://discuss.elastic.co/t/docker-using-old-data-folder-not-working/271280 "2021-04-26T16:59:38Z")
Hello, for some issues my Docker container with Elasticsearch has been re-created, this came with the lost of all my indexes (i have two nodes in my cluster and since the container was recreated, the cluster ID was gener… --- ## [Elastic Cloud Output not working Tshoot](https://discuss.elastic.co/t/elastic-cloud-output-not-working-tshoot/270465)
**Author:** [@Leon21](https://discuss.elastic.co/u/Leon21)\ **Replies:** 5\ **Last updated:** [April 26, 2021, 4:29pm UTC](https://discuss.elastic.co/t/elastic-cloud-output-not-working-tshoot/270465 "2021-04-26T16:29:45Z")
Hi I'm trying to push my events from a filebeat module into elastic cloud but could not find them into ES in my filebeat\* indexes. filebeat module gets events from a tcp socket (filebeat listen to it successfully) and … --- ## [Event correlation query not generating detection alerts](https://discuss.elastic.co/t/event-correlation-query-not-generating-detection-alerts/271224)
**Author:** [@Nil\_Battey\_Sannata](https://discuss.elastic.co/u/Nil_Battey_Sannata)\ **Replies:** 3\ **Last updated:** [April 26, 2021, 4:10pm UTC](https://discuss.elastic.co/t/event-correlation-query-not-generating-detection-alerts/271224 "2021-04-26T16:10:43Z")
This is prebuilt detection rule with event correlation. If I am executing query manually then it showing the correct hits. But this is not generating any alerts. Same story with every rule with event correlation.… --- ## [How to include the ctx.payload.value in the email text for watcher alert?](https://discuss.elastic.co/t/how-to-include-the-ctx-payload-value-in-the-email-text-for-watcher-alert/269655)
**Author:** [@Patr123](https://discuss.elastic.co/u/Patr123)\ **Replies:** 23\ **Last updated:** [April 26, 2021, 4:07pm UTC](https://discuss.elastic.co/t/how-to-include-the-ctx-payload-value-in-the-email-text-for-watcher-alert/269655 "2021-04-26T16:07:21Z")
Hello All, I am trying to create a watcher alert for the servers that communicated to elastic in last 48 hrs but not in last 35 mins. I want to include the ctx.payload.value result in my email text but all I am getting … --- ## [Decode\_xml not working](https://discuss.elastic.co/t/decode-xml-not-working/269763)
**Author:** [@witkacy](https://discuss.elastic.co/u/witkacy)\ **Replies:** 5\ **Last updated:** [April 26, 2021, 4:04pm UTC](https://discuss.elastic.co/t/decode-xml-not-working/269763 "2021-04-26T16:04:38Z")
Hello, I'm using winlogbeats version 7.12 and it works great but I have a question. In my windows log in message field I have an XML. What I would like to do is first check if in that XML in specific field there is a s… --- ## [Filebeat decode\_xml processor missing](https://discuss.elastic.co/t/filebeat-decode-xml-processor-missing/268915)
**Author:** [@Francisco\_Peralta\_Gu](https://discuss.elastic.co/u/Francisco_Peralta_Gu)\ **Replies:** 5\ **Last updated:** [April 26, 2021, 4:03pm UTC](https://discuss.elastic.co/t/filebeat-decode-xml-processor-missing/268915 "2021-04-26T16:03:43Z")
Hi. I'm facing issues trying to configure decode\_xml processor in filebeat version 7.11. The error is the following: Failed to start crawler: starting input failed: Error while initializing input: the processor ac… --- ## [Winlogbeat If statements](https://discuss.elastic.co/t/winlogbeat-if-statements/271275)
**Author:** [@d-ring](https://discuss.elastic.co/u/d-ring)\ **Replies:** 0\ **Last updated:** [April 26, 2021, 3:18pm UTC](https://discuss.elastic.co/t/winlogbeat-if-statements/271275 "2021-04-26T15:18:51Z")
I am trying to do some conditional filtering of windows logon events and I am having some difficulty trying to figure out how the if statements work. How I am trying to get things to work is if it is a network logon do … [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1443) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1445)