# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1879 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1880 --- ## [Ingest pipelines with Serial Difference aggregation](https://discuss.elastic.co/t/ingest-pipelines-with-serial-difference-aggregation/241511)
**Author:** [@dg1](https://discuss.elastic.co/u/dg1)\ **Replies:** 5\ **Last updated:** [August 9, 2020, 12:21pm UTC](https://discuss.elastic.co/t/ingest-pipelines-with-serial-difference-aggregation/241511 "2020-08-09T12:21:13Z")
I'm storing quality metrics in elastic. One of those metrics is code coverage (extracted from sonar). I want to calculate the difference in average code coverage for a project between weeks. So this week the average code… --- ## [Manual heap dump in elasticsearch not working](https://discuss.elastic.co/t/manual-heap-dump-in-elasticsearch-not-working/244299)
**Author:** [@Prashanth\_123](https://discuss.elastic.co/u/Prashanth_123)\ **Replies:** 0\ **Last updated:** [August 9, 2020, 10:12am UTC](https://discuss.elastic.co/t/manual-heap-dump-in-elasticsearch-not-working/244299 "2020-08-09T10:12:09Z")
Over last few days noticed more GC in my es server and was trying to take a look at heap .. My ES is running on EC2 , with Amazon linux AMI OS. When i tried to take heap dump with "jmap -dump:format=b,file=/var/lib/e… --- ## [Enable modules.d/suricata.yml module](https://discuss.elastic.co/t/enable-modules-d-suricata-yml-module/244114)
**Author:** [@bn1980](https://discuss.elastic.co/u/bn1980)\ **Replies:** 3\ **Last updated:** [August 9, 2020, 6:10am UTC](https://discuss.elastic.co/t/enable-modules-d-suricata-yml-module/244114 "2020-08-09T06:10:18Z")
I don't appear to have the suricata.yml module. Am i missing something? Installed ELK on a freebsd11.2 machine. --- ## [Logstash filter doesn't return correct results based on conditionals](https://discuss.elastic.co/t/logstash-filter-doesnt-return-correct-results-based-on-conditionals/244234)
**Author:** [@nmoham](https://discuss.elastic.co/u/nmoham)\ **Replies:** 2\ **Last updated:** [August 9, 2020, 1:15am UTC](https://discuss.elastic.co/t/logstash-filter-doesnt-return-correct-results-based-on-conditionals/244234 "2020-08-09T01:15:16Z")
I've have the following conf file in logstash; based on the incoming data from beats containing the environment field, I want to add a new field "\[@metadata\]\[shared\_env\_category\]". I am testing the it using stdin and wi… --- ## [Parsing complex xml](https://discuss.elastic.co/t/parsing-complex-xml/244271)
**Author:** [@zmundo](https://discuss.elastic.co/u/zmundo)\ **Replies:** 1\ **Last updated:** [August 8, 2020, 11:30pm UTC](https://discuss.elastic.co/t/parsing-complex-xml/244271 "2020-08-08T23:30:55Z")
I am working with cnc machine data reporting back in a complex xml structure. The data is only generated when the component changes value. If a machine sits ideal, things like the path component won't report any data,… --- ## [Logstash instantnly shut down after importing .csv file inside pipeline folder, also I dont have permisions inside CLI of logstash container or other containers](https://discuss.elastic.co/t/logstash-instantnly-shut-down-after-importing-csv-file-inside-pipeline-folder-also-i-dont-have-permisions-inside-cli-of-logstash-container-or-other-containers/244268)
**Author:** [@giganttus](https://discuss.elastic.co/u/giganttus)\ **Replies:** 0\ **Last updated:** [August 8, 2020, 8:39pm UTC](https://discuss.elastic.co/t/logstash-instantnly-shut-down-after-importing-csv-file-inside-pipeline-folder-also-i-dont-have-permisions-inside-cli-of-logstash-container-or-other-containers/244268 "2020-08-08T20:39:25Z")
So I configured ELK - Stack with this config. And as result my logstash container shuts down and also while I was trying to fix it through CLI of logstash container I found out that I have no permission of any container… --- ## [How to update a record in ES if storing of field data in \_id will be removed in future?](https://discuss.elastic.co/t/how-to-update-a-record-in-es-if-storing-of-field-data-in-id-will-be-removed-in-future/244265)
**Author:** [@aryaman](https://discuss.elastic.co/u/aryaman)\ **Replies:** 0\ **Last updated:** [August 8, 2020, 8:01pm UTC](https://discuss.elastic.co/t/how-to-update-a-record-in-es-if-storing-of-field-data-in-id-will-be-removed-in-future/244265 "2020-08-08T20:01:21Z")
Hi, I use python Elasticsearch client for sending new documents to Elasticsearch and updating already existing records as well. My client request looks like this: es.index(index='index\_name',ignore=400,id=dict\['Logica… --- ## [Trouble connecting to Microsoft SQL Server in Logstash via jdbc](https://discuss.elastic.co/t/trouble-connecting-to-microsoft-sql-server-in-logstash-via-jdbc/244266)
**Author:** [@logicalblonde](https://discuss.elastic.co/u/logicalblonde)\ **Replies:** 0\ **Last updated:** [August 8, 2020, 8:04pm UTC](https://discuss.elastic.co/t/trouble-connecting-to-microsoft-sql-server-in-logstash-via-jdbc/244266 "2020-08-08T20:04:06Z")
I'm having trouble connecting logstash to MS Sql Server via the jdbc driver. I'm using jdk 11.0.8 and MSS Sql Server jdbc driver 4.2 (jre8). I've had success setting up other machines with 11.0.7 and 4.2, but with this… --- ## [Indexing a threaded conversation](https://discuss.elastic.co/t/indexing-a-threaded-conversation/244257)
**Author:** [@dansei](https://discuss.elastic.co/u/dansei)\ **Replies:** 0\ **Last updated:** [August 8, 2020, 4:26pm UTC](https://discuss.elastic.co/t/indexing-a-threaded-conversation/244257 "2020-08-08T16:26:05Z")
I would like to use Elasticsearch to index and search a threaded conversation, like a blog's comments where there are top-level comments and replies to those comments. I have tried making a single document like this, wi… --- ## [Can I use an existing field in the event to create the columns for a CSV filter?](https://discuss.elastic.co/t/can-i-use-an-existing-field-in-the-event-to-create-the-columns-for-a-csv-filter/244258)
**Author:** [@crickes](https://discuss.elastic.co/u/crickes)\ **Replies:** 2\ **Last updated:** [August 8, 2020, 5:10pm UTC](https://discuss.elastic.co/t/can-i-use-an-existing-field-in-the-event-to-create-the-columns-for-a-csv-filter/244258 "2020-08-08T17:10:27Z")
I've been struugling with this for far too long and I'm giving up and asking for help. I am being sent data via a Kafka topic which contains a field called 'headers' which is a CSV string of column headers, and a field c… --- ## [Need help to build dashboard for daily ingestion rate per host](https://discuss.elastic.co/t/need-help-to-build-dashboard-for-daily-ingestion-rate-per-host/244239)
**Author:** [@prabhakar\_talari](https://discuss.elastic.co/u/prabhakar_talari)\ **Replies:** 4\ **Last updated:** [August 8, 2020, 3:56pm UTC](https://discuss.elastic.co/t/need-help-to-build-dashboard-for-daily-ingestion-rate-per-host/244239 "2020-08-08T15:56:00Z")
Hi All, I am in process of building dashboard which gives me daily ingetion growth per day per host. After reviewing the monitoring index fields I saw there was no any field which Carrie's this data. Could any one help… --- ## [Dump RRD files into logstash to visualise in Kibana](https://discuss.elastic.co/t/dump-rrd-files-into-logstash-to-visualise-in-kibana/244250)
**Author:** [@Himani\_Agarwal](https://discuss.elastic.co/u/Himani_Agarwal)\ **Replies:** 0\ **Last updated:** [August 8, 2020, 12:28pm UTC](https://discuss.elastic.co/t/dump-rrd-files-into-logstash-to-visualise-in-kibana/244250 "2020-08-08T12:28:56Z")
Hello, I have RRD(Round Robin Database) files which contains the statistics of a remote server(collected using collectd). The requirement is to put this data on elastic search for analysis purpose. One of the solution … --- ## [Unable to allocate replica shard after node restart](https://discuss.elastic.co/t/unable-to-allocate-replica-shard-after-node-restart/243738)
**Author:** [@Chimu](https://discuss.elastic.co/u/Chimu)\ **Replies:** 8\ **Last updated:** [August 8, 2020, 12:22pm UTC](https://discuss.elastic.co/t/unable-to-allocate-replica-shard-after-node-restart/243738 "2020-08-08T12:22:43Z")
I have an Elasticsearch (v5.6.10) cluster with 3 nodes. Node A : Master Node B : Master + Data Node C : Master + Data There are 6 shards per data node with replication set as 1. All 6 primary nodes are in Node B and a… --- ## [How to remove repeting string in message?](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195)
**Author:** [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Replies:** 11\ **Last updated:** [August 8, 2020, 7:20am UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195 "2020-08-08T07:20:28Z")
I have a syslog5424\_msg field which gets repeating JSON string, example log {"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development… --- ## [ELK target cluster and redactions](https://discuss.elastic.co/t/elk-target-cluster-and-redactions/244189)
**Author:** [@griffer98](https://discuss.elastic.co/u/griffer98)\ **Replies:** 1\ **Last updated:** [August 8, 2020, 5:25am UTC](https://discuss.elastic.co/t/elk-target-cluster-and-redactions/244189 "2020-08-08T05:25:54Z")
I am planning on having multiple clusters and some files that go through the stack will need ip redactions and so on. What I want to do is have some way for logstash to determine, based on certain requirements, which clu… --- ## ["Refresh every" field gets reset to 0.01 seconds](https://discuss.elastic.co/t/refresh-every-field-gets-reset-to-0-01-seconds/242364)
**Author:** [@sanjaysubramanian](https://discuss.elastic.co/u/sanjaysubramanian)\ **Replies:** 3\ **Last updated:** [August 8, 2020, 12:28am UTC](https://discuss.elastic.co/t/refresh-every-field-gets-reset-to-0-01-seconds/242364 "2020-08-08T00:28:15Z")
Hi all I am not able to reproduce this use case every time. However every once in a while when I open an existing visualization in Kibana, I observe that the graph never displays. And then one time I observed that t… --- ## [\[ERROR\]\[logstash.config.sourceloader\] No configuration found in the configured sources](https://discuss.elastic.co/t/error-logstash-config-sourceloader-no-configuration-found-in-the-configured-sources/244230)
**Author:** [@Umang\_Mahant](https://discuss.elastic.co/u/Umang_Mahant)\ **Replies:** 2\ **Last updated:** [August 7, 2020, 10:17pm UTC](https://discuss.elastic.co/t/error-logstash-config-sourceloader-no-configuration-found-in-the-configured-sources/244230 "2020-08-07T22:17:31Z")
Hi Masters, Please help me with this small query. Trying to load excel (Csv) to Kibana from logstash. I have not changed/ created any YML . geting the below error - Z:\\\>logstash -f logstash\_tc.conf --debug … --- ## [Ingest manager won't let kibana start](https://discuss.elastic.co/t/ingest-manager-wont-let-kibana-start/244222)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 2\ **Last updated:** [August 7, 2020, 9:57pm UTC](https://discuss.elastic.co/t/ingest-manager-wont-let-kibana-start/244222 "2020-08-07T21:57:40Z")
when I enable ingest manager in kibana.yml file like below xpack.ingestManager.enabled: true kibana won't start and on browser get following message. is this a bug? Version: 7.8.0 Build: 31997 Error: Start lifecycle … --- ## [Total hits for search involving searchafter](https://discuss.elastic.co/t/total-hits-for-search-involving-searchafter/244223)
**Author:** [@jaekim06450](https://discuss.elastic.co/u/jaekim06450)\ **Replies:** 0\ **Last updated:** [August 7, 2020, 8:36pm UTC](https://discuss.elastic.co/t/total-hits-for-search-involving-searchafter/244223 "2020-08-07T20:36:37Z")
Hello, I am currently using Search After component of Search API for Java. The question I have involves the total hits that are tracked as part of each query. Let's say there's an index that contains 100 records for a … --- ## [Doc.length starts counting approximately starting from 41 tokens](https://discuss.elastic.co/t/doc-length-starts-counting-approximately-starting-from-41-tokens/244215)
**Author:** [@Alex\_D](https://discuss.elastic.co/u/Alex_D)\ **Replies:** 0\ **Last updated:** [August 7, 2020, 7:37pm UTC](https://discuss.elastic.co/t/doc-length-starts-counting-approximately-starting-from-41-tokens/244215 "2020-08-07T19:37:08Z")
Hi all ! I founded that doc.length variable starts counting approximately starting from 41 tokens in a document. Could anyone prompt, is it expected behavior ? I am trying to use scripted similarity and it's important… --- ## [Kibana Dashboards not getting loaded using setup.dashboards.file option](https://discuss.elastic.co/t/kibana-dashboards-not-getting-loaded-using-setup-dashboards-file-option/244210)
**Author:** [@Rakesh\_Katakam](https://discuss.elastic.co/u/Rakesh_Katakam)\ **Replies:** 1\ **Last updated:** [August 7, 2020, 6:33pm UTC](https://discuss.elastic.co/t/kibana-dashboards-not-getting-loaded-using-setup-dashboards-file-option/244210 "2020-08-07T18:33:37Z")
Hi , Instead of loading all dashboards to Kibana, I want to load only kafka specific dashboards . so I zipped below two json file and placed on my local drive. Metricbeat-kafka-overview.json Metricbeat-zookeeper-over… --- ## [Canvas Drop Down Filters](https://discuss.elastic.co/t/canvas-drop-down-filters/243790)
**Author:** [@aacharya](https://discuss.elastic.co/u/aacharya)\ **Replies:** 2\ **Last updated:** [August 7, 2020, 5:40pm UTC](https://discuss.elastic.co/t/canvas-drop-down-filters/243790 "2020-08-07T17:40:05Z")
I am trying to filter my data in Canvas using two drop downs. When someone selects an option from the first dropdown, the second one is automatically filtered to only contain options within their first selection. I als… --- ## [Seeing both 7.7 and 7.8 indices for all beats](https://discuss.elastic.co/t/seeing-both-7-7-and-7-8-indices-for-all-beats/243906)
**Author:** [@srpine](https://discuss.elastic.co/u/srpine)\ **Replies:** 2\ **Last updated:** [August 7, 2020, 5:27pm UTC](https://discuss.elastic.co/t/seeing-both-7-7-and-7-8-indices-for-all-beats/243906 "2020-08-07T17:27:08Z")
I recently upgraded to 7.8 and after that it started creating 7.8.0 indices and I thought that it would close out the existing 7.7.0 indices at some point but here I am days later and indices for 7.7.0 and 7.8.0 are stil… --- ## [Beat processor for mapping your network](https://discuss.elastic.co/t/beat-processor-for-mapping-your-network/244208)
**Author:** [@jasonwomack](https://discuss.elastic.co/u/jasonwomack)\ **Replies:** 0\ **Last updated:** [August 7, 2020, 5:19pm UTC](https://discuss.elastic.co/t/beat-processor-for-mapping-your-network/244208 "2020-08-07T17:19:59Z")
I think I've read every geoip and map network post I can find and I have one thing I need clarified. To clarify I'm trying to configure the yml of a beat agent, not Logstash. We have right around 50 facilities with dif… --- ## [Default found\_snapshot cycles every 5m unsuccessfully (set to 30m)](https://discuss.elastic.co/t/default-found-snapshot-cycles-every-5m-unsuccessfully-set-to-30m/244104)
**Author:** [@Erica\_Case](https://discuss.elastic.co/u/Erica_Case)\ **Replies:** 0\ **Last updated:** [August 6, 2020, 11:22pm UTC](https://discuss.elastic.co/t/default-found-snapshot-cycles-every-5m-unsuccessfully-set-to-30m/244104 "2020-08-06T23:22:32Z")
Without any configuration changes, the elastic cloudwatch snapshot runs every 5 minutes (event though the frequency is set to 30m). Despite this frequent running, monitoring says the cluster is "healthy with warnings" … --- ## [Kubernetes logs to filebeats](https://discuss.elastic.co/t/kubernetes-logs-to-filebeats/243682)
**Author:** [@sachithmuhandiram](https://discuss.elastic.co/u/sachithmuhandiram)\ **Replies:** 5\ **Last updated:** [August 7, 2020, 4:28pm UTC](https://discuss.elastic.co/t/kubernetes-logs-to-filebeats/243682 "2020-08-07T16:28:44Z")
I have ELK stack running in CentOS-7 environment, and I try to get logs from my Kubernetes cluster. for that, I installed filebeats as described. And those pods are running. I can see Obeservability-\> Logs shows kubern… --- ## [Getting new uuid every hour in logstash pipeline](https://discuss.elastic.co/t/getting-new-uuid-every-hour-in-logstash-pipeline/243406)
**Author:** [@Gopal\_Chauhan](https://discuss.elastic.co/u/Gopal_Chauhan)\ **Replies:** 2\ **Last updated:** [August 7, 2020, 4:18pm UTC](https://discuss.elastic.co/t/getting-new-uuid-every-hour-in-logstash-pipeline/243406 "2020-08-07T16:18:40Z")
Hi, I want to assign one unique id to all event/messages processed by logstash pipeline with in one hour. how can i do that? is there any timer which i can use to check that one hour has passes and now need to change t… --- ## [Error on filebeat to monitor systemlogs](https://discuss.elastic.co/t/error-on-filebeat-to-monitor-systemlogs/244196)
**Author:** [@javig12](https://discuss.elastic.co/u/javig12)\ **Replies:** 0\ **Last updated:** [August 7, 2020, 4:10pm UTC](https://discuss.elastic.co/t/error-on-filebeat-to-monitor-systemlogs/244196 "2020-08-07T16:10:32Z")
Hello community, Recently I've installed filebeat to monitore systemlogs on linux server, and have this error: 2020-08-07T11:57:14.546-0400 INFO instance/beat.go:280 Setup Beat: filebeat; Version: 6.8.11 2020-08-07T11… --- ## [Logstash is sending variables to elasticsearch](https://discuss.elastic.co/t/logstash-is-sending-variables-to-elasticsearch/244082)
**Author:** [@shreddoctor](https://discuss.elastic.co/u/shreddoctor)\ **Replies:** 10\ **Last updated:** [August 7, 2020, 3:19pm UTC](https://discuss.elastic.co/t/logstash-is-sending-variables-to-elasticsearch/244082 "2020-08-07T15:19:49Z")
I have an issue where logstash is sending the variables to elasticsearch instead of enumerating the contents of the variable. The variables in question are coming from filebeat and I'm using these variables to create 2 d… --- ## [Cluster state update taking more than 10 seconds warnings](https://discuss.elastic.co/t/cluster-state-update-taking-more-than-10-seconds-warnings/243768)
**Author:** [@luiz.ooliveira](https://discuss.elastic.co/u/luiz.ooliveira)\ **Replies:** 7\ **Last updated:** [August 7, 2020, 2:47pm UTC](https://discuss.elastic.co/t/cluster-state-update-taking-more-than-10-seconds-warnings/243768 "2020-08-07T14:47:59Z")
Hi, I'm getting the following warning every time I update a cluster setting: took \[10.6s\], which is over \[10s\], to compute cluster state update for \[update-settings\] Is there any way to identify where it is taking mor… [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1878) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1880)