# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1498 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1499 --- ## [When will translog file be deleted from disk?](https://discuss.elastic.co/t/when-will-translog-file-be-deleted-from-disk/267944)
**Author:** [@CHU\_XU](https://discuss.elastic.co/u/CHU_XU)\ **Replies:** 3\ **Last updated:** [March 22, 2021, 8:02am UTC](https://discuss.elastic.co/t/when-will-translog-file-be-deleted-from-disk/267944 "2021-03-22T08:02:27Z")
Hi there, I found in the data directory, there many translog files like \*.tlog or \*.ckp have been existing for few days. According to the document, there's a retention setting with default 12 hours. But I'm using 7.5 v… --- ## [Elastic cloud email reporting](https://discuss.elastic.co/t/elastic-cloud-email-reporting/267685)
**Author:** [@accateo](https://discuss.elastic.co/u/accateo)\ **Replies:** 2\ **Last updated:** [March 22, 2021, 8:01am UTC](https://discuss.elastic.co/t/elastic-cloud-email-reporting/267685 "2021-03-22T08:01:46Z")
Hello, I am using Elastic Cloud and I was trying to set up a Watcher to send a pdf (generated from a dashboard) via email. I have some questions: can I create a watcher who, every 7 days, automatically sends this repo… --- ## [Filebeat netflow module do not start after update to 7.11](https://discuss.elastic.co/t/filebeat-netflow-module-do-not-start-after-update-to-7-11/264037)
**Author:** [@UweW](https://discuss.elastic.co/u/UweW)\ **Replies:** 7\ **Last updated:** [March 22, 2021, 7:53am UTC](https://discuss.elastic.co/t/filebeat-netflow-module-do-not-start-after-update-to-7-11/264037 "2021-03-22T07:53:53Z")
Hi, after updating the stack from 7.10.2 to 7.11 the netflow module will not start anymore. The only message we see every 10 seconds in journal is filebeat\[1976620\]: 2021-02-11T17:40:01.242+0100 ERROR \[r… --- ## [How to deploy alert on incoming events](https://discuss.elastic.co/t/how-to-deploy-alert-on-incoming-events/267920)
**Author:** [@\_omar](https://discuss.elastic.co/u/_omar)\ **Replies:** 1\ **Last updated:** [March 22, 2021, 5:44am UTC](https://discuss.elastic.co/t/how-to-deploy-alert-on-incoming-events/267920 "2021-03-22T05:44:20Z")
Hi I'm a new user of 'elastic', and need to deploy an alert that notify each event with 'geoip.country\_name' tag value distinct than 'peru'. The alert have to be in real time, like the mostly siem solutions do, so far i… --- ## [Is there any way to score nested documents as if they're native fields of the main document?](https://discuss.elastic.co/t/is-there-any-way-to-score-nested-documents-as-if-theyre-native-fields-of-the-main-document/267934)
**Author:** [@Yaron\_Golan](https://discuss.elastic.co/u/Yaron_Golan)\ **Replies:** 0\ **Last updated:** [March 22, 2021, 3:20am UTC](https://discuss.elastic.co/t/is-there-any-way-to-score-nested-documents-as-if-theyre-native-fields-of-the-main-document/267934 "2021-03-22T03:20:09Z")
Is it possible to get similar scoring methodologies for nested document fields normal fields? See searches for "awful" (nested) and "amazing" (direct) below. DELETE /testindex PUT /testindex { "mappings": { "pro… --- ## [How to create pipeline definition from command line?](https://discuss.elastic.co/t/how-to-create-pipeline-definition-from-command-line/267139)
**Author:** [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\ **Replies:** 2\ **Last updated:** [March 22, 2021, 3:44am UTC](https://discuss.elastic.co/t/how-to-create-pipeline-definition-from-command-line/267139 "2021-03-22T03:44:30Z")
I want to automate the process for preparing below pipeline. i.e by command line , api , etc This is because I want to send logs from filebeat to logstash instead of sending directly to elasticsearch . GET \_ingest/p… --- ## [Empty Values in DateTimeStamp - Ignore Empty Values & Perform Calculation](https://discuss.elastic.co/t/empty-values-in-datetimestamp-ignore-empty-values-perform-calculation/267721)
**Author:** [@prohit](https://discuss.elastic.co/u/prohit)\ **Replies:** 3\ **Last updated:** [March 22, 2021, 3:12am UTC](https://discuss.elastic.co/t/empty-values-in-datetimestamp-ignore-empty-values-perform-calculation/267721 "2021-03-22T03:12:52Z")
Hi All, Ver: 7.11 (Painless script) I have datetimestamp fields in number type, to convert into date format i have used doc\['actualdatetimestamp'\].value with the default format in painless script it worked Need Help … --- ## [100% cpu system time used on hdd data node](https://discuss.elastic.co/t/100-cpu-system-time-used-on-hdd-data-node/267532)
**Author:** [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Replies:** 2\ **Last updated:** [March 22, 2021, 2:44am UTC](https://discuss.elastic.co/t/100-cpu-system-time-used-on-hdd-data-node/267532 "2021-03-22T02:44:38Z")
The ES version is 7.11.1 We use 16C64G vm which has 4 physical hdd disks(striped lvm volume) as warm data node. The issue is that the cpu system time of a random vm offen suddenly rises up to 100%, and then the vm keep… --- ## [STIG - The DBMS must isolate security functions from non-security functions](https://discuss.elastic.co/t/stig-the-dbms-must-isolate-security-functions-from-non-security-functions/267744)
**Author:** [@Dan\_Bitter](https://discuss.elastic.co/u/Dan_Bitter)\ **Replies:** 3\ **Last updated:** [March 22, 2021, 2:28am UTC](https://discuss.elastic.co/t/stig-the-dbms-must-isolate-security-functions-from-non-security-functions/267744 "2021-03-22T02:28:01Z")
I have a list of STIGs to implement for Elasticsearch, and I'm not sure how to verify the needs of this one. I read about the keystore, which I think helps, but I don't think it covers all the points. I found setup p… --- ## [Logstash config file without input section](https://discuss.elastic.co/t/logstash-config-file-without-input-section/267931)
**Author:** [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\ **Replies:** 1\ **Last updated:** [March 22, 2021, 2:24am UTC](https://discuss.elastic.co/t/logstash-config-file-without-input-section/267931 "2021-03-22T02:24:55Z")
Hi All, I am wondering, shall we use a logstash config file without the input section? Apparently, it works without complain. I tried that using settings like below: input { stdin{} } filter { ruby { # here i … --- ## [Packetbeat: Trying to find utilization across network](https://discuss.elastic.co/t/packetbeat-trying-to-find-utilization-across-network/267368)
**Author:** [@Sandeep\_Raju](https://discuss.elastic.co/u/Sandeep_Raju)\ **Replies:** 1\ **Last updated:** [March 22, 2021, 2:03am UTC](https://discuss.elastic.co/t/packetbeat-trying-to-find-utilization-across-network/267368 "2021-03-22T02:03:58Z")
Hi all, I'm trying to find Network devices in network with over 70% utilization. I'm unable to find the data in my packetbeat index. Any way to go about to do visualization. Thanks in Advance. --- ## [Packetbeat : Detecting Top applications usage on Network](https://discuss.elastic.co/t/packetbeat-detecting-top-applications-usage-on-network/267366)
**Author:** [@Sandeep\_Raju](https://discuss.elastic.co/u/Sandeep_Raju)\ **Replies:** 1\ **Last updated:** [March 22, 2021, 2:03am UTC](https://discuss.elastic.co/t/packetbeat-detecting-top-applications-usage-on-network/267366 "2021-03-22T02:03:40Z")
Hi all, I'm using packetbeat and building visualizations based on network data. I'm trying to see top applications usage across network. I'm not getting any data regarding this. Any solution for this. Thanks in adva… --- ## [How to stop logstash when ruby filter encountered error](https://discuss.elastic.co/t/how-to-stop-logstash-when-ruby-filter-encountered-error/267899)
**Author:** [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\ **Replies:** 4\ **Last updated:** [March 22, 2021, 2:02am UTC](https://discuss.elastic.co/t/how-to-stop-logstash-when-ruby-filter-encountered-error/267899 "2021-03-22T02:02:57Z")
Hi, May i know how to stop logstash from indexing when error found inside ruby filter? i have following config file which will call external python script to process data and insert the records into elasticsearch. It … --- ## [Indices name Can't use IP?](https://discuss.elastic.co/t/indices-name-cant-use-ip/267439)
**Author:** [@jun\_li](https://discuss.elastic.co/u/jun_li)\ **Replies:** 1\ **Last updated:** [March 22, 2021, 2:02am UTC](https://discuss.elastic.co/t/indices-name-cant-use-ip/267439 "2021-03-22T02:02:22Z")
I did a test to create index: Add Index PUT /access.log.2020.04.07/\_doc/1 { "domain":"www.amazon.com", "status":200 } #There is no problem when I query this index, GET \_cat/indices yellow open access.log.2020.04.0… --- ## [How the real data is stored in data disk?](https://discuss.elastic.co/t/how-the-real-data-is-stored-in-data-disk/267476)
**Author:** [@Jax\_dev](https://discuss.elastic.co/u/Jax_dev)\ **Replies:** 1\ **Last updated:** [March 22, 2021, 1:58am UTC](https://discuss.elastic.co/t/how-the-real-data-is-stored-in-data-disk/267476 "2021-03-22T01:58:46Z")
I have executed below api GET "/\_cat/shards?v" it has shown all the indices in which one of the indices has taken 302 GB. When I have checked the mounted disk ( mentioned in path.data ) which elastic is being used, it s… --- ## [How much disk storage required to take backup of index which is having 300GB?](https://discuss.elastic.co/t/how-much-disk-storage-required-to-take-backup-of-index-which-is-having-300gb/267424)
**Author:** [@Jax\_dev](https://discuss.elastic.co/u/Jax_dev)\ **Replies:** 12\ **Last updated:** [March 22, 2021, 1:55am UTC](https://discuss.elastic.co/t/how-much-disk-storage-required-to-take-backup-of-index-which-is-having-300gb/267424 "2021-03-22T01:55:57Z")
If we want to take a back of one of the indices which is having 300GB, enabling compression - true. How much space snapshot will occupy? How do we take snapshot to azure blob storage? Elastic server cluster is on azure… --- ## [How to fix Unassigned state for my indexes?](https://discuss.elastic.co/t/how-to-fix-unassigned-state-for-my-indexes/267560)
**Author:** [@andy1122](https://discuss.elastic.co/u/andy1122)\ **Replies:** 1\ **Last updated:** [March 22, 2021, 1:54am UTC](https://discuss.elastic.co/t/how-to-fix-unassigned-state-for-my-indexes/267560 "2021-03-22T01:54:08Z")
I have ElasticSearch configured in Docker and loading data from Magento into ES via indexer. When quering ES indexes I can see all the data is there but when I try to access it from my localhost:8080 where my connection … --- ## [Help With Validating This Heartbeat Config](https://discuss.elastic.co/t/help-with-validating-this-heartbeat-config/267874)
**Author:** [@LongBeachHXC](https://discuss.elastic.co/u/LongBeachHXC)\ **Replies:** 2\ **Last updated:** [March 22, 2021, 1:53am UTC](https://discuss.elastic.co/t/help-with-validating-this-heartbeat-config/267874 "2021-03-22T01:53:14Z")
I am having a tough time starting a heartbeat container. I am bind mounting the config file into the path /usr/share/heartbeat/heartbeat.yml. My config file is below. I'm sure this is something simple but I haven't been … --- ## [How to create a document using the .NET library?](https://discuss.elastic.co/t/how-to-create-a-document-using-the-net-library/267584)
**Author:** [@jay2021](https://discuss.elastic.co/u/jay2021)\ **Replies:** 0\ **Last updated:** [March 17, 2021, 11:20pm UTC](https://discuss.elastic.co/t/how-to-create-a-document-using-the-net-library/267584 "2021-03-17T23:20:33Z")
Hello, I am trying to create a document using the .NET library. Here is the code that I attempted var newLogResponse = await elasticClient.CreateAsync(new ElasticJobLog() { Create… --- ## [Indices don't have primary shards while taking snapshot](https://discuss.elastic.co/t/indices-dont-have-primary-shards-while-taking-snapshot/267619)
**Author:** [@Venkatesh\_Kumar](https://discuss.elastic.co/u/Venkatesh_Kumar)\ **Replies:** 1\ **Last updated:** [March 22, 2021, 1:44am UTC](https://discuss.elastic.co/t/indices-dont-have-primary-shards-while-taking-snapshot/267619 "2021-03-22T01:44:22Z")
Hi , I am trying to take snapshot using curl -X PUT "http://localhost:9200/\_snapshot/repo1/snap1?wait\_for\_completion=true&pretty" -H 'Content-Type: application/json' -d" { \\"indices\\": \[ \\"apm-7.10.1-transaction-00… --- ## [Filebeat misp module](https://discuss.elastic.co/t/filebeat-misp-module/267843)
**Author:** [@Charles100](https://discuss.elastic.co/u/Charles100)\ **Replies:** 2\ **Last updated:** [March 22, 2021, 1:09am UTC](https://discuss.elastic.co/t/filebeat-misp-module/267843 "2021-03-22T01:09:16Z")
Hi, I'm playing the filebeat MISP module and getting this error when starting filebeat : : ERROR fileset/factory.go:97 Error creating input: (assert) value of type 'string' not convertible into unsupported go type '… --- ## [Elastic and kibana 7.12 (or higher) compatibility with logstash , beats etc older 7.x](https://discuss.elastic.co/t/elastic-and-kibana-7-12-or-higher-compatibility-with-logstash-beats-etc-older-7-x/267850)
**Author:** [@xyz2](https://discuss.elastic.co/u/xyz2)\ **Replies:** 1\ **Last updated:** [March 22, 2021, 12:47am UTC](https://discuss.elastic.co/t/elastic-and-kibana-7-12-or-higher-compatibility-with-logstash-beats-etc-older-7-x/267850 "2021-03-22T00:47:26Z")
HI, Is elasticsearch and kibana 7.12 or 7.13 compatible with logstash, metricbeat, filebeat, heartbeat 7.x older version ? I see the that it is compatible until 7.11 though. --- ## ["reason"=\>"failed to parse field \[Time\] of type \[date\] in document with id 'xxxxxxxxxxxxxxx'](https://discuss.elastic.co/t/reason-failed-to-parse-field-time-of-type-date-in-document-with-id-xxxxxxxxxxxxxxx/267473)
**Author:** [@BlackCat](https://discuss.elastic.co/u/BlackCat)\ **Replies:** 2\ **Last updated:** [March 22, 2021, 12:26am UTC](https://discuss.elastic.co/t/reason-failed-to-parse-field-time-of-type-date-in-document-with-id-xxxxxxxxxxxxxxx/267473 "2021-03-22T00:26:35Z")
I set config of logstash, But happen warning message. Please, how to do resolve. I want to send below log to Elasticsearch. ////////////////// 2021-03-17 10:27:50.115 +0900 26240 main : INFO com.tableausoftware.acti… --- ## [Yum repos returning: Not Found](https://discuss.elastic.co/t/yum-repos-returning-not-found/267924)
**Author:** [@lachlan.simpson](https://discuss.elastic.co/u/lachlan.simpson)\ **Replies:** 3\ **Last updated:** [March 21, 2021, 11:50pm UTC](https://discuss.elastic.co/t/yum-repos-returning-not-found/267924 "2021-03-21T23:50:12Z")
Following the documentation for installation, I'm getting not found on the regular and oss repositories? I'm not the first to experience this in 2021. Has the OS installation method been deprecated but not removed from… --- ## [Elastic Search Null pointer exception in two different join types while fetching their parents](https://discuss.elastic.co/t/elastic-search-null-pointer-exception-in-two-different-join-types-while-fetching-their-parents/267922)
**Author:** [@Syed\_ali\_abbas\_shah](https://discuss.elastic.co/u/Syed_ali_abbas_shah)\ **Replies:** 0\ **Last updated:** [March 21, 2021, 11:00pm UTC](https://discuss.elastic.co/t/elastic-search-null-pointer-exception-in-two-different-join-types-while-fetching-their-parents/267922 "2021-03-21T23:00:58Z")
I have very simple mapping in Elastic Search 6.x version, and following is my mapping with two different types bookmarks user can also be bookmarked and post can also be bookmarked, { "properties": { "type": { … --- ## [Storing value and attaching it later](https://discuss.elastic.co/t/storing-value-and-attaching-it-later/267892)
**Author:** [@thaideval](https://discuss.elastic.co/u/thaideval)\ **Replies:** 2\ **Last updated:** [March 21, 2021, 4:10pm UTC](https://discuss.elastic.co/t/storing-value-and-attaching-it-later/267892 "2021-03-21T16:10:14Z")
Hello, so i have an event like this: A B С B С .. B С And i can parse it getting "A" value + making an array of "B" and "C". That's how i am handling it right now. But i got curious - if it's possible to store … --- ## [Best way to handle "circuit\_breaking\_exception"](https://discuss.elastic.co/t/best-way-to-handle-circuit-breaking-exception/267913)
**Author:** [@Leibo](https://discuss.elastic.co/u/Leibo)\ **Replies:** 0\ **Last updated:** [March 21, 2021, 4:13pm UTC](https://discuss.elastic.co/t/best-way-to-handle-circuit-breaking-exception/267913 "2021-03-21T16:13:50Z")
Hi, Our micro-service is written in Java. We are facing the "circuit\_breaking\_exception" exception occasionally while trying to index documents from multiple threads. How is the best way to handle it without changing … --- ## [Possiblities of bucket level subtraction](https://discuss.elastic.co/t/possiblities-of-bucket-level-subtraction/267911)
**Author:** [@venkat\_swaminathan](https://discuss.elastic.co/u/venkat_swaminathan)\ **Replies:** 0\ **Last updated:** [March 21, 2021, 3:46pm UTC](https://discuss.elastic.co/t/possiblities-of-bucket-level-subtraction/267911 "2021-03-21T15:46:36Z")
Good Day, I am collecting metric information from my provider and pushing them to ES through a python script. all I need to do is subtract "MetricName" , (i.e: Count - 5xx) with respect to time on total system availabi… --- ## [Kibana: show logs with a specifc field name value only](https://discuss.elastic.co/t/kibana-show-logs-with-a-specifc-field-name-value-only/267773)
**Author:** [@h\_q](https://discuss.elastic.co/u/h_q)\ **Replies:** 2\ **Last updated:** [March 21, 2021, 3:46pm UTC](https://discuss.elastic.co/t/kibana-show-logs-with-a-specifc-field-name-value-only/267773 "2021-03-21T15:46:36Z")
I started ingesting Kubernetes logs into ElasticSearch, using the Filebeat Input Container: Filebeat -\> Logstash -\> ElasticSearch My filebeat.yml looks like this: - type: container paths: - /var/lib/docker/conta… --- ## [Extracting a field further after its been parsed by Grok Filter](https://discuss.elastic.co/t/extracting-a-field-further-after-its-been-parsed-by-grok-filter/267883)
**Author:** [@rmoss25](https://discuss.elastic.co/u/rmoss25)\ **Replies:** 3\ **Last updated:** [March 21, 2021, 12:44pm UTC](https://discuss.elastic.co/t/extracting-a-field-further-after-its-been-parsed-by-grok-filter/267883 "2021-03-21T12:44:48Z")
HI, Is there a way to parse a field further after its already been parsed by grok? So after I parse out DN from the main message, I now want to parse DN further as it contains a username. So I want to parse out test.T… [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1497) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1499)