# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1922 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1923 --- ## [Logstash date filter can not convert certain range of unix time](https://discuss.elastic.co/t/logstash-date-filter-can-not-convert-certain-range-of-unix-time/241425)
**Author:** [@Rex\_lin](https://discuss.elastic.co/u/Rex_lin)\ **Replies:** 2\ **Last updated:** [July 17, 2020, 6:09am UTC](https://discuss.elastic.co/t/logstash-date-filter-can-not-convert-certain-range-of-unix-time/241425 "2020-07-17T06:09:46Z")
I have met some date filter problem, when tried to convert my unix time to seconds. As my picture, when I input "2903299199" , I except the result is "2061-12-31T23:59:59+00:00 in ISO 8601, however it returned failure… --- ## [Would there be an impact / difference of Big and Small Indices?](https://discuss.elastic.co/t/would-there-be-an-impact-difference-of-big-and-small-indices/241560)
**Author:** [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Replies:** 8\ **Last updated:** [July 17, 2020, 6:05am UTC](https://discuss.elastic.co/t/would-there-be-an-impact-difference-of-big-and-small-indices/241560 "2020-07-17T06:05:55Z")
Hello and good day! Just like what the title says, I humbly request for you guys' expertise and a healthy, bountiful discussion regarding this.... My team has 2 indices called "web" and "socialmedia", both of them got … --- ## [Monitoring execution failed on cluster in ElasticSearch SaaS service](https://discuss.elastic.co/t/monitoring-execution-failed-on-cluster-in-elasticsearch-saas-service/241568)
**Author:** [@jeromeb](https://discuss.elastic.co/u/jeromeb)\ **Replies:** 2\ **Last updated:** [July 17, 2020, 5:58am UTC](https://discuss.elastic.co/t/monitoring-execution-failed-on-cluster-in-elasticsearch-saas-service/241568 "2020-07-17T05:58:55Z")
Hi I have a cluster on ElasticSearch Service (the managed service solution) in version 7.7. It has 2 data nodes (AWS.DATA.HIGHIO.I3 8GB) + a Master (AWS.MASTER.R4). Am trying to enable Monitoring on it but I am facing … --- ## [Still getting 'max open shards' error after re-indexing and decreasing shards to 850](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369)
**Author:** [@anda](https://discuss.elastic.co/u/anda)\ **Replies:** 9\ **Last updated:** [July 17, 2020, 5:42am UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369 "2020-07-17T05:42:49Z")
Hello, I'm running ELK 7.4 and have been getting this error in Logstash logs for basically every event that's being processed: \[2020-07-15T15:24:06,789\]\[WARN \]\[logstash.outputs.elasticsearch\]\[enm\] Could not index event… --- ## [RabbitMQ MetricBeat messages totals](https://discuss.elastic.co/t/rabbitmq-metricbeat-messages-totals/241446)
**Author:** [@mdymel](https://discuss.elastic.co/u/mdymel)\ **Replies:** 2\ **Last updated:** [July 17, 2020, 5:38am UTC](https://discuss.elastic.co/t/rabbitmq-metricbeat-messages-totals/241446 "2020-07-17T05:38:11Z")
Is it possible to visualize total message counts in Kibana when using RabbitMQ MetricBeat? The RMQ Management Plugin displays it at the top of the overview: But I can't find this information in the node metrics. --- ## [Mapping Value is not changing after updating](https://discuss.elastic.co/t/mapping-value-is-not-changing-after-updating/241437)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 2\ **Last updated:** [July 17, 2020, 5:35am UTC](https://discuss.elastic.co/t/mapping-value-is-not-changing-after-updating/241437 "2020-07-17T05:35:39Z")
"mappings": { "properties": { "client.ip":{"type": "ip"}, "host.ip":{"type": "ip"}, "server.ip":{"type": "ip"}, "source.ip":{"type": "ip"}, "destination.ip":{"type": "ip"}, "geoip": { "dynamic": true, "properti… --- ## [How to check filebeat log delivery rate](https://discuss.elastic.co/t/how-to-check-filebeat-log-delivery-rate/241458)
**Author:** [@mahi7](https://discuss.elastic.co/u/mahi7)\ **Replies:** 2\ **Last updated:** [July 17, 2020, 5:30am UTC](https://discuss.elastic.co/t/how-to-check-filebeat-log-delivery-rate/241458 "2020-07-17T05:30:57Z")
Hello , I have started using filebeat recently , How can i check on what rate my filebeat is sending data to my logstash server. This is my filebeat.yml file filebeat: prospectors: - paths: - /hom… --- ## [CircuitBreakingException Data too large](https://discuss.elastic.co/t/circuitbreakingexception-data-too-large/237953)
**Author:** [@Barak](https://discuss.elastic.co/u/Barak)\ **Replies:** 3\ **Last updated:** [July 17, 2020, 3:33am UTC](https://discuss.elastic.co/t/circuitbreakingexception-data-too-large/237953 "2020-07-17T03:33:55Z")
Hi, I have an ES stack running on AWS spot instances, so shard reallocation happens quite frequently. Occasionally I receive the following error message and then the shard remains unassigned without further allocation … --- ## [Do you need to keep the small version always in elk and kibana](https://discuss.elastic.co/t/do-you-need-to-keep-the-small-version-always-in-elk-and-kibana/241571)
**Author:** [@111351](https://discuss.elastic.co/u/111351)\ **Replies:** 1\ **Last updated:** [July 17, 2020, 3:28am UTC](https://discuss.elastic.co/t/do-you-need-to-keep-the-small-version-always-in-elk-and-kibana/241571 "2020-07-17T03:28:36Z")
Do you need to keep the small version always in elk and kibana if I use logstash5.6.16 and elasticsearch5.4 will there be a problem thanks for you --- ## [How to set default mapping to “keyword” for dynamically added properties in Elasticsearch](https://discuss.elastic.co/t/how-to-set-default-mapping-to-keyword-for-dynamically-added-properties-in-elasticsearch/241485)
**Author:** [@gitterhh](https://discuss.elastic.co/u/gitterhh)\ **Replies:** 1\ **Last updated:** [July 17, 2020, 3:26am UTC](https://discuss.elastic.co/t/how-to-set-default-mapping-to-keyword-for-dynamically-added-properties-in-elasticsearch/241485 "2020-07-17T03:26:13Z")
In our application we add new documents which can have new unknown properties. Is it possible to set default mapping to "keyword" for dynamically added new properties in Elasticsearch? --- ## [Aggregate filter gets different output even with same input](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394)
**Author:** [@Aberry](https://discuss.elastic.co/u/Aberry)\ **Replies:** 4\ **Last updated:** [July 17, 2020, 2:33am UTC](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394 "2020-07-17T02:33:42Z")
I encountered similar problem again while using a mass of test data. This time to set pipeline.workers=1 and pipeline.java\_execution = false in logstash.yml cannot help anymore. I have 2 input files like below - inboun… --- ## [Shards unbalanced distribution](https://discuss.elastic.co/t/shards-unbalanced-distribution/241569)
**Author:** [@LuckyNemo](https://discuss.elastic.co/u/LuckyNemo)\ **Replies:** 1\ **Last updated:** [July 17, 2020, 2:18am UTC](https://discuss.elastic.co/t/shards-unbalanced-distribution/241569 "2020-07-17T02:18:11Z")
ES version 6.0.0 8 nodes(5 datanode 3 ingests,all are 8c16g config) I have a index (5 primary 1 replica) mapping like this { "myindex": { "mappings": { "order": { "properties": { "addTime… --- ## [Convert winbeats back to Raw windows event](https://discuss.elastic.co/t/convert-winbeats-back-to-raw-windows-event/241567)
**Author:** [@bcl](https://discuss.elastic.co/u/bcl)\ **Replies:** 0\ **Last updated:** [July 17, 2020, 1:56am UTC](https://discuss.elastic.co/t/convert-winbeats-back-to-raw-windows-event/241567 "2020-07-17T01:56:16Z")
Hi All, We currently use ELK in the following way winbeats,filebeats,syslog -\> logstash ingest nodes -\> kafka cluster -\> logstash grok nodes -\> Elastic cluster We have a requirement to send winbeats logs to our securi… --- ## [ES 7.4 Field \[start\_time\] of type \[keyword\] does not support custom formats](https://discuss.elastic.co/t/es-7-4-field-start-time-of-type-keyword-does-not-support-custom-formats/241554)
**Author:** [@ghirsch](https://discuss.elastic.co/u/ghirsch)\ **Replies:** 5\ **Last updated:** [July 17, 2020, 1:55am UTC](https://discuss.elastic.co/t/es-7-4-field-start-time-of-type-keyword-does-not-support-custom-formats/241554 "2020-07-17T01:55:10Z")
Hi, I have an elasticsearch index with dynamic field mapping. I am inserting a log into the index with a field start\_time in the RFC3339 format (e.g. "2020-07-15T23:45:16Z") and until today this field was getting discove… --- ## [Ml detector with exclude frequent enabled](https://discuss.elastic.co/t/ml-detector-with-exclude-frequent-enabled/241387)
**Author:** [@saraKM](https://discuss.elastic.co/u/saraKM)\ **Replies:** 5\ **Last updated:** [July 16, 2020, 11:40pm UTC](https://discuss.elastic.co/t/ml-detector-with-exclude-frequent-enabled/241387 "2020-07-16T23:40:08Z")
Hi, I have two jobs (Each job has two detectors with max and high\_mean functions) working on same data and exclude frequent enabled for one. I have seen some high severity anomalies raised when exclude frequent is none,… --- ## [Reindex 350K data from RestHighLevelClient without timeout & shard fail](https://discuss.elastic.co/t/reindex-350k-data-from-resthighlevelclient-without-timeout-shard-fail/241467)
**Author:** [@sandeep\_rana](https://discuss.elastic.co/u/sandeep_rana)\ **Replies:** 1\ **Last updated:** [July 16, 2020, 11:17pm UTC](https://discuss.elastic.co/t/reindex-350k-data-from-resthighlevelclient-without-timeout-shard-fail/241467 "2020-07-16T23:17:37Z")
Why the timeout or fail shared while reindex 350k data with same mapping and settings --- ## [Elasticsearch 7.x with Puppet 3.x](https://discuss.elastic.co/t/elasticsearch-7-x-with-puppet-3-x/241529)
**Author:** [@maatthias](https://discuss.elastic.co/u/maatthias)\ **Replies:** 0\ **Last updated:** [July 16, 2020, 6:40pm UTC](https://discuss.elastic.co/t/elasticsearch-7-x-with-puppet-3-x/241529 "2020-07-16T18:40:10Z")
We are interested in using puppet to instrument our ELK stack. The ELK cluster is currently running 7.6.2. Puppet is older version 3.8.7. Is anyone aware of any successful puppet configurations in this arrangement?… --- ## [Datestamp to timestamp](https://discuss.elastic.co/t/datestamp-to-timestamp/241550)
**Author:** [@merlock](https://discuss.elastic.co/u/merlock)\ **Replies:** 2\ **Last updated:** [July 16, 2020, 11:12pm UTC](https://discuss.elastic.co/t/datestamp-to-timestamp/241550 "2020-07-16T23:12:35Z")
hello all, i have the next format of log 06/17/2020 00:00:00 - 1.6.250.107 o00044696e20022001200940005 my filter match =\> { "message" =\> ".%{DATESTAMP:LOGTIME}.-.\*%{IP}.\*w(?.{10})(?\.{5})(?.{3})(?\ **Author:** [@Yamini\_Shashank](https://discuss.elastic.co/u/Yamini_Shashank)\ **Replies:** 3\ **Last updated:** [July 16, 2020, 10:39pm UTC](https://discuss.elastic.co/t/elastic-cluster-on-azure-cloud-linux-one-server-fails-to-startup-while-the-other-is-successful/241021 "2020-07-16T22:39:58Z") Here is a configuration of a 2 node elastic cluster: node.max\_local\_storage\_nodes: 2 cluster.name: elasticsearch node.name: server2 path.data: /etc/elasticsearch/data path.logs: /etc/elasticsearch/logs discovery.se… --- ## [How apply a security of data and services into Elastcisearch](https://discuss.elastic.co/t/how-apply-a-security-of-data-and-services-into-elastcisearch/241428)
**Author:** [@vero](https://discuss.elastic.co/u/vero)\ **Replies:** 3\ **Last updated:** [July 16, 2020, 10:39pm UTC](https://discuss.elastic.co/t/how-apply-a-security-of-data-and-services-into-elastcisearch/241428 "2020-07-16T22:39:29Z")
Hi, Do you have please some ideas on how we securise our data into Elasticsearch ? As my information we don't have a DataLake. On the other hand, how we securise also the data shifting between services, could we apply… --- ## [Heartbeat to Logstash disappears in Elasticsearch (Indices)](https://discuss.elastic.co/t/heartbeat-to-logstash-disappears-in-elasticsearch-indices/241359)
**Author:** [@Asinus1223](https://discuss.elastic.co/u/Asinus1223)\ **Replies:** 3\ **Last updated:** [July 16, 2020, 10:38pm UTC](https://discuss.elastic.co/t/heartbeat-to-logstash-disappears-in-elasticsearch-indices/241359 "2020-07-16T22:38:55Z")
How can I get Heartbeat to output to Logstash where I can manipulate it some? Right now the only way I can anything to appear in Elasticsearch/Kibana is to output directly to Elasticsearch and keep the default index nam… --- ## [Make logstash filter by name](https://discuss.elastic.co/t/make-logstash-filter-by-name/241526)
**Author:** [@elk6](https://discuss.elastic.co/u/elk6)\ **Replies:** 3\ **Last updated:** [July 16, 2020, 9:49pm UTC](https://discuss.elastic.co/t/make-logstash-filter-by-name/241526 "2020-07-16T21:49:59Z")
I have a log file called "/var/log/commands.log" that I'm trying to separate into fields with logstash & grok. I've got it working. Now, I'm trying to make logstash only do this to the file "/var/log/commands.log" and no… --- ## [Alias name duplication](https://discuss.elastic.co/t/alias-name-duplication/241364)
**Author:** [@Ugon](https://discuss.elastic.co/u/Ugon)\ **Replies:** 2\ **Last updated:** [July 16, 2020, 9:17pm UTC](https://discuss.elastic.co/t/alias-name-duplication/241364 "2020-07-16T21:17:12Z")
Hello :slight\_smile: After restart of one of my clusters, data nodes fail to start because of alias name duplication. On my other clusters I have similar situations (alias name pointing to multiple indices) and i can re… --- ## [Need help using a bucket\_script with single-value or filters filters](https://discuss.elastic.co/t/need-help-using-a-bucket-script-with-single-value-or-filters-filters/241544)
**Author:** [@teebu](https://discuss.elastic.co/u/teebu)\ **Replies:** 0\ **Last updated:** [July 16, 2020, 8:56pm UTC](https://discuss.elastic.co/t/need-help-using-a-bucket-script-with-single-value-or-filters-filters/241544 "2020-07-16T20:56:41Z")
I'm trying to compute count of a nested field where active and inactive is not 0 for both values. But I can't get my aggs working inside the bucket\_script. Initially I tried using the active\_entities" : { "filter" but I … --- ## [Logstash Netflow Codec Issues](https://discuss.elastic.co/t/logstash-netflow-codec-issues/241500)
**Author:** [@csgeek](https://discuss.elastic.co/u/csgeek)\ **Replies:** 1\ **Last updated:** [July 16, 2020, 8:26pm UTC](https://discuss.elastic.co/t/logstash-netflow-codec-issues/241500 "2020-07-16T20:26:26Z")
I'm having issues handling netflow data through the logstash codec. I was wondering if anyone had any insight that could help. Logstash version: 7.6.2 (I'm happy to upgrade if need be not tied to a specific version) f… --- ## [How to access aggregations from within custom request handler](https://discuss.elastic.co/t/how-to-access-aggregations-from-within-custom-request-handler/226445)
**Author:** [@ssimmons](https://discuss.elastic.co/u/ssimmons)\ **Replies:** 4\ **Last updated:** [July 16, 2020, 8:23pm UTC](https://discuss.elastic.co/t/how-to-access-aggregations-from-within-custom-request-handler/226445 "2020-07-16T20:23:32Z")
Is it possible to access the aggregations from within a custom visualization request handler? I'm currently not seeing the aggs object contained within the object that is passed into the custom request handler. I'm tryin… --- ## [Tag \_grokparsefailure when parsing logs via logstash](https://discuss.elastic.co/t/tag-grokparsefailure-when-parsing-logs-via-logstash/241489)
**Author:** [@Rahul\_Aggarwal](https://discuss.elastic.co/u/Rahul_Aggarwal)\ **Replies:** 3\ **Last updated:** [July 16, 2020, 8:18pm UTC](https://discuss.elastic.co/t/tag-grokparsefailure-when-parsing-logs-via-logstash/241489 "2020-07-16T20:18:58Z")
Below are 2 logs of same cloudfront distribution, patterns of logs same but when processing with logstash same filter showing tag \_grokparsefailure for first line but no any failure for second line, Please help to remove… --- ## [Logstash Patterns - Firewalls - Best Place to Find it](https://discuss.elastic.co/t/logstash-patterns-firewalls-best-place-to-find-it/239107)
**Author:** [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Replies:** 4\ **Last updated:** [July 16, 2020, 8:15pm UTC](https://discuss.elastic.co/t/logstash-patterns-firewalls-best-place-to-find-it/239107 "2020-07-16T20:15:43Z")
My use case is Cisco ASA firewall logs but I think these questions apply more broadly. I am trying do the parsing of Cisco ASA logs in logstash, not using Filebeat. I'd like to not reinvent the wheel so where can I f… --- ## [Filebeat, Cisco ASA and ECS fields](https://discuss.elastic.co/t/filebeat-cisco-asa-and-ecs-fields/238381)
**Author:** [@rossw](https://discuss.elastic.co/u/rossw)\ **Replies:** 5\ **Last updated:** [July 16, 2020, 7:40pm UTC](https://discuss.elastic.co/t/filebeat-cisco-asa-and-ecs-fields/238381 "2020-07-16T19:40:38Z")
Filebeat 7.7, Cisco ASA logs ASA syslog -\> logstash for filtering -\> filebeat (as original raw syslog) -\> cisco module/asa -\> logstash -\> ES According to the recommendations from Elastic, the firewall should be the "ob… --- ## [Implications of using serial ids for documents](https://discuss.elastic.co/t/implications-of-using-serial-ids-for-documents/241355)
**Author:** [@terell](https://discuss.elastic.co/u/terell)\ **Replies:** 3\ **Last updated:** [July 16, 2020, 7:27pm UTC](https://discuss.elastic.co/t/implications-of-using-serial-ids-for-documents/241355 "2020-07-16T19:27:02Z")
I'm planning on linking documents I write to ES to records in my relational database by simply taking the record id and writing it to the document id. This means the IDs for my documents will be serial (ie. 1,2,3,4,5) in… [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1921) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1923)