# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1635 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1636 --- ## [How to filter and narrow the data before aggregation](https://discuss.elastic.co/t/how-to-filter-and-narrow-the-data-before-aggregation/259147)
**Author:** [@jun\_miao](https://discuss.elastic.co/u/jun_miao)\ **Replies:** 2\ **Last updated:** [December 21, 2020, 1:44am UTC](https://discuss.elastic.co/t/how-to-filter-and-narrow-the-data-before-aggregation/259147 "2020-12-21T01:44:37Z")
Hello, all I have a query which group by groupId. { "aggs":{ "result":{ "terms":{ "field":"groupId", "size": 100 } } }, "size": 0 } Because the index co… --- ## [Should we create New Index or Use the same index for all application logs(number of app ~ 5)](https://discuss.elastic.co/t/should-we-create-new-index-or-use-the-same-index-for-all-application-logs-number-of-app-5/259011)
**Author:** [@SHOAIB\_AHMED](https://discuss.elastic.co/u/SHOAIB_AHMED)\ **Replies:** 3\ **Last updated:** [December 20, 2020, 11:52pm UTC](https://discuss.elastic.co/t/should-we-create-new-index-or-use-the-same-index-for-all-application-logs-number-of-app-5/259011 "2020-12-20T23:52:13Z")
I have about 5 apps to monitor using log parsing with ELK. Does it make sense to create new index for each application log or only one index is sufficient to store all app logs. The logs for these apps would also contai… --- ## [Q: ILM and Beats index-template thoughts in a lab 7.10.x setup](https://discuss.elastic.co/t/q-ilm-and-beats-index-template-thoughts-in-a-lab-7-10-x-setup/259208)
**Author:** [@Andreas\_Falk](https://discuss.elastic.co/u/Andreas_Falk)\ **Replies:** 0\ **Last updated:** [December 20, 2020, 11:51pm UTC](https://discuss.elastic.co/t/q-ilm-and-beats-index-template-thoughts-in-a-lab-7-10-x-setup/259208 "2020-12-20T23:51:50Z")
Hi, We are setting up a new 7.10.1 cluster and are struggling with some ILM and beat stuff :slight\_smile: First up, when doing a simple template setup through Filebeat docker run \\ docker.elastic.co/beats/filebeat:7… --- ## [ES hadoop Spark query returns too many partitions](https://discuss.elastic.co/t/es-hadoop-spark-query-returns-too-many-partitions/259201)
**Author:** [@Greg\_Jensen](https://discuss.elastic.co/u/Greg_Jensen)\ **Replies:** 0\ **Last updated:** [December 20, 2020, 8:04pm UTC](https://discuss.elastic.co/t/es-hadoop-spark-query-returns-too-many-partitions/259201 "2020-12-20T20:04:51Z")
I have a Java Spark application that is querying 15K \_ids. The query is against an alias, that is backed by 15 or more indexes, each with ~ 9 shards. This query is taking far longer then is necessary because based on the… --- ## [Parsing ip address coming at random places inside logs](https://discuss.elastic.co/t/parsing-ip-address-coming-at-random-places-inside-logs/259177)
**Author:** [@Muhammad\_Faisal](https://discuss.elastic.co/u/Muhammad_Faisal)\ **Replies:** 4\ **Last updated:** [December 20, 2020, 4:05pm UTC](https://discuss.elastic.co/t/parsing-ip-address-coming-at-random-places-inside-logs/259177 "2020-12-20T16:05:12Z")
hi guys, input log message contains IP address but the occurence of ip is random inside message and also some times the message will not contain any ip ...how can we extract this ip address using grok or other filters. … --- ## [My index contains all geo data but not showing in list while creating Maps from that index](https://discuss.elastic.co/t/my-index-contains-all-geo-data-but-not-showing-in-list-while-creating-maps-from-that-index/259196)
**Author:** [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\ **Replies:** 8\ **Last updated:** [December 20, 2020, 7:31pm UTC](https://discuss.elastic.co/t/my-index-contains-all-geo-data-but-not-showing-in-list-while-creating-maps-from-that-index/259196 "2020-12-20T19:31:36Z")
I am performing Geo lookup via Logstash and added geoip filter to get required data. I am getting data in logs but some how it is not getting listed in indexes while configuring layers in Map. Here is my conf file. inpu… --- ## [Need to parse logs and enrich some values from mysql data at logstats](https://discuss.elastic.co/t/need-to-parse-logs-and-enrich-some-values-from-mysql-data-at-logstats/259176)
**Author:** [@abhi.coerian](https://discuss.elastic.co/u/abhi.coerian)\ **Replies:** 3\ **Last updated:** [December 20, 2020, 3:58pm UTC](https://discuss.elastic.co/t/need-to-parse-logs-and-enrich-some-values-from-mysql-data-at-logstats/259176 "2020-12-20T15:58:07Z")
Need to parse logs on logstash and enrich some values from mysql db based on the log message and generate json message . e.x JOB application\_1603918538928\_3286731 QUEUE default USER xyz Need to fetch details of USER :… --- ## [CircuitBreaking Exception](https://discuss.elastic.co/t/circuitbreaking-exception/258568)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 17\ **Last updated:** [December 20, 2020, 1:21pm UTC](https://discuss.elastic.co/t/circuitbreaking-exception/258568 "2020-12-20T13:21:11Z")
I am using ELK cluster(3 master+3 data +2 coordinate) + kafka cluster.I am also using winlogbeat and packetbeat at client side.In kibana discover i saw that winlogbeat data was coming slow and packetbeat data was coming … --- ## [Term query with keyword mapping](https://discuss.elastic.co/t/term-query-with-keyword-mapping/259173)
**Author:** [@klapvee](https://discuss.elastic.co/u/klapvee)\ **Replies:** 1\ **Last updated:** [December 20, 2020, 1:17pm UTC](https://discuss.elastic.co/t/term-query-with-keyword-mapping/259173 "2020-12-20T13:17:36Z")
Hi, a very simple mappping: { "mappings": { "chat": { "properties": { "checksum": { "type": "keyword" }, "created": { "type": "date", "store": true… --- ## [IPv6 endpoint for S3 repository](https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164)
**Author:** [@NominaSumpta](https://discuss.elastic.co/u/NominaSumpta)\ **Replies:** 5\ **Last updated:** [December 20, 2020, 10:17am UTC](https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164 "2020-12-20T10:17:53Z")
How do I use an IPv6 endpoint for an S3 repository? Using a hostname that resolves to an IPv6 address and an IPv4 address results in: { "error": { "root\_cause": \[ { "type": "repository\_verification\_… --- ## [Logs including emojis](https://discuss.elastic.co/t/logs-including-emojis/258946)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 6\ **Last updated:** [December 20, 2020, 9:29am UTC](https://discuss.elastic.co/t/logs-including-emojis/258946 "2020-12-20T09:29:07Z")
Hello everyone! is there a way to parse this log with emojis? Could it be shown in kibana or logstash output? I do not have a clue how to parse the emojis in logstash pipeline. "emoji\_code":"🌸" Can someone please help … --- ## [Pattern for this log](https://discuss.elastic.co/t/pattern-for-this-log/259184)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 6\ **Last updated:** [December 20, 2020, 9:24am UTC](https://discuss.elastic.co/t/pattern-for-this-log/259184 "2020-12-20T09:24:36Z")
Hello everyone! Can someone please help me with writing a grok pattern for this log line? I am not able to seperate its fields. {"name":"test","test\_arr":\[{"key\_word":"ok","test\_code":"👌","h\_code":"1234.png"}\],"create\_d… --- ## [Cannot resolve method 'index' in 'RestClient'](https://discuss.elastic.co/t/cannot-resolve-method-index-in-restclient/259188)
**Author:** [@Murigipudi\_Revanth](https://discuss.elastic.co/u/Murigipudi_Revanth)\ **Replies:** 2\ **Last updated:** [December 20, 2020, 8:23am UTC](https://discuss.elastic.co/t/cannot-resolve-method-index-in-restclient/259188 "2020-12-20T08:23:50Z")
Hello, I'm using ElasticSearch 7.10 and trying to index a very sample document following the documentation from here - https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-document-index… --- ## [High CPU Utilization](https://discuss.elastic.co/t/high-cpu-utilization/259174)
**Author:** [@Gokul6](https://discuss.elastic.co/u/Gokul6)\ **Replies:** 1\ **Last updated:** [December 20, 2020, 6:29am UTC](https://discuss.elastic.co/t/high-cpu-utilization/259174 "2020-12-20T06:29:51Z")
Hi, we have a single node cluster for elasticsearch running in elastic cloud. for the past two days the cpu utilization is 100%. what might be the issue here ?? please help :slightly\_smiling\_face: --- ## [Increased node timeouts after 7.7.0 to 7.10.1 upgrade](https://discuss.elastic.co/t/increased-node-timeouts-after-7-7-0-to-7-10-1-upgrade/258888)
**Author:** [@Aharon](https://discuss.elastic.co/u/Aharon)\ **Replies:** 2\ **Last updated:** [December 19, 2020, 7:58pm UTC](https://discuss.elastic.co/t/increased-node-timeouts-after-7-7-0-to-7-10-1-upgrade/258888 "2020-12-19T19:58:52Z")
We have a largish production cluster that is experiencing a high number of node timeouts (on our hot indexing nodes) since we upgraded to 7.10.1 (from 7.7.0). Prior to the upgrade, we did not experience these timeouts. O… --- ## [Elastic search is not working into ubuntu 18.04](https://discuss.elastic.co/t/elastic-search-is-not-working-into-ubuntu-18-04/259169)
**Author:** [@Balwinder\_Singh](https://discuss.elastic.co/u/Balwinder_Singh)\ **Replies:** 1\ **Last updated:** [December 19, 2020, 7:16pm UTC](https://discuss.elastic.co/t/elastic-search-is-not-working-into-ubuntu-18-04/259169 "2020-12-19T19:16:21Z")
Dear All, Please help me to sort out below mention problem. Dec 17 21:05:14 zammad091212020-Virtual-Machine systemd\[1\]: Starting Elasticsearch... Dec 17 21:06:29 zammad091212020-Virtual-Machine systemd\[1\]: elasticsear… --- ## [Removing ongoing duplicates](https://discuss.elastic.co/t/removing-ongoing-duplicates/259157)
**Author:** [@cyberzlo](https://discuss.elastic.co/u/cyberzlo)\ **Replies:** 1\ **Last updated:** [December 19, 2020, 3:36pm UTC](https://discuss.elastic.co/t/removing-ongoing-duplicates/259157 "2020-12-19T15:36:14Z")
This is what I need however I would like do it on input due parsing jsons. What I mean this tutorial show how do it post-factum, but I would like do it every time I recieve new data basing for example on data from last… --- ## [Missing fields in Kibana](https://discuss.elastic.co/t/missing-fields-in-kibana/256519)
**Author:** [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)\ **Replies:** 1\ **Last updated:** [December 19, 2020, 11:16am UTC](https://discuss.elastic.co/t/missing-fields-in-kibana/256519 "2020-12-19T11:16:40Z")
OS : Centos 7.4, ELK Version: 7.6.2 I am trying to count how many hosts are configured in the ELK stack and for some reason beat.hostname.keyword is not pulling any data. If i use the default metric without any other f… --- ## [Query to search multiple values in a single field with wildcard startswith condition](https://discuss.elastic.co/t/query-to-search-multiple-values-in-a-single-field-with-wildcard-startswith-condition/259155)
**Author:** [@subham\_p](https://discuss.elastic.co/u/subham_p)\ **Replies:** 0\ **Last updated:** [December 19, 2020, 9:52am UTC](https://discuss.elastic.co/t/query-to-search-multiple-values-in-a-single-field-with-wildcard-startswith-condition/259155 "2020-12-19T09:52:50Z")
Diseases: “fever, headache, cold cough” Diseases:” fever, infection” Diseases:” fever,covid\_19” I want to find out the (diseases:” fever,covid\_19”) with in a must with wildcard condition starts with. I have tried wit… --- ## [How to make input processor process all data?](https://discuss.elastic.co/t/how-to-make-input-processor-process-all-data/259067)
**Author:** [@dukeyang](https://discuss.elastic.co/u/dukeyang)\ **Replies:** 2\ **Last updated:** [December 19, 2020, 6:56am UTC](https://discuss.elastic.co/t/how-to-make-input-processor-process-all-data/259067 "2020-12-19T06:56:59Z")
In my application, there are multiple logs that need to process, I want every log has its own processors. BUT processors under specific input only process data collected by this input; - type: log paths: - ./data… --- ## [How to Program APPs in ELK?](https://discuss.elastic.co/t/how-to-program-apps-in-elk/259141)
**Author:** [@xenon-xenon](https://discuss.elastic.co/u/xenon-xenon)\ **Replies:** 1\ **Last updated:** [December 19, 2020, 5:02am UTC](https://discuss.elastic.co/t/how-to-program-apps-in-elk/259141 "2020-12-19T05:02:27Z")
Hi everyone, I just got familiar with ELK , My question is how I can program APPs in ELK (I think most of it is in Kibana) what programming language should I know and how can I learn programming APPs in Kibana and ELK… --- ## [Accessing two various indexes in canvas datatable](https://discuss.elastic.co/t/accessing-two-various-indexes-in-canvas-datatable/259134)
**Author:** [@Alfar](https://discuss.elastic.co/u/Alfar)\ **Replies:** 2\ **Last updated:** [December 19, 2020, 4:45am UTC](https://discuss.elastic.co/t/accessing-two-various-indexes-in-canvas-datatable/259134 "2020-12-19T04:45:21Z")
I want to populate a data table in canvas with this kind of query SELECT student\_id,name,address,department\_id FROM student WHERE department\_id IN ( SELECT department\_id FROM department WHERE department\_name ="Engine… --- ## [Host.os is missing in generated template when host is expected in a custom schema using ECS Tooling 1.6 and 1.7](https://discuss.elastic.co/t/host-os-is-missing-in-generated-template-when-host-is-expected-in-a-custom-schema-using-ecs-tooling-1-6-and-1-7/258786)
**Author:** [@mxu](https://discuss.elastic.co/u/mxu)\ **Replies:** 4\ **Last updated:** [December 18, 2020, 11:46pm UTC](https://discuss.elastic.co/t/host-os-is-missing-in-generated-template-when-host-is-expected-in-a-custom-schema-using-ecs-tooling-1-6-and-1-7/258786 "2020-12-18T23:46:18Z")
Hi, I am using ECS Tooling (1.6 and 1.7) to generate template, adding "host" to a custom schema "Link". Added host.yml as following. In the resulted template.json, while host.geo exists, host.os is missing. After I c… --- ## [How to delete documents older than 10 days - cant use daily indices](https://discuss.elastic.co/t/how-to-delete-documents-older-than-10-days-cant-use-daily-indices/259119)
**Author:** [@Jana\_P](https://discuss.elastic.co/u/Jana_P)\ **Replies:** 6\ **Last updated:** [December 18, 2020, 11:13pm UTC](https://discuss.elastic.co/t/how-to-delete-documents-older-than-10-days-cant-use-daily-indices/259119 "2020-12-18T23:13:14Z")
Hello, I'm trying to figure out how to delete documents older than 10 days. It sounds simple (curator etc.) - but my situation is quite complicated - I have 3 nodes cluster for DEV logs from about 30 environments. So … --- ## [Watcher not letting me pull a timestamp from a log and use it to search from there](https://discuss.elastic.co/t/watcher-not-letting-me-pull-a-timestamp-from-a-log-and-use-it-to-search-from-there/258636)
**Author:** [@cah-cborders](https://discuss.elastic.co/u/cah-cborders)\ **Replies:** 0\ **Last updated:** [December 14, 2020, 7:51pm UTC](https://discuss.elastic.co/t/watcher-not-letting-me-pull-a-timestamp-from-a-log-and-use-it-to-search-from-there/258636 "2020-12-14T19:51:20Z")
I'm trying to set up a Watcher in Kibana that will let me find an log with the message "Exit Status 1", then use the timestamp from that log to chain a new search and go back 20 logs or so from there so I can kind of cre… --- ## [Canvas Group By Month with Chronological order](https://discuss.elastic.co/t/canvas-group-by-month-with-chronological-order/259066)
**Author:** [@willemdh](https://discuss.elastic.co/u/willemdh)\ **Replies:** 2\ **Last updated:** [December 18, 2020, 9:35pm UTC](https://discuss.elastic.co/t/canvas-group-by-month-with-chronological-order/259066 "2020-12-18T21:35:12Z")
Hello, I'm trying for some time to create a graph in canvas of unique users per month, ordered chronologically (oldest month of last year first). Whatever ESSQL query I'm trying, it seems to always order alphabetically. … --- ## [How to create alert in watcher advance? for the basic metrics of CPU, MEMORY](https://discuss.elastic.co/t/how-to-create-alert-in-watcher-advance-for-the-basic-metrics-of-cpu-memory/257705)
**Author:** [@aandrewsllp](https://discuss.elastic.co/u/aandrewsllp)\ **Replies:** 1\ **Last updated:** [December 18, 2020, 9:04pm UTC](https://discuss.elastic.co/t/how-to-create-alert-in-watcher-advance-for-the-basic-metrics-of-cpu-memory/257705 "2020-12-18T21:04:15Z")
How do I add more fields in the message of my alert by watcher --- ## [Is there a way to show alerts as thresholds in visualizations?](https://discuss.elastic.co/t/is-there-a-way-to-show-alerts-as-thresholds-in-visualizations/257726)
**Author:** [@Matheus\_Webler](https://discuss.elastic.co/u/Matheus_Webler)\ **Replies:** 1\ **Last updated:** [December 18, 2020, 8:46pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-show-alerts-as-thresholds-in-visualizations/257726 "2020-12-18T20:46:17Z")
Hello there! I really enjoy the new alerting tools on Kibana, but it is not optimal the way I need to maintain thresholds that are shown in dashboards/visualizations and alerts separately. It is really helpful to see at… --- ## [Elasticsearch bucket\_script agg](https://discuss.elastic.co/t/elasticsearch-bucket-script-agg/259138)
**Author:** [@Alexkl](https://discuss.elastic.co/u/Alexkl)\ **Replies:** 0\ **Last updated:** [December 18, 2020, 8:23pm UTC](https://discuss.elastic.co/t/elasticsearch-bucket-script-agg/259138 "2020-12-18T20:23:03Z")
Hello, There is something i don't understand. I want to get a daily hit ratio calculated on the hit/miss stored in a particular field on my index. I tried to reproduce the doc example adding an extra agg. When i try … --- ## [Correct way to setup the beats index patterns?](https://discuss.elastic.co/t/correct-way-to-setup-the-beats-index-patterns/258360)
**Author:** [@jclemons7](https://discuss.elastic.co/u/jclemons7)\ **Replies:** 5\ **Last updated:** [December 18, 2020, 8:12pm UTC](https://discuss.elastic.co/t/correct-way-to-setup-the-beats-index-patterns/258360 "2020-12-18T20:12:41Z")
Hello and thanks in advance for the guidance. I've been trying to figure this out on my own for a while, but I'm not having any luck so I thought I'd just finally ask. What is the CORRECT way to setup the indexes for t… [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1634) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1636)