# \#none **URL:** https://discuss.elastic.co/tag/none.md?no_tags=true&page=582 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 583 --- ## [Best practices for dynamic expiration](https://discuss.elastic.co/t/best-practices-for-dynamic-expiration/309957)
**Author:** [@terramar](https://discuss.elastic.co/u/terramar)\ **Replies:** 1\ **Last updated:** [July 19, 2022, 12:24am UTC](https://discuss.elastic.co/t/best-practices-for-dynamic-expiration/309957 "2022-07-19T00:24:18Z")
I'm not sure if that title makes much sense. Right now, I have a fair amount of data coming in through logstash - about 7-10GB/day, and it all needs to stick around for 60 days. I currently write it to an index ("index-… --- ## [About the Logs category](https://discuss.elastic.co/t/about-the-logs-category/156755)
**Author:** [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Replies:** 0
Open source log monitoring The Elastic Stack (sometimes known as the ELK Stack) is the most popular free and open logging platform. This is the place for any questions you may have on log monitoring. --- ## [Documentation for reporting with basic licence is wrong](https://discuss.elastic.co/t/documentation-for-reporting-with-basic-licence-is-wrong/309924)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [July 18, 2022, 10:50pm UTC](https://discuss.elastic.co/t/documentation-for-reporting-with-basic-licence-is-wrong/309924 "2022-07-18T22:50:27Z")
Hi the API call in this section of the documentation doesnt work in the actual version of elasticsearch. what will be the right API call to set reporting with basic licence? --- ## [Logstash pipeline DLQ issue](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920)
**Author:** [@Harper\_S1](https://discuss.elastic.co/u/Harper_S1)\ **Replies:** 9\ **Last updated:** [July 18, 2022, 7:31pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920 "2022-07-18T19:31:37Z")
Hi, I am upgrading my logstash from 7.x.x to 8.2.0. Problem is, logstash is not able to run the pipeline, showing the following error. \[2022-07-18T11:08:52,606\]\[ERROR\]\[org.logstash.common.io.DeadLetterQueueWriter\]\[main… --- ## [Better to have more indices or more shards per index?](https://discuss.elastic.co/t/better-to-have-more-indices-or-more-shards-per-index/309937)
**Author:** [@Jeremy\_Andrews](https://discuss.elastic.co/u/Jeremy_Andrews)\ **Replies:** 2\ **Last updated:** [July 18, 2022, 6:58pm UTC](https://discuss.elastic.co/t/better-to-have-more-indices-or-more-shards-per-index/309937 "2022-07-18T18:58:35Z")
Let's say I have 200 GB of data and I'll split it between 4 shards of 50 GB each. Is there a difference in terms of performance whether I have 4 indices with 1 shard each, versus having one index with 4 primary shards? --- ## [Logstash syslog input filter default grok patterns](https://discuss.elastic.co/t/logstash-syslog-input-filter-default-grok-patterns/309938)
**Author:** [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Replies:** 1\ **Last updated:** [July 18, 2022, 6:54pm UTC](https://discuss.elastic.co/t/logstash-syslog-input-filter-default-grok-patterns/309938 "2022-07-18T18:54:47Z")
What are the default grok patterns for the syslog input filter? The data is from old snare, failing messages are formatted something like this: \<135\> 07/18/2022:18:14:52 GMT HOSTNAME syslog\_message. Thanks --- ## [Logstash cannot parse user\_agent field of nginx](https://discuss.elastic.co/t/logstash-cannot-parse-user-agent-field-of-nginx/309934)
**Author:** [@kosmylo](https://discuss.elastic.co/u/kosmylo)\ **Replies:** 3\ **Last updated:** [July 18, 2022, 6:51pm UTC](https://discuss.elastic.co/t/logstash-cannot-parse-user-agent-field-of-nginx/309934 "2022-07-18T18:51:43Z")
I have nginx logs with the following format: 192.168.0.1 - - \[18/Jul/2022:11:20:28 +0000\] "GET / HTTP/1.1" 200 15 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Sa… --- ## [Logstash Filter JSON syslog message field is not getting parsed](https://discuss.elastic.co/t/logstash-filter-json-syslog-message-field-is-not-getting-parsed/309864)
**Author:** [@Giridharan\_C](https://discuss.elastic.co/u/Giridharan_C)\ **Replies:** 3\ **Last updated:** [July 18, 2022, 6:12pm UTC](https://discuss.elastic.co/t/logstash-filter-json-syslog-message-field-is-not-getting-parsed/309864 "2022-07-18T18:12:22Z")
I cannot parse the incoming Syslog by JSON. The message field is not getting parsed. I tried JSON filter using addfield and also with mutate but no luck. I used GROK to parse specific fields but the message field has key… --- ## [Logstash grok pattern for custom nginx access log](https://discuss.elastic.co/t/logstash-grok-pattern-for-custom-nginx-access-log/309933)
**Author:** [@kosmylo](https://discuss.elastic.co/u/kosmylo)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 5:46pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-for-custom-nginx-access-log/309933 "2022-07-18T17:46:09Z")
I have a custom format of nginx access.log and I want to parse it with logstash. The format of the access log is the following: log\_format main '$remote\_addr - $remote\_user \[$time\_local\] "$request" ' … --- ## [Crawling Microsoft Stream Videos Site Issue](https://discuss.elastic.co/t/crawling-microsoft-stream-videos-site-issue/309823)
**Author:** [@itsraj](https://discuss.elastic.co/u/itsraj)\ **Replies:** 5\ **Last updated:** [July 18, 2022, 5:00pm UTC](https://discuss.elastic.co/t/crawling-microsoft-stream-videos-site-issue/309823 "2022-07-18T17:00:22Z")
Hello Team, I am trying to crawl the Microsoft Streams site in Enterprise Search. But getting below error. Please help me to crawl the password-protected URL below to crawl videos. Microsoft Stream --- ## [Localhost path to folder to automatically ingest existent files](https://discuss.elastic.co/t/localhost-path-to-folder-to-automatically-ingest-existent-files/309707)
**Author:** [@marius03](https://discuss.elastic.co/u/marius03)\ **Replies:** 2\ **Last updated:** [July 18, 2022, 4:18pm UTC](https://discuss.elastic.co/t/localhost-path-to-folder-to-automatically-ingest-existent-files/309707 "2022-07-18T16:18:23Z")
Hi, I have installed Elasticsearch with Kibana on Localhost on Linux Mint, everything works perfectly but I can't seem to set it to automatically ingest files from certain folders. I want it to automatically ingest all… --- ## [Implementing Search After to fetch hits greater than 10000](https://discuss.elastic.co/t/implementing-search-after-to-fetch-hits-greater-than-10000/309913)
**Author:** [@Namita\_Jaokar](https://discuss.elastic.co/u/Namita_Jaokar)\ **Replies:** 1\ **Last updated:** [July 18, 2022, 2:59pm UTC](https://discuss.elastic.co/t/implementing-search-after-to-fetch-hits-greater-than-10000/309913 "2022-07-18T14:59:18Z")
Hi team, I am using elasticsearch in java to query hits that are greater than maximum default of 10000. For the same I came across using search after along with sort. I tried the query by sorting it on the basis of @ti… --- ## [Receiving partial messages in elastic](https://discuss.elastic.co/t/receiving-partial-messages-in-elastic/309730)
**Author:** [@ytld](https://discuss.elastic.co/u/ytld)\ **Replies:** 4\ **Last updated:** [July 18, 2022, 2:56pm UTC](https://discuss.elastic.co/t/receiving-partial-messages-in-elastic/309730 "2022-07-18T14:56:03Z")
We have a problem in our ELK-stack of which I'm unsure how to solve. We're running 8.3 version of the ELK-stack on Centos 7 machines and whereas everything worked fine before, since this week we're only seeing partial m… --- ## [Searching on aggregated nested fields](https://discuss.elastic.co/t/searching-on-aggregated-nested-fields/309909)
**Author:** [@Robina\_Dhingra1](https://discuss.elastic.co/u/Robina_Dhingra1)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 1:57pm UTC](https://discuss.elastic.co/t/searching-on-aggregated-nested-fields/309909 "2022-07-18T13:57:23Z")
Hi Team Elastic, We have a requirement where we created continuous transform to keep track of field status of our trades data. The transform aggregates the data and produces documents in this form - { "\_index":… --- ## [First filter then run nested query](https://discuss.elastic.co/t/first-filter-then-run-nested-query/309905)
**Author:** [@Michal\_Bogusz](https://discuss.elastic.co/u/Michal_Bogusz)\ **Replies:** 2\ **Last updated:** [July 18, 2022, 1:51pm UTC](https://discuss.elastic.co/t/first-filter-then-run-nested-query/309905 "2022-07-18T13:51:03Z")
I have document with mapping: { "properties": { "title": {"type": "text"}, "added": {"type": "date"}, "pdf\_url": {"type": "text"}, "blocks": {"type": "nested"} } } I plan to have… --- ## [Query Time Dynamic synonym](https://discuss.elastic.co/t/query-time-dynamic-synonym/309898)
**Author:** [@pavithra\_arul](https://discuss.elastic.co/u/pavithra_arul)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 12:38pm UTC](https://discuss.elastic.co/t/query-time-dynamic-synonym/309898 "2022-07-18T12:38:38Z")
Hi Team, Trying to achieve dynamic synonym expansion at Query time using Rest API call Trying to update the synonym.txt file dynamically using Rest API. (or) updating the synonyms list via Rest call without even creat… --- ## [How to "reindex" a field with "source" disabled and "store: true"?](https://discuss.elastic.co/t/how-to-reindex-a-field-with-source-disabled-and-store-true/309817)
**Author:** [@huuyafwww](https://discuss.elastic.co/u/huuyafwww)\ **Replies:** 4\ **Last updated:** [July 18, 2022, 12:52pm UTC](https://discuss.elastic.co/t/how-to-reindex-a-field-with-source-disabled-and-store-true/309817 "2022-07-18T12:52:15Z")
I am currently looking to optimize the performance of indexing and search on fields with very large text in them. And I have already read the following articles and previous forum posts. Searching in a large text fiel… --- ## [Aggregate sum till specific threshold reached](https://discuss.elastic.co/t/aggregate-sum-till-specific-threshold-reached/309897)
**Author:** [@brampurnot](https://discuss.elastic.co/u/brampurnot)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 12:27pm UTC](https://discuss.elastic.co/t/aggregate-sum-till-specific-threshold-reached/309897 "2022-07-18T12:27:16Z")
Hi all, We are storing products in an index. Each product has a price and a specific quantity. Now we want to get the list of cheapest products up till a maximum quantity of 100. The total quantity should be a sum aggr… --- ## [Available fields are not visible in the Discovery panel](https://discuss.elastic.co/t/available-fields-are-not-visible-in-the-discovery-panel/309781)
**Author:** [@random\_dash](https://discuss.elastic.co/u/random_dash)\ **Replies:** 1\ **Last updated:** [July 18, 2022, 12:25pm UTC](https://discuss.elastic.co/t/available-fields-are-not-visible-in-the-discovery-panel/309781 "2022-07-18T12:25:05Z")
Hello! I have some indices where a part of their corresponding template is like this: when I query the indices in the Dev Tools, I am able to retrieve desired data and see the above fields. For example you can c… --- ## [Hide Metricbeat Dashboard](https://discuss.elastic.co/t/hide-metricbeat-dashboard/309399)
**Author:** [@Diego\_Lima](https://discuss.elastic.co/u/Diego_Lima)\ **Replies:** 1\ **Last updated:** [July 18, 2022, 12:23pm UTC](https://discuss.elastic.co/t/hide-metricbeat-dashboard/309399 "2022-07-18T12:23:05Z")
Hello guys! I would like to know how I can hide only the metricbeat dashboard, for my anonymous login user. Note: I didn't upload an example image, because I'm getting the following error here: "Sorry, there was an err… --- ## [Elastic charts with angular js](https://discuss.elastic.co/t/elastic-charts-with-angular-js/309656)
**Author:** [@frankgif](https://discuss.elastic.co/u/frankgif)\ **Replies:** 1\ **Last updated:** [July 18, 2022, 12:22pm UTC](https://discuss.elastic.co/t/elastic-charts-with-angular-js/309656 "2022-07-18T12:22:13Z")
How to integrate elastic charts with Angular application. Without having react framework --- ## [Automation of Data Analysis](https://discuss.elastic.co/t/automation-of-data-analysis/309879)
**Author:** [@elkstack2](https://discuss.elastic.co/u/elkstack2)\ **Replies:** 1\ **Last updated:** [July 18, 2022, 12:17pm UTC](https://discuss.elastic.co/t/automation-of-data-analysis/309879 "2022-07-18T12:17:46Z")
Hi all! I am interning at an organisation and have been asked to research automation in the ELK stack - I don't really know what I'm doing so any help would be appreciated! One of the increasingly large challenges my or… --- ## [Elastic v8.3 Install Errors MacOS](https://discuss.elastic.co/t/elastic-v8-3-install-errors-macos/308690)
**Author:** [@Sameer\_Malik](https://discuss.elastic.co/u/Sameer_Malik)\ **Replies:** 3\ **Last updated:** [July 18, 2022, 11:59am UTC](https://discuss.elastic.co/t/elastic-v8-3-install-errors-macos/308690 "2022-07-18T11:59:24Z")
Elasticsearch Version 8.3.1 OS Version macOS 12.3.1 Problem Description I'm very new to Elasticsearch. I'm trying to install ES v8.3.1 on my Mac by following the instructions on the download website. However, once I ru… --- ## [Group by date with aggr and apply sub aggr with group by date in elastic search](https://discuss.elastic.co/t/group-by-date-with-aggr-and-apply-sub-aggr-with-group-by-date-in-elastic-search/309894)
**Author:** [@surendrakandira](https://discuss.elastic.co/u/surendrakandira)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 11:49am UTC](https://discuss.elastic.co/t/group-by-date-with-aggr-and-apply-sub-aggr-with-group-by-date-in-elastic-search/309894 "2022-07-18T11:49:37Z")
How to implement below C# entity logic in Elasticsearch var queryable1 = queryable.GroupBy(g =\> g.Date).Select(x =\> new GroupChartDto() { DailyPnL = x.Sum(s =\> s.DailyPnL), Date = x.Key, AggregateSum = queryable.Whe… --- ## [In ES 7.8 parent breaker is tripping a lot and causing unallocation of shards](https://discuss.elastic.co/t/in-es-7-8-parent-breaker-is-tripping-a-lot-and-causing-unallocation-of-shards/309635)
**Author:** [@sreekanth](https://discuss.elastic.co/u/sreekanth)\ **Replies:** 2\ **Last updated:** [July 18, 2022, 10:58am UTC](https://discuss.elastic.co/t/in-es-7-8-parent-breaker-is-tripping-a-lot-and-causing-unallocation-of-shards/309635 "2022-07-18T10:58:08Z")
We have an 8 node cluster and our load (mainly bulk ingest) is pretty high. Earlier the same load was handled well by 6 nodes in ES6.8. Now after moving to 7.8, we see many replica shards get unallocated during load. al… --- ## [Overriding logstash access "remote\_host" with client IP - Spring Boot](https://discuss.elastic.co/t/overriding-logstash-access-remote-host-with-client-ip-spring-boot/309887)
**Author:** [@Wes1](https://discuss.elastic.co/u/Wes1)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 10:57am UTC](https://discuss.elastic.co/t/overriding-logstash-access-remote-host-with-client-ip-spring-boot/309887 "2022-07-18T10:57:59Z")
Good day, Does anyone know how to override "remote\_host" field within logstash access. Currently we are logging the load balancer's IP. I would like to use the "X-Forwarded-For" header. Thank you. --- ## [Group by date with aggr and apply sub aggr with group by date in elastic search](https://discuss.elastic.co/t/group-by-date-with-aggr-and-apply-sub-aggr-with-group-by-date-in-elastic-search/309877)
**Author:** [@surendrakandira](https://discuss.elastic.co/u/surendrakandira)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 10:23am UTC](https://discuss.elastic.co/t/group-by-date-with-aggr-and-apply-sub-aggr-with-group-by-date-in-elastic-search/309877 "2022-07-18T10:23:20Z")
I am trying to apply filter with aggregations. c# code , I want to apply this logic in elastic search queryable = queryable.OrderBy(o =\> o.Date).ToList(); var queryable1 = queryable.GroupBy(g =\> g… --- ## [Cannot connect to elastic cluster on VPN machine](https://discuss.elastic.co/t/cannot-connect-to-elastic-cluster-on-vpn-machine/309848)
**Author:** [@Michal\_Bogusz](https://discuss.elastic.co/u/Michal_Bogusz)\ **Replies:** 2\ **Last updated:** [July 18, 2022, 7:43am UTC](https://discuss.elastic.co/t/cannot-connect-to-elastic-cluster-on-vpn-machine/309848 "2022-07-18T07:43:47Z")
Hi, I am new to Elasticsearch. What I managed to do so far is to bring up ElasticSearch container and successfully connect to it using both http connection(via Postman) and using python client. However, now I want to br… --- ## [Logstash http filter with JWT](https://discuss.elastic.co/t/logstash-http-filter-with-jwt/309856)
**Author:** [@pszemesy](https://discuss.elastic.co/u/pszemesy)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 7:42am UTC](https://discuss.elastic.co/t/logstash-http-filter-with-jwt/309856 "2022-07-18T07:42:10Z")
Hi, I have to creat a Logstash pipeline where I have to enrich the data from Remedy. But Remedy requires JWT based authentication. Are there any way to use JWT token based authentication in http filter? Thanks. --- ## [Duplicate document](https://discuss.elastic.co/t/duplicate-document/309827)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 3\ **Last updated:** [July 18, 2022, 7:24am UTC](https://discuss.elastic.co/t/duplicate-document/309827 "2022-07-18T07:24:17Z")
I have few servers running metricbeats and all of them sends data to proxy server on port xyz then proxy redirects them to three logstash server in load balance manner. all three logstash runs same configuration via pi… [Previous page](https://discuss.elastic.co/tag/none.md?no_tags=true&page=581) [Next page](https://discuss.elastic.co/tag/none.md?no_tags=true&page=583)