# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=121 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 122 --- ## [If we can call API with designated cookie to retrieve the log before outputing to downstream in Filebeat](https://discuss.elastic.co/t/if-we-can-call-api-with-designated-cookie-to-retrieve-the-log-before-outputing-to-downstream-in-filebeat/372208)
**Author:** [@Jason9988](https://discuss.elastic.co/u/Jason9988)\ **Replies:** 0\ **Last updated:** [December 19, 2024, 8:15am UTC](https://discuss.elastic.co/t/if-we-can-call-api-with-designated-cookie-to-retrieve-the-log-before-outputing-to-downstream-in-filebeat/372208 "2024-12-19T08:15:22Z")
if we can call API with designated cookie to retrieve the log before outputting to downstream in Filebeat ? Scenarios: My server have generated the first batch 1000 records log with cookie information I need to extrac… --- ## [Reindex from remote. From v8.5.1 to v7.17.1 error](https://discuss.elastic.co/t/reindex-from-remote-from-v8-5-1-to-v7-17-1-error/372153)
**Author:** [@Khanh\_Tran13579](https://discuss.elastic.co/u/Khanh_Tran13579)\ **Replies:** 5\ **Last updated:** [December 19, 2024, 8:27am UTC](https://discuss.elastic.co/t/reindex-from-remote-from-v8-5-1-to-v7-17-1-error/372153 "2024-12-19T08:27:43Z")
I am trying to reindex from the remote server. The source version is 8.5.1 and the dest version is 7.17.1. I got the error below. Please help me explain this error and how to solve it. { "error": { "root\_cause": \[… --- ## [Issue with Logstash, ILM and Data stream](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897)
**Author:** [@fio23](https://discuss.elastic.co/u/fio23)\ **Replies:** 5\ **Last updated:** [December 19, 2024, 7:28am UTC](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897 "2024-12-19T07:28:34Z")
Hello all, I'm currently trying to create a data stream with a corresponding ILM policy enabled, and then have Logstash to forward logs to that data stream. See below steps I follow: Create Component Template PUT \_co… --- ## [Parsing confluence audit json into Elastic with logstash](https://discuss.elastic.co/t/parsing-confluence-audit-json-into-elastic-with-logstash/371798)
**Author:** [@Logistic\_dilated](https://discuss.elastic.co/u/Logistic_dilated)\ **Replies:** 2\ **Last updated:** [December 19, 2024, 3:02am UTC](https://discuss.elastic.co/t/parsing-confluence-audit-json-into-elastic-with-logstash/371798 "2024-12-19T03:02:26Z")
Hello, I'm testing scraping our company confluence (self hosted) audit api so permission changes etc. are in ELK (also self hosted). I've got an http\_poller pipeline that is successfully polling the api, however the ev… --- ## [Filebeat input stream missing log lines when file update in very infrequent](https://discuss.elastic.co/t/filebeat-input-stream-missing-log-lines-when-file-update-in-very-infrequent/372178)
**Author:** [@Shaishav](https://discuss.elastic.co/u/Shaishav)\ **Replies:** 0\ **Last updated:** [December 18, 2024, 5:34pm UTC](https://discuss.elastic.co/t/filebeat-input-stream-missing-log-lines-when-file-update-in-very-infrequent/372178 "2024-12-18T17:34:04Z")
Set up: Kubernetes pod saves the log file on host machine at location /var/log/app\_logs Filebeat 8.16.2 is configured as Daemonset to read from logfile and enrich the data Filebeat.yml filebeat.inputs: - type: filest… --- ## [API logs not coming after setting up Elastic Agent , Beats logs are coming](https://discuss.elastic.co/t/api-logs-not-coming-after-setting-up-elastic-agent-beats-logs-are-coming/372081)
**Author:** [@ErGeek](https://discuss.elastic.co/u/ErGeek)\ **Replies:** 14\ **Last updated:** [December 18, 2024, 6:18pm UTC](https://discuss.elastic.co/t/api-logs-not-coming-after-setting-up-elastic-agent-beats-logs-are-coming/372081 "2024-12-18T18:18:41Z")
Hi All, We have setup an Elastic Agent to send logs from an Application API to Kafka. We are able to establish the connection using the Elastic Agent. But instead of the API logs, we are getting Beat logs (Filebeat and… --- ## [Switch between the "document" and "summary" columns in Kibana Discover](https://discuss.elastic.co/t/switch-between-the-document-and-summary-columns-in-kibana-discover/372063)
**Author:** [@debuggedme](https://discuss.elastic.co/u/debuggedme)\ **Replies:** 6\ **Last updated:** [December 18, 2024, 3:10pm UTC](https://discuss.elastic.co/t/switch-between-the-document-and-summary-columns-in-kibana-discover/372063 "2024-12-18T15:10:46Z")
How to switch between the "document" and "summary" columns in Kibana Discover? In version 8.15, I saw "document," but in 8.16, I see "summary," and it is empty. The default column set to \_source doesn't seem to take … --- ## [Enhancing Ingestion Rate of Elastic Agent Reading from Azure Event Hub](https://discuss.elastic.co/t/enhancing-ingestion-rate-of-elastic-agent-reading-from-azure-event-hub/372162)
**Author:** [@RajuParipelly](https://discuss.elastic.co/u/RajuParipelly)\ **Replies:** 3\ **Last updated:** [December 18, 2024, 2:52pm UTC](https://discuss.elastic.co/t/enhancing-ingestion-rate-of-elastic-agent-reading-from-azure-event-hub/372162 "2024-12-18T14:52:07Z")
I have successfully deployed an Elastic Agent that interfaces with an Azure Event Hub. The current configuration processes approximately 5 to 10 million logs per hour, with individual log sizes ranging between 5 to 20 KB… --- ## [Snapshot restore data stream problems](https://discuss.elastic.co/t/snapshot-restore-data-stream-problems/372170)
**Author:** [@Dave\_Houser](https://discuss.elastic.co/u/Dave_Houser)\ **Replies:** 0\ **Last updated:** [December 18, 2024, 2:48pm UTC](https://discuss.elastic.co/t/snapshot-restore-data-stream-problems/372170 "2024-12-18T14:48:02Z")
Running 8.13 I am working on restoring a data stream from a snapshot. I am doing this in a surgical way as I need to be able to write to indexes in snapshots while the restore of all other indexes happens in the backgro… --- ## [Metricbeat issue: failed to parse: Limit of total fields \[10000\] has been exceeded while adding new fields](https://discuss.elastic.co/t/metricbeat-issue-failed-to-parse-limit-of-total-fields-10000-has-been-exceeded-while-adding-new-fields/369405)
**Author:** [@alexns](https://discuss.elastic.co/u/alexns)\ **Replies:** 5\ **Last updated:** [December 18, 2024, 2:06pm UTC](https://discuss.elastic.co/t/metricbeat-issue-failed-to-parse-limit-of-total-fields-10000-has-been-exceeded-while-adding-new-fields/369405 "2024-12-18T14:06:33Z")
We use the AWS module to push SQS and RDS metrics to Metricbeat. Everything is setup with ECK on Kubernetes and Helm. Recently, probably after updating to 8.15.2, these metrics weren't being pushed anymore. The metricb… --- ## [Set system time as timestamp](https://discuss.elastic.co/t/set-system-time-as-timestamp/372079)
**Author:** [@Taras\_Mikityuk](https://discuss.elastic.co/u/Taras_Mikityuk)\ **Replies:** 3\ **Last updated:** [December 18, 2024, 12:39pm UTC](https://discuss.elastic.co/t/set-system-time-as-timestamp/372079 "2024-12-18T12:39:55Z")
I'm collecting logs using elastic agent. The problem is, in @timestamp field I have time when event actually happened, but I also need time when log arrived in logstash. I was thinking to use ruby(ruby { code =\> … --- ## [Traditional Group By With Aggregations & Pagination](https://discuss.elastic.co/t/traditional-group-by-with-aggregations-pagination/372151)
**Author:** [@jespin](https://discuss.elastic.co/u/jespin)\ **Replies:** 2\ **Last updated:** [December 18, 2024, 11:59am UTC](https://discuss.elastic.co/t/traditional-group-by-with-aggregations-pagination/372151 "2024-12-18T11:59:28Z")
Hi all, We are looking to rewrite our current Elastic search implementation as it's not fit for purpose. We have a number of hotels which we index in as a single document with all of the packages available for that hot… --- ## [Logstash SNMP Interface Monitoring Configuration with Bandwidth Utilization Calculation](https://discuss.elastic.co/t/logstash-snmp-interface-monitoring-configuration-with-bandwidth-utilization-calculation/372068)
**Author:** [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Replies:** 0\ **Last updated:** [December 17, 2024, 12:34pm UTC](https://discuss.elastic.co/t/logstash-snmp-interface-monitoring-configuration-with-bandwidth-utilization-calculation/372068 "2024-12-17T12:34:45Z")
Hi everyone, I'd like to share my Logstash configuration for monitoring network interfaces via SNMP with bandwidth utilization calculation. This configuration collects interface statistics from network devices (tested w… --- ## [Semantic Search in Java](https://discuss.elastic.co/t/semantic-search-in-java/372141)
**Author:** [@erkosh](https://discuss.elastic.co/u/erkosh)\ **Replies:** 1\ **Last updated:** [December 18, 2024, 9:09am UTC](https://discuss.elastic.co/t/semantic-search-in-java/372141 "2024-12-18T09:09:15Z")
I want to add a semantic search feature to my project but cannot find a good method. I tried using ELSER, but the libraries are deprecated. Is there any way to add semantic search in Java Spring? --- ## [Delete .monitoring index as it's unassigned elk 8.7](https://discuss.elastic.co/t/delete-monitoring-index-as-its-unassigned-elk-8-7/372144)
**Author:** [@amoghbarve](https://discuss.elastic.co/u/amoghbarve)\ **Replies:** 0\ **Last updated:** [December 18, 2024, 8:12am UTC](https://discuss.elastic.co/t/delete-monitoring-index-as-its-unassigned-elk-8-7/372144 "2024-12-18T08:12:16Z")
Hi All... Need one guidance I am using Elasticsearch 8.7 and my cluster status is yellow Upon checking I find that the . monitoring index is in unassigned status . Can it be deleted or dropped ? Is there any impact ? … --- ## [Snapshot Repository is 8x larger than total clustered data!?!?!](https://discuss.elastic.co/t/snapshot-repository-is-8x-larger-than-total-clustered-data/371625)
**Author:** [@ivanlawrence](https://discuss.elastic.co/u/ivanlawrence)\ **Replies:** 1\ **Last updated:** [December 18, 2024, 8:08am UTC](https://discuss.elastic.co/t/snapshot-repository-is-8x-larger-than-total-clustered-data/371625 "2024-12-18T08:08:16Z")
My cluster has a total usage reported by kibana (kibana/app/monitoring#/overview) as 1TB (102 indices with one replica.) Snapshots go to backblaze (as the s3 default client) which is reporting 8TB stored. I was snappin… --- ## [How to modify the content of the field in source?](https://discuss.elastic.co/t/how-to-modify-the-content-of-the-field-in-source/372097)
**Author:** [@yoopaan](https://discuss.elastic.co/u/yoopaan)\ **Replies:** 1\ **Last updated:** [December 18, 2024, 3:18am UTC](https://discuss.elastic.co/t/how-to-modify-the-content-of-the-field-in-source/372097 "2024-12-18T03:18:04Z")
I want to process the query return value of the document in the source code and implement the following operations: index: { "id": 1001, "title":"abcd" } return: { "id": 1001, "title": "bcde" } That is, replace the… --- ## [Path.data: is ignored in filebeat.yml ===\> Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)](https://discuss.elastic.co/t/path-data-is-ignored-in-filebeat-yml-exiting-data-path-already-locked-by-another-beat-please-make-sure-that-multiple-beats-are-not-sharing-the-same-data-path-path-data/372026)
**Author:** [@Oow4chah](https://discuss.elastic.co/u/Oow4chah)\ **Replies:** 1\ **Last updated:** [December 18, 2024, 1:51am UTC](https://discuss.elastic.co/t/path-data-is-ignored-in-filebeat-yml-exiting-data-path-already-locked-by-another-beat-please-make-sure-that-multiple-beats-are-not-sharing-the-same-data-path-path-data/372026 "2024-12-18T01:51:23Z")
Hi path.data is ignored in filebeat.yml I am on RHEL9 if that matters. I try to start 2 different filebeat processes. the process already running: # ps -ef | grep filebeat root 552625 519242 0 14:32 pts/3 … --- ## [Ecs metrics](https://discuss.elastic.co/t/ecs-metrics/371727)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 17\ **Last updated:** [December 17, 2024, 10:41pm UTC](https://discuss.elastic.co/t/ecs-metrics/371727 "2024-12-17T22:41:39Z")
I have a metricbeat running on one place which is generating following metrics host.network.ingress.bytes but at different place (same version metricbeat) but different elastic cluster is not producting this metric. Ho… --- ## [No data from logfile on a new index](https://discuss.elastic.co/t/no-data-from-logfile-on-a-new-index/371998)
**Author:** [@moep](https://discuss.elastic.co/u/moep)\ **Replies:** 1\ **Last updated:** [December 17, 2024, 8:48pm UTC](https://discuss.elastic.co/t/no-data-from-logfile-on-a-new-index/371998 "2024-12-17T20:48:45Z")
Hello, I have running the ELK Stack in a Docker-compose setup. thats my setup: ~/local-ELK/logstash   main  pwd /home/moep/local-ELK/logstash Thats my filter in /home/moep/local-ELK/logstash/pipeline/weechat.co… --- ## [Nested object behavior](https://discuss.elastic.co/t/nested-object-behavior/372087)
**Author:** [@Carlos\_Barros](https://discuss.elastic.co/u/Carlos_Barros)\ **Replies:** 2\ **Last updated:** [December 17, 2024, 7:52pm UTC](https://discuss.elastic.co/t/nested-object-behavior/372087 "2024-12-17T19:52:09Z")
Hi, I have the situation exemplified below. I wonder if any of you can tell why when I search in the fields of the nested object "partes" I can't find the document that contains "fulano" and "beltrano" in the nested obje… --- ## [DLQ Errors after upgrading ELK (Elasticsearch and Kibana) from 7.17 to 8.15.2](https://discuss.elastic.co/t/dlq-errors-after-upgrading-elk-elasticsearch-and-kibana-from-7-17-to-8-15-2/372083)
**Author:** [@ErGeek](https://discuss.elastic.co/u/ErGeek)\ **Replies:** 1\ **Last updated:** [December 17, 2024, 6:22pm UTC](https://discuss.elastic.co/t/dlq-errors-after-upgrading-elk-elasticsearch-and-kibana-from-7-17-to-8-15-2/372083 "2024-12-17T18:22:06Z")
Hi All, Our log management process starts from Beats -\> MSK (Kafka) -\> Logstash -\> Elasticsearch -\> Kibana. We recently upgraded our Elastic cluster (application-test-cluster) in October. However, since the upgrade, we … --- ## [Does a utility function exist for converting SearchRequest into a RequestItem for elasticsearch-java?](https://discuss.elastic.co/t/does-a-utility-function-exist-for-converting-searchrequest-into-a-requestitem-for-elasticsearch-java/372071)
**Author:** [@Hans\_Kruse](https://discuss.elastic.co/u/Hans_Kruse)\ **Replies:** 1\ **Last updated:** [December 17, 2024, 4:18pm UTC](https://discuss.elastic.co/t/does-a-utility-function-exist-for-converting-searchrequest-into-a-requestitem-for-elasticsearch-java/372071 "2024-12-17T16:18:25Z")
Does a utility function exist for converting a SearchRequest into a RequestItem for performing multi search requests with the elasticsearch-java. The Java highlevel client allowed to create a multisearch request usi… --- ## [Duplicate Logs Issue Due to next\_start\_date Configuration in Google Workspace Admin Module](https://discuss.elastic.co/t/duplicate-logs-issue-due-to-next-start-date-configuration-in-google-workspace-admin-module/372082)
**Author:** [@Ninas](https://discuss.elastic.co/u/Ninas)\ **Replies:** 0\ **Last updated:** [December 17, 2024, 3:54pm UTC](https://discuss.elastic.co/t/duplicate-logs-issue-due-to-next-start-date-configuration-in-google-workspace-admin-module/372082 "2024-12-17T15:54:50Z")
While using the Google Workspace Admin module for Filebeat, I've encountered an issue where duplicate logs are being received. The problem seems to originate from the url.params.startTime configuration, particularly wit… --- ## [Anomaly detection for web site visitor surge (sparse data)](https://discuss.elastic.co/t/anomaly-detection-for-web-site-visitor-surge-sparse-data/372033)
**Author:** [@J\_Reinhardt](https://discuss.elastic.co/u/J_Reinhardt)\ **Replies:** 1\ **Last updated:** [December 17, 2024, 3:31pm UTC](https://discuss.elastic.co/t/anomaly-detection-for-web-site-visitor-surge-sparse-data/372033 "2024-12-17T15:31:20Z")
I'm trying to use elastic anomaly detection to identify when a surge in website visitor activity occurs for a particular IP address. The data set is really simple, it just shows the number of times an IP address has vis… --- ## [Double processor in ingest pipeline](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768)
**Author:** [@Cristina\_Marletta\_Li](https://discuss.elastic.co/u/Cristina_Marletta_Li)\ **Replies:** 30\ **Last updated:** [December 17, 2024, 3:24pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768 "2024-12-17T15:24:28Z")
Hi all, it is possible to create a custom ingest pipeline associated with an integration and a specific Data Stream with 2 processors. The second one should process a field created by the first one. --- ## [Kibana canvas timing out after upgrade](https://discuss.elastic.co/t/kibana-canvas-timing-out-after-upgrade/372040)
**Author:** [@Stephen\_IAnson](https://discuss.elastic.co/u/Stephen_IAnson)\ **Replies:** 1\ **Last updated:** [December 17, 2024, 2:10pm UTC](https://discuss.elastic.co/t/kibana-canvas-timing-out-after-upgrade/372040 "2024-12-17T14:10:34Z")
Hi, I've just upgraded from elasticsearch+kibana 8.13.3 to 8.16.1. The upgrade seemed to go smoothly, but now I notice my canvas dashboards show clocks for ages and many elements (using essql with elasticsearch) then f… --- ## [Self managed cluster](https://discuss.elastic.co/t/self-managed-cluster/372052)
**Author:** [@searchwithme](https://discuss.elastic.co/u/searchwithme)\ **Replies:** 1\ **Last updated:** [December 17, 2024, 1:02pm UTC](https://discuss.elastic.co/t/self-managed-cluster/372052 "2024-12-17T13:02:49Z")
A few questions on setting up a self managed cluster: I will be ingesting 100 gb data per day, want to do queries for a month. How to set up hot, warm, cold, frozen tier with self managed cluster? If I have to set up j… --- ## [How to translate HighLevel client Bucket order to elasticsearch-java](https://discuss.elastic.co/t/how-to-translate-highlevel-client-bucket-order-to-elasticsearch-java/371840)
**Author:** [@Hans\_Kruse](https://discuss.elastic.co/u/Hans_Kruse)\ **Replies:** 3\ **Last updated:** [December 17, 2024, 10:36am UTC](https://discuss.elastic.co/t/how-to-translate-highlevel-client-bucket-order-to-elasticsearch-java/371840 "2024-12-17T10:36:20Z")
In our Elasticsearch highlevel client based code we have some code that uses the BucketOrder.key(...) and BucketOrder.count(...) factory methods. What are equivalent constructs for the the elasticsearch-java client SDK?… --- ## [Shard allocation fails when splitting index on multi-node cluster](https://discuss.elastic.co/t/shard-allocation-fails-when-splitting-index-on-multi-node-cluster/372029)
**Author:** [@techytushar](https://discuss.elastic.co/u/techytushar)\ **Replies:** 1\ **Last updated:** [December 16, 2024, 10:35pm UTC](https://discuss.elastic.co/t/shard-allocation-fails-when-splitting-index-on-multi-node-cluster/372029 "2024-12-16T22:35:05Z")
I have a 3 node Elasticsearch cluster and an index of size 40GB on it which only has a single shard. The single shard of the index reside on one of the nodes. I am trying to split that index into 6 shards. The problem i… [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=120) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=122)