# Latest **URL:** https://discuss.elastic.co/latest.md?page=1400 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1401 --- ## [Filebeat vs Kinesis agent](https://discuss.elastic.co/t/filebeat-vs-kinesis-agent/288550)
**Author:** [@Nehray](https://discuss.elastic.co/u/Nehray)\ **Replies:** 2\ **Last updated:** [November 6, 2021, 10:35pm UTC](https://discuss.elastic.co/t/filebeat-vs-kinesis-agent/288550 "2021-11-06T22:35:38Z")
Hi all, I am very new to Elasticsearch. I am looking ways to ingest logs into ES. I am looking for answers why should I choose between filebeat vs Kinesis agent. --- ## [Bug on logstash input for gz files](https://discuss.elastic.co/t/bug-on-logstash-input-for-gz-files/288547)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 0\ **Last updated:** [November 6, 2021, 7:05pm UTC](https://discuss.elastic.co/t/bug-on-logstash-input-for-gz-files/288547 "2021-11-06T19:05:42Z")
Hi I've observed that logstash with input plugin doesn't proceed in the correct way gz files in sincedb. I saw that logstash doesn't put all logs info to sincedb but only the header without names. If I have restarted lo… --- ## [Is timestamp field also indexed while storing time siries data like logs into an index?](https://discuss.elastic.co/t/is-timestamp-field-also-indexed-while-storing-time-siries-data-like-logs-into-an-index/288438)
**Author:** [@shi](https://discuss.elastic.co/u/shi)\ **Replies:** 2\ **Last updated:** [November 6, 2021, 7:52pm UTC](https://discuss.elastic.co/t/is-timestamp-field-also-indexed-while-storing-time-siries-data-like-logs-into-an-index/288438 "2021-11-06T19:52:41Z")
Is timestamp field in a logentry of logfile also indexed (ie, made searchable by means of inverted index data structure) while storing time series data like logs into an index/data stream, provided we are not explicitly… --- ## [If the intended value itself contains the 'field split', can we use kv processor?](https://discuss.elastic.co/t/if-the-intended-value-itself-contains-the-field-split-can-we-use-kv-processor/288540)
**Author:** [@shi](https://discuss.elastic.co/u/shi)\ **Replies:** 6\ **Last updated:** [November 6, 2021, 7:51pm UTC](https://discuss.elastic.co/t/if-the-intended-value-itself-contains-the-field-split-can-we-use-kv-processor/288540 "2021-11-06T19:51:29Z")
The following sample log entry , is in the form of key=value pairs action="blocked" direction="outgoing" msg="from ip is in blocklist" proto=6 I have tried the following kv processor in ingest pipeline for the log e… --- ## [How to make logstash to automatically detect new pipleines configuration and execute them?](https://discuss.elastic.co/t/how-to-make-logstash-to-automatically-detect-new-pipleines-configuration-and-execute-them/288541)
**Author:** [@Husnain](https://discuss.elastic.co/u/Husnain)\ **Replies:** 4\ **Last updated:** [November 6, 2021, 7:06pm UTC](https://discuss.elastic.co/t/how-to-make-logstash-to-automatically-detect-new-pipleines-configuration-and-execute-them/288541 "2021-11-06T19:06:47Z")
Hii.I am using logstash to ingest data from four different data sources from i.e Elasticsearch,mongodb,textfile and mysql.My requirement is,that I will specify a directory path and logstash will read pipeline configurati… --- ## [Timestamp not being overwriten (ECS)](https://discuss.elastic.co/t/timestamp-not-being-overwriten-ecs/288544)
**Author:** [@vcaleffi](https://discuss.elastic.co/u/vcaleffi)\ **Replies:** 2\ **Last updated:** [November 6, 2021, 6:41pm UTC](https://discuss.elastic.co/t/timestamp-not-being-overwriten-ecs/288544 "2021-11-06T18:41:03Z")
Hello, I'm very new to filebeat and I'm trying to import logs created by my application (PHP - symfony). The application uses monolog and ECS formatter to write to a log file. Monolog configuration: monolog: handl… --- ## [Serval indices were created with different timestamp upon using new winlogbeat template](https://discuss.elastic.co/t/serval-indices-were-created-with-different-timestamp-upon-using-new-winlogbeat-template/288539)
**Author:** [@Alaxwora](https://discuss.elastic.co/u/Alaxwora)\ **Replies:** 1\ **Last updated:** [November 6, 2021, 4:06pm UTC](https://discuss.elastic.co/t/serval-indices-were-created-with-different-timestamp-upon-using-new-winlogbeat-template/288539 "2021-11-06T16:06:50Z")
Hello, I noted serval indices were created upon using my new winlogbeat template, and I already set ILM to rollover after 30 days or upon index size reaches 10 Giga but it seems that I made a mistake. the following scr… --- ## [How to send logs from different filebeat to logstash](https://discuss.elastic.co/t/how-to-send-logs-from-different-filebeat-to-logstash/288305)
**Author:** [@root\_linux](https://discuss.elastic.co/u/root_linux)\ **Replies:** 2\ **Last updated:** [November 6, 2021, 2:27pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-different-filebeat-to-logstash/288305 "2021-11-06T14:27:05Z")
I have configured Elasticsearch, kibana and logstash on same server. I want to send logs from multiple filebeat clients to logstash with different index name. How can I configure it? --- ## [I can not send logs to elastic cloud](https://discuss.elastic.co/t/i-can-not-send-logs-to-elastic-cloud/288411)
**Author:** [@Hugo\_Pacini](https://discuss.elastic.co/u/Hugo_Pacini)\ **Replies:** 5\ **Last updated:** [November 6, 2021, 2:16pm UTC](https://discuss.elastic.co/t/i-can-not-send-logs-to-elastic-cloud/288411 "2021-11-06T14:16:55Z")
I'm trying to send logs from Serilogs to Elastic Cloud but doesn't work and I can not see errors. Can I send logs to test using Postman? The URL is https://my-custom-domain.es.eastus2.azure.elastic-cloud.com:9243 var u… --- ## [Ingestion performance dropping by 10x daily](https://discuss.elastic.co/t/ingestion-performance-dropping-by-10x-daily/288537)
**Author:** [@TanguyB](https://discuss.elastic.co/u/TanguyB)\ **Replies:** 3\ **Last updated:** [November 6, 2021, 11:19am UTC](https://discuss.elastic.co/t/ingestion-performance-dropping-by-10x-daily/288537 "2021-11-06T11:19:46Z")
Hello, :wave: We are using Fluentbit to ingest Kubernetes cluster logs into Elasticsearch. Not sure what I am setting wrong, but every day, as the index grows, ingestion is worsening from 550k op/s to 40k op/s... Inge… --- ## [Large shard size](https://discuss.elastic.co/t/large-shard-size/288458)
**Author:** [@caseydm](https://discuss.elastic.co/u/caseydm)\ **Replies:** 3\ **Last updated:** [November 6, 2021, 4:57am UTC](https://discuss.elastic.co/t/large-shard-size/288458 "2021-11-06T04:57:56Z")
I am setting up a new Elasticsearch cluster that will hold around 300 million records in a single index to start. The documents are academic paper data. Right now I have two nodes, so I set my shards as 2 primary and 1 r… --- ## [Java Msgs in Log](https://discuss.elastic.co/t/java-msgs-in-log/287967)
**Author:** [@SteveP](https://discuss.elastic.co/u/SteveP)\ **Replies:** 8\ **Last updated:** [November 6, 2021, 2:13am UTC](https://discuss.elastic.co/t/java-msgs-in-log/287967 "2021-11-06T02:13:16Z")
Hello all, I have installed Elasticsearch for the first time and I am seeing msgs that a terminally deprecated method in java.lang.system has been called. Should I be concerned with this, even though it looks like Elast… --- ## [Elastic Stack 7.14.1 Security Update](https://discuss.elastic.co/t/elastic-stack-7-14-1-security-update/283077)
**Author:** [@douglasday](https://discuss.elastic.co/u/douglasday)\ **Replies:** 0\ **Last updated:** [September 1, 2021, 4:10pm UTC](https://discuss.elastic.co/t/elastic-stack-7-14-1-security-update/283077 "2021-09-01T16:10:55Z")
Kibana code execution issue (ESA-2021-21) It was discovered that a user with fleet admin permissions could upload a malicious package. Due to using an older version of the js-yaml library, this package would be loaded i… --- ## [Fleet server setup documentation incorrect when using quickstart + (elastic with TLS)](https://discuss.elastic.co/t/fleet-server-setup-documentation-incorrect-when-using-quickstart-elastic-with-tls/288526)
**Author:** [@tfriesen](https://discuss.elastic.co/u/tfriesen)\ **Replies:** 0\ **Last updated:** [November 5, 2021, 8:38pm UTC](https://discuss.elastic.co/t/fleet-server-setup-documentation-incorrect-when-using-quickstart-elastic-with-tls/288526 "2021-11-05T20:38:48Z")
The documentation for setting up a fleet server, both the online guide and via the 'Add a fleet server', don't account for a situation where Elasticsearch is configured with TLS, but you want to use Quickstart for settin… --- ## [Python Client - unable to remove "read\_only" block](https://discuss.elastic.co/t/python-client-unable-to-remove-read-only-block/288032)
**Author:** [@philhagen](https://discuss.elastic.co/u/philhagen)\ **Replies:** 9\ **Last updated:** [November 5, 2021, 8:29pm UTC](https://discuss.elastic.co/t/python-client-unable-to-remove-read-only-block/288032 "2021-11-05T20:29:18Z")
When using the official Python client, I can't seem to find a way to change an index to normal "read-write" status once it's been marked "read\_only". This function works fine to go from read-write to read\_only, but I wa… --- ## [Build a Query based off the result of another query](https://discuss.elastic.co/t/build-a-query-based-off-the-result-of-another-query/288174)
**Author:** [@Wonjun\_Cho](https://discuss.elastic.co/u/Wonjun_Cho)\ **Replies:** 2\ **Last updated:** [November 5, 2021, 7:48pm UTC](https://discuss.elastic.co/t/build-a-query-based-off-the-result-of-another-query/288174 "2021-11-05T19:48:51Z")
Hi guys, I would like to know if it is possible to make a query based off the result of another query. For example, there are indices for school and students. School { "settings": { ... }, "mappings": { "propertie… --- ## [Some cgroup metrics are missing from output](https://discuss.elastic.co/t/some-cgroup-metrics-are-missing-from-output/288524)
**Author:** [@vPilot](https://discuss.elastic.co/u/vPilot)\ **Replies:** 0\ **Last updated:** [November 5, 2021, 7:44pm UTC](https://discuss.elastic.co/t/some-cgroup-metrics-are-missing-from-output/288524 "2021-11-05T19:44:30Z")
I've got several systems that I have recently started collecting data on via Metricbeat, and am running into an issue on several of them. Particularly, I have a couple of systems that are not reporting back system.proces… --- ## [JDBC Input plugin installation](https://discuss.elastic.co/t/jdbc-input-plugin-installation/288510)
**Author:** [@SANDRA\_ROY\_ARICATT](https://discuss.elastic.co/u/SANDRA_ROY_ARICATT)\ **Replies:** 1\ **Last updated:** [November 5, 2021, 7:00pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-installation/288510 "2021-11-05T19:00:06Z")
I have the new Mac M1 Silicon. I need to get data from MySQL workbench into Elasticsearch. I came to know that it can be achieved using Logstash, which in turn uses JDBC input plugin for getting data from database into… --- ## [How do I add a filter to an Index Threshold alerting rule and alert per server?](https://discuss.elastic.co/t/how-do-i-add-a-filter-to-an-index-threshold-alerting-rule-and-alert-per-server/288519)
**Author:** [@nsouth](https://discuss.elastic.co/u/nsouth)\ **Replies:** 0\ **Last updated:** [November 5, 2021, 6:51pm UTC](https://discuss.elastic.co/t/how-do-i-add-a-filter-to-an-index-threshold-alerting-rule-and-alert-per-server/288519 "2021-11-05T18:51:41Z")
I want to be alerted when our logs indicate that the average duration of commands is over 1 second. However, I cannot figure out how to do the following: Filter the query to only log events which have duration informa… --- ## [Missing Beats data in Kibana Stack monitoring interface](https://discuss.elastic.co/t/missing-beats-data-in-kibana-stack-monitoring-interface/288514)
**Author:** [@Fangy](https://discuss.elastic.co/u/Fangy)\ **Replies:** 1\ **Last updated:** [November 5, 2021, 6:31pm UTC](https://discuss.elastic.co/t/missing-beats-data-in-kibana-stack-monitoring-interface/288514 "2021-11-05T18:31:32Z")
Hello. I'm trying to monitor beat agents on a brand new cluster. Metricbeat is configured to send data to a separate one-node monitoring cluster. Elastic nodes and Kibana monitoring data are displayed correctly, but th… --- ## [Painless script: can not write type \[class java.math.BigDecimal\]](https://discuss.elastic.co/t/painless-script-can-not-write-type-class-java-math-bigdecimal/288517)
**Author:** [@catelling](https://discuss.elastic.co/u/catelling)\ **Replies:** 0\ **Last updated:** [November 5, 2021, 6:30pm UTC](https://discuss.elastic.co/t/painless-script-can-not-write-type-class-java-math-bigdecimal/288517 "2021-11-05T18:30:43Z")
I am facing this exception on my discover page when trying to view my logs. Previously, I had a field involving BigDecimal and worked fine. this was the script: if(!(doc\[‘obyts.keyword’\].size()==0)) { String b\_str… --- ## [Dealing with invalid json](https://discuss.elastic.co/t/dealing-with-invalid-json/288179)
**Author:** [@caseydm](https://discuss.elastic.co/u/caseydm)\ **Replies:** 6\ **Last updated:** [November 5, 2021, 6:01pm UTC](https://discuss.elastic.co/t/dealing-with-invalid-json/288179 "2021-11-05T18:01:32Z")
I'm trying to import some json data that is in a column in redshift. Some of the records have a field that includes invalid json escape sequences, such as this: "work\_title": "The Discrete and Semi-continuous Fr\\'echet … --- ## [Logstash group time series by decade](https://discuss.elastic.co/t/logstash-group-time-series-by-decade/288462)
**Author:** [@caseydm](https://discuss.elastic.co/u/caseydm)\ **Replies:** 4\ **Last updated:** [November 5, 2021, 5:48pm UTC](https://discuss.elastic.co/t/logstash-group-time-series-by-decade/288462 "2021-11-05T17:48:03Z")
I am ingesting academic articles and splitting the index by year: output { elasticsearch { hosts =\> \["${ES\_HOST\_PROD}"\] index =\> "works-%{year}" user =\> "${ES\_USER\_PROD}" password =\> … --- ## [Continuous Transform query, does it update?](https://discuss.elastic.co/t/continuous-transform-query-does-it-update/288443)
**Author:** [@iambatman](https://discuss.elastic.co/u/iambatman)\ **Replies:** 4\ **Last updated:** [November 5, 2021, 5:26pm UTC](https://discuss.elastic.co/t/continuous-transform-query-does-it-update/288443 "2021-11-05T17:26:23Z")
If you use a query when creating a continuous transform will it only ever work on the query subset that is created when the transform is started? { "bool": { "should": \[ { "match": { "state… --- ## [Recurring Date - What is the best patterns?](https://discuss.elastic.co/t/recurring-date-what-is-the-best-patterns/288515)
**Author:** [@Remy\_Letient](https://discuss.elastic.co/u/Remy_Letient)\ **Replies:** 0\ **Last updated:** [November 5, 2021, 5:19pm UTC](https://discuss.elastic.co/t/recurring-date-what-is-the-best-patterns/288515 "2021-11-05T17:19:11Z")
Hi, I need to represent "recurring event" like "each month" with the goal to create a calendar view of them. The events can be unlimited in time. What is the best way to represent this kind of data to permit query like… --- ## [Add tag to filebeat modules](https://discuss.elastic.co/t/add-tag-to-filebeat-modules/288314)
**Author:** [@rodri.gz](https://discuss.elastic.co/u/rodri.gz)\ **Replies:** 5\ **Last updated:** [November 5, 2021, 4:53pm UTC](https://discuss.elastic.co/t/add-tag-to-filebeat-modules/288314 "2021-11-05T16:53:46Z")
hello! I am trying to send the logs with the filebeat mysql module to my logstash and it already obtains the pipeline and does the parsing correctly. How can I differentiate both inputs ( Slowlog and errror ) to apply… --- ## [Elastic Agent Crashes Status Code: 0](https://discuss.elastic.co/t/elastic-agent-crashes-status-code-0/287264)
**Author:** [@Ramguru\_Nageswaran](https://discuss.elastic.co/u/Ramguru_Nageswaran)\ **Replies:** 1\ **Last updated:** [November 5, 2021, 4:42pm UTC](https://discuss.elastic.co/t/elastic-agent-crashes-status-code-0/287264 "2021-11-05T16:42:09Z")
09:32:28.601 elastic\_agent \[elastic\_agent\]\[error\] Could not communicate with fleet-server Checking API will retry, error: status code: 0, fleet-server returned an error: , message: Unknown resource. 09:41:24.952 elastic\_… --- ## [Span breakdown metrics and service name auto-discovery (Time spent by span type is empty)](https://discuss.elastic.co/t/span-breakdown-metrics-and-service-name-auto-discovery-time-spent-by-span-type-is-empty/288498)
**Author:** [@Matthias\_W](https://discuss.elastic.co/u/Matthias_W)\ **Replies:** 4\ **Last updated:** [November 5, 2021, 3:56pm UTC](https://discuss.elastic.co/t/span-breakdown-metrics-and-service-name-auto-discovery-time-spent-by-span-type-is-empty/288498 "2021-11-05T15:56:58Z")
Hi, just wanted to confirm if this is expected behavior: If I use the Elastic APM Agent in a servlet environment (e.g. JAX-RS running on a Wildfly) and don't set the service\_name, then the span breakdown metrics won't … --- ## [How to make a nested term query](https://discuss.elastic.co/t/how-to-make-a-nested-term-query/288506)
**Author:** [@Araf\_Shihab](https://discuss.elastic.co/u/Araf_Shihab)\ **Replies:** 0\ **Last updated:** [November 5, 2021, 2:58pm UTC](https://discuss.elastic.co/t/how-to-make-a-nested-term-query/288506 "2021-11-05T14:58:59Z")
Hey Elasticsearch community. I have a nested query as such { "size": 20, "\_source": \["tags"\], "sort": \[ { "@timestamp": { "order": "desc" } } \], "query": { "nested" : { "path" : "tags… --- ## [\[Filebeat\] Override Module Ingest Pipeline at Runtime](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432)
**Author:** [@DougR](https://discuss.elastic.co/u/DougR)\ **Replies:** 6\ **Last updated:** [November 5, 2021, 2:50pm UTC](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432 "2021-11-05T14:50:43Z")
TL;DR Can I override the ingest pipeline for a filebeat module at runtime from modules.d/module.yml? The Rest of the Story I use the apache module with filebeat to capture logs on the Apache servers in our environment. … [Previous page](https://discuss.elastic.co/latest.md?page=1399) [Next page](https://discuss.elastic.co/latest.md?page=1401)