# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=10 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 11 --- ## [Remove the values/fields from Logstash](https://discuss.elastic.co/t/remove-the-values-fields-from-logstash/375711)
**Author:** [@Gowtham1](https://discuss.elastic.co/u/Gowtham1)\ **Replies:** 3\ **Last updated:** [March 11, 2025, 6:28pm UTC](https://discuss.elastic.co/t/remove-the-values-fields-from-logstash/375711 "2025-03-11T18:28:26Z")
Hi, I want to remove this values %{\[@METADATA\]\[HOST\]\[1\]} from the field named "FLOOR" . Please see the below Logstash pipeline config. We have split the fields and then the field name %{\[@METADATA\]\[HOST\]\[1\]} coming as… --- ## [Filter the values based on particular string in Logstash](https://discuss.elastic.co/t/filter-the-values-based-on-particular-string-in-logstash/375717)
**Author:** [@Gowtham1](https://discuss.elastic.co/u/Gowtham1)\ **Replies:** 4\ **Last updated:** [March 11, 2025, 6:27pm UTC](https://discuss.elastic.co/t/filter-the-values-based-on-particular-string-in-logstash/375717 "2025-03-11T18:27:52Z")
Hi, We have a field named "AP\_NAME" this field value contains a common names like WIRAP and WIRWM but some additional names also coming to this field so I want put a condition in logstash to get only this particular WIR… --- ## [Grok ecs\_compatibility option](https://discuss.elastic.co/t/grok-ecs-compatibility-option/375702)
**Author:** [@frans-wtax](https://discuss.elastic.co/u/frans-wtax)\ **Replies:** 2\ **Last updated:** [March 11, 2025, 12:44pm UTC](https://discuss.elastic.co/t/grok-ecs-compatibility-option/375702 "2025-03-11T12:44:56Z")
Hey all, I noticed that my grok filter that uses the COMBINEDAPACHELOG pattern (logstash-patterns-core/patterns/ecs-v1/httpd at main · logstash-plugins/logstash-patterns-core · GitHub) doesn't actually return the \[user\]… --- ## [Facing Issue while parsing nested JSON](https://discuss.elastic.co/t/facing-issue-while-parsing-nested-json/375595)
**Author:** [@madhavi\_pdb](https://discuss.elastic.co/u/madhavi_pdb)\ **Replies:** 6\ **Last updated:** [March 10, 2025, 2:53pm UTC](https://discuss.elastic.co/t/facing-issue-while-parsing-nested-json/375595 "2025-03-10T14:53:08Z")
Hello Community, I have been facing parsing issue for past 24 hrs, I could not understand if log stash does not allow JSON parsing, adding new fields without GROK, KV ,Flattening SCENARIO: INPUT LOG: { "accountname… --- ## [Beats Input Batch Size Warning](https://discuss.elastic.co/t/beats-input-batch-size-warning/375640)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 0\ **Last updated:** [March 10, 2025, 9:54am UTC](https://discuss.elastic.co/t/beats-input-batch-size-warning/375640 "2025-03-10T09:54:49Z")
Hi, We're seeing this warning in our logstash logs: \[2025-03-10T09:26:40,147\]\[WARN \]\[org.logstash.beats.V2Batch\]\[base-input\]\[beats-input\] Received batch of size 67137414 bytes that is too large to fit into the pre-allo… --- ## [Unable to parse message field in logstash using grok filter](https://discuss.elastic.co/t/unable-to-parse-message-field-in-logstash-using-grok-filter/375516)
**Author:** [@huzaifa224](https://discuss.elastic.co/u/huzaifa224)\ **Replies:** 0\ **Last updated:** [March 6, 2025, 12:19pm UTC](https://discuss.elastic.co/t/unable-to-parse-message-field-in-logstash-using-grok-filter/375516 "2025-03-06T12:19:22Z")
I have set up a Fleet Server in Kibana, and my Meraki Firewall is sending VPN data to Logstash. I’ve configured a Logstash filter to process the logs and store the filtered data at a custom path. Additionally, I have in… --- ## [Logstash file output crashes when writing to append only filesystem](https://discuss.elastic.co/t/logstash-file-output-crashes-when-writing-to-append-only-filesystem/375569)
**Author:** [@hilsonp](https://discuss.elastic.co/u/hilsonp)\ **Replies:** 0\ **Last updated:** [March 7, 2025, 5:16pm UTC](https://discuss.elastic.co/t/logstash-file-output-crashes-when-writing-to-append-only-filesystem/375569 "2025-03-07T17:16:51Z")
Hello, We have a logstash pipeline which output is configured to write to an append only nfs filesystem (WORM - NetApp SnapLock Volume Append Mode (VAM)). File are being written but all of a sudden, the pipeline will c… --- ## [Optimistic concurrency control in ElasticSearch Output plugin](https://discuss.elastic.co/t/optimistic-concurrency-control-in-elasticsearch-output-plugin/375506)
**Author:** [@HDDTHR](https://discuss.elastic.co/u/HDDTHR)\ **Replies:** 3\ **Last updated:** [March 6, 2025, 1:36pm UTC](https://discuss.elastic.co/t/optimistic-concurrency-control-in-elasticsearch-output-plugin/375506 "2025-03-06T13:36:50Z")
Is there a mechanism to use if\_seq\_no and if\_primary\_term in the Elasticsearch output plugin? My use-case involves using Elasticsearch as an input (including the \_seq\_no, and \_primary\_term fields), and updating the docu… --- ## [Logstash Pipeline](https://discuss.elastic.co/t/logstash-pipeline/375396)
**Author:** [@JosephR](https://discuss.elastic.co/u/JosephR)\ **Replies:** 18\ **Last updated:** [March 5, 2025, 5:52pm UTC](https://discuss.elastic.co/t/logstash-pipeline/375396 "2025-03-05T17:52:19Z")
I have quite a few logstash conf files written out that have to be kicked off manually at the moment. Is there a way to cause them to run automatically? I need to basically tie what I bring in to a dashboard that consta… --- ## [JMX plugin logstash. Configuration for retreive process.pid , process.nb\_thread?](https://discuss.elastic.co/t/jmx-plugin-logstash-configuration-for-retreive-process-pid-process-nb-thread/375457)
**Author:** [@martel](https://discuss.elastic.co/u/martel)\ **Replies:** 0\ **Last updated:** [March 5, 2025, 12:40pm UTC](https://discuss.elastic.co/t/jmx-plugin-logstash-configuration-for-retreive-process-pid-process-nb-thread/375457 "2025-03-05T12:40:15Z")
i want add metrics to the JMX information from process. i want add pid, the number of thread for this pid, etc... i cant try the good query: "queries" : \[ { "object\_name" : "java.lang:type=Memory", "object\_alias" :… --- ## [Convert the code in Logstash](https://discuss.elastic.co/t/convert-the-code-in-logstash/375284)
**Author:** [@Gowtham1](https://discuss.elastic.co/u/Gowtham1)\ **Replies:** 3\ **Last updated:** [March 5, 2025, 11:03am UTC](https://discuss.elastic.co/t/convert-the-code-in-logstash/375284 "2025-03-05T11:03:40Z")
Hi, I'm to the Logstash, How can I convert the below codes in Logstash? Code 1: def categorize\_rssi(rssi): if rssi \<= -30 and rssi \> -50: return "Excellent Signal" elif rssi \<= -50 and rssi \> -60: return "Good Sig… --- ## [How to search for specific items in nested logs](https://discuss.elastic.co/t/how-to-search-for-specific-items-in-nested-logs/375437)
**Author:** [@random\_typescript](https://discuss.elastic.co/u/random_typescript)\ **Replies:** 0\ **Last updated:** [March 5, 2025, 9:08am UTC](https://discuss.elastic.co/t/how-to-search-for-specific-items-in-nested-logs/375437 "2025-03-05T09:08:24Z")
When I search for logs with nested JSON objects like below where "combined" is an object. Some of the search terms I am using for searching this are not working. For example when trying to search for logs with the "l… --- ## [Logstash has a lag in pushing events to Elastic](https://discuss.elastic.co/t/logstash-has-a-lag-in-pushing-events-to-elastic/374798)
**Author:** [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Replies:** 18\ **Last updated:** [March 5, 2025, 7:28am UTC](https://discuss.elastic.co/t/logstash-has-a-lag-in-pushing-events-to-elastic/374798 "2025-03-05T07:28:14Z")
logstash config input { file { path =\> "C:/Program Files (x86)/db/Server/logs/oracle.log" type =\> "localhost\_access\_log" start\_position =\> "beginning" ignore\_older =\> 86400 # ignore files older than 24 … --- ## [Logstash categorizing all Sonicwall events as "Emergency"](https://discuss.elastic.co/t/logstash-categorizing-all-sonicwall-events-as-emergency/374319)
**Author:** [@savante21](https://discuss.elastic.co/u/savante21)\ **Replies:** 2\ **Last updated:** [March 4, 2025, 2:51pm UTC](https://discuss.elastic.co/t/logstash-categorizing-all-sonicwall-events-as-emergency/374319 "2025-03-04T14:51:21Z")
I am using logstash to capture syslogs from a sonicwall firewall in order to transmit to Wazuh cloud. Our default configuration is : input { syslog { port =\> "514" type =\> "sonicwall" } } output { fi… --- ## [Need help with grok expression logstash](https://discuss.elastic.co/t/need-help-with-grok-expression-logstash/375313)
**Author:** [@Mohamed\_NOUISSEL](https://discuss.elastic.co/u/Mohamed_NOUISSEL)\ **Replies:** 4\ **Last updated:** [March 3, 2025, 5:54pm UTC](https://discuss.elastic.co/t/need-help-with-grok-expression-logstash/375313 "2025-03-03T17:54:51Z")
Hello, I am new to Logstash and Grok filters. My sample logs of glassfish server look like this: \[2025-03-02T17:17:00.722133Z\] \[GF 7.0.12\] \[INFO\] \[\] \[jakarta.enterprise.logging.stdout\] \[tid: \_ThreadID=693 \_ThreadName… --- ## [Logstash multiple conf files configuration is not working when passing fields between conf files](https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299)
**Author:** [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Replies:** 6\ **Last updated:** [March 3, 2025, 5:02pm UTC](https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299 "2025-03-03T17:02:37Z")
Hello, Currently a I have a single file as below which will do the following requirements: Step1: Get bearer token every minute Step2: Use bearer token and get Prometheus data stpe3: parsing Prometheus data Below si… --- ## [Logstash JDBC Input PREPARE on every run](https://discuss.elastic.co/t/logstash-jdbc-input-prepare-on-every-run/375312)
**Author:** [@dwjvaughan](https://discuss.elastic.co/u/dwjvaughan)\ **Replies:** 2\ **Last updated:** [March 3, 2025, 3:56pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-prepare-on-every-run/375312 "2025-03-03T15:56:41Z")
Hi, Since upgrading from Logstash 8.7.1 to 8.17.2 I've noticed that the PREPARE statement is run every time (followed by the EXECUTE), where as previously PREPARE would run once at start up, and EXECUTE would run every … --- ## [Logstash output](https://discuss.elastic.co/t/logstash-output/375237)
**Author:** [@JosephR](https://discuss.elastic.co/u/JosephR)\ **Replies:** 10\ **Last updated:** [March 3, 2025, 2:09pm UTC](https://discuss.elastic.co/t/logstash-output/375237 "2025-03-03T14:09:40Z")
I wrote a conf file and when it reads my ip addresses from the DB it outputs them as an integer instead of the ip address. I tried to do a mutate like the following: filter { mutate { convert =\> { "sourceipv4" =\> "st… --- ## [Logstash: Can we add http inputs dynamically in a filter section to config file from env variables](https://discuss.elastic.co/t/logstash-can-we-add-http-inputs-dynamically-in-a-filter-section-to-config-file-from-env-variables/375200)
**Author:** [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Replies:** 8\ **Last updated:** [March 1, 2025, 12:40am UTC](https://discuss.elastic.co/t/logstash-can-we-add-http-inputs-dynamically-in-a-filter-section-to-config-file-from-env-variables/375200 "2025-03-01T00:40:46Z")
Hello, Logstash: Can we add http inputs dynamically in a filter section to config file from env variables. Here is my requirement: Below filter contains the URL to get Prometheus data for a server endpoint. Now I need … --- ## [Logstash http poller input & filter flow not working properly](https://discuss.elastic.co/t/logstash-http-poller-input-filter-flow-not-working-properly/375183)
**Author:** [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Replies:** 4\ **Last updated:** [February 28, 2025, 4:41pm UTC](https://discuss.elastic.co/t/logstash-http-poller-input-filter-flow-not-working-properly/375183 "2025-02-28T16:41:51Z")
Hello, Below is my requirement: First http call with user & password will return bearer token, token expires every 30mins. So need to call the URL every 30mins for updated token. Second: Get token from above and poll … --- ## [Keep copy of file sent by Logstash to Google Cloud Storage bucket](https://discuss.elastic.co/t/keep-copy-of-file-sent-by-logstash-to-google-cloud-storage-bucket/375160)
**Author:** [@Taz](https://discuss.elastic.co/u/Taz)\ **Replies:** 0\ **Last updated:** [February 27, 2025, 1:28pm UTC](https://discuss.elastic.co/t/keep-copy-of-file-sent-by-logstash-to-google-cloud-storage-bucket/375160 "2025-02-27T13:28:21Z")
Hi. We have a requirement to keep a copy of each log file sent to Google Cloud Storage bucket for a couple of days (crazy business requirement). Sending the log files to Google Cloud works great. Can't figure out how t… --- ## [Aggregation rule](https://discuss.elastic.co/t/aggregation-rule/375011)
**Author:** [@cybersc\_1](https://discuss.elastic.co/u/cybersc_1)\ **Replies:** 7\ **Last updated:** [February 27, 2025, 1:30pm UTC](https://discuss.elastic.co/t/aggregation-rule/375011 "2025-02-27T13:30:10Z")
Hi there! I have plenty of events that look like that: Feb 25 15:05:50 xfirewall CEF:0|infotecs|xf|5.4|62|Non-encrypted forwarded IP packet passed|5|start=1740477743000 end=1740477743000 src=\*\*\* dst=\*\*\* spt=53 dpt=3882… --- ## [Sending and Filtering Logs to Syslog Server](https://discuss.elastic.co/t/sending-and-filtering-logs-to-syslog-server/375100)
**Author:** [@kleioemre](https://discuss.elastic.co/u/kleioemre)\ **Replies:** 2\ **Last updated:** [February 26, 2025, 4:41pm UTC](https://discuss.elastic.co/t/sending-and-filtering-logs-to-syslog-server/375100 "2025-02-26T16:41:11Z")
Hi, I'm trying send logs to syslog server using output plugin but couldn't get any success. I want to send only certain pod logs and add the labels. Is this config accurate? output { if \[kubernetes\]\[pod\]\[name\] =~ /^p… --- ## [Split hostname into new fields](https://discuss.elastic.co/t/split-hostname-into-new-fields/375022)
**Author:** [@Gowtham1](https://discuss.elastic.co/u/Gowtham1)\ **Replies:** 3\ **Last updated:** [February 25, 2025, 5:07pm UTC](https://discuss.elastic.co/t/split-hostname-into-new-fields/375022 "2025-02-25T17:07:54Z")
Hi, I'm new to Logstash,I'm trying to split the hostname into new fields like the below example hostname = gbldi-f01-switch split to the below fields SITE\_ID = gbldi FLOOR = f01 CATEGORY = switch Please see the bel… --- ## [Using a Single Logstash template.conf Across Multiple Pipelines with Dynamic Variable Replacement](https://discuss.elastic.co/t/using-a-single-logstash-template-conf-across-multiple-pipelines-with-dynamic-variable-replacement/374853)
**Author:** [@Dante1](https://discuss.elastic.co/u/Dante1)\ **Replies:** 6\ **Last updated:** [February 24, 2025, 5:05pm UTC](https://discuss.elastic.co/t/using-a-single-logstash-template-conf-across-multiple-pipelines-with-dynamic-variable-replacement/374853 "2025-02-24T17:05:06Z")
I need to create 1000 pipelines that will all use the same template file, template.conf, as a template. For example, in the pipelines.yml file I have these pipelines: - pipeline.id: pip\_01 path.config: "/etc/logstash… --- ## [How to Resolve \`QueueRuntimeException: Data to be written is bigger than page capacity\` in Logstash Exec Input Plugin?](https://discuss.elastic.co/t/how-to-resolve-queueruntimeexception-data-to-be-written-is-bigger-than-page-capacity-in-logstash-exec-input-plugin/374930)
**Author:** [@A\_Bhise](https://discuss.elastic.co/u/A_Bhise)\ **Replies:** 2\ **Last updated:** [February 24, 2025, 1:52pm UTC](https://discuss.elastic.co/t/how-to-resolve-queueruntimeexception-data-to-be-written-is-bigger-than-page-capacity-in-logstash-exec-input-plugin/374930 "2025-02-24T13:52:37Z")
Hello Logstash Community, I’m encountering a QueueRuntimeException in Logstash that says:"data to be written is bigger than page capacity" Configurations are: logstash.yml pipeline.ordered: auto config.support\_esc… --- ## [Not able to parse prometheus metrics to json format](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447)
**Author:** [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Replies:** 34\ **Last updated:** [February 23, 2025, 7:18pm UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447 "2025-02-23T19:18:38Z")
Hello, Able to get Prometheus metrics as a message using http\_poller. But not able to convert to json. Below is the same output of Prometheus format # TYPE http\_requests\_total counter http\_requests\_total{code="200",m… --- ## [LS to ES via mTLS RSA only?](https://discuss.elastic.co/t/ls-to-es-via-mtls-rsa-only/374849)
**Author:** [@natharran](https://discuss.elastic.co/u/natharran)\ **Replies:** 1\ **Last updated:** [February 21, 2025, 2:38pm UTC](https://discuss.elastic.co/t/ls-to-es-via-mtls-rsa-only/374849 "2025-02-21T14:38:55Z")
Hello all, I have a working configuration where Logstash authenticates to Elasticsearch via mTLS scheme using it's default RSA key. I now need to replace this key via another one that is generated by out Openshift opera… --- ## [Discrepancy in Log Filtering in Logstash](https://discuss.elastic.co/t/discrepancy-in-log-filtering-in-logstash/374821)
**Author:** [@ErGeek](https://discuss.elastic.co/u/ErGeek)\ **Replies:** 12\ **Last updated:** [February 20, 2025, 7:07pm UTC](https://discuss.elastic.co/t/discrepancy-in-log-filtering-in-logstash/374821 "2025-02-20T19:07:05Z")
Hi All, We are currently using the filter below in Logstash to process logs. Filter - filter { # Parse the JSON string from the message field into "parsed\_msg" json { source =\> "message" target =\> "parsed\_… --- ## [Long message is truncated by logstash](https://discuss.elastic.co/t/long-message-is-truncated-by-logstash/374691)
**Author:** [@Misakiz](https://discuss.elastic.co/u/Misakiz)\ **Replies:** 5\ **Last updated:** [February 19, 2025, 9:53am UTC](https://discuss.elastic.co/t/long-message-is-truncated-by-logstash/374691 "2025-02-19T09:53:50Z")
\