# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=219 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 220 --- ## [Configure logstash to recieve logs from Openshift 4 and fluentdForward](https://discuss.elastic.co/t/configure-logstash-to-recieve-logs-from-openshift-4-and-fluentdforward/276012)
**Author:** [@sudden74](https://discuss.elastic.co/u/sudden74)\ **Replies:** 0\ **Last updated:** [June 15, 2021, 2:00pm UTC](https://discuss.elastic.co/t/configure-logstash-to-recieve-logs-from-openshift-4-and-fluentdforward/276012 "2021-06-15T14:00:41Z")
We have configured the fluentdForward on Openshift 4.7 to send logs to an external ELK stack. Logs are arriving to logstash but we are not able to decode the messages correctly. It looks like the special characters are r… --- ## [Filter messages in logstash with different index](https://discuss.elastic.co/t/filter-messages-in-logstash-with-different-index/275990)
**Author:** [@learner21](https://discuss.elastic.co/u/learner21)\ **Replies:** 0\ **Last updated:** [June 15, 2021, 11:45am UTC](https://discuss.elastic.co/t/filter-messages-in-logstash-with-different-index/275990 "2021-06-15T11:45:11Z")
I am getting 3 types of messages in logstash, based on that i need to parse and save in elasticSearch with 3 different index Message 1 =\> "2021-05-26T09:55:36.091040+00:00 10.13.14.11 \[S=294230650\] \[ID=fbf282:30:1115898… --- ## [Logstash multiple pipelines](https://discuss.elastic.co/t/logstash-multiple-pipelines/275984)
**Author:** [@sreedhar1](https://discuss.elastic.co/u/sreedhar1)\ **Replies:** 0\ **Last updated:** [June 15, 2021, 11:25am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines/275984 "2021-06-15T11:25:30Z")
Hi All, I am using Logstash to populate data from sql server to Elasticsearch. I have created multiple .conf files to populate data in ES. Created pipeline to run all these .conf files. Question: do we need to create m… --- ## [Logstash is not creating ingest pipelines](https://discuss.elastic.co/t/logstash-is-not-creating-ingest-pipelines/275953)
**Author:** [@Mocem](https://discuss.elastic.co/u/Mocem)\ **Replies:** 2\ **Last updated:** [June 15, 2021, 9:35am UTC](https://discuss.elastic.co/t/logstash-is-not-creating-ingest-pipelines/275953 "2021-06-15T09:35:42Z")
Hi All, For the past few days I'm struggling with loading the pipelines from filebeat. Basically nothing is happening. It works directly to Elastic but not when logstash is in between. Once I create the ingest pipeline … --- ## [Received an event that has a different character encoding than you configured. Log4j2 to Logstash](https://discuss.elastic.co/t/received-an-event-that-has-a-different-character-encoding-than-you-configured-log4j2-to-logstash/275952)
**Author:** [@Rashmika\_Gamage](https://discuss.elastic.co/u/Rashmika_Gamage)\ **Replies:** 0\ **Last updated:** [June 15, 2021, 7:08am UTC](https://discuss.elastic.co/t/received-an-event-that-has-a-different-character-encoding-than-you-configured-log4j2-to-logstash/275952 "2021-06-15T07:08:21Z")
I am trying to get spark logs to logstash with custom log4j2 properties. My log4j2 properties file #Define the log4j configuration for local application #log4j.rootLogger=INFO, server #We will use socket appender log4j… --- ## [Will the data be lost when the network is disconnected?](https://discuss.elastic.co/t/will-the-data-be-lost-when-the-network-is-disconnected/275946)
**Author:** [@aurantiacus](https://discuss.elastic.co/u/aurantiacus)\ **Replies:** 0\ **Last updated:** [June 15, 2021, 6:43am UTC](https://discuss.elastic.co/t/will-the-data-be-lost-when-the-network-is-disconnected/275946 "2021-06-15T06:43:16Z")
Hello, I am using beat version 6.8.0 and Logstash version 6.8.0 I am sending data from "beat"(auditbeat,filebeat,packetbeat,winlogbeat) to "logstash A" and from "logstash A" to another "logstash B". What happens to th… --- ## [Logstash http output plugin customize building json array for batching](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535)
**Author:** [@RahulGS](https://discuss.elastic.co/u/RahulGS)\ **Replies:** 8\ **Last updated:** [June 15, 2021, 4:11am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535 "2021-06-15T04:11:17Z")
Hello , We have a pipeline running on logstash 6.6.2 with input(redis)-\>filter(ruby)-\>output(http) how do you batch http output using custom format ? we can't use format="json\_batch" because it will construct a simple… --- ## [Logstash plain-log file is getting following lines of errors and causing no logs appear in Kibana. Please help!](https://discuss.elastic.co/t/logstash-plain-log-file-is-getting-following-lines-of-errors-and-causing-no-logs-appear-in-kibana-please-help/275922)
**Author:** [@devashishsingh](https://discuss.elastic.co/u/devashishsingh)\ **Replies:** 0\ **Last updated:** [June 15, 2021, 4:07am UTC](https://discuss.elastic.co/t/logstash-plain-log-file-is-getting-following-lines-of-errors-and-causing-no-logs-appear-in-kibana-please-help/275922 "2021-06-15T04:07:38Z")
\[2021-06-15T11:52:59,022\]\[WARN \]\[io.netty.channel.AbstractChannelHandlerContext\]\[main\]\[841e111a9a3864d5c6aa677dd9865551e728aa14cc10a6e208f20d80a45001cd\] The exceptionCaught() event that was failed to submit was: java.ut… --- ## [Sync data from MongoDB using logstash](https://discuss.elastic.co/t/sync-data-from-mongodb-using-logstash/275918)
**Author:** [@Bach\_Tran](https://discuss.elastic.co/u/Bach_Tran)\ **Replies:** 0\ **Last updated:** [June 15, 2021, 2:45am UTC](https://discuss.elastic.co/t/sync-data-from-mongodb-using-logstash/275918 "2021-06-15T02:45:24Z")
I'm using Logstash 6.8.0 to push my Message collection to Elasticsearch 6.8.0. This is my logstash file: input { jdbc { jdbc\_connection\_string =\> "jdbc:mongodb://127.0.0.1:27017/vnpost\_chatdb" jdbc\_driver\_libr… --- ## [KV Filter registered without jruby interruptible regular expressions enabled](https://discuss.elastic.co/t/kv-filter-registered-without-jruby-interruptible-regular-expressions-enabled/275916)
**Author:** [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Replies:** 1\ **Last updated:** [June 15, 2021, 2:29am UTC](https://discuss.elastic.co/t/kv-filter-registered-without-jruby-interruptible-regular-expressions-enabled/275916 "2021-06-15T02:29:25Z")
When logstash starts it logs this for each KV I have configured this logs: Jun 15 01:49:10 hostname logstash\[3879863\]: \[2021-06-15T01:49:10,651\]\[WARN \]\[logstash.filters.kv \] KV Filter registered without jruby inter… --- ## [Need help with splitting a log using logstash split filter](https://discuss.elastic.co/t/need-help-with-splitting-a-log-using-logstash-split-filter/275906)
**Author:** [@kajira](https://discuss.elastic.co/u/kajira)\ **Replies:** 4\ **Last updated:** [June 15, 2021, 12:49am UTC](https://discuss.elastic.co/t/need-help-with-splitting-a-log-using-logstash-split-filter/275906 "2021-06-15T00:49:50Z")
Hi, I receive json logs with the structure as shown below on logstash from a remote server. { "timestamp: "...". "message": { "records": \[ { result ... }, { result ... }, … --- ## [Logstash multiple pipeline configuration using logstash helm chart](https://discuss.elastic.co/t/logstash-multiple-pipeline-configuration-using-logstash-helm-chart/275880)
**Author:** [@arijitc](https://discuss.elastic.co/u/arijitc)\ **Replies:** 0\ **Last updated:** [June 14, 2021, 5:47pm UTC](https://discuss.elastic.co/t/logstash-multiple-pipeline-configuration-using-logstash-helm-chart/275880 "2021-06-14T17:47:37Z")
While configuring multiple pipeline using logstash (version 7.13.1) helm chart , the values.yaml is having a section # Allows you to add any pipeline files in /usr/share/logstash/pipeline/ ### \*\*\*warn\*\*\* there is a hard… --- ## [Split and store tags](https://discuss.elastic.co/t/split-and-store-tags/275866)
**Author:** [@rodrigue](https://discuss.elastic.co/u/rodrigue)\ **Replies:** 2\ **Last updated:** [June 14, 2021, 5:43pm UTC](https://discuss.elastic.co/t/split-and-store-tags/275866 "2021-06-14T17:43:58Z")
Hello, Sorry to bother you but I tried everything... Il simply would like to take tags from my json and store it as a tag in logstash... Here is my configuration input { http { port =\> "5046" tags… --- ## [Logstash TCP input splits events unintentionally](https://discuss.elastic.co/t/logstash-tcp-input-splits-events-unintentionally/275811)
**Author:** [@cyber\_crab](https://discuss.elastic.co/u/cyber_crab)\ **Replies:** 3\ **Last updated:** [June 14, 2021, 4:49pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-splits-events-unintentionally/275811 "2021-06-14T16:49:48Z")
Hi guys, I have an issue with one of the Logstash pipelines in our stack. The incoming CEF logs from Cyberark sometimes get split into two events for no apparent reason. I ran tcpdump on the port to which CYA is suppose… --- ## [Aggregate filter timeouts and input completion](https://discuss.elastic.co/t/aggregate-filter-timeouts-and-input-completion/275719)
**Author:** [@jratliff](https://discuss.elastic.co/u/jratliff)\ **Replies:** 3\ **Last updated:** [June 14, 2021, 4:42pm UTC](https://discuss.elastic.co/t/aggregate-filter-timeouts-and-input-completion/275719 "2021-06-14T16:42:55Z")
I am using logstash to read events from elasticsearch and the aggregate filter to condense them. Because the data I am condensing has no clear start/stop events, we are using the timeout and inactivity\_timeout settings … --- ## [Parsing XML document contained in a field](https://discuss.elastic.co/t/parsing-xml-document-contained-in-a-field/275037)
**Author:** [@fmaginga](https://discuss.elastic.co/u/fmaginga)\ **Replies:** 4\ **Last updated:** [June 14, 2021, 4:40pm UTC](https://discuss.elastic.co/t/parsing-xml-document-contained-in-a-field/275037 "2021-06-14T16:40:10Z")
We have written a logstash configuration file with kv filter plugin to extract key-value pairs successfully. Some of the values is the keys contains XML documents as example below. What is the better way to parse the XML… --- ## [Logstash could not start ' java.lang.IllegalStateException: Logstash stopped processing because of an error'](https://discuss.elastic.co/t/logstash-could-not-start-java-lang-illegalstateexception-logstash-stopped-processing-because-of-an-error/275805)
**Author:** [@devashishsingh](https://discuss.elastic.co/u/devashishsingh)\ **Replies:** 0\ **Last updated:** [June 14, 2021, 8:01am UTC](https://discuss.elastic.co/t/logstash-could-not-start-java-lang-illegalstateexception-logstash-stopped-processing-because-of-an-error/275805 "2021-06-14T08:01:51Z")
Hello, I experienced an error as below while restarting the Logstash plain-log \[ERROR\]\[org.logstash.Logstash \] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit Thi… --- ## [Write Logstash Data to a File While Preserving Original File Name](https://discuss.elastic.co/t/write-logstash-data-to-a-file-while-preserving-original-file-name/275841)
**Author:** [@Rod\_Guiamoy](https://discuss.elastic.co/u/Rod_Guiamoy)\ **Replies:** 0\ **Last updated:** [June 14, 2021, 1:50pm UTC](https://discuss.elastic.co/t/write-logstash-data-to-a-file-while-preserving-original-file-name/275841 "2021-06-14T13:50:53Z")
Basically I'd like to have Logstash capture the original file name and reusing it when writing the logs to another location. As if you've just copy/pasted it. Use case would be for IIS logs where using \*.log captures all… --- ## [Error parsing timestamp](https://discuss.elastic.co/t/error-parsing-timestamp/275795)
**Author:** [@lani](https://discuss.elastic.co/u/lani)\ **Replies:** 1\ **Last updated:** [June 14, 2021, 1:31pm UTC](https://discuss.elastic.co/t/error-parsing-timestamp/275795 "2021-06-14T13:31:01Z")
Hi everyone, i have a weird log parsing error I failed to solve, maybe someone here can assist me. I have a docker container shipping logs in this (simple?) format: 2021-06-14 08:30:14 ERROR Some message I am shippin… --- ## [When does Logstash update last\_run\_metadata\_path?](https://discuss.elastic.co/t/when-does-logstash-update-last-run-metadata-path/275833)
**Author:** [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\ **Replies:** 2\ **Last updated:** [June 14, 2021, 1:21pm UTC](https://discuss.elastic.co/t/when-does-logstash-update-last-run-metadata-path/275833 "2021-06-14T13:21:07Z")
Hi all, I am working with Logstash 7.13.0 & ES 7.13.0 to populate and maintain an index with records from an Oracle DB. The maintenance is achieved by running an update Logstash service every minute. The condition to be… --- ## [Aggregate multiple nested (recursive) logstash](https://discuss.elastic.co/t/aggregate-multiple-nested-recursive-logstash/275621)
**Author:** [@maxi99](https://discuss.elastic.co/u/maxi99)\ **Replies:** 10\ **Last updated:** [June 14, 2021, 1:03pm UTC](https://discuss.elastic.co/t/aggregate-multiple-nested-recursive-logstash/275621 "2021-06-14T13:03:05Z")
I am using logstash with input jdbc, and would like to embed one object inside another with aggregate. How can I use add recursive? Ie add an object inside another object? My jdbc input would return output similar (as … --- ## [Error message LEAK: ByteBuf.release() in /var/log/logstash-plain.log](https://discuss.elastic.co/t/error-message-leak-bytebuf-release-in-var-log-logstash-plain-log/275824)
**Author:** [@jgkootstra](https://discuss.elastic.co/u/jgkootstra)\ **Replies:** 0\ **Last updated:** [June 14, 2021, 10:38am UTC](https://discuss.elastic.co/t/error-message-leak-bytebuf-release-in-var-log-logstash-plain-log/275824 "2021-06-14T10:38:23Z")
io.netty.util.ResourceLeakDetector\] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. for more information. What does th… --- ## [JSON Filtering In Logstash To Remove Unwanted Fields](https://discuss.elastic.co/t/json-filtering-in-logstash-to-remove-unwanted-fields/275697)
**Author:** [@pkward](https://discuss.elastic.co/u/pkward)\ **Replies:** 2\ **Last updated:** [June 13, 2021, 9:38pm UTC](https://discuss.elastic.co/t/json-filtering-in-logstash-to-remove-unwanted-fields/275697 "2021-06-13T21:38:18Z")
I want to ingest JSON PE file data that has 40+ fields w/nested fields. I only want the first few fields. Is there a way to either drop the unwanted fields or can I explicitly state which fields I want? --- ## [Parsing mySql slow query logs using Logstash](https://discuss.elastic.co/t/parsing-mysql-slow-query-logs-using-logstash/275518)
**Author:** [@learningelk](https://discuss.elastic.co/u/learningelk)\ **Replies:** 6\ **Last updated:** [June 13, 2021, 5:07pm UTC](https://discuss.elastic.co/t/parsing-mysql-slow-query-logs-using-logstash/275518 "2021-06-13T17:07:10Z")
I want to parse the mysql slow query logs using logstash and I want to replace modifieddate \> '2021-06-08 08:13:48' with some generix text like modifieddate \> 'NNNN' Below is the log sample : # Time: 2021-06-10T06:30:… --- ## [How can ser grok if log have different content from the same equipment?](https://discuss.elastic.co/t/how-can-ser-grok-if-log-have-different-content-from-the-same-equipment/275541)
**Author:** [@111418](https://discuss.elastic.co/u/111418)\ **Replies:** 5\ **Last updated:** [June 13, 2021, 3:54pm UTC](https://discuss.elastic.co/t/how-can-ser-grok-if-log-have-different-content-from-the-same-equipment/275541 "2021-06-13T15:54:59Z")
Jun 2 16:45:49 tp-id01.test.corp zorp/scb\_ssh\[1379\]: core.alerting(3): (svc/wwVRLvYoUMToHT2Xz9vEbj/ssh\_alert:11/ssh): Audit event was recognized; rule='PatternMatcherRule', type='adp.event.command', pattern='user', acti… --- ## [Logs hit value is different](https://discuss.elastic.co/t/logs-hit-value-is-different/275675)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 7\ **Last updated:** [June 13, 2021, 10:40am UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675 "2021-06-13T10:40:30Z")
I have two elasticsearch cluster one elk cluster version is 7.6 and other elk cluster version is 7.11 The former elasticsearch cluster is simple it filebeat send the logs to logstash and logstash to elasticsearch, the … --- ## [Logstash grok nginx log](https://discuss.elastic.co/t/logstash-grok-nginx-log/275583)
**Author:** [@rdnaz](https://discuss.elastic.co/u/rdnaz)\ **Replies:** 2\ **Last updated:** [June 13, 2021, 8:19am UTC](https://discuss.elastic.co/t/logstash-grok-nginx-log/275583 "2021-06-13T08:19:36Z")
Hi dears, I want to grok nginx access log but i cant get the seperated field of the message , i tried to use overwrite and doesn't make the solution. my Log format is below : \ Jun 10, 2021 @ 11:07:43.648 host:lo… --- ## [Need to load balance Logstash](https://discuss.elastic.co/t/need-to-load-balance-logstash/275619)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 8\ **Last updated:** [June 13, 2021, 6:45am UTC](https://discuss.elastic.co/t/need-to-load-balance-logstash/275619 "2021-06-13T06:45:58Z")
I'm currently running a single logstash and interested in creating several for resilience in case one goes down, and for improving performance. I'm ingesting data currently with the http input plugin. I'm thinking about… --- ## [Received fatal alert: handshake\_failure. How to Solve it?](https://discuss.elastic.co/t/received-fatal-alert-handshake-failure-how-to-solve-it/275749)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 0\ **Last updated:** [June 13, 2021, 5:18am UTC](https://discuss.elastic.co/t/received-fatal-alert-handshake-failure-how-to-solve-it/275749 "2021-06-13T05:18:12Z")
Hi All While running LS, i am getting this below error. My HTTP endpoint uses TLS 1.3 It works perfectly for HTTP endpoint with TLS V1.2 looks like LS package does not support for open jruby-openssl In the Gem.lock f… --- ## [Logstash drop based on size](https://discuss.elastic.co/t/logstash-drop-based-on-size/275610)
**Author:** [@mcosta](https://discuss.elastic.co/u/mcosta)\ **Replies:** 2\ **Last updated:** [June 11, 2021, 9:11pm UTC](https://discuss.elastic.co/t/logstash-drop-based-on-size/275610 "2021-06-11T21:11:12Z")
Hi all, Is it possible to drop a message based on it's size (ex: above 1 MB)? I'm reading from tcp port and writing into Kafka topic. All messages bigger then 1 MB give errors similar to this and then logstash retry t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=218) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=220)