# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=285 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 286 --- ## [Json template format logstash 7.6.2](https://discuss.elastic.co/t/json-template-format-logstash-7-6-2/254328)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 0\ **Last updated:** [November 4, 2020, 8:30pm UTC](https://discuss.elastic.co/t/json-template-format-logstash-7-6-2/254328 "2020-11-04T20:30:28Z")
Hi All, I am new to applying json templates. I have ELK Stack 7.6.2 My template looks like below: { "mappings": { "\_doc": { "\_all": { "enabled": true, "norms": { "enabled": fals… --- ## [How to remove part of name in multiple fields with Ruby in logstash?](https://discuss.elastic.co/t/how-to-remove-part-of-name-in-multiple-fields-with-ruby-in-logstash/254263)
**Author:** [@alytkowski](https://discuss.elastic.co/u/alytkowski)\ **Replies:** 9\ **Last updated:** [November 5, 2020, 1:57pm UTC](https://discuss.elastic.co/t/how-to-remove-part-of-name-in-multiple-fields-with-ruby-in-logstash/254263 "2020-11-05T13:57:47Z")
I have a source of metrics, which are received by logstash and sent out to Elastic later. There are multiple fields starting with "prometheus.metrics.ems". Can someone help with the ruby code or some other method to cut… --- ## [Logstash Input TCP error Netty libraies](https://discuss.elastic.co/t/logstash-input-tcp-error-netty-libraies/254407)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 0\ **Last updated:** [November 5, 2020, 12:58pm UTC](https://discuss.elastic.co/t/logstash-input-tcp-error-netty-libraies/254407 "2020-11-05T12:58:54Z")
HI Team, Recently we upgraded our logstash-input-tcp from 5.0.9 to 5.2.0 there are SSL errors in the log file. \[2020-11-04T00:00:51,384\]\[ERROR\]\[logstash.inputs.tcp \] Error in Netty pipeline: io.netty.handler.code… --- ## [Pass the schedule dynamically to logstash config file](https://discuss.elastic.co/t/pass-the-schedule-dynamically-to-logstash-config-file/254351)
**Author:** [@Shanmuka\_Chowdary](https://discuss.elastic.co/u/Shanmuka_Chowdary)\ **Replies:** 5\ **Last updated:** [November 5, 2020, 9:17am UTC](https://discuss.elastic.co/t/pass-the-schedule-dynamically-to-logstash-config-file/254351 "2020-11-05T09:17:31Z")
I'm Indexing data from mssql using logstash with schedule like below jdbc { jdbc\_driver\_library =\> "C:\\Program Files\\Microsoft JDBC Driver 4.2 for SQL Server\\sqljdbc\_4.2\\enu\\jre8\\sqljdbc42.jar" … --- ## [Elastiflow / logstash netflow plugin - reduce logging to syslog](https://discuss.elastic.co/t/elastiflow-logstash-netflow-plugin-reduce-logging-to-syslog/254219)
**Author:** [@davidsm](https://discuss.elastic.co/u/davidsm)\ **Replies:** 2\ **Last updated:** [November 5, 2020, 9:04am UTC](https://discuss.elastic.co/t/elastiflow-logstash-netflow-plugin-reduce-logging-to-syslog/254219 "2020-11-05T09:04:56Z")
Greetings! I'm using elastiflow with the logstash netflow plugin. Logs can get quite noisy with expected DNS timeouts and waiting for flow templates. These are logged in both /var/log/logstash/logstash-plain.log as well… --- ## [Filter with select in logstash](https://discuss.elastic.co/t/filter-with-select-in-logstash/253732)
**Author:** [@juuuhuuu](https://discuss.elastic.co/u/juuuhuuu)\ **Replies:** 8\ **Last updated:** [November 5, 2020, 8:15am UTC](https://discuss.elastic.co/t/filter-with-select-in-logstash/253732 "2020-11-05T08:15:02Z")
Hi Community, I would like to have a filter to dissect my logs and aggregate them like this SELECT COUNT(CustomerID), Country FROM Customers GROUP BY Country; Could you please help me? =\> in the logname are c… --- ## [Hadling duplicated records in elastic using logstash configuration](https://discuss.elastic.co/t/hadling-duplicated-records-in-elastic-using-logstash-configuration/254365)
**Author:** [@prathibha](https://discuss.elastic.co/u/prathibha)\ **Replies:** 0\ **Last updated:** [November 5, 2020, 7:56am UTC](https://discuss.elastic.co/t/hadling-duplicated-records-in-elastic-using-logstash-configuration/254365 "2020-11-05T07:56:02Z")
I would like to segregate duplicated records in elastic -kibana by using a field with flag value like 'yes ' or ' no'. So the idea is to add a field from logstash to each documents and value of the field will be 'yes' i… --- ## [Logstash failing to send to Kafka](https://discuss.elastic.co/t/logstash-failing-to-send-to-kafka/254104)
**Author:** [@tamilarasanbravo](https://discuss.elastic.co/u/tamilarasanbravo)\ **Replies:** 5\ **Last updated:** [November 5, 2020, 5:36am UTC](https://discuss.elastic.co/t/logstash-failing-to-send-to-kafka/254104 "2020-11-05T05:36:16Z")
Hi Team, I am parsing Gihthub Log details from Logstash to Kafka and the logstash keeps throwing the below issue. After a certain time, I need to restart logstash to get rid of those warnings. Once initiating the docke… --- ## [JDBC through a SSH tunnel](https://discuss.elastic.co/t/jdbc-through-a-ssh-tunnel/254341)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 0\ **Last updated:** [November 5, 2020, 1:25am UTC](https://discuss.elastic.co/t/jdbc-through-a-ssh-tunnel/254341 "2020-11-05T01:25:05Z")
Im getting data from a PostgreSQL DB through an SSH tunnel: autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -L 9999:localhost:5432 root@mypostgresip The data is requested using Logstash JDBC pointin… --- ## [Grok debugger error for multiple patterns](https://discuss.elastic.co/t/grok-debugger-error-for-multiple-patterns/254335)
**Author:** [@Carlos\_Fernando\_Palm](https://discuss.elastic.co/u/Carlos_Fernando_Palm)\ **Replies:** 1\ **Last updated:** [November 4, 2020, 10:29pm UTC](https://discuss.elastic.co/t/grok-debugger-error-for-multiple-patterns/254335 "2020-11-04T22:29:42Z")
Hello there. This works: Also this works: Their combination however does not work: Any idea why? --- ## [Syslog Filter Plugin](https://discuss.elastic.co/t/syslog-filter-plugin/254187)
**Author:** [@Clay\_Curtis](https://discuss.elastic.co/u/Clay_Curtis)\ **Replies:** 1\ **Last updated:** [November 4, 2020, 4:53pm UTC](https://discuss.elastic.co/t/syslog-filter-plugin/254187 "2020-11-04T16:53:27Z")
I am building a logging pipeline and there is a syslog collector that is depositing raw syslog messages into a Kafka topic. I would usually accept syslog off the wire by using the Logstash syslog input plugin which handl… --- ## [File Input from directory with 100K files](https://discuss.elastic.co/t/file-input-from-directory-with-100k-files/254195)
**Author:** [@surphix](https://discuss.elastic.co/u/surphix)\ **Replies:** 4\ **Last updated:** [November 4, 2020, 2:40pm UTC](https://discuss.elastic.co/t/file-input-from-directory-with-100k-files/254195 "2020-11-04T14:40:19Z")
Hey first time posting here and looking for some understandings. I have a logstash config that is trying to read a directory containing over 100,000 files. I've ran trace logs and even with sincedb\_path set to /dev/null… --- ## [Logstash - JDBC ERROR](https://discuss.elastic.co/t/logstash-jdbc-error/254160)
**Author:** [@peledev](https://discuss.elastic.co/u/peledev)\ **Replies:** 4\ **Last updated:** [November 4, 2020, 12:07pm UTC](https://discuss.elastic.co/t/logstash-jdbc-error/254160 "2020-11-04T12:07:18Z")
Hello , Im using logstash in a linux server I have installed the latest update for logstash on my server And also jdbc installed Ever since the update im unable to run jdbc output I get an error in the log : 'Prefo… --- ## [Decoding base64 logs when input is in JSON format?](https://discuss.elastic.co/t/decoding-base64-logs-when-input-is-in-json-format/254042)
**Author:** [@Mudabbir](https://discuss.elastic.co/u/Mudabbir)\ **Replies:** 3\ **Last updated:** [November 4, 2020, 11:52am UTC](https://discuss.elastic.co/t/decoding-base64-logs-when-input-is-in-json-format/254042 "2020-11-04T11:52:38Z")
Greetings ! I am unable to decode my base64 logs that are coming from nxlog server. Here is my logstash configuration. The error i am getting is shown below The log field is in base64 I have tried cipher filt… --- ## [Logstash not generating any output: "Class path contains multiple SLF4J bindings." error](https://discuss.elastic.co/t/logstash-not-generating-any-output-class-path-contains-multiple-slf4j-bindings-error/254262)
**Author:** [@Silvia\_Koleva](https://discuss.elastic.co/u/Silvia_Koleva)\ **Replies:** 0\ **Last updated:** [November 4, 2020, 11:34am UTC](https://discuss.elastic.co/t/logstash-not-generating-any-output-class-path-contains-multiple-slf4j-bindings-error/254262 "2020-11-04T11:34:17Z")
Hello, I am trying to configure Logstash to collect data from filebeat, where itgets some SharePoint ULS logs and then push them to Elasticsearch with applying some filters. The issue I have is when I run the logstash s… --- ## [Creating a Logstash Module](https://discuss.elastic.co/t/creating-a-logstash-module/254261)
**Author:** [@Avarjana](https://discuss.elastic.co/u/Avarjana)\ **Replies:** 0\ **Last updated:** [November 4, 2020, 11:31am UTC](https://discuss.elastic.co/t/creating-a-logstash-module/254261 "2020-11-04T11:31:56Z")
I need to create a Logstash module like : these. I want to know, Is that possible and if so, where can I find a guide? If not possible, what are the workarounds to bundle my ELK project to be available as an integratio… --- ## [How to work with event "message" in Java filter plugin?](https://discuss.elastic.co/t/how-to-work-with-event-message-in-java-filter-plugin/254258)
**Author:** [@toni1](https://discuss.elastic.co/u/toni1)\ **Replies:** 0\ **Last updated:** [November 4, 2020, 11:17am UTC](https://discuss.elastic.co/t/how-to-work-with-event-message-in-java-filter-plugin/254258 "2020-11-04T11:17:18Z")
Hi, I am developing a Java Filter Plugin to merge different CSVs, and as I'm trying to process them, I need to extract the message field (which contains the whole CSV line). How am I supposed to do so using Java langua… --- ## [Unexpected indices being created by Logstash](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230)
**Author:** [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\ **Replies:** 9\ **Last updated:** [November 4, 2020, 10:36am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230 "2020-11-04T10:36:13Z")
I'm using Logstash to ingest logs stored on AWS S3. My configuration file looks like this: input { s3 { "access\_key\_id" =\> "REDACTED" "secret\_access\_key" =\> "REDACTED" … --- ## [Can't create indices](https://discuss.elastic.co/t/cant-create-indices/254204)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 6\ **Last updated:** [November 4, 2020, 10:08am UTC](https://discuss.elastic.co/t/cant-create-indices/254204 "2020-11-04T10:08:37Z")
setup elk version 7.9.3 cannot create index logstash show me warning \[logstash.outputs.elasticsearch\]\[main\]\[27fb9d77028509eff9b3e4ec584ce0c8528c70f76c962e4e94e144a725a5b964\] Could not index event to Elasticsearch. {:st… --- ## [Logstash output best practice](https://discuss.elastic.co/t/logstash-output-best-practice/253690)
**Author:** [@anon42972578](https://discuss.elastic.co/u/anon42972578)\ **Replies:** 3\ **Last updated:** [November 4, 2020, 9:12am UTC](https://discuss.elastic.co/t/logstash-output-best-practice/253690 "2020-11-04T09:12:38Z")
Hello, we want to create a good scaling elastic cluster with: at least 3 master nodes coord nodes a lot of data nodes machine learning nodes ingest node ( for cases, where using logstash is not possible) also we have… --- ## [How to create a separate index which has only required fields](https://discuss.elastic.co/t/how-to-create-a-separate-index-which-has-only-required-fields/253125)
**Author:** [@Ronnie16](https://discuss.elastic.co/u/Ronnie16)\ **Replies:** 3\ **Last updated:** [November 4, 2020, 5:26am UTC](https://discuss.elastic.co/t/how-to-create-a-separate-index-which-has-only-required-fields/253125 "2020-11-04T05:26:54Z")
I have ELK cluster setup and i delete indices older than 10 days but i want to create a separate index which only has 1 required field so that it is always present . The main index has huge data and multiple fields, i o… --- ## [Error logstash stop command](https://discuss.elastic.co/t/error-logstash-stop-command/254212)
**Author:** [@peledev](https://discuss.elastic.co/u/peledev)\ **Replies:** 1\ **Last updated:** [November 4, 2020, 6:42am UTC](https://discuss.elastic.co/t/error-logstash-stop-command/254212 "2020-11-04T06:42:59Z")
every time i stop logstash service i get failed eventualy it stops the service but not proprly this is whar i get: \[root@vrl-logapp2 logstash\]# logstash\_status ● logstash.service - logstash Loaded: loaded (/etc/syst… --- ## [Logstash Server Location](https://discuss.elastic.co/t/logstash-server-location/254115)
**Author:** [@Karl\_Ofeiche](https://discuss.elastic.co/u/Karl_Ofeiche)\ **Replies:** 3\ **Last updated:** [November 4, 2020, 6:30am UTC](https://discuss.elastic.co/t/logstash-server-location/254115 "2020-11-04T06:30:40Z")
Hello all, I'm building an ELK-Kubernetes environment for an enterprise, and I'm installing Beats on their PCs, servers, network components... and the data center where I'll store and configure my ElasticSearch and Kiba… --- ## [How to parse WebSphere SystemErr log in logstash](https://discuss.elastic.co/t/how-to-parse-websphere-systemerr-log-in-logstash/252323)
**Author:** [@eubarkana](https://discuss.elastic.co/u/eubarkana)\ **Replies:** 14\ **Last updated:** [November 4, 2020, 6:18am UTC](https://discuss.elastic.co/t/how-to-parse-websphere-systemerr-log-in-logstash/252323 "2020-11-04T06:18:37Z")
IBM WebSphere logs include SystemOut and SystemErr logs for a JVM. Each message in SystemOut is enabled with timestamp that can be parsed with logstash. Exceptions in this log has stack trace but does not include timesta… --- ## [OAuth2.0 support for http\_poller input plugin](https://discuss.elastic.co/t/oauth2-0-support-for-http-poller-input-plugin/254199)
**Author:** [@pankaj](https://discuss.elastic.co/u/pankaj)\ **Replies:** 1\ **Last updated:** [November 4, 2020, 5:34am UTC](https://discuss.elastic.co/t/oauth2-0-support-for-http-poller-input-plugin/254199 "2020-11-04T05:34:59Z")
Hello Team I would like to know how to configure the OAuth2.0 in http\_poller input . --- ## [Regex performance with logstash using more general match](https://discuss.elastic.co/t/regex-performance-with-logstash-using-more-general-match/253917)
**Author:** [@vinci](https://discuss.elastic.co/u/vinci)\ **Replies:** 9\ **Last updated:** [November 3, 2020, 10:34pm UTC](https://discuss.elastic.co/t/regex-performance-with-logstash-using-more-general-match/253917 "2020-11-03T22:34:50Z")
Hi, I was wondering about regex performance in general and in particular with logstash and grok. I'm currently trying to parse firewall logs sent through the syslog protocol which mainly contains keys and values (a key… --- ## [Logstash does not send logs into the file](https://discuss.elastic.co/t/logstash-does-not-send-logs-into-the-file/254165)
**Author:** [@Andex](https://discuss.elastic.co/u/Andex)\ **Replies:** 4\ **Last updated:** [November 3, 2020, 4:35pm UTC](https://discuss.elastic.co/t/logstash-does-not-send-logs-into-the-file/254165 "2020-11-03T16:35:34Z")
Hi, from filebeat, i send log to logstash, if I listen in tcpdump I see the logs arrive but Logstash don't send Log tothe file i specified in the pipelines, this is the Pipelines : input { beats { port =\> 5044 } } … --- ## [Error while running file configuration in Logstash - \*\*ERROR at line 4, column 19 (byte 62) after input {\\r\\nhttp\_poller {\\r\\n urls =\> {\\r\\n users ", :backtrace=\>\["C:/ELK/logstash-7.9.2/logstash-core/lib/logstash/compiler.rb:32:in \`compile\_imperative'"](https://discuss.elastic.co/t/error-while-running-file-configuration-in-logstash-error-at-line-4-column-19-byte-62-after-input-r-nhttp-poller-r-n-urls-r-n-users-backtrace-c-elk-logstash-7-9-2-logstash-core-lib-logstash-compiler-rbin-compile-imperative/254099)
**Author:** [@salma\_widiarti](https://discuss.elastic.co/u/salma_widiarti)\ **Replies:** 1\ **Last updated:** [November 3, 2020, 3:45pm UTC](https://discuss.elastic.co/t/error-while-running-file-configuration-in-logstash-error-at-line-4-column-19-byte-62-after-input-r-nhttp-poller-r-n-urls-r-n-users-backtrace-c-elk-logstash-7-9-2-logstash-core-lib-logstash-compiler-rbin-compile-imperative/254099 "2020-11-03T15:45:37Z")
Hi all, Can any guide me to create file configuration logstash with API? I've been using this syntax but i found error. input { http\_poller { urls =\> { users = { method =\> get url =\> "https://api.id/api/categories"… --- ## [Script Params vs Constant Hash in Script](https://discuss.elastic.co/t/script-params-vs-constant-hash-in-script/254163)
**Author:** [@git-blame](https://discuss.elastic.co/u/git-blame)\ **Replies:** 0\ **Last updated:** [November 3, 2020, 2:26pm UTC](https://discuss.elastic.co/t/script-params-vs-constant-hash-in-script/254163 "2020-11-03T14:26:41Z")
I need to translate from one set of values to another. I would use the translate filter plugin but I need to keep the original value if no translation is found instead of nil. So my code is: TRANSLATE\_MAP = { "foo" =\> … --- ## [Logstash of first node getting stopped after adding second node into cluster, added ilm\_enabled=false](https://discuss.elastic.co/t/logstash-of-first-node-getting-stopped-after-adding-second-node-into-cluster-added-ilm-enabled-false/253518)
**Author:** [@kkhadka](https://discuss.elastic.co/u/kkhadka)\ **Replies:** 3\ **Last updated:** [November 3, 2020, 2:01pm UTC](https://discuss.elastic.co/t/logstash-of-first-node-getting-stopped-after-adding-second-node-into-cluster-added-ilm-enabled-false/253518 "2020-11-03T14:01:43Z")
The ERROR logs are as follow: \[2020-10-12T17:52:25,998\]\[ERROR\]\[logstash.outputs.elasticsearch\]\[events\] Failed to install template. {:message=\>"Elasticsearch Unreachable: \[http://164.\*\*.185.202:9200/\]\[Manticore::SocketE… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=284) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=286)