# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=490 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 491 --- ## [KV filter plugin install](https://discuss.elastic.co/t/kv-filter-plugin-install/195952)
**Author:** [@dcrocker](https://discuss.elastic.co/u/dcrocker)\ **Replies:** 0\ **Last updated:** [August 20, 2019, 1:57pm UTC](https://discuss.elastic.co/t/kv-filter-plugin-install/195952 "2019-08-20T13:57:42Z")
I am new to using logstash. I need to install the kv filter but having issues. I get an error when I try to run the following command: gem 'logstash-filter-kv', '~\> 4.4' The error is: ERROR: While executing gem ...… --- ## [Unable to pass the custom pattern](https://discuss.elastic.co/t/unable-to-pass-the-custom-pattern/195953)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 1\ **Last updated:** [August 20, 2019, 2:06pm UTC](https://discuss.elastic.co/t/unable-to-pass-the-custom-pattern/195953 "2019-08-20T14:06:41Z")
Hi Team, I am trying to parse the time which is anchored Eg \ I have created custom pattern it in different location patterns\_dir =\> \[ "/etc/logstash/custompaterns/patterns" \] WEBLOG\_TIM… --- ## [Logstash: parsing AWS WAF log](https://discuss.elastic.co/t/logstash-parsing-aws-waf-log/195925)
**Author:** [@Igor\_Belyakov](https://discuss.elastic.co/u/Igor_Belyakov)\ **Replies:** 4\ **Last updated:** [August 20, 2019, 1:58pm UTC](https://discuss.elastic.co/t/logstash-parsing-aws-waf-log/195925 "2019-08-20T13:58:20Z")
Hi. Can i use HTTP filter (or other filter) to parse some fields from json AWS WAF logs from s3 bucket? logstash.conf: input{ s3 { bucket =\> "XXXX" access\_key\_id =\> "XXXX" secret\_access\_key =\> "XXXX" region =\> "XX… --- ## [Problem with combining xmlfilter and ruby code to compute a time difference](https://discuss.elastic.co/t/problem-with-combining-xmlfilter-and-ruby-code-to-compute-a-time-difference/195924)
**Author:** [@Franco](https://discuss.elastic.co/u/Franco)\ **Replies:** 2\ **Last updated:** [August 20, 2019, 1:24pm UTC](https://discuss.elastic.co/t/problem-with-combining-xmlfilter-and-ruby-code-to-compute-a-time-difference/195924 "2019-08-20T13:24:44Z")
Hi all, I have an xml log which looks like this: \ **Author:** [@Efstathios\_Xagoraris](https://discuss.elastic.co/u/Efstathios_Xagoraris)\ **Replies:** 2\ **Last updated:** [August 20, 2019, 1:21pm UTC](https://discuss.elastic.co/t/rabbitmq-input-plugin-with-empty-payload/195927 "2019-08-20T13:21:37Z") Hello, I am wondering if other users are experiencing the same issue. I have a rabbitmq server with the event exchange plugin enabled. I would like to use logstash to consume the messages: The configuration is straight… --- ## [Forcing Logstash to use TLS 1\_2](https://discuss.elastic.co/t/forcing-logstash-to-use-tls-1-2/195869)
**Author:** [@Zerobot](https://discuss.elastic.co/u/Zerobot)\ **Replies:** 3\ **Last updated:** [August 20, 2019, 12:23pm UTC](https://discuss.elastic.co/t/forcing-logstash-to-use-tls-1-2/195869 "2019-08-20T12:23:02Z")
Hi! Is it possible to force Logstash (TCP\_INPUT) into using TLS v.1.2 only? Beats Input is actually using 1\_2 but sadly my TCP input is using 1.0, 1.1 and 1.2. Also, while starting Logstash I see: ssl.enabled.pr… --- ## [\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main](https://discuss.elastic.co/t/error-logstash-agent-failed-to-execute-action-action-logstash-create-pipeline-id-main/195915)
**Author:** [@jainabhishek0406](https://discuss.elastic.co/u/jainabhishek0406)\ **Replies:** 1\ **Last updated:** [August 20, 2019, 11:56am UTC](https://discuss.elastic.co/t/error-logstash-agent-failed-to-execute-action-action-logstash-create-pipeline-id-main/195915 "2019-08-20T11:56:38Z")
kindly help to resolve this issue. PS C:\\Users\\abhishek12.jain\\ELK\_Data\\logstash-6.4.3\> .\\bin/logstash.bat -f bin/logstash-sample.conf Sending Logstash logs to C:/Users/abhishek12.jain/ELK\_Data/logstash-6.4.3/logs whic… --- ## [Logstash http input plugin configuration of same URL on multiple nodes](https://discuss.elastic.co/t/logstash-http-input-plugin-configuration-of-same-url-on-multiple-nodes/195817)
**Author:** [@iamlearner123](https://discuss.elastic.co/u/iamlearner123)\ **Replies:** 1\ **Last updated:** [August 20, 2019, 11:41am UTC](https://discuss.elastic.co/t/logstash-http-input-plugin-configuration-of-same-url-on-multiple-nodes/195817 "2019-08-20T11:41:56Z")
Hello, i have two logstash instances running and i have configured one of the instance with HTTP input plugin to fetch the data from a URL. I want of find out whether is it possible to configure both the logstash nodes… --- ## [Kibana not recognizing logstash date filter](https://discuss.elastic.co/t/kibana-not-recognizing-logstash-date-filter/195816)
**Author:** [@userR](https://discuss.elastic.co/u/userR)\ **Replies:** 1\ **Last updated:** [August 20, 2019, 11:41am UTC](https://discuss.elastic.co/t/kibana-not-recognizing-logstash-date-filter/195816 "2019-08-20T11:41:04Z")
I have dates formatted as such: 2013-05-02 08:02:22.622+0000 And my date filter is the following: date { match =\> \["logdate", "YYYY-MM-DD HH:mm:ss.SSSZZZZ"\] target =\> "logdate" } However, when I create indices… --- ## [Conf files getting mixed when uploading](https://discuss.elastic.co/t/conf-files-getting-mixed-when-uploading/195803)
**Author:** [@cjb312](https://discuss.elastic.co/u/cjb312)\ **Replies:** 1\ **Last updated:** [August 20, 2019, 11:38am UTC](https://discuss.elastic.co/t/conf-files-getting-mixed-when-uploading/195803 "2019-08-20T11:38:14Z")
I'm trying to ingest a number of csv files to view on Kibana but am encountering an issue of fields carrying over from the previous file. I have multiple .conf files formatted like so: input { file { path =\> … --- ## [Logstash and Elasticsearch Ingest Node](https://discuss.elastic.co/t/logstash-and-elasticsearch-ingest-node/195890)
**Author:** [@stevesimpson](https://discuss.elastic.co/u/stevesimpson)\ **Replies:** 0\ **Last updated:** [August 20, 2019, 10:42am UTC](https://discuss.elastic.co/t/logstash-and-elasticsearch-ingest-node/195890 "2019-08-20T10:42:54Z")
Hi, In our setup we are collecting log files using Filebeat and sending these into Logstash for filtering/enrichment then onto Elasticsearch. This is working as intended (although having some unrelated issues with ILM :… --- ## [Logstash parse log nginx not index refresh](https://discuss.elastic.co/t/logstash-parse-log-nginx-not-index-refresh/195881)
**Author:** [@jfernandez](https://discuss.elastic.co/u/jfernandez)\ **Replies:** 0\ **Last updated:** [August 20, 2019, 10:07am UTC](https://discuss.elastic.co/t/logstash-parse-log-nginx-not-index-refresh/195881 "2019-08-20T10:07:22Z")
Hello, I just finished my first installation ELK and I discover the differents elements that compose it. My first exercise is to parse nginx logs via Logstash and grok filter. To do so, I created a configuration file i… --- ## ["ignoring JAVA\_OPTS=-Xmx1024m -Xms1024m; pass JVM parameters via LS\_JAVA\_OPTS" error](https://discuss.elastic.co/t/ignoring-java-opts-xmx1024m-xms1024m-pass-jvm-parameters-via-ls-java-opts-error/195877)
**Author:** [@whalien52](https://discuss.elastic.co/u/whalien52)\ **Replies:** 0\ **Last updated:** [August 20, 2019, 9:43am UTC](https://discuss.elastic.co/t/ignoring-java-opts-xmx1024m-xms1024m-pass-jvm-parameters-via-ls-java-opts-error/195877 "2019-08-20T09:43:30Z")
I just downloaded the latest version of Logstash 7.3.0 and my aim is to send some data from SQL server to Elasticsearch via Logstash. Whenever I am running the logstash bat file, I am again and again getting the same er… --- ## [Netflow\_module error](https://discuss.elastic.co/t/netflow-module-error/195723)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 15\ **Last updated:** [August 20, 2019, 9:05am UTC](https://discuss.elastic.co/t/netflow-module-error/195723 "2019-08-20T09:05:47Z")
Hi I am trying to use logstash netflow module but when I am executing "bin/logstash --modules netflow -M netflow.var.input.udp.port=2055" then I am getting error saying: "\[WARN \] 2019-08-19 16:26:18.548 \[Ruby-0-Thread-4… --- ## [How to extract json from the response field in the output of dead letter queue?](https://discuss.elastic.co/t/how-to-extract-json-from-the-response-field-in-the-output-of-dead-letter-queue/195306)
**Author:** [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\ **Replies:** 2\ **Last updated:** [August 19, 2019, 12:01pm UTC](https://discuss.elastic.co/t/how-to-extract-json-from-the-response-field-in-the-output-of-dead-letter-queue/195306 "2019-08-19T12:01:39Z")
hi guys , my scripts is as follows :- input { dead\_letter\_queue { path =\> "${LS\_HOME}/data/dead\_letter\_queue" pipeline\_id =\> "main" } } filter { ruby { path =\> "${LS\_HOME}/config/failed-index-cre… --- ## [Is there anyway for logstash to join two csv files(m:m relationship) based on a common field, and ingest the data into one single index](https://discuss.elastic.co/t/is-there-anyway-for-logstash-to-join-two-csv-files-m-m-relationship-based-on-a-common-field-and-ingest-the-data-into-one-single-index/195835)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 0\ **Last updated:** [August 20, 2019, 3:21am UTC](https://discuss.elastic.co/t/is-there-anyway-for-logstash-to-join-two-csv-files-m-m-relationship-based-on-a-common-field-and-ingest-the-data-into-one-single-index/195835 "2019-08-20T03:21:15Z")
Hi Team, I have 2 csv files with be column info Columns of 1.csv \` caseid, classname, features, actions, author, module, prerequisite, steps, ation \` 2. Columns of 2.csv \` module, features, ation, ui, ktf1,kt… --- ## [Can I substring event data's value and use this in elasitcsearch index name in logstash config?](https://discuss.elastic.co/t/can-i-substring-event-datas-value-and-use-this-in-elasitcsearch-index-name-in-logstash-config/195012)
**Author:** [@teemoGod](https://discuss.elastic.co/u/teemoGod)\ **Replies:** 3\ **Last updated:** [August 20, 2019, 4:20am UTC](https://discuss.elastic.co/t/can-i-substring-event-datas-value-and-use-this-in-elasitcsearch-index-name-in-logstash-config/195012 "2019-08-20T04:20:39Z")
following is my config input { file { path =\> "/\*.log" start\_position =\> "beginning" sincedb\_path =\> "/sincedb.log" exclude =\> \["\*current.log", "\*sincedb.log"\] } } filter { json { source =\> "m… --- ## [Parse repeated tags in xml using split filter](https://discuss.elastic.co/t/parse-repeated-tags-in-xml-using-split-filter/194231)
**Author:** [@Arshdeep\_Sandhu](https://discuss.elastic.co/u/Arshdeep_Sandhu)\ **Replies:** 6\ **Last updated:** [August 20, 2019, 3:51am UTC](https://discuss.elastic.co/t/parse-repeated-tags-in-xml-using-split-filter/194231 "2019-08-20T03:51:22Z")
I am trying to parse repeated tags in xml using split filter to get result as below example: If my logstash is: \ \ \ \ \ **Author:** [@xchai](https://discuss.elastic.co/u/xchai)\ **Replies:** 10\ **Last updated:** [August 20, 2019, 12:30am UTC](https://discuss.elastic.co/t/logstash-log-showing-errorcipher-functionsinvalid-nonce/189418 "2019-08-20T00:30:05Z") Hi, we are testing ELK 6.6.1 on s390x platform (Linux on Z). After starting the ELK 6.6.1 stack, everything seems working fine and we can see logs on Kibana UI, but logstash log has following entries from time to time: … --- ## [Pipeline aborted due to error - Invalid argument](https://discuss.elastic.co/t/pipeline-aborted-due-to-error-invalid-argument/195824)
**Author:** [@pgbollwerk](https://discuss.elastic.co/u/pgbollwerk)\ **Replies:** 0\ **Last updated:** [August 20, 2019, 12:12am UTC](https://discuss.elastic.co/t/pipeline-aborted-due-to-error-invalid-argument/195824 "2019-08-20T00:12:29Z")
I am quite new to the ELK stack, and I'm trying to help take over a config someone else built. I'm running into an issue recently with one of my azureblob pipelines not working properly. I've spent a few hours digging … --- ## [File input, from NetApp CIFS share, not reading single file](https://discuss.elastic.co/t/file-input-from-netapp-cifs-share-not-reading-single-file/195391)
**Author:** [@eclipsed450](https://discuss.elastic.co/u/eclipsed450)\ **Replies:** 10\ **Last updated:** [August 19, 2019, 9:25pm UTC](https://discuss.elastic.co/t/file-input-from-netapp-cifs-share-not-reading-single-file/195391 "2019-08-19T21:25:30Z")
Hi, I've made quite a few other posts, and have successfully ingested more than 1 billion messages in the last couple weeks, but we've come to the conclusion that Logstash just isn't keeping up with the backlog of messa… --- ## [Date filter for minor changes in format](https://discuss.elastic.co/t/date-filter-for-minor-changes-in-format/195259)
**Author:** [@userR](https://discuss.elastic.co/u/userR)\ **Replies:** 4\ **Last updated:** [August 19, 2019, 9:07pm UTC](https://discuss.elastic.co/t/date-filter-for-minor-changes-in-format/195259 "2019-08-19T21:07:47Z")
Mar 3, 2019 9:40:30,734 PM Mar 11, 2019 11:20:10,114 PM I have to parse logs with the following formats. The first date does not have a 0 in front of the 3. This also applies to the hours portion. I am unsure how I wou… --- ## [Syslog out plugin connection error and UDP drops](https://discuss.elastic.co/t/syslog-out-plugin-connection-error-and-udp-drops/195809)
**Author:** [@Jayanthi](https://discuss.elastic.co/u/Jayanthi)\ **Replies:** 0\ **Last updated:** [August 19, 2019, 8:57pm UTC](https://discuss.elastic.co/t/syslog-out-plugin-connection-error-and-udp-drops/195809 "2019-08-19T20:57:23Z")
We have 4 logstash servers currently just taking ~4000/sec. Works fine almost all the time, but few times couple server have UDP drops. At the same time, i see following exception logged. syslog tcp output exception: c… --- ## [Actually i am going metricbeat through logstash not getting kibana dashboard with all metrics](https://discuss.elastic.co/t/actually-i-am-going-metricbeat-through-logstash-not-getting-kibana-dashboard-with-all-metrics/195214)
**Author:** [@vishnuvardhan](https://discuss.elastic.co/u/vishnuvardhan)\ **Replies:** 5\ **Last updated:** [August 19, 2019, 8:50pm UTC](https://discuss.elastic.co/t/actually-i-am-going-metricbeat-through-logstash-not-getting-kibana-dashboard-with-all-metrics/195214 "2019-08-19T20:50:56Z")
actually i am going metricbeat through logstash not getting kibana dashboard with all metrics But i will go elasticsearch get the all the metrics . why it is not happen ?? any settings we need to change go through the l… --- ## [Select all tables from database MYSQL](https://discuss.elastic.co/t/select-all-tables-from-database-mysql/195804)
**Author:** [@Farouk6072](https://discuss.elastic.co/u/Farouk6072)\ **Replies:** 0\ **Last updated:** [August 19, 2019, 7:51pm UTC](https://discuss.elastic.co/t/select-all-tables-from-database-mysql/195804 "2019-08-19T19:51:36Z")
Hi how i can select all tables from database ? --- ## [Implementing Event dependent configurations using Java output plugins](https://discuss.elastic.co/t/implementing-event-dependent-configurations-using-java-output-plugins/195780)
**Author:** [@Xucito](https://discuss.elastic.co/u/Xucito)\ **Replies:** 4\ **Last updated:** [August 19, 2019, 6:05pm UTC](https://discuss.elastic.co/t/implementing-event-dependent-configurations-using-java-output-plugins/195780 "2019-08-19T18:05:56Z")
Following from the issue below: I am looking to access configurations that are evaluated based on the event https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html output { file { p… --- ## [How to simplify logic or format code for grok filter?](https://discuss.elastic.co/t/how-to-simplify-logic-or-format-code-for-grok-filter/193696)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 2\ **Last updated:** [August 19, 2019, 2:38pm UTC](https://discuss.elastic.co/t/how-to-simplify-logic-or-format-code-for-grok-filter/193696 "2019-08-19T14:38:20Z")
Hello, I want to find out if there's any way to simplify the below code: filter { mutate { add\_field =\> { "\[@metadata\]\[copyOfMessage\]" =\> "%{\[message\]}" } } mutate { … --- ## [Logstash file descriptors keep increasing](https://discuss.elastic.co/t/logstash-file-descriptors-keep-increasing/192433)
**Author:** [@faika](https://discuss.elastic.co/u/faika)\ **Replies:** 2\ **Last updated:** [August 19, 2019, 8:09am UTC](https://discuss.elastic.co/t/logstash-file-descriptors-keep-increasing/192433 "2019-08-19T08:09:14Z")
i'am using logstash version logstash 6.4.3. The problem is the number of file descriptors that keeps increasing and each time i restart logstash it gets back to normal then increases again. Is that normal behavior ? --- ## [Get only specific fields from a data file](https://discuss.elastic.co/t/get-only-specific-fields-from-a-data-file/195466)
**Author:** [@ROok](https://discuss.elastic.co/u/ROok)\ **Replies:** 2\ **Last updated:** [August 19, 2019, 1:48pm UTC](https://discuss.elastic.co/t/get-only-specific-fields-from-a-data-file/195466 "2019-08-19T13:48:55Z")
Hi all I'm new in elasticsearch and I don't have much knowledge about it. I have a data file that contains about a hundred fields but I only need 25 fields. However, I'm not sure which logstash filter to use to exctra… --- ## [\[7.2.0\] offline plugin installation fails](https://discuss.elastic.co/t/7-2-0-offline-plugin-installation-fails/195651)
**Author:** [@sveldhuisen](https://discuss.elastic.co/u/sveldhuisen)\ **Replies:** 1\ **Last updated:** [August 19, 2019, 12:45pm UTC](https://discuss.elastic.co/t/7-2-0-offline-plugin-installation-fails/195651 "2019-08-19T12:45:28Z")
I'm trying to get my offline plugin pack installed with Logstash 7.2.0, but that fails with an error: NoMethodError: undefined method \`each' for nil:NilClass I first created my offline plugin pack succesfully: # bin/lo… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=489) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=491)