# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=559 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 560 --- ## [Logstash logging problem](https://discuss.elastic.co/t/logstash-logging-problem/178169)
**Author:** [@ahmetkumas1](https://discuss.elastic.co/u/ahmetkumas1)\ **Replies:** 15\ **Last updated:** [April 24, 2019, 3:22pm UTC](https://discuss.elastic.co/t/logstash-logging-problem/178169 "2019-04-24T15:22:47Z")
Hello everyone, I'm having trouble with logstash when I want to log a CSV file into elasticsearch and my config file is below. elastic search version:6.5.4 Logstash version:6.6.0 There is that kind of warning \[WARN … --- ## [Couldnot decode flowset for first flow packet](https://discuss.elastic.co/t/couldnot-decode-flowset-for-first-flow-packet/176943)
**Author:** [@certesrajesh](https://discuss.elastic.co/u/certesrajesh)\ **Replies:** 2\ **Last updated:** [April 24, 2019, 2:15pm UTC](https://discuss.elastic.co/t/couldnot-decode-flowset-for-first-flow-packet/176943 "2019-04-24T14:15:20Z")
Logstash receives the attached packets (template and flow packets) but it seems to drop the flow packet with the error \[WARN \]\[logstash.codecs.netflow \] Can't (yet) decode flowset id 2048 from observation domain id… --- ## [Jdbcstreaming logstash filter plugin - parameters option is not working](https://discuss.elastic.co/t/jdbcstreaming-logstash-filter-plugin-parameters-option-is-not-working/178149)
**Author:** [@Gomathi\_Meena](https://discuss.elastic.co/u/Gomathi_Meena)\ **Replies:** 1\ **Last updated:** [April 24, 2019, 2:13pm UTC](https://discuss.elastic.co/t/jdbcstreaming-logstash-filter-plugin-parameters-option-is-not-working/178149 "2019-04-24T14:13:54Z")
Hi , I have a data in MySQL .So using Jdbc\_streaming filter i can able to filter and parse to elastic search. While configuring the options for Jdbc\_streaming filter , parameters option is not working for me . What wa… --- ## [Input from two csv files and compare](https://discuss.elastic.co/t/input-from-two-csv-files-and-compare/178238)
**Author:** [@nandha\_88](https://discuss.elastic.co/u/nandha_88)\ **Replies:** 1\ **Last updated:** [April 24, 2019, 2:12pm UTC](https://discuss.elastic.co/t/input-from-two-csv-files-and-compare/178238 "2019-04-24T14:12:59Z")
Hi Team, I am planning to get the input from two csf files, assign the columns value and if the device column value match , I need the print the ip address in the output . Please share the codes to compare and print n… --- ## [Live reload of mmdb files in a logstash filter](https://discuss.elastic.co/t/live-reload-of-mmdb-files-in-a-logstash-filter/178158)
**Author:** [@Louis\_Ruben](https://discuss.elastic.co/u/Louis_Ruben)\ **Replies:** 1\ **Last updated:** [April 24, 2019, 1:46pm UTC](https://discuss.elastic.co/t/live-reload-of-mmdb-files-in-a-logstash-filter/178158 "2019-04-24T13:46:11Z")
Hello, Is there a way to replace an mmdb file which is used in a filter without stopping and starting back logstash? For ex. if I want to use an updated version of an mmdb file. I'm using the version 6.4.3 of logstash a… --- ## [Logstash environment filter plugin](https://discuss.elastic.co/t/logstash-environment-filter-plugin/178170)
**Author:** [@Gomathi\_Meena](https://discuss.elastic.co/u/Gomathi_Meena)\ **Replies:** 1\ **Last updated:** [April 24, 2019, 1:41pm UTC](https://discuss.elastic.co/t/logstash-environment-filter-plugin/178170 "2019-04-24T13:41:25Z")
Hi . I have been trying to know the purpose of environment filter for a while But at last it came with no effect in my data index in elastic search Can anyone tell me the purpose of environment filter and what the mist… --- ## [Allocate time field to a different field](https://discuss.elastic.co/t/allocate-time-field-to-a-different-field/178189)
**Author:** [@shrikantgulia](https://discuss.elastic.co/u/shrikantgulia)\ **Replies:** 1\ **Last updated:** [April 24, 2019, 1:33pm UTC](https://discuss.elastic.co/t/allocate-time-field-to-a-different-field/178189 "2019-04-24T13:33:25Z")
In Logstash config file i renamed received to a completely new field i.e created where the format of received field is eg 2019-04-23 13:35:46 received is IST format and my kibana is UTC format rename =\> \[ "received", … --- ## [Extract fields from a log](https://discuss.elastic.co/t/extract-fields-from-a-log/178231)
**Author:** [@ameni](https://discuss.elastic.co/u/ameni)\ **Replies:** 2\ **Last updated:** [April 24, 2019, 1:25pm UTC](https://discuss.elastic.co/t/extract-fields-from-a-log/178231 "2019-04-24T13:25:40Z")
hi this is a log coming from a firewalll i need to extract from that message IP\_srs, src\_Port , IP\_dst , dst\_port and make them as fields please help me --- ## [Grok Configuration error](https://discuss.elastic.co/t/grok-configuration-error/178103)
**Author:** [@Shaooli](https://discuss.elastic.co/u/Shaooli)\ **Replies:** 2\ **Last updated:** [April 24, 2019, 1:14pm UTC](https://discuss.elastic.co/t/grok-configuration-error/178103 "2019-04-24T13:14:43Z")
Hello, i am very new to this, i guess this is a simple one, but i still need your help. when running logstash i get the below configuration error: \[2019-04-23T22:27:14,022\]\[ERROR\]\[logstash.agent \] Failed to e… --- ## [Resolving deprecation warnings using Default Logstash config to allow us to upgrade](https://discuss.elastic.co/t/resolving-deprecation-warnings-using-default-logstash-config-to-allow-us-to-upgrade/178245)
**Author:** [@cowensel](https://discuss.elastic.co/u/cowensel)\ **Replies:** 0\ **Last updated:** [April 24, 2019, 12:18pm UTC](https://discuss.elastic.co/t/resolving-deprecation-warnings-using-default-logstash-config-to-allow-us-to-upgrade/178245 "2019-04-24T12:18:56Z")
I am currently going through the Elasticsearch deprecation logs to check what we may need to resolve prior to upgrading our ELK Stack from v6.0.0 where I have found the following but have no clue how to resolve. We have … --- ## [Logstash 7.0.0 testing config file](https://discuss.elastic.co/t/logstash-7-0-0-testing-config-file/178241)
**Author:** [@davidbien](https://discuss.elastic.co/u/davidbien)\ **Replies:** 0\ **Last updated:** [April 24, 2019, 12:13pm UTC](https://discuss.elastic.co/t/logstash-7-0-0-testing-config-file/178241 "2019-04-24T12:13:55Z")
I'm have a config file(.conf) which I use currently to run logstash on docker. In the same directory I also have a simple pipelines.yml and logstash.yml files. When running these in docker I get no errors whatsoever and … --- ## [If conditional with multiple outputs](https://discuss.elastic.co/t/if-conditional-with-multiple-outputs/178201)
**Author:** [@davidbien](https://discuss.elastic.co/u/davidbien)\ **Replies:** 6\ **Last updated:** [April 24, 2019, 11:34am UTC](https://discuss.elastic.co/t/if-conditional-with-multiple-outputs/178201 "2019-04-24T11:34:33Z")
Hi, I have the following configuration in my logstash atm. input { redis { host =\> "\[elasticb.0001.us1.cache.amazonaws.com\]" port =\> 6379 key =\> "filebeat" data\_type =\> "list" } }… --- ## [Logstash consumer thread poll issue](https://discuss.elastic.co/t/logstash-consumer-thread-poll-issue/178218)
**Author:** [@sukku77](https://discuss.elastic.co/u/sukku77)\ **Replies:** 0\ **Last updated:** [April 24, 2019, 10:20am UTC](https://discuss.elastic.co/t/logstash-consumer-thread-poll-issue/178218 "2019-04-24T10:20:31Z")
Hi , I am getting below exception in logstash logs which has consumer threads running to listen kafka topics. Below is the error trace: \[2019-04-24T09:56:21,012\]\[WARN \]\[org.apache.kafka.clients.consumer.internals… --- ## [Kibana not showing other than .log files like .out , .txt , etc](https://discuss.elastic.co/t/kibana-not-showing-other-than-log-files-like-out-txt-etc/178217)
**Author:** [@bhushan18](https://discuss.elastic.co/u/bhushan18)\ **Replies:** 0\ **Last updated:** [April 24, 2019, 10:09am UTC](https://discuss.elastic.co/t/kibana-not-showing-other-than-log-files-like-out-txt-etc/178217 "2019-04-24T10:09:19Z")
My Pipeline is like FB--\> Logstash --\> EL --\> Kibana I am able to find all .log files in Kibana but not able to search any files who dont have extension .log Filebeat.yml input configurations are : filebeat.inputs: E… --- ## [Why my logstash keeps on restarting!](https://discuss.elastic.co/t/why-my-logstash-keeps-on-restarting/174535)
**Author:** [@Gomathi\_Meena](https://discuss.elastic.co/u/Gomathi_Meena)\ **Replies:** 8\ **Last updated:** [April 24, 2019, 5:22am UTC](https://discuss.elastic.co/t/why-my-logstash-keeps-on-restarting/174535 "2019-04-24T05:22:33Z")
Hi, I'm new to logstash.I have installed logstash and started it using sudo service logstash start ,it works perfectly until today morning. I don't have any idea what happened after that ! Now i tried the same configur… --- ## [Unbale to add filed in logstash for json source](https://discuss.elastic.co/t/unbale-to-add-filed-in-logstash-for-json-source/178099)
**Author:** [@Charan\_Adabala](https://discuss.elastic.co/u/Charan_Adabala)\ **Replies:** 2\ **Last updated:** [April 24, 2019, 2:08am UTC](https://discuss.elastic.co/t/unbale-to-add-filed-in-logstash-for-json-source/178099 "2019-04-24T02:08:02Z")
Hi, I am trying to add new filed based on the filed exists in the event, but I'm unable add can you please help on that. Event: { "@version": "1", "@timestamp": "2019-04-23T18:04:19.680Z", "headers": { "http\_versi… --- ## [Logstash cloudwatch plugin no cloudfront metric](https://discuss.elastic.co/t/logstash-cloudwatch-plugin-no-cloudfront-metric/178131)
**Author:** [@nt-josh](https://discuss.elastic.co/u/nt-josh)\ **Replies:** 0\ **Last updated:** [April 23, 2019, 11:51pm UTC](https://discuss.elastic.co/t/logstash-cloudwatch-plugin-no-cloudfront-metric/178131 "2019-04-23T23:51:19Z")
Hi, I am using logstash 6.6 docker container with elastic and kibana. for right now I am just messing with the logstash part. I am trying to output the cloudwatch metrics for CloudFront to stdout debug and unfortunatel… --- ## [Logstash 6.4.0 stuck on: Successfully started Logstash API endpoint {:port=\>9600}](https://discuss.elastic.co/t/logstash-6-4-0-stuck-on-successfully-started-logstash-api-endpoint-port-9600/177732)
**Author:** [@nomoneynoproblems](https://discuss.elastic.co/u/nomoneynoproblems)\ **Replies:** 9\ **Last updated:** [April 23, 2019, 10:59pm UTC](https://discuss.elastic.co/t/logstash-6-4-0-stuck-on-successfully-started-logstash-api-endpoint-port-9600/177732 "2019-04-23T22:59:06Z")
Hi, I use Jenkins to update two different indexes on one server with two different Logstash configuration scripts. A custom Python script ssh's into the server and starts Logstash with the appropriate yml config file an… --- ## [Logstash Netflow Binding to specific Interface](https://discuss.elastic.co/t/logstash-netflow-binding-to-specific-interface/178127)
**Author:** [@etsai7](https://discuss.elastic.co/u/etsai7)\ **Replies:** 0\ **Last updated:** [April 23, 2019, 10:53pm UTC](https://discuss.elastic.co/t/logstash-netflow-binding-to-specific-interface/178127 "2019-04-23T22:53:26Z")
Hello, I currently have Logstash set up on a server multiple NICs and am trying to bind Logstash to a specific IP instead of the default one when running the netflow module. How can I go about binding it to a specific … --- ## [Output to e-mail address or web hook](https://discuss.elastic.co/t/output-to-e-mail-address-or-web-hook/177975)
**Author:** [@duncOps](https://discuss.elastic.co/u/duncOps)\ **Replies:** 1\ **Last updated:** [April 23, 2019, 9:25pm UTC](https://discuss.elastic.co/t/output-to-e-mail-address-or-web-hook/177975 "2019-04-23T21:25:16Z")
Morning folks, Are there any modules I can use in my Logstash pipeline filter to output certain logs to either an e-mail address or send a web request? Thanks in advance, Duncan --- ## [Access array in ruby loop](https://discuss.elastic.co/t/access-array-in-ruby-loop/178028)
**Author:** [@manunc](https://discuss.elastic.co/u/manunc)\ **Replies:** 4\ **Last updated:** [April 23, 2019, 8:55pm UTC](https://discuss.elastic.co/t/access-array-in-ruby-loop/178028 "2019-04-23T20:55:02Z")
Hello, I would like to create new fields based on an array coming from a splitted field and associate the value of each fields dtab = data.split(/(\\w{20})(\\w{20})(\\w{20})(\\w{20})(\\w{20})(\\w{20})(\\w{20})(\\w{20… --- ## [Grok Filter with optional fields fails](https://discuss.elastic.co/t/grok-filter-with-optional-fields-fails/178057)
**Author:** [@harrytewkesbury](https://discuss.elastic.co/u/harrytewkesbury)\ **Replies:** 1\ **Last updated:** [April 23, 2019, 7:21pm UTC](https://discuss.elastic.co/t/grok-filter-with-optional-fields-fails/178057 "2019-04-23T19:21:56Z")
Hi, For IPTables, we log lines similar to this: Apr 23 14:32:22 hostname1 kernel: \[64719873.568549\] IPTABLES-INPUT DROP :IN=br0-PROD OUT= MAC=ff:ff:ff:ff:ff:ff:52:54:00:b2:6c:4e:08:00 SRC=10.3.9.21 DST=10.3.9.255 LEN=7… --- ## [Error Starting logstash When Using path.config in pipelines.yml](https://discuss.elastic.co/t/error-starting-logstash-when-using-path-config-in-pipelines-yml/178096)
**Author:** [@thadc](https://discuss.elastic.co/u/thadc)\ **Replies:** 0\ **Last updated:** [April 23, 2019, 6:37pm UTC](https://discuss.elastic.co/t/error-starting-logstash-when-using-path-config-in-pipelines-yml/178096 "2019-04-23T18:37:21Z")
Hi, I have a very simple pipelines.yml file defined with a single pipeline. It looks like this: - pipeline.id: testPipe1 path.config: "/tmp/test.conf" pipeline.workers: 1 when starting logstash I received the followi… --- ## [How do I set the file permission for files created with the CSV output plugin?](https://discuss.elastic.co/t/how-do-i-set-the-file-permission-for-files-created-with-the-csv-output-plugin/178092)
**Author:** [@snowboardcto](https://discuss.elastic.co/u/snowboardcto)\ **Replies:** 0\ **Last updated:** [April 23, 2019, 6:13pm UTC](https://discuss.elastic.co/t/how-do-i-set-the-file-permission-for-files-created-with-the-csv-output-plugin/178092 "2019-04-23T18:13:51Z")
Is it possible to set the file permission for files created with the CSV output plugin? Files are owned by the logstash user and I'd like to define another user who has permission. --- ## [Logstash kv filter plugin query](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894)
**Author:** [@Neeraj\_Jain](https://discuss.elastic.co/u/Neeraj_Jain)\ **Replies:** 6\ **Last updated:** [April 23, 2019, 5:45pm UTC](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894 "2019-04-23T17:45:30Z")
Hi, I am trying to use logstash to read a file containing data as below. I want to push this data as key/value to elastic search. pid: 2601 start: 0 stacksize: 0 breaksize: 25 command: app1 arguments: APP01 pid: 2393 … --- ## [Logstash TCP Input expected behavior when event source connection reset](https://discuss.elastic.co/t/logstash-tcp-input-expected-behavior-when-event-source-connection-reset/178070)
**Author:** [@freakingid](https://discuss.elastic.co/u/freakingid)\ **Replies:** 2\ **Last updated:** [April 23, 2019, 5:19pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-expected-behavior-when-event-source-connection-reset/178070 "2019-04-23T17:19:16Z")
When using the Logstash TCP Input Plugin, what is the expected behavior if the network connection between event source and Logstash is interrupted? When the plugin is in client mode, will Logstash attempt to reconnect … --- ## [Installing logstash 6.7 with OpenJDK-11](https://discuss.elastic.co/t/installing-logstash-6-7-with-openjdk-11/177771)
**Author:** [@epacke](https://discuss.elastic.co/u/epacke)\ **Replies:** 4\ **Last updated:** [April 23, 2019, 5:18pm UTC](https://discuss.elastic.co/t/installing-logstash-6-7-with-openjdk-11/177771 "2019-04-23T17:18:54Z")
Hi there Now that Oracle has decided to make the Java installation more troublesome I decided to try Openjdk instead. It says on the Elastic support matrix that both ElasticSearch and Logstash supports Openjdk-11. Howe… --- ## [Configure s3 data pipeline to logstash](https://discuss.elastic.co/t/configure-s3-data-pipeline-to-logstash/178081)
**Author:** [@cbeprem](https://discuss.elastic.co/u/cbeprem)\ **Replies:** 0\ **Last updated:** [April 23, 2019, 5:06pm UTC](https://discuss.elastic.co/t/configure-s3-data-pipeline-to-logstash/178081 "2019-04-23T17:06:29Z")
Hi ALL, I have a s3 bucket which i have data on daily basis named after date for eg 22april 23april 24april and a csv file in each folder . how do i automate it. For a particular file there is document but for dif… --- ## [Logstash filter error (Pipeline aborted due to error )](https://discuss.elastic.co/t/logstash-filter-error-pipeline-aborted-due-to-error/178062)
**Author:** [@Dhaoui\_Ahmed](https://discuss.elastic.co/u/Dhaoui_Ahmed)\ **Replies:** 1\ **Last updated:** [April 23, 2019, 3:50pm UTC](https://discuss.elastic.co/t/logstash-filter-error-pipeline-aborted-due-to-error/178062 "2019-04-23T15:50:33Z")
Hello am running ELK stack( forwarding netscreen firewall logs ) with logstash input and output config no filter ,, all good until i have added a filter and i got this error in logstash logfile Pipeline aborted due to … --- ## [Logstash 6.5.4 Changing UDP Listener Address](https://discuss.elastic.co/t/logstash-6-5-4-changing-udp-listener-address/177912)
**Author:** [@etsai7](https://discuss.elastic.co/u/etsai7)\ **Replies:** 0\ **Last updated:** [April 22, 2019, 11:39pm UTC](https://discuss.elastic.co/t/logstash-6-5-4-changing-udp-listener-address/177912 "2019-04-22T23:39:19Z")
Hello, Currently I am trying to change the UDP listener for the netflow module on Logstash from 0.0.0.0:2055 to 10.10.10.10:2055 for example. I have tried changing the logstash.yml file to the following: modules: … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=558) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=560)