# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=835 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 836 --- ## [ArgumentError when using Kafka input Avro codec](https://discuss.elastic.co/t/argumenterror-when-using-kafka-input-avro-codec/116975)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 0\ **Last updated:** [January 25, 2018, 6:31am UTC](https://discuss.elastic.co/t/argumenterror-when-using-kafka-input-avro-codec/116975 "2018-01-25T06:31:08Z")
Hi everyone, I am using ELK GA 6.0.0. I have avro records in my Kafka topic and I am trying to push them into elasticsearch using Logstash Kafka input plugin using Avro Codec. Below is my configuration, its pretty simpl… --- ## [The collection of syslog incorrect](https://discuss.elastic.co/t/the-collection-of-syslog-incorrect/116948)
**Author:** [@antony.y](https://discuss.elastic.co/u/antony.y)\ **Replies:** 0\ **Last updated:** [January 25, 2018, 1:43am UTC](https://discuss.elastic.co/t/the-collection-of-syslog-incorrect/116948 "2018-01-25T01:43:35Z")
I am now using tcp 8514 collect the our cisco switches' syslogs,In most cases it does work,but i found in my device it does not work.I can not even see any information in my logs.My configuration is as below: input { … --- ## [Can Logstash pull gzipped files from s3](https://discuss.elastic.co/t/can-logstash-pull-gzipped-files-from-s3/116412)
**Author:** [@red888](https://discuss.elastic.co/u/red888)\ **Replies:** 3\ **Last updated:** [January 24, 2018, 11:42pm UTC](https://discuss.elastic.co/t/can-logstash-pull-gzipped-files-from-s3/116412 "2018-01-24T23:42:16Z")
I have gzipped files on S3 I want to push into ES. there is an S3 input plugin: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html There is a gzip input plugin: https://www.elastic.co/guide/en/logs… --- ## [Logstash to elastic output SSL](https://discuss.elastic.co/t/logstash-to-elastic-output-ssl/116703)
**Author:** [@elasticus](https://discuss.elastic.co/u/elasticus)\ **Replies:** 10\ **Last updated:** [January 24, 2018, 10:26pm UTC](https://discuss.elastic.co/t/logstash-to-elastic-output-ssl/116703 "2018-01-24T22:26:57Z")
I have configured SSL for all ELASTIC cluster nodes Blowed if I can find any reliable information regarding how to subsequently configure the logstash output plugin once I restart my ES cluster with ssl enabled. I 'thi… --- ## [Issue with jdbc for mariadb](https://discuss.elastic.co/t/issue-with-jdbc-for-mariadb/116255)
**Author:** [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\ **Replies:** 6\ **Last updated:** [January 24, 2018, 8:00pm UTC](https://discuss.elastic.co/t/issue-with-jdbc-for-mariadb/116255 "2018-01-24T20:00:07Z")
Hi, I'm honestly trying to browse for similar issue related to this topic, but not able to find any related..so, please excuse if my search engine is wrong, but most likely i will need a help with troubleshooting, why m… --- ## [Logstash JMS input connecting and disconnecting on and on](https://discuss.elastic.co/t/logstash-jms-input-connecting-and-disconnecting-on-and-on/116900)
**Author:** [@roblopes](https://discuss.elastic.co/u/roblopes)\ **Replies:** 0\ **Last updated:** [January 24, 2018, 4:48pm UTC](https://discuss.elastic.co/t/logstash-jms-input-connecting-and-disconnecting-on-and-on/116900 "2018-01-24T16:48:29Z")
Hi, I am using Logstash v6 with the logstash-input-jms plugin, but I am using TIBCO EMS v7.0. I was doing a POC in a windows environment, and everything is works fine there, but I am now moving the elasticsearch stack … --- ## [Elasticsearch plugin settings in conf are overridden -- why?](https://discuss.elastic.co/t/elasticsearch-plugin-settings-in-conf-are-overridden-why/116705)
**Author:** [@cgorla](https://discuss.elastic.co/u/cgorla)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 3:41pm UTC](https://discuss.elastic.co/t/elasticsearch-plugin-settings-in-conf-are-overridden-why/116705 "2018-01-23T15:41:18Z")
tl;dr Basically, I've specified the elasticsearch plugin in Output, but the host setting is getting overriden -- anyone have ideas why? Here's my output: output { elasticsearch { document\_id =\> … --- ## [My certain fields are lost in logstash](https://discuss.elastic.co/t/my-certain-fields-are-lost-in-logstash/116890)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 0\ **Last updated:** [January 24, 2018, 4:06pm UTC](https://discuss.elastic.co/t/my-certain-fields-are-lost-in-logstash/116890 "2018-01-24T16:06:26Z")
Hi Guys, I have kind of honeypot created on elasticstack and ran for almost 12 hours which unfortunately filled up my hard disk space hence what I did is deleted certain indices using DELETE API from console. However th… --- ## [Is it me or logstash doesn't support special chars for monitoring password?](https://discuss.elastic.co/t/is-it-me-or-logstash-doesnt-support-special-chars-for-monitoring-password/114744)
**Author:** [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Replies:** 6\ **Last updated:** [January 24, 2018, 4:00pm UTC](https://discuss.elastic.co/t/is-it-me-or-logstash-doesnt-support-special-chars-for-monitoring-password/114744 "2018-01-24T16:00:26Z")
I have a password with ! and @ chars and for the life of me I cannot get it to connect to xpack for monitoring. I keep getting 401. I switched to a password that does not use ! and @ and it worked... --- ## [Multiline pattern for system logs](https://discuss.elastic.co/t/multiline-pattern-for-system-logs/116473)
**Author:** [@aditi.puri](https://discuss.elastic.co/u/aditi.puri)\ **Replies:** 2\ **Last updated:** [January 24, 2018, 2:44pm UTC](https://discuss.elastic.co/t/multiline-pattern-for-system-logs/116473 "2018-01-24T14:44:38Z")
i am trying to capture the full error description in my system log file.using the below pattern "^\<\>%{SPACE}\[%{MONTH} %{MONTHDAY}%{TIME}\]". But i am not bale to capture the error description. Please help in getting me t… --- ## [Error "Setting "xpack.management.enabled" hasn't been registered"](https://discuss.elastic.co/t/error-setting-xpack-management-enabled-hasnt-been-registered/116862)
**Author:** [@80c1141eb6d0c564dd90](https://discuss.elastic.co/u/80c1141eb6d0c564dd90)\ **Replies:** 0\ **Last updated:** [January 24, 2018, 12:16pm UTC](https://discuss.elastic.co/t/error-setting-xpack-management-enabled-hasnt-been-registered/116862 "2018-01-24T12:16:22Z")
Updated licence from trial from basic, installed x-pack in all of ELK stack. Than added xpack.management.enabled: true xpack.management.elasticsearch.url: "http://127.0.0.1:9200" and there is an error in Logstash log: … --- ## [Parsing xml multiline file](https://discuss.elastic.co/t/parsing-xml-multiline-file/115810)
**Author:** [@davidcsi](https://discuss.elastic.co/u/davidcsi)\ **Replies:** 3\ **Last updated:** [January 24, 2018, 2:21pm UTC](https://discuss.elastic.co/t/parsing-xml-multiline-file/115810 "2018-01-24T14:21:20Z")
Hello guys, I'm trying to parse xml files from S3. I get the file just fine, but i want the xml file to be processed completely like: \ \ **Author:** [@Ged](https://discuss.elastic.co/u/Ged)\ **Replies:** 9\ **Last updated:** [January 24, 2018, 2:16pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292 "2018-01-24T14:16:30Z") Hi, Elasticsearch 5.5.0 Logstash 5.5.0 logstash-codec-multiline 3.0.5 logstash-codec-json 3.0.3. Another issue when reading file (NFS) containing JSON lines using above. From time to time getting such an error: :e… --- ## [Filebeeat sending log, but logstash not running, connection refused!](https://discuss.elastic.co/t/filebeeat-sending-log-but-logstash-not-running-connection-refused/116665)
**Author:** [@omid](https://discuss.elastic.co/u/omid)\ **Replies:** 10\ **Last updated:** [January 24, 2018, 2:10pm UTC](https://discuss.elastic.co/t/filebeeat-sending-log-but-logstash-not-running-connection-refused/116665 "2018-01-24T14:10:20Z")
hi im used filebeat to many server, shipped nginx log to logstash in this time and months my elk server is very good worked but, my 1 line added grok pattern to syslog-filter.conf, and restart logstash ,,, my elk and… --- ## [Grok Output to comma separated](https://discuss.elastic.co/t/grok-output-to-comma-separated/116502)
**Author:** [@Vivek\_Raj](https://discuss.elastic.co/u/Vivek_Raj)\ **Replies:** 1\ **Last updated:** [January 24, 2018, 11:20am UTC](https://discuss.elastic.co/t/grok-output-to-comma-separated/116502 "2018-01-24T11:20:51Z")
Hi Everyone, How can i convert my Grok output to comma separated My Grok Pattern (Nginx access log) %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] \\ "(?:%{WORD:verb} %{NOTSPACE:request}(?:… --- ## [ES logger](https://discuss.elastic.co/t/es-logger/116848)
**Author:** [@hasti\_gh](https://discuss.elastic.co/u/hasti_gh)\ **Replies:** 0\ **Last updated:** [January 24, 2018, 10:43am UTC](https://discuss.elastic.co/t/es-logger/116848 "2018-01-24T10:43:34Z")
hi i have a problem with ES logger.i use slf4j logger in my unit Test with Junit and add dependency of jcl-over-slf4j and logback-classic in my project. but when i try to connect with my client (transport Client) in Es … --- ## [Count Event and delay from logstash to elasticsearch](https://discuss.elastic.co/t/count-event-and-delay-from-logstash-to-elasticsearch/116847)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 0\ **Last updated:** [January 24, 2018, 10:34am UTC](https://discuss.elastic.co/t/count-event-and-delay-from-logstash-to-elasticsearch/116847 "2018-01-24T10:34:37Z")
I have a large delay with events stored in elasticsearch and events received. I would like to tune logstash but before to start I have to know the difference between events received into a file and events loaded in elast… --- ## [Simple example wanted: read and output json files to elasticsearch](https://discuss.elastic.co/t/simple-example-wanted-read-and-output-json-files-to-elasticsearch/115410)
**Author:** [@jali](https://discuss.elastic.co/u/jali)\ **Replies:** 3\ **Last updated:** [January 24, 2018, 9:47am UTC](https://discuss.elastic.co/t/simple-example-wanted-read-and-output-json-files-to-elasticsearch/115410 "2018-01-24T09:47:10Z")
Hi! Basic stuff but cannot get this to work. I have a directory which contains bunch of json files (and new ones arriving every 10mins). The json files need to be read from dir and outputted to elastic. I want to read … --- ## [Conditionals not working on numeric fields](https://discuss.elastic.co/t/conditionals-not-working-on-numeric-fields/116426)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 2\ **Last updated:** [January 24, 2018, 9:04am UTC](https://discuss.elastic.co/t/conditionals-not-working-on-numeric-fields/116426 "2018-01-24T09:04:51Z")
Hi, I'm having an issue with conditionals and numeric fields. The config below only works if I first convert my fields to a string. I found a blog that mentions if (\[foo\]) { ... } should work for numeric fields but … --- ## [Sudo systemctl start doesn't start logstash](https://discuss.elastic.co/t/sudo-systemctl-start-doesnt-start-logstash/116832)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [January 24, 2018, 9:04am UTC](https://discuss.elastic.co/t/sudo-systemctl-start-doesnt-start-logstash/116832 "2018-01-24T09:04:25Z")
Hi, I've moved Logstash to a AWS EC2 instance but running sudo systemctl start logstash.service doesn't do anything. systemctl status shows the process as active but nothing is showing up in the logstash logs. Running … --- ## [Start with Logstash with Java](https://discuss.elastic.co/t/start-with-logstash-with-java/116829)
**Author:** [@SawyerFord](https://discuss.elastic.co/u/SawyerFord)\ **Replies:** 0\ **Last updated:** [January 24, 2018, 8:44am UTC](https://discuss.elastic.co/t/start-with-logstash-with-java/116829 "2018-01-24T08:44:40Z")
I am new to Logstash with Java, and we would like to use JSON logging which Logstash enables us. But I think Logstash stores all the logs in elastic search where presently I would like to test with a standalone applicati… --- ## [Kafka to elasticsearch not going data](https://discuss.elastic.co/t/kafka-to-elasticsearch-not-going-data/116824)
**Author:** [@Shekhar\_Pandey](https://discuss.elastic.co/u/Shekhar_Pandey)\ **Replies:** 0\ **Last updated:** [January 24, 2018, 8:26am UTC](https://discuss.elastic.co/t/kafka-to-elasticsearch-not-going-data/116824 "2018-01-24T08:26:15Z")
Hello.. I am transferring data from kafka to elastic through logstash...below is my conf file details input { kafka { bootstrap\_servers =\> "localhost:9092" topics =\> \["test"\] } } output { stdout { codec =\> json… --- ## [Filebeat connections stuck in CLOSE\_WAIT](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747)
**Author:** [@dwjvaughan](https://discuss.elastic.co/u/dwjvaughan)\ **Replies:** 7\ **Last updated:** [January 24, 2018, 8:14am UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747 "2018-01-24T08:14:35Z")
I have multiple servers running filebeat sending logs to two logstash servers (all in AWS), after a couple of days the logstash servers stop ingesting and report "Too many files open". An lsof shows thousands of connecti… --- ## [I want to add a field with filename](https://discuss.elastic.co/t/i-want-to-add-a-field-with-filename/114049)
**Author:** [@Asier\_Saluena](https://discuss.elastic.co/u/Asier_Saluena)\ **Replies:** 7\ **Last updated:** [January 24, 2018, 7:53am UTC](https://discuss.elastic.co/t/i-want-to-add-a-field-with-filename/114049 "2018-01-24T07:53:06Z")
Hi you all, i'm starting a project with elk stack and we have few domains working on the same machine, which sends his apache logs to elk. I want to extract the filename to filter by this field on kibana. I'm trying so… --- ## [Import csv file to elastic search](https://discuss.elastic.co/t/import-csv-file-to-elastic-search/116747)
**Author:** [@ankur\_singla](https://discuss.elastic.co/u/ankur_singla)\ **Replies:** 3\ **Last updated:** [January 24, 2018, 7:11am UTC](https://discuss.elastic.co/t/import-csv-file-to-elastic-search/116747 "2018-01-24T07:11:07Z")
I am trying to ulpoad csv file from logstash to my elastic cloud cluster using below config input { stdin { type =\> "stdin-type" } file { path =\> \["/pipelines/Data.csv"\] start\_position =\> "beginning" } } filter { csv {… --- ## [Rename field](https://discuss.elastic.co/t/rename-field/116799)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 2\ **Last updated:** [January 24, 2018, 6:46am UTC](https://discuss.elastic.co/t/rename-field/116799 "2018-01-24T06:46:05Z")
Winlogbeat fields name: "event\_data": { "ProcessName" "LogonGuid" "IpPort" "SubjectLogonId" "TargetLogonGuid" "SubjectUserName" "TargetInfo" "TargetServerName" "SubjectDomainName" "IpAddress" "TargetUserName" … --- ## [Kafka input stop running after some minutes](https://discuss.elastic.co/t/kafka-input-stop-running-after-some-minutes/116812)
**Author:** [@Lu\_Jacky](https://discuss.elastic.co/u/Lu_Jacky)\ **Replies:** 0\ **Last updated:** [January 24, 2018, 6:25am UTC](https://discuss.elastic.co/t/kafka-input-stop-running-after-some-minutes/116812 "2018-01-24T06:25:38Z")
i am using logstash-input-kafka 8.0.4,logstash 6.1.1 kafka server version 1.0 logstash config kafka { group\_id =\> "b2bconsumer-group" topics =\> "b2border" consumer\_threads =\> 1 # number (optional)， default: 1 dec… --- ## [Fatal Error starting Logstash 6.1.1](https://discuss.elastic.co/t/fatal-error-starting-logstash-6-1-1/114497)
**Author:** [@danielePerri](https://discuss.elastic.co/u/danielePerri)\ **Replies:** 7\ **Last updated:** [January 24, 2018, 5:51am UTC](https://discuss.elastic.co/t/fatal-error-starting-logstash-6-1-1/114497 "2018-01-24T05:51:07Z")
Hi all, I have installed Elasticsearch, Logstash and Kibana with X-packs, in version 6.1.1 During the start of Logstash, I receive this FATAL: \[FATAL\] 2018-01-08 10:49:01.377 \[main\] runner - An unexpected error occurr… --- ## [JMX input plugin usage](https://discuss.elastic.co/t/jmx-input-plugin-usage/116795)
**Author:** [@naveen\_K.N](https://discuss.elastic.co/u/naveen_K.N)\ **Replies:** 0\ **Last updated:** [January 24, 2018, 4:35am UTC](https://discuss.elastic.co/t/jmx-input-plugin-usage/116795 "2018-01-24T04:35:39Z")
Hi Everyone, im using Logstash's JMX Plugin to get JVM metrics of server, where i enabled JMX port and Remote connection access. i created JVM-Monoitoring.conf config file as follows: input { jmx{ path =\> "C:\\Users… --- ## [CSV Filter - Quote character parse failure](https://discuss.elastic.co/t/csv-filter-quote-character-parse-failure/116611)
**Author:** [@NerdSec](https://discuss.elastic.co/u/NerdSec)\ **Replies:** 1\ **Last updated:** [January 23, 2018, 11:25am UTC](https://discuss.elastic.co/t/csv-filter-quote-character-parse-failure/116611 "2018-01-23T11:25:05Z")
I have a CSV file and the format is something like this: "102","60","Open","I hope this works out for \\"random.guy@gmail.com\\"" When i parse this using the CSV filter i get the following error: \[2018-01-23T13:11:58,52… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=834) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=836)