# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=748 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 749 --- ## [Logstash 6.2.4 with x-pack tries to connect to ES on localhost when it's not configured to and ignores credentials for the real cluster](https://discuss.elastic.co/t/logstash-6-2-4-with-x-pack-tries-to-connect-to-es-on-localhost-when-its-not-configured-to-and-ignores-credentials-for-the-real-cluster/132850)
**Author:** [@Emily1](https://discuss.elastic.co/u/Emily1)\ **Replies:** 2\ **Last updated:** [June 1, 2018, 10:04pm UTC](https://discuss.elastic.co/t/logstash-6-2-4-with-x-pack-tries-to-connect-to-es-on-localhost-when-its-not-configured-to-and-ignores-credentials-for-the-real-cluster/132850 "2018-06-01T22:04:51Z")
Hi, I have been running an Elastic stack for over a year and was recently trying out X-Pack. When I added X-Pack, Logstash started trying to connect to ES on localhost even though my logstash instance is not configured… --- ## [Issues with multiple pipeline](https://discuss.elastic.co/t/issues-with-multiple-pipeline/134263)
**Author:** [@Saketh\_Chandra\_Kolis](https://discuss.elastic.co/u/Saketh_Chandra_Kolis)\ **Replies:** 1\ **Last updated:** [June 1, 2018, 6:06pm UTC](https://discuss.elastic.co/t/issues-with-multiple-pipeline/134263 "2018-06-01T18:06:02Z")
this is error i keep getting: PS C:\\logstash-5.4.1\> bin\\logstash --path.settings="C:\\logstash-5.4.1\\config\\logstash.yml" WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logsta… --- ## [Need help in creating grok patterns for haproxy logs](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180)
**Author:** [@nareshreddyp](https://discuss.elastic.co/u/nareshreddyp)\ **Replies:** 7\ **Last updated:** [June 1, 2018, 4:08pm UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180 "2018-06-01T16:08:40Z")
Hello, I am new to logstash and trying to filter the haproxy logs. Example of my haproxy logs. \<134\>0 2018-05-29T03:45:29+02:00 localhost epic-webservicefrontend 18645 - \[type=haproxy\] \[18645\] \[1527558316.602\] 3/3/1/… --- ## [How to prevent empty lines](https://discuss.elastic.co/t/how-to-prevent-empty-lines/133490)
**Author:** [@npontes](https://discuss.elastic.co/u/npontes)\ **Replies:** 25\ **Last updated:** [June 1, 2018, 9:05am UTC](https://discuss.elastic.co/t/how-to-prevent-empty-lines/133490 "2018-06-01T09:05:28Z")
Hello, I'm currently using Logstash 2.1.1 to retrieve data from Elasticsearch 1.4.4 and put it on a CSV to be imported to a database. My problem is that there are lines where I only have commas (Like this: ",,,,,,,,,,")… --- ## [Log4net UDP alternative](https://discuss.elastic.co/t/log4net-udp-alternative/133804)
**Author:** [@cowensel](https://discuss.elastic.co/u/cowensel)\ **Replies:** 3\ **Last updated:** [June 1, 2018, 1:36pm UTC](https://discuss.elastic.co/t/log4net-udp-alternative/133804 "2018-06-01T13:36:57Z")
We currently use log4net that uses UDP to parse data over the network to logstash where we have recently found that it is silently dropping messages as the UDP appender cannot keep up. Can anyone advise if there are any … --- ## [NoMethodError: undefined method \`\<' for nil:NilClass\>](https://discuss.elastic.co/t/nomethoderror-undefined-method-for-nil-nilclass/133869)
**Author:** [@Akash\_Kumar\_Dutta](https://discuss.elastic.co/u/Akash_Kumar_Dutta)\ **Replies:** 1\ **Last updated:** [June 1, 2018, 1:28pm UTC](https://discuss.elastic.co/t/nomethoderror-undefined-method-for-nil-nilclass/133869 "2018-06-01T13:28:59Z")
The config for my logstash looks like: input { kafka { bootstrap\_servers =\> "kafka-0.kafka-svc.ns-adutta.svc.cluster.local:9093,kafka-1.kafka-svc.ns-adutta.svc.cluster.local:9093,kafka-2.kafka-svc.ns-adutt… --- ## [Event.remove not working](https://discuss.elastic.co/t/event-remove-not-working/134147)
**Author:** [@Akash\_Kumar\_Dutta](https://discuss.elastic.co/u/Akash_Kumar_Dutta)\ **Replies:** 4\ **Last updated:** [June 1, 2018, 12:57pm UTC](https://discuss.elastic.co/t/event-remove-not-working/134147 "2018-06-01T12:57:37Z")
I am using the following ruby filter to update the fields. def filter(event) keys = event.to\_hash.keys keys.each{|key| if ( key.start\_with? '\_' and key != "\_@timestamp") value = event.get(key) newke… --- ## [Fingerprint plugin: make base64encode url friendly](https://discuss.elastic.co/t/fingerprint-plugin-make-base64encode-url-friendly/134215)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 0\ **Last updated:** [June 1, 2018, 11:57am UTC](https://discuss.elastic.co/t/fingerprint-plugin-make-base64encode-url-friendly/134215 "2018-06-01T11:57:26Z")
I am using the fingerprint plugin to create custom \_id fields that should be base64 encoded: fingerprint { key =\> "somekey" base64encode =\> true method =\> "SHA512" source =\> \[ "username" \] target =\>… --- ## [Remove duplicates in logstash logs](https://discuss.elastic.co/t/remove-duplicates-in-logstash-logs/134057)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 3\ **Last updated:** [June 1, 2018, 7:56am UTC](https://discuss.elastic.co/t/remove-duplicates-in-logstash-logs/134057 "2018-06-01T07:56:03Z")
Hello, I use filebeat to send logs to logstash and then I use Logstash to parse them. For exemple this line : 31/5/2018 01:06:24.073 (TACHE) 30/5/2018/D T UE\_TECXXX\_J\_TN\_SSH002\_DELCLONE\_LPSACAS1(14033)/UE\_TECXXX\_X\_LL\_OP… --- ## [Get specific fields from csv file](https://discuss.elastic.co/t/get-specific-fields-from-csv-file/134200)
**Author:** [@kingprakatheesh](https://discuss.elastic.co/u/kingprakatheesh)\ **Replies:** 2\ **Last updated:** [June 1, 2018, 12:06pm UTC](https://discuss.elastic.co/t/get-specific-fields-from-csv-file/134200 "2018-06-01T12:06:56Z")
i am haing a csv file having 100 fields. i want to parse only 5 fields so i used Ruby filter and framed my conf. Upon executing getting {} in all lines instead of fields Conf file input{ file { path =\> path\_to\_file … --- ## [Should Kafka Input handle Exceptions during poll()?](https://discuss.elastic.co/t/should-kafka-input-handle-exceptions-during-poll/134210)
**Author:** [@JohanRask](https://discuss.elastic.co/u/JohanRask)\ **Replies:** 0\ **Last updated:** [June 1, 2018, 11:42am UTC](https://discuss.elastic.co/t/should-kafka-input-handle-exceptions-during-poll/134210 "2018-06-01T11:42:32Z")
Hi, Kafka clients producer and consumer does good job in making sure that any errors are not propagated up to logstash. However, some errors can slip through and especially during a cluster restart. Our problem is that… --- ## [The best logstash-to-logstash protocol](https://discuss.elastic.co/t/the-best-logstash-to-logstash-protocol/134184)
**Author:** [@thepax](https://discuss.elastic.co/u/thepax)\ **Replies:** 1\ **Last updated:** [June 1, 2018, 11:17am UTC](https://discuss.elastic.co/t/the-best-logstash-to-logstash-protocol/134184 "2018-06-01T11:17:22Z")
Hello, I've got several regional logstash'es (beats, syslog, snmptrap, netflow inputs, persistent queue) which transfer data to a central logstash for processing. What is the best protocol for transferring structured d… --- ## [Use API field as a variable in logstash input](https://discuss.elastic.co/t/use-api-field-as-a-variable-in-logstash-input/133689)
**Author:** [@AurelienG](https://discuss.elastic.co/u/AurelienG)\ **Replies:** 5\ **Last updated:** [June 1, 2018, 9:10am UTC](https://discuss.elastic.co/t/use-api-field-as-a-variable-in-logstash-input/133689 "2018-06-01T09:10:14Z")
Hello, What I am trying to do seems quite similar to what @dorj1234 was asking help for a few months ago in this topic but the answer provided is not enough for me so I hope you won't mind me opening a similar topic. W… --- ## [Sqlite](https://discuss.elastic.co/t/sqlite/134182)
**Author:** [@ToMas-hack](https://discuss.elastic.co/u/ToMas-hack)\ **Replies:** 0\ **Last updated:** [June 1, 2018, 9:07am UTC](https://discuss.elastic.co/t/sqlite/134182 "2018-06-01T09:07:46Z")
Good morning. Help to "read" the data of a sqlite? input { sqlite { path =\> "/media/elcaos/Backup/ComplementosUbuntu/Backup/Cloud/Cloud.sqlite" type =\> cloud } } output { # Send the information to elasticsearch … --- ## [Logstash parse only JSON messages](https://discuss.elastic.co/t/logstash-parse-only-json-messages/134103)
**Author:** [@ArunANayagam](https://discuss.elastic.co/u/ArunANayagam)\ **Replies:** 2\ **Last updated:** [June 1, 2018, 9:04am UTC](https://discuss.elastic.co/t/logstash-parse-only-json-messages/134103 "2018-06-01T09:04:46Z")
Hi, I want to parse various different messages being logged into cloudwatch. And I am planning to use logstash-input-cloudwatch-logs plugin. Now there are a lot of non-json messages written into cloudwatch, I want to i… --- ## [How to filter my logs](https://discuss.elastic.co/t/how-to-filter-my-logs/133292)
**Author:** [@kimfut](https://discuss.elastic.co/u/kimfut)\ **Replies:** 8\ **Last updated:** [June 1, 2018, 8:53am UTC](https://discuss.elastic.co/t/how-to-filter-my-logs/133292 "2018-06-01T08:53:13Z")
hello everyone , i am trying to filter my logs to something more structured , mostly the message with url part , i tried a lot of grok filter that i found online but everytime i got " \_grokparsefailure" in my logs and no… --- ## [Logstash has fetched the data but not shipped into Elasticsearch](https://discuss.elastic.co/t/logstash-has-fetched-the-data-but-not-shipped-into-elasticsearch/134154)
**Author:** [@Lee\_Jack](https://discuss.elastic.co/u/Lee_Jack)\ **Replies:** 5\ **Last updated:** [June 1, 2018, 8:45am UTC](https://discuss.elastic.co/t/logstash-has-fetched-the-data-but-not-shipped-into-elasticsearch/134154 "2018-06-01T08:45:31Z")
I am using Logstash to fetch data from another host's mysql database and want to ship the data into Elastic database.After starting the Logstash ,I can observe the Logstash has already fetch the new data which I insert … --- ## [Remove\_field with regex as argument](https://discuss.elastic.co/t/remove-field-with-regex-as-argument/134001)
**Author:** [@dr01](https://discuss.elastic.co/u/dr01)\ **Replies:** 3\ **Last updated:** [May 31, 2018, 2:58pm UTC](https://discuss.elastic.co/t/remove-field-with-regex-as-argument/134001 "2018-05-31T14:58:33Z")
I'd like to remove all fields that begin by "foobar". Is there any way to pass a regex to the remove\_field command, or is there any remove command that accepts regexs? --- ## [Grok pattern construction](https://discuss.elastic.co/t/grok-pattern-construction/134142)
**Author:** [@hitman](https://discuss.elastic.co/u/hitman)\ **Replies:** 5\ **Last updated:** [June 1, 2018, 6:25am UTC](https://discuss.elastic.co/t/grok-pattern-construction/134142 "2018-06-01T06:25:45Z")
hi guys.....i m new to logsatsh and i am unable to understand this grok filter.i m providing my logs here...can anyone please provide the grok pattern and explain how? 2018-05-25 10:53:21.7529 TRACE Parameter Name : sch… --- ## [While restarting logstash service , Getting error](https://discuss.elastic.co/t/while-restarting-logstash-service-getting-error/134153)
**Author:** [@Suresh\_Pal](https://discuss.elastic.co/u/Suresh_Pal)\ **Replies:** 2\ **Last updated:** [June 1, 2018, 7:30am UTC](https://discuss.elastic.co/t/while-restarting-logstash-service-getting-error/134153 "2018-06-01T07:30:14Z")
While starting my logstash service, i'm getting below error please help. /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22\] agent - Failed to execute action {:action=\>LogStash::PipelineA… --- ## [Filebeat output not reaching logstash](https://discuss.elastic.co/t/filebeat-output-not-reaching-logstash/133971)
**Author:** [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)\ **Replies:** 7\ **Last updated:** [June 1, 2018, 6:36am UTC](https://discuss.elastic.co/t/filebeat-output-not-reaching-logstash/133971 "2018-06-01T06:36:36Z")
I have logstash, es and kibana inside docker-compose. They are communicating to each other well. But filebeat which is installed outside is responsible for fetching data from a file and sending to logstash. Filebeat is h… --- ## [Couchbase - Kafka - Logstash output invalid characters](https://discuss.elastic.co/t/couchbase-kafka-logstash-output-invalid-characters/134111)
**Author:** [@jscubida](https://discuss.elastic.co/u/jscubida)\ **Replies:** 0\ **Last updated:** [May 31, 2018, 9:48pm UTC](https://discuss.elastic.co/t/couchbase-kafka-logstash-output-invalid-characters/134111 "2018-05-31T21:48:51Z")
Hi. I currently have the following setup: Confluent Open Source 4.1.0 (which includes Kafka) Couchbase 4.5.1-2844 Community Edition (build-2844) Logstash 6.2.4 Data flows from Couchbase -\> Kafka -\> Logstash. According… --- ## [How can i use aggregation in logstash-elasticsearch filter plugin](https://discuss.elastic.co/t/how-can-i-use-aggregation-in-logstash-elasticsearch-filter-plugin/133805)
**Author:** [@yaser](https://discuss.elastic.co/u/yaser)\ **Replies:** 4\ **Last updated:** [May 31, 2018, 9:36pm UTC](https://discuss.elastic.co/t/how-can-i-use-aggregation-in-logstash-elasticsearch-filter-plugin/133805 "2018-05-31T21:36:10Z")
if\[type\] == "mytype" { elasticsearch { hosts =\> \["192.168.1.1:9200"\] index =\> "my\_index" query\_template =\> "/logstash/my\_query.json" … --- ## [Logstash not receiving data from beats](https://discuss.elastic.co/t/logstash-not-receiving-data-from-beats/133169)
**Author:** [@rohitg](https://discuss.elastic.co/u/rohitg)\ **Replies:** 6\ **Last updated:** [May 31, 2018, 7:54pm UTC](https://discuss.elastic.co/t/logstash-not-receiving-data-from-beats/133169 "2018-05-31T19:54:21Z")
I have setup one elastic cluster (Elasticsearch + Logstash + Kibana) on windows server. And installed beats(filebeat+winlogbeat) on 3 windows servers. I am able to view all logs from 2 servers, but not from third one. … --- ## [Grok different types of log in one file](https://discuss.elastic.co/t/grok-different-types-of-log-in-one-file/134015)
**Author:** [@kimfut](https://discuss.elastic.co/u/kimfut)\ **Replies:** 2\ **Last updated:** [May 31, 2018, 5:30pm UTC](https://discuss.elastic.co/t/grok-different-types-of-log-in-one-file/134015 "2018-05-31T17:30:06Z")
hello every one , i have a file which contain 3 types of log : 1 ) X.X.X.X - - \[04/May/2018:16:24:21 +0200\] "GET /WebServiceRequestProxyCenter/api/request-url?country=fr&url=https%3A%2F%2Fwww.google.fr%2Fsearch%3Fq%3D… --- ## [Simple Logstash Date Filter](https://discuss.elastic.co/t/simple-logstash-date-filter/134084)
**Author:** [@AloysiusParedes](https://discuss.elastic.co/u/AloysiusParedes)\ **Replies:** 4\ **Last updated:** [May 31, 2018, 5:19pm UTC](https://discuss.elastic.co/t/simple-logstash-date-filter/134084 "2018-05-31T17:19:19Z")
I am parsing JSON, and one of my fields are: "myDateTime":"20180510T042139.680 GMT" Here is my Logstash Config: input{ file{ type =\> "test" codec =\> json{ } path =\> "/home/my-dev/Documen… --- ## [Trying to have dissect clean up after itself](https://discuss.elastic.co/t/trying-to-have-dissect-clean-up-after-itself/134087)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 2\ **Last updated:** [May 31, 2018, 5:18pm UTC](https://discuss.elastic.co/t/trying-to-have-dissect-clean-up-after-itself/134087 "2018-05-31T17:18:31Z")
I am using dissect to parse a field it has just parsed out of a message. input { generator { message =\> 'aaa bbb/ccc ddd' count =\> 1 } } output { stdout { codec =\> rubydebug } } filter { dissect { mapping =\> … --- ## [Parse snort los](https://discuss.elastic.co/t/parse-snort-los/133834)
**Author:** [@Hanen](https://discuss.elastic.co/u/Hanen)\ **Replies:** 1\ **Last updated:** [May 31, 2018, 3:47pm UTC](https://discuss.elastic.co/t/parse-snort-los/133834 "2018-05-31T15:47:10Z")
Hello Please How can I parse snort logs with grok pattern? 05/30-09:08:58.481608 \[\] \[1:10000001:1\] ICMP test detected \[\] \[Classification: Generic ICMP event\] \[Priority: 3\] {ICMP} @IP -\> @IP Many thanks --- ## [JDBC Input Plugin - JOIN Query issue](https://discuss.elastic.co/t/jdbc-input-plugin-join-query-issue/134063)
**Author:** [@ysumit](https://discuss.elastic.co/u/ysumit)\ **Replies:** 4\ **Last updated:** [May 31, 2018, 2:59pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-join-query-issue/134063 "2018-05-31T14:59:38Z")
Hi Team I am unable to receive any input when I am using an INNER JOIN query in jdbc input plugin of Logstash. Need immediate assistance. Sumit --- ## [Monitoring logstash with port 9600](https://discuss.elastic.co/t/monitoring-logstash-with-port-9600/133807)
**Author:** [@zeenon](https://discuss.elastic.co/u/zeenon)\ **Replies:** 4\ **Last updated:** [May 31, 2018, 1:58pm UTC](https://discuss.elastic.co/t/monitoring-logstash-with-port-9600/133807 "2018-05-31T13:58:56Z")
Hi, I am running logstash 5.6.4 and I have set http.port: 9600-9700 in my logstash.yml configuration file. On the logstash server, all is fine $ telnet 127.0.0.1 9600 Trying 127.0.0.1... Connected to 127.0.0.1. … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=747) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=749)