# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=339 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 340 --- ## [Logstash error : Failed to publish events caused by: write tcp YY.YY.YY.YY:40912-\>XX.XX.XX.XX:5044: write: connection reset by peer](https://discuss.elastic.co/t/logstash-error-failed-to-publish-events-caused-by-write-tcp-yy-yy-yy-yy-40912-xx-xx-xx-xx-write-connection-reset-by-peer/237898)
**Author:** [@Aqel](https://discuss.elastic.co/u/Aqel)\ **Replies:** 0\ **Last updated:** [June 20, 2020, 7:08am UTC](https://discuss.elastic.co/t/logstash-error-failed-to-publish-events-caused-by-write-tcp-yy-yy-yy-yy-40912-xx-xx-xx-xx-write-connection-reset-by-peer/237898 "2020-06-20T07:08:45Z")
I am using filebeat to push my logs to elasticsearch using logstash and the set up was working fine for me before. I am getting Failed to publish events error now. filebeat | 2020-06-20T06:26:03.832969730Z 2020-06… --- ## [Filter Sonicwall Logstash](https://discuss.elastic.co/t/filter-sonicwall-logstash/238146)
**Author:** [@Danilo\_Visual](https://discuss.elastic.co/u/Danilo_Visual)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 7:41pm UTC](https://discuss.elastic.co/t/filter-sonicwall-logstash/238146 "2020-06-22T19:41:28Z")
I'm new to Logstash and I'm trying to filter my Sonicwall log My logstash.conf: input { syslog { type =\> Sonicwall port =\> 9991 } } filter { if \[type\] == "Sonicwall" { kv { exclude\_keys =\> \[ "c", "id", "m", "… --- ## [Ruby time.strftime logstash](https://discuss.elastic.co/t/ruby-time-strftime-logstash/238138)
**Author:** [@Rushabh\_Doshi](https://discuss.elastic.co/u/Rushabh_Doshi)\ **Replies:** 1\ **Last updated:** [June 22, 2020, 7:05pm UTC](https://discuss.elastic.co/t/ruby-time-strftime-logstash/238138 "2020-06-22T19:05:58Z")
Hi I want to convert hour to number , hour is extracted from timestamp as below in logstash: ruby { code =\> "event.set('\[hour\]',event.get('Activity-Time-New').time.strftime('%k') )" } In Kibana i want #hour ,a number … --- ## [Logstash grok filter combinedapachelog does not work](https://discuss.elastic.co/t/logstash-grok-filter-combinedapachelog-does-not-work/238133)
**Author:** [@ademxoy](https://discuss.elastic.co/u/ademxoy)\ **Replies:** 1\ **Last updated:** [June 22, 2020, 5:50pm UTC](https://discuss.elastic.co/t/logstash-grok-filter-combinedapachelog-does-not-work/238133 "2020-06-22T17:50:16Z")
Grok does not seem to create any outputs for COMBINEDAPACHELOG Here is my filter: filter { grok { match =\> { "messages" =\> "%{COMBINEDAPACHELOG}"} } mutate { convert =\> { "bytes" =\> "integer" } } date { mat… --- ## [Logstash input plugin for azure service bus topics](https://discuss.elastic.co/t/logstash-input-plugin-for-azure-service-bus-topics/238103)
**Author:** [@fim](https://discuss.elastic.co/u/fim)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 2:31pm UTC](https://discuss.elastic.co/t/logstash-input-plugin-for-azure-service-bus-topics/238103 "2020-06-22T14:31:26Z")
Hello, Does anybody know about the following logstash plugin? This plugin reads messages from Azure Service Bus Topics but we would like to monitor only meta-data like number of messages in queue. Is this plug-in … --- ## [How to parse dynamic array json through logstash?](https://discuss.elastic.co/t/how-to-parse-dynamic-array-json-through-logstash/238081)
**Author:** [@Rahul\_rr](https://discuss.elastic.co/u/Rahul_rr)\ **Replies:** 1\ **Last updated:** [June 22, 2020, 2:10pm UTC](https://discuss.elastic.co/t/how-to-parse-dynamic-array-json-through-logstash/238081 "2020-06-22T14:10:38Z")
Here is my array json. "service\_lines": \[ { "id": 584942, "is\_approved": null, "aar": { "category": { "id": 42, "name": "Tires" }, "system": { "id": 1, "name": "Tire" }, "subsystem": { "id": 1, "name": "T… --- ## [Logstash multiple jdbc input and clone](https://discuss.elastic.co/t/logstash-multiple-jdbc-input-and-clone/237739)
**Author:** [@ohoroyoi](https://discuss.elastic.co/u/ohoroyoi)\ **Replies:** 3\ **Last updated:** [June 22, 2020, 1:50pm UTC](https://discuss.elastic.co/t/logstash-multiple-jdbc-input-and-clone/237739 "2020-06-22T13:50:25Z")
I want to synchronize 2 postgres tables with 3 elasticsearch indexes. I'm going to split one of the tables and synchronize it with two indexes. so I'm cloning one of those tables. Divide the cloned things into 2 indexe… --- ## [Why I am getting this error when using JDBC input plugin of Logstash](https://discuss.elastic.co/t/why-i-am-getting-this-error-when-using-jdbc-input-plugin-of-logstash/237988)
**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Replies:** 7\ **Last updated:** [June 22, 2020, 1:42pm UTC](https://discuss.elastic.co/t/why-i-am-getting-this-error-when-using-jdbc-input-plugin-of-logstash/237988 "2020-06-22T13:42:01Z")
Here is the query I had put in the logstash conf file: select a.ID, a.TransactionID, b.Result from MyDB.Result a inner join MyDB.ResultData b on a.ID=b.ID where a.ID \< :sql\_last\_value and a.CreatedOn \> '2020-01-01' o… --- ## [Elasticsearch index not created after adding logstash (works fine with filebeat -\> es)](https://discuss.elastic.co/t/elasticsearch-index-not-created-after-adding-logstash-works-fine-with-filebeat-es/237374)
**Author:** [@code\_share](https://discuss.elastic.co/u/code_share)\ **Replies:** 13\ **Last updated:** [June 22, 2020, 12:45pm UTC](https://discuss.elastic.co/t/elasticsearch-index-not-created-after-adding-logstash-works-fine-with-filebeat-es/237374 "2020-06-22T12:45:11Z")
I spin up an ELK stack using docker-compose (see below). I have manged to successfully enable basic authentication (using xpack + certs + built-in passwords) for the ES cluster and connect filebeat to the the ES cluster … --- ## [Syslog to different indexes](https://discuss.elastic.co/t/syslog-to-different-indexes/238050)
**Author:** [@arun\_k](https://discuss.elastic.co/u/arun_k)\ **Replies:** 1\ **Last updated:** [June 22, 2020, 12:41pm UTC](https://discuss.elastic.co/t/syslog-to-different-indexes/238050 "2020-06-22T12:41:23Z")
Hi Team Am using filebeat to collect logs (mainly syslog) from multiple NW devices belongs to multiple tenants How can i configure logstash conf to index this to seperate indexes based on customers? Log source x,y(sys… --- ## [Logststash](https://discuss.elastic.co/t/logststash/238080)
**Author:** [@Eryadav](https://discuss.elastic.co/u/Eryadav)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 11:55am UTC](https://discuss.elastic.co/t/logststash/238080 "2020-06-22T11:55:35Z")
i am using the docker compose to configure the logstash .but while i trying to mount the sincedb path of logstash in host container its not working if anyone can help me with this. \< - /opt/sanjay/logs/aartech:/usr/sha… --- ## [How to add the count of 2 values in logstash](https://discuss.elastic.co/t/how-to-add-the-count-of-2-values-in-logstash/238077)
**Author:** [@Matish\_Bhuyan](https://discuss.elastic.co/u/Matish_Bhuyan)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 11:43am UTC](https://discuss.elastic.co/t/how-to-add-the-count-of-2-values-in-logstash/238077 "2020-06-22T11:43:56Z")
Hi All, I have a requirement in logstash. For an example in fruits keyword i have 3 values 1- Mangoes ---- 5 2- Oranges -----6 3- Brinjal -----7 But in Kibana i need to visualize the sum of only 2 objects as… --- ## [How to parse this xml data each rate plan id data should go to new row or new column dynamically by using xml filter xpath option or any other solution let me know please (edited)](https://discuss.elastic.co/t/how-to-parse-this-xml-data-each-rate-plan-id-data-should-go-to-new-row-or-new-column-dynamically-by-using-xml-filter-xpath-option-or-any-other-solution-let-me-know-please-edited/238073)
**Author:** [@Puneeth\_S\_B\_Gowda1](https://discuss.elastic.co/u/Puneeth_S_B_Gowda1)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 11:30am UTC](https://discuss.elastic.co/t/how-to-parse-this-xml-data-each-rate-plan-id-data-should-go-to-new-row-or-new-column-dynamically-by-using-xml-filter-xpath-option-or-any-other-solution-let-me-know-please-edited/238073 "2020-06-22T11:30:04Z")
my logstash input { beats { port =\> 5044 } } filter { mutate { split =\> { "message" =\> "@$@" }} mutate { add\_field =\> { "response" =\> "%{\[message\]\[1\]}" "request" =\> "%{\[message\]\[0\]}" } } xml { store\_xml =\> "fa… --- ## [Elasticsearch input missing logs](https://discuss.elastic.co/t/elasticsearch-input-missing-logs/238071)
**Author:** [@magnalite](https://discuss.elastic.co/u/magnalite)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 11:06am UTC](https://discuss.elastic.co/t/elasticsearch-input-missing-logs/238071 "2020-06-22T11:06:17Z")
I'm using the Elasticsearch input plugin to reindex through a Logstash pipeline however it appears as though it is mysteriously skipping ~20% of logs! I do not terminate the process myself and wait for it to come to comp… --- ## [After upgrading logstash to 6.6, I get failed authentication due to: SSL handshake failed](https://discuss.elastic.co/t/after-upgrading-logstash-to-6-6-i-get-failed-authentication-due-to-ssl-handshake-failed/238052)
**Author:** [@Zalman\_Goldstein1](https://discuss.elastic.co/u/Zalman_Goldstein1)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 9:27am UTC](https://discuss.elastic.co/t/after-upgrading-logstash-to-6-6-i-get-failed-authentication-due-to-ssl-handshake-failed/238052 "2020-06-22T09:27:58Z")
we have kafka\_2.12-2.3.0, and use logstash. after upgrading to logstash version 6.6, and plugin 7.3.2 logstash-output-kafka, we got an error of The setting ssl in plugin kafka is obsolete and is no longer available. … --- ## [Filter ruby exception occurred with no message](https://discuss.elastic.co/t/filter-ruby-exception-occurred-with-no-message/238051)
**Author:** [@sam\_rui](https://discuss.elastic.co/u/sam_rui)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 9:24am UTC](https://discuss.elastic.co/t/filter-ruby-exception-occurred-with-no-message/238051 "2020-06-22T09:24:35Z")
I encountered same issue with https://discuss.elastic.co/t/ruby-exception-with-no-message/198887. following is logstash log info. following is configuration info. When I get output, the "index\_postfix" can not b… --- ## [SAP .AUD log parsin](https://discuss.elastic.co/t/sap-aud-log-parsin/238041)
**Author:** [@yasin\_mohammed](https://discuss.elastic.co/u/yasin_mohammed)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 8:36am UTC](https://discuss.elastic.co/t/sap-aud-log-parsin/238041 "2020-06-22T08:36:49Z")
Hi... I am trying to parse a .AUD log file of SAP through logstash. The log is in binary. Any clues on what is the encoding to use for SAP .AUD files to read them in logstash. --- ## [Problem beetween elastic and logstash: csv files are saved twice](https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655)
**Author:** [@IneedHelp](https://discuss.elastic.co/u/IneedHelp)\ **Replies:** 5\ **Last updated:** [June 22, 2020, 8:27am UTC](https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655 "2020-06-22T08:27:13Z")
Hello ! I use ELK to parse log file and csv file but yesterday a little problem appeared. All the data in my csv file were saved twice so a 1000 lines document is now a 2000 lines document. In fact you just have to divi… --- ## [Netflow codec. Wrong data](https://discuss.elastic.co/t/netflow-codec-wrong-data/238029)
**Author:** [@swb](https://discuss.elastic.co/u/swb)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 7:52am UTC](https://discuss.elastic.co/t/netflow-codec-wrong-data/238029 "2020-06-22T07:52:21Z")
Hello friends, We want to replace our old system for viewing Netflow (Nfsen) information with ELK. For this, we used the netflow module for Logstash. But we see fewer packets and bytes than it actually is. For example: … --- ## [Logstash output to redis](https://discuss.elastic.co/t/logstash-output-to-redis/237892)
**Author:** [@kase](https://discuss.elastic.co/u/kase)\ **Replies:** 2\ **Last updated:** [June 22, 2020, 8:03am UTC](https://discuss.elastic.co/t/logstash-output-to-redis/237892 "2020-06-22T08:03:09Z")
Hi there, my logstash will get lots of data from other's API, after filter these data, logstash will output to redis. I want to ask some questions: redis is a key-value database, how to let logstash know which is key… --- ## [Winlogbeat index pattern not showing up on kibana](https://discuss.elastic.co/t/winlogbeat-index-pattern-not-showing-up-on-kibana/237608)
**Author:** [@norup](https://discuss.elastic.co/u/norup)\ **Replies:** 1\ **Last updated:** [June 22, 2020, 7:51am UTC](https://discuss.elastic.co/t/winlogbeat-index-pattern-not-showing-up-on-kibana/237608 "2020-06-22T07:51:26Z")
I can see the index has been made here, but it looks like theres 0 documents in it. But for some reason its not showing up here I have run the command .\\winlogbeat.exe setup --index-management -E output.logstash.en… --- ## [Logstash doesn't start due to Charset issue](https://discuss.elastic.co/t/logstash-doesnt-start-due-to-charset-issue/237291)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 5\ **Last updated:** [June 22, 2020, 6:15am UTC](https://discuss.elastic.co/t/logstash-doesnt-start-due-to-charset-issue/237291 "2020-06-22T06:15:00Z")
hi all I installed logstash v. 6.6 on windows. my config file is as following: input { beats { port =\> 5044 } } output { elasticsearch { hosts =\> \["localhost:9200"\] index =\> "my\_log" … --- ## [Logstash multiple ruby filter](https://discuss.elastic.co/t/logstash-multiple-ruby-filter/237758)
**Author:** [@ohoroyoi](https://discuss.elastic.co/u/ohoroyoi)\ **Replies:** 4\ **Last updated:** [June 22, 2020, 5:15am UTC](https://discuss.elastic.co/t/logstash-multiple-ruby-filter/237758 "2020-06-22T05:15:17Z")
I'm bringing table A and B from postgres. I want to apply the ruby filter in different forms. How should I set it up? input { jdbc { tags =\> table A } jdbc { tags =\> table B } } filter { ruby { code=\>… --- ## [Logstash Netflow Codec support for Multiple Definition files](https://discuss.elastic.co/t/logstash-netflow-codec-support-for-multiple-definition-files/237998)
**Author:** [@bniranjanbhat](https://discuss.elastic.co/u/bniranjanbhat)\ **Replies:** 0\ **Last updated:** [June 22, 2020, 4:41am UTC](https://discuss.elastic.co/t/logstash-netflow-codec-support-for-multiple-definition-files/237998 "2020-06-22T04:41:09Z")
Hi All, I am trying to get the netflow codec work with multiple definition files. I understand that "netflow\_definitions" configuration option is a valid file system path. However, when I try to configure this path as… --- ## [Logs from Firewall CheckPoint](https://discuss.elastic.co/t/logs-from-firewall-checkpoint/237835)
**Author:** [@julianksanchez](https://discuss.elastic.co/u/julianksanchez)\ **Replies:** 8\ **Last updated:** [June 21, 2020, 3:37pm UTC](https://discuss.elastic.co/t/logs-from-firewall-checkpoint/237835 "2020-06-21T15:37:23Z")
Hello, I am new in Elastic, and I want to send the logs of our firewalls CheckPoint to logstash. I configurate the log exporter in CheckPoint, however we dont know how received and configurate this information in logsta… --- ## [Error during reindexing via Elasticsearch input plugin](https://discuss.elastic.co/t/error-during-reindexing-via-elasticsearch-input-plugin/237940)
**Author:** [@magnalite](https://discuss.elastic.co/u/magnalite)\ **Replies:** 0\ **Last updated:** [June 20, 2020, 10:02pm UTC](https://discuss.elastic.co/t/error-during-reindexing-via-elasticsearch-input-plugin/237940 "2020-06-20T22:02:41Z")
I'm currently using the Elasticsearch input plugin to reindex an index via a Logstash (7.8.0) pipeline however I keep getting a random error which will stop the Logstash instance (and then get restarted, starting from th… --- ## [Agent less deployment](https://discuss.elastic.co/t/agent-less-deployment/237589)
**Author:** [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Replies:** 2\ **Last updated:** [June 21, 2020, 2:27pm UTC](https://discuss.elastic.co/t/agent-less-deployment/237589 "2020-06-21T14:27:21Z")
Hello, I need to deploy agent less deployment version of ELK stack on a basic license. Basically we want to use IT as a log collector and availability monitoring for Windows/Centos/RedHat servers as well as gather info… --- ## [Count of Log messages on logstash](https://discuss.elastic.co/t/count-of-log-messages-on-logstash/237827)
**Author:** [@Dublin\_2020](https://discuss.elastic.co/u/Dublin_2020)\ **Replies:** 3\ **Last updated:** [June 21, 2020, 2:20pm UTC](https://discuss.elastic.co/t/count-of-log-messages-on-logstash/237827 "2020-06-21T14:20:41Z")
Hi Guys i am new to elastic search , need an insight on how can we get the count of Log messages received in Logstash from rsyslog server. Any idea , this is to check the differnce in logs forwarded from rsyslog and r… --- ## [Import CSV data to Elastic App Search "block in start\_input" Error](https://discuss.elastic.co/t/import-csv-data-to-elastic-app-search-block-in-start-input-error/237937)
**Author:** [@th3Hopper](https://discuss.elastic.co/u/th3Hopper)\ **Replies:** 4\ **Last updated:** [June 20, 2020, 8:40pm UTC](https://discuss.elastic.co/t/import-csv-data-to-elastic-app-search-block-in-start-input-error/237937 "2020-06-20T20:40:42Z")
I'm trying to import CSV data to my Elastic App search but it throwing an error. Error Message:- \[2020-06-21T01:40:17,552\]\[INFO \]\[filewatch.observingtail \]\[main\]\[07a10415268d8655d99cdad82a61cbca125e21687e65bc0296c88f9… --- ## [Logstash aggregate/mutate](https://discuss.elastic.co/t/logstash-aggregate-mutate/237876)
**Author:** [@pixelinc](https://discuss.elastic.co/u/pixelinc)\ **Replies:** 4\ **Last updated:** [June 20, 2020, 3:51pm UTC](https://discuss.elastic.co/t/logstash-aggregate-mutate/237876 "2020-06-20T15:51:31Z")
Hey all, I'm attempting to combine take 2 logs and add a field from a previous log to the other log. I've read around about aggregate and mutate, and all that but I'm not quite understanding how to do this. I have 2 lo… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=338) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=340)