# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=757 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 758 --- ## [Unbale to decode Message using protobuf logstahs plusgin which is encoded using python or ruby?](https://discuss.elastic.co/t/unbale-to-decode-message-using-protobuf-logstahs-plusgin-which-is-encoded-using-python-or-ruby/132414)
**Author:** [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\ **Replies:** 0\ **Last updated:** [May 18, 2018, 4:28am UTC](https://discuss.elastic.co/t/unbale-to-decode-message-using-protobuf-logstahs-plusgin-which-is-encoded-using-python-or-ruby/132414 "2018-05-18T04:28:24Z")
Here is message.proto syntax = "proto3"; message SearchRequest { string query = 1; int32 page\_number = 2; int32 result\_per\_page = 3; } compiled using sudo protoc --python\_out=. message.proto which will gives mes… --- ## [Matching templates with indices](https://discuss.elastic.co/t/matching-templates-with-indices/132460)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 2\ **Last updated:** [May 18, 2018, 11:46am UTC](https://discuss.elastic.co/t/matching-templates-with-indices/132460 "2018-05-18T11:46:36Z")
Can we use wildcards when we want a specific template to be applied in several indices? e.g. is the following applicable? this is from a template file, say my\_template\_file.json { "order": 1, "index\_patterns": "\*m… --- ## [Multiple Outputs](https://discuss.elastic.co/t/multiple-outputs/132471)
**Author:** [@amageek](https://discuss.elastic.co/u/amageek)\ **Replies:** 0\ **Last updated:** [May 18, 2018, 11:00am UTC](https://discuss.elastic.co/t/multiple-outputs/132471 "2018-05-18T11:00:17Z")
Hi, Quick question for you..... We have setup a test environment, which is taking messages from RabbitMQ as in input and sending to both Elastic and Mongo DB. The items going to Elastic will be stipped down, but we'll… --- ## [Couldn't decode protobuf: #\](https://discuss.elastic.co/t/couldnt-decode-protobuf-runtimeerror-protocol-message-tag-had-invalid-wire-type/132439)
**Author:** [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\ **Replies:** 1\ **Last updated:** [May 18, 2018, 8:53am UTC](https://discuss.elastic.co/t/couldnt-decode-protobuf-runtimeerror-protocol-message-tag-had-invalid-wire-type/132439 "2018-05-18T08:53:20Z")
I am not getting why this is happening with my config --- ## [Pepeline.conf](https://discuss.elastic.co/t/pepeline-conf/132426)
**Author:** [@wuwenjie\_cn](https://discuss.elastic.co/u/wuwenjie_cn)\ **Replies:** 1\ **Last updated:** [May 18, 2018, 7:06am UTC](https://discuss.elastic.co/t/pepeline-conf/132426 "2018-05-18T07:06:38Z")
The following is the contents of my Logstash pipeline.conf configuration file： output { elasticsearch { hosts =\> \[ "xxx.xxx.xxx.xxx:9200" \] index =\> "%{\[@metahostname\]}" # How to write dyna… --- ## [Add hits.total to field in Elasticsearch filter](https://discuss.elastic.co/t/add-hits-total-to-field-in-elasticsearch-filter/132413)
**Author:** [@crux](https://discuss.elastic.co/u/crux)\ **Replies:** 0\ **Last updated:** [May 18, 2018, 4:27am UTC](https://discuss.elastic.co/t/add-hits-total-to-field-in-elasticsearch-filter/132413 "2018-05-18T04:27:16Z")
I can’t seem to find a good way to get hits.total from the Elasticsearch response in my logstash pipeline. I’m really looking for a yes or no answer, so I have size set to 0. If I have a document that looks like this: … --- ## [Rename Fields](https://discuss.elastic.co/t/rename-fields/132269)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 1\ **Last updated:** [May 17, 2018, 8:35pm UTC](https://discuss.elastic.co/t/rename-fields/132269 "2018-05-17T20:35:25Z")
Hi, I have made two filters for two firewalls and I want to rename the fiealds so they have the same name for each field, and also I want to put ip\_dest and ip\_source in one IP fieald. How can I do this please? See my … --- ## [UDP worker dies unable to bind to port](https://discuss.elastic.co/t/udp-worker-dies-unable-to-bind-to-port/132353)
**Author:** [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Replies:** 1\ **Last updated:** [May 17, 2018, 8:32pm UTC](https://discuss.elastic.co/t/udp-worker-dies-unable-to-bind-to-port/132353 "2018-05-17T20:32:04Z")
Hi, I am trying to collect netflow from an ASA. My UDP worker dies saying it cannot bind to port. It dies .002 seconds after starting. When I run an lsof -nPi on the port there is nothing connected to the port. When… --- ## [Logstash 6.2 - full persistent queue (wrong mapping?)](https://discuss.elastic.co/t/logstash-6-2-full-persistent-queue-wrong-mapping/132177)
**Author:** [@dorinand](https://discuss.elastic.co/u/dorinand)\ **Replies:** 1\ **Last updated:** [May 17, 2018, 7:21pm UTC](https://discuss.elastic.co/t/logstash-6-2-full-persistent-queue-wrong-mapping/132177 "2018-05-17T19:21:24Z")
My queue is almost full and I see this errors in my log file: \[2018-05-16T00:01:33,334\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_inde… --- ## [Sending \[tags\] to zabbix\_value](https://discuss.elastic.co/t/sending-tags-to-zabbix-value/132387)
**Author:** [@CasMeiron](https://discuss.elastic.co/u/CasMeiron)\ **Replies:** 0\ **Last updated:** [May 17, 2018, 7:19pm UTC](https://discuss.elastic.co/t/sending-tags-to-zabbix-value/132387 "2018-05-17T19:19:18Z")
Hello, I'm tryin' to send the field "\[tags\]" to zabbix using the field but I'm unable to do that. It seems the plugin doesn't accept array because all of my other fields works when I send to zabbix. I could try to use … --- ## [Multiple conf files handling issue in logstash](https://discuss.elastic.co/t/multiple-conf-files-handling-issue-in-logstash/131278)
**Author:** [@Dazith\_Kj](https://discuss.elastic.co/u/Dazith_Kj)\ **Replies:** 11\ **Last updated:** [May 17, 2018, 7:16pm UTC](https://discuss.elastic.co/t/multiple-conf-files-handling-issue-in-logstash/131278 "2018-05-17T19:16:24Z")
Hi All, I have the below setup on my logstash config. File name : custom.conf input { beats { port =\> "5044" ssl =\> true ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt" … --- ## [Filter JSON content in field and parse target with existing mapping](https://discuss.elastic.co/t/filter-json-content-in-field-and-parse-target-with-existing-mapping/132350)
**Author:** [@baba](https://discuss.elastic.co/u/baba)\ **Replies:** 5\ **Last updated:** [May 17, 2018, 4:39pm UTC](https://discuss.elastic.co/t/filter-json-content-in-field-and-parse-target-with-existing-mapping/132350 "2018-05-17T16:39:09Z")
Hi, I'm trying to filter a second type of messages coming in my input queue. The difference between messages is that in the second type of messages the log is embedded in a JSON as value of the "Message" field. I confi… --- ## [Changing logstash config in docker container](https://discuss.elastic.co/t/changing-logstash-config-in-docker-container/131138)
**Author:** [@skaldenmet](https://discuss.elastic.co/u/skaldenmet)\ **Replies:** 2\ **Last updated:** [May 17, 2018, 1:59pm UTC](https://discuss.elastic.co/t/changing-logstash-config-in-docker-container/131138 "2018-05-17T13:59:59Z")
Hi, I would like to reload the logstash configuration files automatically after they have changed. Normally this should happen with bin/logstash –f apache.config --config.reload.automatic but the docker container in log… --- ## [Process VMWare ESXi syslog with multiline events](https://discuss.elastic.co/t/process-vmware-esxi-syslog-with-multiline-events/132117)
**Author:** [@Kris\_Boeckx](https://discuss.elastic.co/u/Kris_Boeckx)\ **Replies:** 6\ **Last updated:** [May 17, 2018, 1:59pm UTC](https://discuss.elastic.co/t/process-vmware-esxi-syslog-with-multiline-events/132117 "2018-05-17T13:59:17Z")
Hi, I'm trying to process syslog events that are send by an vmware ESXi server with logstash. This is working fine except multiline events . Below an example of the raw data that is sent via syslog: \<166\>2018-05-16T08… --- ## [How to connect Logstash with Elastic Cloud Elasticsearch via https](https://discuss.elastic.co/t/how-to-connect-logstash-with-elastic-cloud-elasticsearch-via-https/132071)
**Author:** [@EloaCardoso](https://discuss.elastic.co/u/EloaCardoso)\ **Replies:** 10\ **Last updated:** [May 17, 2018, 1:52pm UTC](https://discuss.elastic.co/t/how-to-connect-logstash-with-elastic-cloud-elasticsearch-via-https/132071 "2018-05-17T13:52:49Z")
I am having problems to connect my local Logstash with an Elastic Cloud Elasticsearch. The following lines are placed within the output plugin in the config file: elasticsearch { codec =\> json manage\_template =\> fal… --- ## [Grokparsefailure Time set up +1hour](https://discuss.elastic.co/t/grokparsefailure-time-set-up-1hour/132058)
**Author:** [@Invictus](https://discuss.elastic.co/u/Invictus)\ **Replies:** 4\ **Last updated:** [May 17, 2018, 1:24pm UTC](https://discuss.elastic.co/t/grokparsefailure-time-set-up-1hour/132058 "2018-05-17T13:24:25Z")
Hi members, so I have a log with three entries. 26.01.2018 15:57:12 Hostname: ApplicationName : INFO - Message 26.01.2018 15:57:12 Hostname: ApplicationName : INFO - Message /configFile=Greedydata /logConfigFolder=Gree… --- ## [Logstash ignores some pipelines](https://discuss.elastic.co/t/logstash-ignores-some-pipelines/132336)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 1\ **Last updated:** [May 17, 2018, 1:10pm UTC](https://discuss.elastic.co/t/logstash-ignores-some-pipelines/132336 "2018-05-17T13:10:10Z")
I am trying to use multiple pipelines and my pipeline.yml is the following: - pipeline.id: main path.config: "/etc/logstash/conf.d/\*.conf" - pipeline.id: p\_mapping\_users path.config: "/etc/logstash/mapping\_pipelines… --- ## [Logstash as a container hosted on docker](https://discuss.elastic.co/t/logstash-as-a-container-hosted-on-docker/132332)
**Author:** [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\ **Replies:** 0\ **Last updated:** [May 17, 2018, 12:40pm UTC](https://discuss.elastic.co/t/logstash-as-a-container-hosted-on-docker/132332 "2018-05-17T12:40:33Z")
Hello I am new to logstash and presently m configuration runs as a conatiner hosted on docker. But I am getting this error Your application used more memory than the safety cap of 1G. Specify -J-Xmx####m to increase i… --- ## [Analysis logstash jvm Tuning and Profiling Logstash Performance question](https://discuss.elastic.co/t/analysis-logstash-jvm-tuning-and-profiling-logstash-performance-question/132013)
**Author:** [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Replies:** 3\ **Last updated:** [May 17, 2018, 12:07pm UTC](https://discuss.elastic.co/t/analysis-logstash-jvm-tuning-and-profiling-logstash-performance-question/132013 "2018-05-17T12:07:41Z")
i test a logstash instance with the visualVM at: https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html#profiling-the-heap cmd is as ./logstash -b 50000 -u 1 -w 16 -f xxx.conf why the JVM use not like a … --- ## [Allow logstash to store multi-line messages](https://discuss.elastic.co/t/allow-logstash-to-store-multi-line-messages/132308)
**Author:** [@SumitV](https://discuss.elastic.co/u/SumitV)\ **Replies:** 1\ **Last updated:** [May 17, 2018, 11:37am UTC](https://discuss.elastic.co/t/allow-logstash-to-store-multi-line-messages/132308 "2018-05-17T11:37:41Z")
In my application, i am using filebeat-\>logstash-\>elasticsearch-\>kibana. Here filebeat is listening to a folder location(/XYZ/\*.log) This log file contains logs where messages can be in the form of multi-line. For exam… --- ## [Error connecting to mysql using jdbc\_static filter](https://discuss.elastic.co/t/error-connecting-to-mysql-using-jdbc-static-filter/132163)
**Author:** [@parallelthought](https://discuss.elastic.co/u/parallelthought)\ **Replies:** 2\ **Last updated:** [May 17, 2018, 11:30am UTC](https://discuss.elastic.co/t/error-connecting-to-mysql-using-jdbc-static-filter/132163 "2018-05-17T11:30:32Z")
I'm trying to connect to a Mysql DB using the jdbc\_static filter plugin. I've observed that the error thrown by the plugin for both non existent (Eg: 127.1.1.1) and correct hostname (Eg: 127.0.0.1) as part of the jdbc c… --- ## [Pipeline not working with logstash-output-elasticsearch plugin when using doc\_as\_upsert](https://discuss.elastic.co/t/pipeline-not-working-with-logstash-output-elasticsearch-plugin-when-using-doc-as-upsert/132314)
**Author:** [@dao](https://discuss.elastic.co/u/dao)\ **Replies:** 0\ **Last updated:** [May 17, 2018, 11:04am UTC](https://discuss.elastic.co/t/pipeline-not-working-with-logstash-output-elasticsearch-plugin-when-using-doc-as-upsert/132314 "2018-05-17T11:04:48Z")
Continuing the discussion from Pipeline not working with logstash-output-elasticsearch plugin when using doc\_as\_upsert =\> true: It seems that I have the same issue. Do you confirm it is a known issue? is there a fix in … --- ## [Logstash-input-azureblob - Only String and Array types are splittable](https://discuss.elastic.co/t/logstash-input-azureblob-only-string-and-array-types-are-splittable/132293)
**Author:** [@charan.gandra](https://discuss.elastic.co/u/charan.gandra)\ **Replies:** 0\ **Last updated:** [May 17, 2018, 9:26am UTC](https://discuss.elastic.co/t/logstash-input-azureblob-only-string-and-array-types-are-splittable/132293 "2018-05-17T09:26:32Z")
Hi, I am trying to use logstash-input-azureblob plugin to push the NSG flow logs into ES. It was working for a week and I could see an excellent dash board with all the data. But all of sudden it stopped working with th… --- ## [Imap emails attachments](https://discuss.elastic.co/t/imap-emails-attachments/131924)
**Author:** [@rouchad\_rouchad](https://discuss.elastic.co/u/rouchad_rouchad)\ **Replies:** 1\ **Last updated:** [May 17, 2018, 9:02am UTC](https://discuss.elastic.co/t/imap-emails-attachments/131924 "2018-05-17T09:02:05Z")
i use elasticSearch 6.2 i want transfer my database(emails) from my gmail to elasticsearch the probleme is that elastcisearch can't read the emails which contains attachements like PDF , XML files i found that i need i… --- ## [How to avoid duplicate log?](https://discuss.elastic.co/t/how-to-avoid-duplicate-log/132099)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 3\ **Last updated:** [May 17, 2018, 8:38am UTC](https://discuss.elastic.co/t/how-to-avoid-duplicate-log/132099 "2018-05-17T08:38:34Z")
Hi, I made two configuration files, one for ASA and one for Fortigate. The problem is that I have duplicate logs on Kibana, how do I solve that? --- ## [How logstash parse log files internally?](https://discuss.elastic.co/t/how-logstash-parse-log-files-internally/132172)
**Author:** [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\ **Replies:** 8\ **Last updated:** [May 17, 2018, 7:31am UTC](https://discuss.elastic.co/t/how-logstash-parse-log-files-internally/132172 "2018-05-17T07:31:07Z")
I am new here for ELK(ElasticSearch,Logstash,kibana). I would like to know how logstash parse log files and upload into an ElasticSearch. Lets take an example, If I want to load around 250GB of log files data into an Ela… --- ## [Logstash FATAL Error](https://discuss.elastic.co/t/logstash-fatal-error/132247)
**Author:** [@Sunillinus](https://discuss.elastic.co/u/Sunillinus)\ **Replies:** 0\ **Last updated:** [May 17, 2018, 3:57am UTC](https://discuss.elastic.co/t/logstash-fatal-error/132247 "2018-05-17T03:57:02Z")
Hi, I'm using logstash 6.2.4 version I ran into this error in logstash when it is executing to upload log file into kibana. Can anyone please let me know what would be the reason for this error. \[2018-05-17T00:25:00,… --- ## [Logstash Input redis](https://discuss.elastic.co/t/logstash-input-redis/132194)
**Author:** [@shivkumar](https://discuss.elastic.co/u/shivkumar)\ **Replies:** 2\ **Last updated:** [May 17, 2018, 3:55am UTC](https://discuss.elastic.co/t/logstash-input-redis/132194 "2018-05-17T03:55:58Z")
Hello I am using logstash 5.6.9. I am simply trying to read from my local redis server and print it to console using logstash, here is my configuration :slight\_smile: input { redis { host =\> 'localhost' data\_type =\> … --- ## [Logstash file output - persisted queue option?](https://discuss.elastic.co/t/logstash-file-output-persisted-queue-option/132233)
**Author:** [@bloke](https://discuss.elastic.co/u/bloke)\ **Replies:** 0\ **Last updated:** [May 17, 2018, 12:56am UTC](https://discuss.elastic.co/t/logstash-file-output-persisted-queue-option/132233 "2018-05-17T00:56:53Z")
Hi, I am wondering, currently I have logstash writing file output and doing ES like such output { if ((\[type\] =~ /^srx$/) and ("noelastic" not in \[tags\])) { elasticsearch { … --- ## [Udp input is crashing](https://discuss.elastic.co/t/udp-input-is-crashing/128485)
**Author:** [@tobstone](https://discuss.elastic.co/u/tobstone)\ **Replies:** 6\ **Last updated:** [May 17, 2018, 12:08am UTC](https://discuss.elastic.co/t/udp-input-is-crashing/128485 "2018-05-17T00:08:18Z")
\[2018-04-18T10:33:55,772\]\[INFO \]\[logstash.inputs.udp \] Starting UDP listener {:address=\>"0.0.0.0:9988"} \[2018-04-18T10:33:55,773\]\[WARN \]\[logstash.inputs.udp \] UDP listener died {:exception=\>#\