# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=692 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 693 --- ## [Logstash output going to syslog](https://discuss.elastic.co/t/logstash-output-going-to-syslog/146603)
**Author:** [@lemont](https://discuss.elastic.co/u/lemont)\ **Replies:** 5\ **Last updated:** [September 1, 2018, 11:34am UTC](https://discuss.elastic.co/t/logstash-output-going-to-syslog/146603 "2018-09-01T11:34:25Z")
Outputs from logstash somehow are going to the linux syslog file eating into the storage. I will have to delete the syslog file periodically. How do I stop logstash output from sending to syslog? --- ## [Couldn't decode protobuf](https://discuss.elastic.co/t/couldnt-decode-protobuf/146937)
**Author:** [@artobstrel](https://discuss.elastic.co/u/artobstrel)\ **Replies:** 0\ **Last updated:** [September 1, 2018, 11:10am UTC](https://discuss.elastic.co/t/couldnt-decode-protobuf/146937 "2018-09-01T11:10:52Z")
After upgrading Logstash 5.6.11 → 6.4.0 when decoding protobuf I get an error: \[WARN \]\[logstash.codecs.protobuf \] Couldn't decode protobuf: #\. Logsta… --- ## [Schedule option with exec module](https://discuss.elastic.co/t/schedule-option-with-exec-module/146821)
**Author:** [@niraj\_pandey](https://discuss.elastic.co/u/niraj_pandey)\ **Replies:** 4\ **Last updated:** [September 1, 2018, 6:31am UTC](https://discuss.elastic.co/t/schedule-option-with-exec-module/146821 "2018-09-01T06:31:13Z")
\[niraj@farmdev02 ~\]$ sudo /usr/share/logstash/bin/logstash -f /tmp/cloud-cost.conf -t netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} \[WARN \] 2018-08-31 00:27:30.366 \[LogStash::Runner\] multil… --- ## [Can I filter the Output data separately?!](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064)
**Author:** [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\ **Replies:** 8\ **Last updated:** [August 31, 2018, 8:11pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064 "2018-08-31T20:11:33Z")
We have the following env: LS 2.0.x; ES 2.0.x and K 4.2.x The input comes from the trace logs that an application produces, we filter the data and then wish to populate two separate (but related) indices: orders and… --- ## [How to remove subfield using pattern](https://discuss.elastic.co/t/how-to-remove-subfield-using-pattern/145101)
**Author:** [@jerry-chen-315](https://discuss.elastic.co/u/jerry-chen-315)\ **Replies:** 2\ **Last updated:** [August 31, 2018, 7:15pm UTC](https://discuss.elastic.co/t/how-to-remove-subfield-using-pattern/145101 "2018-08-31T19:15:44Z")
logstash version 6.2.1 my message like this: {"@timestamp":"2018-08-12T15:53:02Z","hostanme":"server-oem","hostip":"162.12.1.240","tag":"auditlog","event\_id":521244,"EXECVE":{ "type": "EXECVE", "argc": "28", "a0": ""f… --- ## [Call a python script from ElasticSearch daily](https://discuss.elastic.co/t/call-a-python-script-from-elasticsearch-daily/146557)
**Author:** [@aashishgahlawat](https://discuss.elastic.co/u/aashishgahlawat)\ **Replies:** 9\ **Last updated:** [August 31, 2018, 5:34pm UTC](https://discuss.elastic.co/t/call-a-python-script-from-elasticsearch-daily/146557 "2018-08-31T17:34:12Z")
I am using logstash with scheduling to load data from database to ElasticSearch daily, now I have a script that will do some machine learning stuff on my index, how can I call a python script whenever an index is updated… --- ## [Log Grok does not work](https://discuss.elastic.co/t/log-grok-does-not-work/146810)
**Author:** [@aroldev](https://discuss.elastic.co/u/aroldev)\ **Replies:** 1\ **Last updated:** [August 31, 2018, 5:10pm UTC](https://discuss.elastic.co/t/log-grok-does-not-work/146810 "2018-08-31T17:10:01Z")
Hello elastic community, I want to parse some log lines and send it to my elasticsearch engine. My Log looks like this: 2018-08-27T11:56:35:537 | SERVERNAME | 19140 | processname | 27 | 127.0.0.1 | | 0 | Verbose | \*\*\*… --- ## [How to set filter integer settings based on event filed or metadata](https://discuss.elastic.co/t/how-to-set-filter-integer-settings-based-on-event-filed-or-metadata/146877)
**Author:** [@Tomasz\_Macioszek](https://discuss.elastic.co/u/Tomasz_Macioszek)\ **Replies:** 1\ **Last updated:** [August 31, 2018, 3:18pm UTC](https://discuss.elastic.co/t/how-to-set-filter-integer-settings-based-on-event-filed-or-metadata/146877 "2018-08-31T15:18:12Z")
Hi All, I would like to set filter settings - aggregate filter timeout value - based on event field or metadata. I have converted metadata to integere mutate { convert =\> { "\[@metadata\]\[taskID\_timeout\]" =\> "integer" }… --- ## [Logstash max\_open\_files setting not working?](https://discuss.elastic.co/t/logstash-max-open-files-setting-not-working/143208)
**Author:** [@jspriesterbach](https://discuss.elastic.co/u/jspriesterbach)\ **Replies:** 5\ **Last updated:** [August 31, 2018, 2:45pm UTC](https://discuss.elastic.co/t/logstash-max-open-files-setting-not-working/143208 "2018-08-31T14:45:37Z")
Hello, I am using logstash version 6.3 and I can't seem to get the max\_open\_files to stick. According to the proc it is set properly: root@ip-172-31-47-78:/proc/1433# cat limits | grep "Max open files" Max open files … --- ## [Load Balancing for two ELK statck servers](https://discuss.elastic.co/t/load-balancing-for-two-elk-statck-servers/146866)
**Author:** [@nik9](https://discuss.elastic.co/u/nik9)\ **Replies:** 0\ **Last updated:** [August 31, 2018, 12:36pm UTC](https://discuss.elastic.co/t/load-balancing-for-two-elk-statck-servers/146866 "2018-08-31T12:36:28Z")
Hi team, I had done the elk stack setup for qa in single node. And we are able to get logs from different nodes using filebeat on elk server. For Advance i want to use load balancer to distribute logs load to 2 servers… --- ## [Using Logstash need to ingest lat lon values into elasticsearch](https://discuss.elastic.co/t/using-logstash-need-to-ingest-lat-lon-values-into-elasticsearch/146816)
**Author:** [@SRINIVASARAOPOTNUR](https://discuss.elastic.co/u/SRINIVASARAOPOTNUR)\ **Replies:** 3\ **Last updated:** [August 31, 2018, 11:46am UTC](https://discuss.elastic.co/t/using-logstash-need-to-ingest-lat-lon-values-into-elasticsearch/146816 "2018-08-31T11:46:17Z")
Hi Team, We are using below versions to ingest the lat lon values into Elasticsearch using logstash config file. Logstash -6.3.2 Elasticsearch - 6.3.2 Kibana - 6.3.2 Data : ( File Name : example.csv ) |cid | |… --- ## [Aggregate filter logstash- pulling field from another event](https://discuss.elastic.co/t/aggregate-filter-logstash-pulling-field-from-another-event/146850)
**Author:** [@thiyagarajan.vasudev](https://discuss.elastic.co/u/thiyagarajan.vasudev)\ **Replies:** 0\ **Last updated:** [August 31, 2018, 10:56am UTC](https://discuss.elastic.co/t/aggregate-filter-logstash-pulling-field-from-another-event/146850 "2018-08-31T10:56:00Z")
I am sending logs from filebeat to logstash and in logstash I am doing filter based on regex and storing it in elastic search My logstash config: input { beats { port =\> 5002 } } filter { grok { add\_tag =\> \[ "… --- ## [Unable to retrieve logs sent to same index](https://discuss.elastic.co/t/unable-to-retrieve-logs-sent-to-same-index/146849)
**Author:** [@jissjoseph82](https://discuss.elastic.co/u/jissjoseph82)\ **Replies:** 0\ **Last updated:** [August 31, 2018, 10:50am UTC](https://discuss.elastic.co/t/unable-to-retrieve-logs-sent-to-same-index/146849 "2018-08-31T10:50:01Z")
We have logstash setup that pushes alb and cloudfront logs from S3 and sends to elasticsearch. The index used is same. The logstash runs on docker and we do see that logstash starts and and registers the buckets and pro… --- ## [Dead Letter Queue (DLQ) issue](https://discuss.elastic.co/t/dead-letter-queue-dlq-issue/146757)
**Author:** [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Replies:** 1\ **Last updated:** [August 31, 2018, 10:02am UTC](https://discuss.elastic.co/t/dead-letter-queue-dlq-issue/146757 "2018-08-31T10:02:42Z")
Hello, I'm using Logstash 6.3.2 and I've enabled the DLQ but it doesn't appear to be indexing the events back into Elasticsearch; the log files in the DLQ data directory just keep growing and all i'm seeing in logstash-… --- ## [Logstash is not connecting on muliple port for different beats](https://discuss.elastic.co/t/logstash-is-not-connecting-on-muliple-port-for-different-beats/146379)
**Author:** [@Nirpendra\_Kumar](https://discuss.elastic.co/u/Nirpendra_Kumar)\ **Replies:** 11\ **Last updated:** [August 31, 2018, 9:34am UTC](https://discuss.elastic.co/t/logstash-is-not-connecting-on-muliple-port-for-different-beats/146379 "2018-08-31T09:34:42Z")
I have 3 server with beat service installed , passing the logs to logstash on different ports 5044,5045,4046 with different index .but all indexes have the same data . --- ## [Can't filter Unicode logs at all](https://discuss.elastic.co/t/cant-filter-unicode-logs-at-all/146701)
**Author:** [@yolt](https://discuss.elastic.co/u/yolt)\ **Replies:** 2\ **Last updated:** [August 31, 2018, 9:32am UTC](https://discuss.elastic.co/t/cant-filter-unicode-logs-at-all/146701 "2018-08-31T09:32:17Z")
Hello, I need help to find out how to set Filebeat or Logstash or both... to be able successfully filter logs by message from files which have set encoding as Unicode I will provide screenshots to describe problem as… --- ## [Message contains full log line](https://discuss.elastic.co/t/message-contains-full-log-line/146692)
**Author:** [@Ico\_Re](https://discuss.elastic.co/u/Ico_Re)\ **Replies:** 2\ **Last updated:** [August 31, 2018, 8:42am UTC](https://discuss.elastic.co/t/message-contains-full-log-line/146692 "2018-08-31T08:42:05Z")
Hi I have the following filter defined in the logstash.conf filter { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:timestamp} (?\\\\[.\*\\\])? %{LOGLEVEL:level} %{DATA:classname} %{GREEDYDATA:mess… --- ## [How to use data from multiple csv files to single elasticseach index using logstash](https://discuss.elastic.co/t/how-to-use-data-from-multiple-csv-files-to-single-elasticseach-index-using-logstash/146831)
**Author:** [@Pramod08](https://discuss.elastic.co/u/Pramod08)\ **Replies:** 0\ **Last updated:** [August 31, 2018, 8:18am UTC](https://discuss.elastic.co/t/how-to-use-data-from-multiple-csv-files-to-single-elasticseach-index-using-logstash/146831 "2018-08-31T08:18:43Z")
Hi, I'm having two different csv files first file contain concurrency data for online sources (A) and second contain for internal system (B). How to add this data single elasticseach index File A pattern Timestamp,… --- ## [Pass parameters between inputs (jdbc)](https://discuss.elastic.co/t/pass-parameters-between-inputs-jdbc/146787)
**Author:** [@franklin.samboni](https://discuss.elastic.co/u/franklin.samboni)\ **Replies:** 1\ **Last updated:** [August 31, 2018, 7:32am UTC](https://discuss.elastic.co/t/pass-parameters-between-inputs-jdbc/146787 "2018-08-31T07:32:02Z")
Hello, I configured Logstash to read one jdbc input (mysql) and put information in elasticsearch, all this run ok. But now I need some parameters of a second input (other mysql database) that must be passed to SQL stat… --- ## [Template not taking effect for ES output](https://discuss.elastic.co/t/template-not-taking-effect-for-es-output/145095)
**Author:** [@kaushik.vankayala](https://discuss.elastic.co/u/kaushik.vankayala)\ **Replies:** 5\ **Last updated:** [August 31, 2018, 7:24am UTC](https://discuss.elastic.co/t/template-not-taking-effect-for-es-output/145095 "2018-08-31T07:24:14Z")
Hello, I am trying to change the type of a field, so i am using a template option in my elastic search output plugin. My output field is like; output { #stdout { codec =\> rubydebug } if "uat-" in \[app\] { elastics… --- ## [Parse nested JSON using Ruby Filter](https://discuss.elastic.co/t/parse-nested-json-using-ruby-filter/146789)
**Author:** [@Samvid\_Kulkarni](https://discuss.elastic.co/u/Samvid_Kulkarni)\ **Replies:** 0\ **Last updated:** [August 30, 2018, 11:40pm UTC](https://discuss.elastic.co/t/parse-nested-json-using-ruby-filter/146789 "2018-08-30T23:40:32Z")
I have this json log and i am trying to parse it using ruby filter but i am having hard time as I dont have any knowledge of ruby. here is my log. Can anyone show me some proper way to parse nested JSON please? Is ther… --- ## [Logstash Geoip Filter with a Custom Database](https://discuss.elastic.co/t/logstash-geoip-filter-with-a-custom-database/146778)
**Author:** [@lens](https://discuss.elastic.co/u/lens)\ **Replies:** 0\ **Last updated:** [August 30, 2018, 9:28pm UTC](https://discuss.elastic.co/t/logstash-geoip-filter-with-a-custom-database/146778 "2018-08-30T21:28:39Z")
We are looking into using the geoip filter to basically search through IP addresses and blocks to get information that is not actually geographic. We've followed the instructions for creating a new .mmdb file (https://bl… --- ## [Begin LogStash](https://discuss.elastic.co/t/begin-logstash/146585)
**Author:** [@marcos.vbp](https://discuss.elastic.co/u/marcos.vbp)\ **Replies:** 9\ **Last updated:** [August 30, 2018, 9:05pm UTC](https://discuss.elastic.co/t/begin-logstash/146585 "2018-08-30T21:05:07Z")
Hi All, I'm starting with logstash, and studying alone. I was trying to make a simple filter to display in kiban, for example the number of requests that an apache receives, I put two apaches of tests to show in a grap… --- ## [Logstash output plugin not writing data sequentially](https://discuss.elastic.co/t/logstash-output-plugin-not-writing-data-sequentially/146178)
**Author:** [@kaushik.vankayala](https://discuss.elastic.co/u/kaushik.vankayala)\ **Replies:** 4\ **Last updated:** [August 30, 2018, 6:34pm UTC](https://discuss.elastic.co/t/logstash-output-plugin-not-writing-data-sequentially/146178 "2018-08-30T18:34:08Z")
Hi There, I am using logstash as means of backup tool for the logs of my application. The conf file i created is simply an input with a file plugin and an output with a file plugin. So basically i have a shell script r… --- ## [Logstash 6.3.2: sqs to bq output error](https://discuss.elastic.co/t/logstash-6-3-2-sqs-to-bq-output-error/146762)
**Author:** [@Harshad\_Velapure](https://discuss.elastic.co/u/Harshad_Velapure)\ **Replies:** 0\ **Last updated:** [August 30, 2018, 6:01pm UTC](https://discuss.elastic.co/t/logstash-6-3-2-sqs-to-bq-output-error/146762 "2018-08-30T18:01:15Z")
Hello I am trying to push data to BQ from sqs I have below configuration in logstash file json { source =\> "Message" target =\> "parsed" } split { field =\> "parsed" } mutate{ rename =\> { "\[parsed\]\[timestamp\]" … --- ## [Curator Leaving Indices with Status of Deleted](https://discuss.elastic.co/t/curator-leaving-indices-with-status-of-deleted/146732)
**Author:** [@Luis\_Alejandro\_Galan](https://discuss.elastic.co/u/Luis_Alejandro_Galan)\ **Replies:** 2\ **Last updated:** [August 30, 2018, 4:05pm UTC](https://discuss.elastic.co/t/curator-leaving-indices-with-status-of-deleted/146732 "2018-08-30T16:05:14Z")
hello! I have just finished the set up of a curator and it successfully deletes the indices with the correct pattern. However, I noticed that under Monitoring --\> Indices: --\> deleted index the name of the file still ex… --- ## [Logstash Docker](https://discuss.elastic.co/t/logstash-docker/146740)
**Author:** [@avisual](https://discuss.elastic.co/u/avisual)\ **Replies:** 0\ **Last updated:** [August 30, 2018, 4:03pm UTC](https://discuss.elastic.co/t/logstash-docker/146740 "2018-08-30T16:03:45Z")
I am trying to get multi-pipelines running under the official docker image. I am using envsubst to replace all the Env in a template. but when passing in environment variables from docker it fails with: Ignoring the 'p… --- ## [\[SOLVED\]Logstash configuration problem identification](https://discuss.elastic.co/t/solved-logstash-configuration-problem-identification/146739)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 1\ **Last updated:** [August 30, 2018, 3:53pm UTC](https://discuss.elastic.co/t/solved-logstash-configuration-problem-identification/146739 "2018-08-30T15:53:40Z")
Hello . I cannot figure out the problem in this configuration I have created: I get error: \[2018-08-30T17:43:41,301\]\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Crea… --- ## [\[SOLVED\] Trying monitoring logstash in kibana/elasticsearch](https://discuss.elastic.co/t/solved-trying-monitoring-logstash-in-kibana-elasticsearch/146528)
**Author:** [@alexolivan](https://discuss.elastic.co/u/alexolivan)\ **Replies:** 3\ **Last updated:** [August 30, 2018, 3:09pm UTC](https://discuss.elastic.co/t/solved-trying-monitoring-logstash-in-kibana-elasticsearch/146528 "2018-08-30T15:09:32Z")
Hi again forum! I've been again playing with ELK stuff these days, I've upgraded my 3 VM baby cluster lab from 5.4 to 6.3, and started adding new data onto it ... and I've realized that it is possible to monitor logstas… --- ## [Ruby test framework documentation](https://discuss.elastic.co/t/ruby-test-framework-documentation/146721)
**Author:** [@darefilz](https://discuss.elastic.co/u/darefilz)\ **Replies:** 0\ **Last updated:** [August 30, 2018, 2:12pm UTC](https://discuss.elastic.co/t/ruby-test-framework-documentation/146721 "2018-08-30T14:12:13Z")
Hi, I want to write a ruby filter for logstash including some tests. I've read the examples and another blog, but I can't find any documentation about how to write such tests. Can anybody help me with that? Regards [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=691) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=693)