# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=758 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 759 --- ## [Multiple Pipelines using docker instance of logstash 6.2.4](https://discuss.elastic.co/t/multiple-pipelines-using-docker-instance-of-logstash-6-2-4/131546)
**Author:** [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Replies:** 3\ **Last updated:** [May 17, 2018, 12:07am UTC](https://discuss.elastic.co/t/multiple-pipelines-using-docker-instance-of-logstash-6-2-4/131546 "2018-05-17T00:07:26Z")
Using logstash docker image for 6.2.4, we are trying to run multiple pipelines. But it is not working. We always see "main" pipeline getting started. With exact same logstash.yml and pipeline.yml settings, the non-dock… --- ## [CIDR, location lookup in the most efficient way](https://discuss.elastic.co/t/cidr-location-lookup-in-the-most-efficient-way/131232)
**Author:** [@geertn444](https://discuss.elastic.co/u/geertn444)\ **Replies:** 2\ **Last updated:** [May 16, 2018, 8:59pm UTC](https://discuss.elastic.co/t/cidr-location-lookup-in-the-most-efficient-way/131232 "2018-05-16T20:59:57Z")
I want to map IPs in a certain range to a site. I don't want to map the IP ranges inside logstash config files, but in external files. CIDR has this possibility using network\_path So now i have a logic like this: if \[f… --- ## [Fingerprint filter- Key from Document](https://discuss.elastic.co/t/fingerprint-filter-key-from-document/131254)
**Author:** [@SKumarMN](https://discuss.elastic.co/u/SKumarMN)\ **Replies:** 1\ **Last updated:** [May 16, 2018, 8:22pm UTC](https://discuss.elastic.co/t/fingerprint-filter-key-from-document/131254 "2018-05-16T20:22:32Z")
Hi All, I am using fingerprint filter with SHA256, I want to use key for the plugin dynamically (.i.e based on the document attribute) As shown below group\_id is a document attribute and tenant\_id is the document attrib… --- ## [Why does this date filter get a \_dateparsefailure?](https://discuss.elastic.co/t/why-does-this-date-filter-get-a-dateparsefailure/132199)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 2\ **Last updated:** [May 16, 2018, 7:54pm UTC](https://discuss.elastic.co/t/why-does-this-date-filter-get-a-dateparsefailure/132199 "2018-05-16T19:54:59Z")
Why does this date filter get a \_dateparsefailure? input { generator { message =\> 'N' count =\> 1 } } output { stdout { codec =\> rubydebug } } filter { mutate { add\_field =\> { "DateReported" =\> "1292015 1745" } } dat… --- ## [Remove a field before being sent to output](https://discuss.elastic.co/t/remove-a-field-before-being-sent-to-output/132175)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 2\ **Last updated:** [May 16, 2018, 7:35pm UTC](https://discuss.elastic.co/t/remove-a-field-before-being-sent-to-output/132175 "2018-05-16T19:35:26Z")
I am trying to add a custom \_id field as follows: My filter.conf filter { prune { whitelist\_names =\> \["^http\_user$"\] } mutate { rename =\> { "http\_user" =\> "username" } add\_field =\> \[ 'custom\_id… --- ## [Pipeline conf file processes wrong log](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159)
**Author:** [@shine](https://discuss.elastic.co/u/shine)\ **Replies:** 7\ **Last updated:** [May 16, 2018, 7:08pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159 "2018-05-16T19:08:45Z")
My logs are not being processed by the correct conf file. Customer B logs are being processed by "cust\_log\_a.conf" instead of "cust\_log\_b.conf" pipeline.yml pipeline.id: customer\_logs\_a path.config: "/etc/logstash/co… --- ## [Twitter Sample API](https://discuss.elastic.co/t/twitter-sample-api/131942)
**Author:** [@kelevra](https://discuss.elastic.co/u/kelevra)\ **Replies:** 4\ **Last updated:** [May 16, 2018, 6:01pm UTC](https://discuss.elastic.co/t/twitter-sample-api/131942 "2018-05-16T18:01:36Z")
Hi, I was wondering is it possible to filter and send only the tweets in English to Elasticsearch while getting the data with Sample API? --- ## [Logstash Google Pub Sub - results in error java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String](https://discuss.elastic.co/t/logstash-google-pub-sub-results-in-error-java-lang-nosuchmethoderror-com-google-common-base-preconditions-checkargument-zljava-lang-string/132001)
**Author:** [@Pinkesh\_Valdria](https://discuss.elastic.co/u/Pinkesh_Valdria)\ **Replies:** 1\ **Last updated:** [May 16, 2018, 5:51pm UTC](https://discuss.elastic.co/t/logstash-google-pub-sub-results-in-error-java-lang-nosuchmethoderror-com-google-common-base-preconditions-checkargument-zljava-lang-string/132001 "2018-05-16T17:51:01Z")
I installed logstash on a debian OS and then installed the Google Pub Sub plugin . Simple pipeline test works fine, but using Google pub/sub as input results in below error. last line of stacktrace: \[ERROR\] 2018-05-1… --- ## [Logstash 6.2.4 Slow Performance Kinesis Input when using json codec](https://discuss.elastic.co/t/logstash-6-2-4-slow-performance-kinesis-input-when-using-json-codec/132188)
**Author:** [@amarsh](https://discuss.elastic.co/u/amarsh)\ **Replies:** 0\ **Last updated:** [May 16, 2018, 5:14pm UTC](https://discuss.elastic.co/t/logstash-6-2-4-slow-performance-kinesis-input-when-using-json-codec/132188 "2018-05-16T17:14:35Z")
I'm seeing significant performance impacts when using codec =\> "json" with the Kinesis input plugin and wanted to see if anyone has recommendations or suggestions for speeding up the processing. On a 4 CPU 16 GB box run… --- ## [Could not find logstsh.yml](https://discuss.elastic.co/t/could-not-find-logstsh-yml/131843)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 1\ **Last updated:** [May 16, 2018, 5:02pm UTC](https://discuss.elastic.co/t/could-not-find-logstsh-yml/131843 "2018-05-16T17:02:58Z")
Hello. WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults /log4j2.properties. Using defaul… --- ## [Trying to use the Google\_pubsub input plugin](https://discuss.elastic.co/t/trying-to-use-the-google-pubsub-input-plugin/131728)
**Author:** [@Victor\_Yampolsky](https://discuss.elastic.co/u/Victor_Yampolsky)\ **Replies:** 4\ **Last updated:** [May 16, 2018, 4:26pm UTC](https://discuss.elastic.co/t/trying-to-use-the-google-pubsub-input-plugin/131728 "2018-05-16T16:26:02Z")
I use the bitnami elk for gcp. I didn't installed x-pack and just want to pull logs from google pub/sub to es: input{ google\_pub\_sub { project\_id =\> proj123 project\_id =\> "projects/proj123/topics/logs" subscript… --- ## [Fingerprint plugin not working](https://discuss.elastic.co/t/fingerprint-plugin-not-working/132167)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 2\ **Last updated:** [May 16, 2018, 4:11pm UTC](https://discuss.elastic.co/t/fingerprint-plugin-not-working/132167 "2018-05-16T16:11:17Z")
My filter.conf filter { prune { whitelist\_names =\> \["^http\_user$"\] } mutate { rename =\> { "http\_user" =\> "username" } add\_field =\> \[ 'custom\_id', '0' \] } fingerprint { key =\> "somekey"… --- ## [Cannot install local plugin without internet connection](https://discuss.elastic.co/t/cannot-install-local-plugin-without-internet-connection/130673)
**Author:** [@smbullet](https://discuss.elastic.co/u/smbullet)\ **Replies:** 7\ **Last updated:** [May 16, 2018, 3:12pm UTC](https://discuss.elastic.co/t/cannot-install-local-plugin-without-internet-connection/130673 "2018-05-16T15:12:27Z")
I'm trying to install a locally built plugin but for some reason it's trying to connect to artifacts.elastic.co: $ sudo bin/logstash-plugin install /path/to/my/plugin-1.0.0-java.gem ERROR: Something went wrong when inst… --- ## [Ingesting both 'standard' timestamp syslog and ISO8601 timestamped syslog](https://discuss.elastic.co/t/ingesting-both-standard-timestamp-syslog-and-iso8601-timestamped-syslog/132128)
**Author:** [@nathharp](https://discuss.elastic.co/u/nathharp)\ **Replies:** 4\ **Last updated:** [May 16, 2018, 3:01pm UTC](https://discuss.elastic.co/t/ingesting-both-standard-timestamp-syslog-and-iso8601-timestamped-syslog/132128 "2018-05-16T15:01:24Z")
As per https://www.elastic.co/guide/en/logstash/current/logstash-config-for-filebeat-modules.html we are happily ingesting Filebeat \> logstash \> elasticsearch for system logs that use the 'standard' syslog timestamp (eg… --- ## [How to ignore the same log contents in the different log files?](https://discuss.elastic.co/t/how-to-ignore-the-same-log-contents-in-the-different-log-files/131757)
**Author:** [@alicia1](https://discuss.elastic.co/u/alicia1)\ **Replies:** 4\ **Last updated:** [May 16, 2018, 2:40pm UTC](https://discuss.elastic.co/t/how-to-ignore-the-same-log-contents-in-the-different-log-files/131757 "2018-05-16T14:40:23Z")
Hi, We have logs files in different repertories, same logs contents can find several times in different files. How to ignore the same log contents in the different log files in order to send only once a same log conten… --- ## [Max\_files\_open location](https://discuss.elastic.co/t/max-files-open-location/132151)
**Author:** [@rsaeks](https://discuss.elastic.co/u/rsaeks)\ **Replies:** 0\ **Last updated:** [May 16, 2018, 2:30pm UTC](https://discuss.elastic.co/t/max-files-open-location/132151 "2018-05-16T14:30:49Z")
I've spent some time searching through the forums (and Google - but many sources are 2+ years old and the paths have changed) to find the locations to set max\_files\_open for logstash since I'm hitting the 4095 maximum le… --- ## [Elasticsearch to elasticsearch index transfer](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953)
**Author:** [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\ **Replies:** 6\ **Last updated:** [May 16, 2018, 1:34pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953 "2018-05-16T13:34:29Z")
I am trying to move an index from my development cluster to our production cluster via logstash. I am getting the below response in the logs that looks like a web response. The development has no security so I'm not real… --- ## [Logstash go down](https://discuss.elastic.co/t/logstash-go-down/131982)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 2\ **Last updated:** [May 16, 2018, 12:26pm UTC](https://discuss.elastic.co/t/logstash-go-down/131982 "2018-05-16T12:26:22Z")
I have a file periodically updated but when logstash reach the end of file it stops. How can I maintain the process active? this is the logstash-plain.log file: \[2018-05-13T06:00:11,220\]\[WARN \]\[logstash.outputs.tcp \] t… --- ## [Ruby to combine Object Array](https://discuss.elastic.co/t/ruby-to-combine-object-array/132050)
**Author:** [@Dhivya\_Buvanesh](https://discuss.elastic.co/u/Dhivya_Buvanesh)\ **Replies:** 1\ **Last updated:** [May 16, 2018, 12:16pm UTC](https://discuss.elastic.co/t/ruby-to-combine-object-array/132050 "2018-05-16T12:16:08Z")
Need Help with ruby code ( I have no knowledge with ruby) on how to combine nested array objects as one Dict ? , as I am facing issues with elastic getting"Object in arrays are not well supported Error" , Also I tried "… --- ## [Logstash Data Enhancement](https://discuss.elastic.co/t/logstash-data-enhancement/131949)
**Author:** [@satya\_sandeep](https://discuss.elastic.co/u/satya_sandeep)\ **Replies:** 3\ **Last updated:** [May 16, 2018, 12:10pm UTC](https://discuss.elastic.co/t/logstash-data-enhancement/131949 "2018-05-16T12:10:48Z")
I have a kafka -ES pipeline running , i want to add new fields basing on a certain field e.g i have the following parameters from kafka: { "id":1, "name":"name1"} now depending on the id i want to add department to … --- ## [VPC flow logs](https://discuss.elastic.co/t/vpc-flow-logs/131548)
**Author:** [@randomuserid](https://discuss.elastic.co/u/randomuserid)\ **Replies:** 6\ **Last updated:** [May 16, 2018, 12:04pm UTC](https://discuss.elastic.co/t/vpc-flow-logs/131548 "2018-05-16T12:04:34Z")
So I'm parsing VPC flow logs (the AWS flow logs) using the csv filter with the space char as the separator b/c I thought this would be efficient and I want to convert certain numeric fields to integers and, ideally, IPs … --- ## [Logstash grok don't parse value](https://discuss.elastic.co/t/logstash-grok-dont-parse-value/132039)
**Author:** [@povisx1](https://discuss.elastic.co/u/povisx1)\ **Replies:** 4\ **Last updated:** [May 16, 2018, 11:31am UTC](https://discuss.elastic.co/t/logstash-grok-dont-parse-value/132039 "2018-05-16T11:31:13Z")
Hello, I have the problem that when I try to use grok regex then I can't take others values like LOGLEVEL and so on maybe someone can help me there is my code: input { beats { port =\> 5043 } } filter { if \[host\] … --- ## [Event generated after timeout is not being pushed to elasticsearch](https://discuss.elastic.co/t/event-generated-after-timeout-is-not-being-pushed-to-elasticsearch/132096)
**Author:** [@vchandrashekar](https://discuss.elastic.co/u/vchandrashekar)\ **Replies:** 0\ **Last updated:** [May 16, 2018, 10:40am UTC](https://discuss.elastic.co/t/event-generated-after-timeout-is-not-being-pushed-to-elasticsearch/132096 "2018-05-16T10:40:55Z")
Hi All, I have following logstash configuration: input { file { type =\> "fail" path =\> "/filepath/ag.txt" } } filter { grok { match =\> \[ "message", "%{LOGLEVEL:loglevel} - %{NOTSPACE:user\_id} - %{GREEDYDATA:msg\_… --- ## [\[FATAL\] 2018-05-16 09:39:07.354 \[LogStash::Runner\] runner - The given configuration is invalid. Reason: Expected one of #, =\> at line 15, column 10 (byte 170) after](https://discuss.elastic.co/t/fatal-2018-05-16-09-39-07-354-logstash-runner-runner-the-given-configuration-is-invalid-reason-expected-one-of-at-line-15-column-10-byte-170-after/132051)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 3\ **Last updated:** [May 16, 2018, 10:08am UTC](https://discuss.elastic.co/t/fatal-2018-05-16-09-39-07-354-logstash-runner-runner-the-given-configuration-is-invalid-reason-expected-one-of-at-line-15-column-10-byte-170-after/132051 "2018-05-16T10:08:54Z")
Hello, i've an error in my syslog pipeline but i'm unable to find it ! Can someone help me ? Error : \[FATAL\] 2018-05-16 09:39:07.354 \[LogStash::Runner\] runner - The given configuration is invalid. Reason: Expected one… --- ## [Red Hat Enterprise sudo service logstash start](https://discuss.elastic.co/t/red-hat-enterprise-sudo-service-logstash-start/131913)
**Author:** [@solo](https://discuss.elastic.co/u/solo)\ **Replies:** 4\ **Last updated:** [May 16, 2018, 9:03am UTC](https://discuss.elastic.co/t/red-hat-enterprise-sudo-service-logstash-start/131913 "2018-05-16T09:03:13Z")
Im running Red Hat Enterprise 6.9 After installing logsatash I try to start it sudo service logstash start logstash: unrecognized service Anyone know the issue? --- ## [Read Message Key from Azure-Event-Hub](https://discuss.elastic.co/t/read-message-key-from-azure-event-hub/131721)
**Author:** [@gghdr](https://discuss.elastic.co/u/gghdr)\ **Replies:** 2\ **Last updated:** [May 16, 2018, 9:01am UTC](https://discuss.elastic.co/t/read-message-key-from-azure-event-hub/131721 "2018-05-16T09:01:52Z")
Hi, i'm using the plugin logstash-input-azureeventhub (https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureeventhub) to read messages from Azure Event Hub queue. It's possible to r… --- ## [Protobuf plugin to convert to Google protobuf logs to readable format](https://discuss.elastic.co/t/protobuf-plugin-to-convert-to-google-protobuf-logs-to-readable-format/132066)
**Author:** [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\ **Replies:** 0\ **Last updated:** [May 16, 2018, 8:50am UTC](https://discuss.elastic.co/t/protobuf-plugin-to-convert-to-google-protobuf-logs-to-readable-format/132066 "2018-05-16T08:50:42Z")
Hi, I am not getting how to use protobuf in confi file of Logstash? I have gone thought https://www.elastic.co/guide/en/logstash/current/plugins-codecs-protobuf.html#plugins-codecs-protobuf-class\_name But not able to … --- ## [Minimal logstash pipeline for beats (6.x)](https://discuss.elastic.co/t/minimal-logstash-pipeline-for-beats-6-x/131729)
**Author:** [@pas](https://discuss.elastic.co/u/pas)\ **Replies:** 5\ **Last updated:** [May 16, 2018, 8:32am UTC](https://discuss.elastic.co/t/minimal-logstash-pipeline-for-beats-6-x/131729 "2018-05-16T08:32:42Z")
Hi, I am currently in the beginning of learning the ELK-stack, so have me excused for asking for the basics. My first goal is to set up a functional beats(winlogbeat and filebeat), logstash, kibana and elasticsearch en… --- ## [I cannot see my all log in kibana](https://discuss.elastic.co/t/i-cannot-see-my-all-log-in-kibana/131555)
**Author:** [@NIloufarVafaei](https://discuss.elastic.co/u/NIloufarVafaei)\ **Replies:** 6\ **Last updated:** [May 16, 2018, 8:28am UTC](https://discuss.elastic.co/t/i-cannot-see-my-all-log-in-kibana/131555 "2018-05-16T08:28:19Z")
i have foldre full of log file and my logstash config is : input { file { path =\> "/home/adanic/kilid-logs/\*.log" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } output { elasticsearch { hosts =\> … --- ## [Logstash error when dictionary(CSV) has \\r or \\n at end of line](https://discuss.elastic.co/t/logstash-error-when-dictionary-csv-has-r-or-n-at-end-of-line/131703)
**Author:** [@Toni\_Riera](https://discuss.elastic.co/u/Toni_Riera)\ **Replies:** 3\ **Last updated:** [May 16, 2018, 8:25am UTC](https://discuss.elastic.co/t/logstash-error-when-dictionary-csv-has-r-or-n-at-end-of-line/131703 "2018-05-16T08:25:50Z")
Hello there, I'm developing a .conf related to security analysis with ELK Stack, and saw the article that Blueliv made on January '18. What i'm trying to do is load a csv file wich will serve as a dictionary to match ma… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=757) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=759)