# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=980 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 981 --- ## [Using Csv filter on a Split value](https://discuss.elastic.co/t/using-csv-filter-on-a-split-value/86094)
**Author:** [@sahar\_q](https://discuss.elastic.co/u/sahar_q)\ **Replies:** 1\ **Last updated:** [May 21, 2017, 12:01am UTC](https://discuss.elastic.co/t/using-csv-filter-on-a-split-value/86094 "2017-05-21T00:01:07Z")
I used mutate and split in order to split the data im getting, i want to pass each of the value i spilted on the csv filter but i dont get how to do it, any help? this is my current config: filter{ mutate { split =\> … --- ## [Loading logs, located in one machine into elasticsearch, setup in another machine, using logstash](https://discuss.elastic.co/t/loading-logs-located-in-one-machine-into-elasticsearch-setup-in-another-machine-using-logstash/86197)
**Author:** [@gautamv](https://discuss.elastic.co/u/gautamv)\ **Replies:** 1\ **Last updated:** [May 20, 2017, 11:41pm UTC](https://discuss.elastic.co/t/loading-logs-located-in-one-machine-into-elasticsearch-setup-in-another-machine-using-logstash/86197 "2017-05-20T23:41:54Z")
Hi, I have my logs and logstash running on the one EC2 machine (M1), so I read my logs placed on my local machine with this config: input { file{ path =\> "/path/to/logs/in/M1" start\_position =\> "… --- ## [Logstash filter for django logs not working](https://discuss.elastic.co/t/logstash-filter-for-django-logs-not-working/86209)
**Author:** [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\ **Replies:** 1\ **Last updated:** [May 20, 2017, 11:38pm UTC](https://discuss.elastic.co/t/logstash-filter-for-django-logs-not-working/86209 "2017-05-20T23:38:53Z")
filter { if\[type\] == "django" { grok { patterns\_dir =\> \["/opt/logstash/patterns/"\] match =\> \[ "message" , "%{LOGLEVEL:loglevel} %{DJANGOTIMESTAMP:timestamp},{INT:pid} %{WORD:origin} %{INT… --- ## [LogStash Out Filters](https://discuss.elastic.co/t/logstash-out-filters/86252)
**Author:** [@pfabrizi](https://discuss.elastic.co/u/pfabrizi)\ **Replies:** 1\ **Last updated:** [May 20, 2017, 11:37pm UTC](https://discuss.elastic.co/t/logstash-out-filters/86252 "2017-05-20T23:37:55Z")
I an mew to the ELK stack, but I have an understanding of the how it works. I was attempting to run a python script from my output using the exec command but when I do this I generate an error: Cannot create pipeline {:r… --- ## [Logstash load error](https://discuss.elastic.co/t/logstash-load-error/86363)
**Author:** [@mankeys](https://discuss.elastic.co/u/mankeys)\ **Replies:** 1\ **Last updated:** [May 20, 2017, 11:24pm UTC](https://discuss.elastic.co/t/logstash-load-error/86363 "2017-05-20T23:24:51Z")
I'm new to logstash and elasticsearch. I have installed elasticsearch and put data and search word, that's good. I have installed logstash on My windows7 PC and run. but failed there is logstash load error I have log… --- ## [Grok parse failure for django logs](https://discuss.elastic.co/t/grok-parse-failure-for-django-logs/86370)
**Author:** [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\ **Replies:** 1\ **Last updated:** [May 20, 2017, 11:24pm UTC](https://discuss.elastic.co/t/grok-parse-failure-for-django-logs/86370 "2017-05-20T23:24:27Z")
This is 1 of my log entries, INFO 2017-05-16 17:24:11,690 views 14463 139643033982720 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral\_code%3DP5E This is my pattern , DJANGOTIMESTAMP %{… --- ## [Error while reading CSV file from Amazon s3](https://discuss.elastic.co/t/error-while-reading-csv-file-from-amazon-s3/86502)
**Author:** [@asharma](https://discuss.elastic.co/u/asharma)\ **Replies:** 1\ **Last updated:** [May 20, 2017, 10:20pm UTC](https://discuss.elastic.co/t/error-while-reading-csv-file-from-amazon-s3/86502 "2017-05-20T22:20:10Z")
my configuration file looks something like this : input{ s3{ bucket =\> “s3://bucketaddress/tranlogfull.csv000” access\_key\_id =\> “\*\*\*\*\*\*\*\*\*\*\*” secret\_access\_key =\> “\*\*\*\*\*\*\*\*\*\*\*” region =\> “us-west-2” } } f… --- ## [Filtering rules in Logstash](https://discuss.elastic.co/t/filtering-rules-in-logstash/86463)
**Author:** [@arvindsing](https://discuss.elastic.co/u/arvindsing)\ **Replies:** 1\ **Last updated:** [May 20, 2017, 6:54pm UTC](https://discuss.elastic.co/t/filtering-rules-in-logstash/86463 "2017-05-20T18:54:19Z")
I have configured few filebeats machines and they all are sending logs ( whatever configured). For instance I have applied filtering for /var/log/auth.log. Now I am getting too much of data. I want to record only this ty… --- ## [Backdating @timestamp into ES data](https://discuss.elastic.co/t/backdating-timestamp-into-es-data/86531)
**Author:** [@chuchurocket27](https://discuss.elastic.co/u/chuchurocket27)\ **Replies:** 1\ **Last updated:** [May 20, 2017, 3:56pm UTC](https://discuss.elastic.co/t/backdating-timestamp-into-es-data/86531 "2017-05-20T15:56:16Z")
I have a SQL table that I plan on ingesting via Logstash into Elasticsearch. The entire table needs to be sucked in, but the date in which the data was originally generated is important to me - that is, I want the @times… --- ## [Logstash elasticsearch pipeline processing rate](https://discuss.elastic.co/t/logstash-elasticsearch-pipeline-processing-rate/84789)
**Author:** [@Sharat\_Jagannath](https://discuss.elastic.co/u/Sharat_Jagannath)\ **Replies:** 3\ **Last updated:** [May 20, 2017, 4:41am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-pipeline-processing-rate/84789 "2017-05-20T04:41:40Z")
i have a basic question on logstash elasticsearch pipeline processing rate. When i set flush\_size to a value, does logstash wait until that size is reached before it starts sending it to es? if the size is not reached w… --- ## [Logstash index keeps increasing all the time](https://discuss.elastic.co/t/logstash-index-keeps-increasing-all-the-time/86343)
**Author:** [@John\_06](https://discuss.elastic.co/u/John_06)\ **Replies:** 1\ **Last updated:** [May 20, 2017, 12:20am UTC](https://discuss.elastic.co/t/logstash-index-keeps-increasing-all-the-time/86343 "2017-05-20T00:20:48Z")
I am experiencing the following ES behavior: I copied a sample log file to a particular directory so Filebeat was able to read it and forward to Kafka. Then Logstash read that topic with logs data and sent it to ES. I … --- ## [New installation questions](https://discuss.elastic.co/t/new-installation-questions/85410)
**Author:** [@Diggy](https://discuss.elastic.co/u/Diggy)\ **Replies:** 1\ **Last updated:** [May 19, 2017, 11:00pm UTC](https://discuss.elastic.co/t/new-installation-questions/85410 "2017-05-19T23:00:55Z")
Hello, all. Let me apologize first, as this is a cross-post. I haven't gotten complete answers to my questions in the previous post, and this is extremely important to me. Hope you don't mind too much. Up to now, I'… --- ## [Grok filter](https://discuss.elastic.co/t/grok-filter/85444)
**Author:** [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)\ **Replies:** 5\ **Last updated:** [May 19, 2017, 7:03pm UTC](https://discuss.elastic.co/t/grok-filter/85444 "2017-05-19T19:03:08Z")
Hi , I am trying to construct grok filter to parse data (tomcat log ) but having issues . I am using input : \[2017-05-09 16:27:53,156\] :|: INFO :|: dfprdsndl22.df.jabodo.com :|: 6771ebd6eb814dc28e479908d542b6f… --- ## [Grok for - \[BT:CHROME, BV:55, BL:en, CC:CZ\]](https://discuss.elastic.co/t/grok-for-bt-chrome-bv-55-bl-en-cc-cz/85962)
**Author:** [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)\ **Replies:** 4\ **Last updated:** [May 19, 2017, 6:48pm UTC](https://discuss.elastic.co/t/grok-for-bt-chrome-bv-55-bl-en-cc-cz/85962 "2017-05-19T18:48:30Z")
Hi , i am trying to capture the while block in the grok along with the square brackets \[\] . The grok filter field should be displaying the data with the pattern inside it . Can someone please advise what should be grok f… --- ## [ERROR logstash.agent - Cannot load an invalid configuration](https://discuss.elastic.co/t/error-logstash-agent-cannot-load-an-invalid-configuration/86441)
**Author:** [@teejayuu](https://discuss.elastic.co/u/teejayuu)\ **Replies:** 2\ **Last updated:** [May 19, 2017, 3:34pm UTC](https://discuss.elastic.co/t/error-logstash-agent-cannot-load-an-invalid-configuration/86441 "2017-05-19T15:34:11Z")
Hi ELK installed on Windows 2012 R2 and trying to capture Exchange message tracking. Everything is in D:\\ELK and have ELK 5.3.2. Running the command: D:\\ELK\\logstash-5.3.2\\bin\\logstash -f D:\\ELK\\logstash-5.3.2\\01-in… --- ## [Logstash.conf location in Windows](https://discuss.elastic.co/t/logstash-conf-location-in-windows/86142)
**Author:** [@teejayuu](https://discuss.elastic.co/u/teejayuu)\ **Replies:** 5\ **Last updated:** [May 19, 2017, 3:26pm UTC](https://discuss.elastic.co/t/logstash-conf-location-in-windows/86142 "2017-05-19T15:26:55Z")
Hi, Have just started with ELK and following a couple of guides have got it working (I think). I am moving on to the next step of monitoring Exchange traffic and following this guide (it uses an older version) https:/… --- ## [Logstash jdbc - Exception Access Violation jvm.dll](https://discuss.elastic.co/t/logstash-jdbc-exception-access-violation-jvm-dll/86224)
**Author:** [@Alexis\_HONORAT](https://discuss.elastic.co/u/Alexis_HONORAT)\ **Replies:** 1\ **Last updated:** [May 19, 2017, 3:11pm UTC](https://discuss.elastic.co/t/logstash-jdbc-exception-access-violation-jvm-dll/86224 "2017-05-19T15:11:06Z")
Hello, I'm very new to all this Elasticsearch and logstash stuff. I'm trying to import my SQLServer Database to Elastic Search, and i don't understand well how Logstash work. I found that's i need to use the jdbc plug… --- ## [Small Help with gsub command to replace a special character with space](https://discuss.elastic.co/t/small-help-with-gsub-command-to-replace-a-special-character-with-space/86447)
**Author:** [@fathi\_rida](https://discuss.elastic.co/u/fathi_rida)\ **Replies:** 0\ **Last updated:** [May 19, 2017, 2:47pm UTC](https://discuss.elastic.co/t/small-help-with-gsub-command-to-replace-a-special-character-with-space/86447 "2017-05-19T14:47:02Z")
Hello, I have data as below, I like to replace "-" with space using gsub but I could not get correct output. data : 1111-4-1-12-00-11 i want new data Like So: data : 1111-4-1 12-00-11 Thanks in Advance --- ## [Parsing a keyword from message](https://discuss.elastic.co/t/parsing-a-keyword-from-message/86434)
**Author:** [@Bhargav\_Narine](https://discuss.elastic.co/u/Bhargav_Narine)\ **Replies:** 2\ **Last updated:** [May 19, 2017, 2:35pm UTC](https://discuss.elastic.co/t/parsing-a-keyword-from-message/86434 "2017-05-19T14:35:35Z")
Hi, I am new to ELK and trying to parse a specific keyword from message of the log entry and if that matches, creating a separate index in elasticsearch. My output config is like below output { if "ALARM" in \[logm… --- ## [Ruby exception occurred: undefined method \`+' for nil:NilClass](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-for-nil-nilclass/86437)
**Author:** [@Pierre\_Vincent\_Ledou](https://discuss.elastic.co/u/Pierre_Vincent_Ledou)\ **Replies:** 1\ **Last updated:** [May 19, 2017, 2:18pm UTC](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-for-nil-nilclass/86437 "2017-05-19T14:18:18Z")
I have the following ruby filter: mutate { convert =\> \["body\_size", "integer"\] convert =\> \["header\_size", "integer"\] } # Sum of body and header size ruby { code =\> "event.set('bytes', (event.get('body\_size') + ev… --- ## [Mutate logstash netflow codec variable name to user defined names](https://discuss.elastic.co/t/mutate-logstash-netflow-codec-variable-name-to-user-defined-names/86352)
**Author:** [@tharu85](https://discuss.elastic.co/u/tharu85)\ **Replies:** 1\ **Last updated:** [May 19, 2017, 2:09pm UTC](https://discuss.elastic.co/t/mutate-logstash-netflow-codec-variable-name-to-user-defined-names/86352 "2017-05-19T14:09:45Z")
I need to change default logstash netflow codec variable name to my own names . How can I do it. I have tried it as below. In the output, for stdout { codec =\> rubydebug } shows rename variables names, but gelf outp… --- ## [Grok does not work](https://discuss.elastic.co/t/grok-does-not-work/85995)
**Author:** [@Igor\_Gerasimow](https://discuss.elastic.co/u/Igor_Gerasimow)\ **Replies:** 10\ **Last updated:** [May 19, 2017, 1:37pm UTC](https://discuss.elastic.co/t/grok-does-not-work/85995 "2017-05-19T13:37:50Z")
Hello, i had such log row ( for example ) 10.128.0.23 - - \[16/May/2017:12:24:16 -0000\] "GET /pp/api/personalprofile/update\_status/na/Sweaty%20ASol HTTP/1.1" 101 361" - and I use such grok for parsing %{IP:clientip}… --- ## [Exessive field mapping Twitter to Elasticsearch](https://discuss.elastic.co/t/exessive-field-mapping-twitter-to-elasticsearch/86401)
**Author:** [@kubersakkie](https://discuss.elastic.co/u/kubersakkie)\ **Replies:** 3\ **Last updated:** [May 19, 2017, 1:12pm UTC](https://discuss.elastic.co/t/exessive-field-mapping-twitter-to-elasticsearch/86401 "2017-05-19T13:12:34Z")
Hey guys I am using the ElasticStack\_twitter configuration to ingest tweets. (ES, Logstash and Kibana are on version 5.2) I have applied the exact same config and templates from this commit. When I view the data… --- ## [Jdbc: record\_last\_run always saves 2017-01-01 as last run date](https://discuss.elastic.co/t/jdbc-record-last-run-always-saves-2017-01-01-as-last-run-date/85972)
**Author:** [@sivabalan19](https://discuss.elastic.co/u/sivabalan19)\ **Replies:** 3\ **Last updated:** [May 19, 2017, 12:45pm UTC](https://discuss.elastic.co/t/jdbc-record-last-run-always-saves-2017-01-01-as-last-run-date/85972 "2017-05-19T12:45:39Z")
Hi , I am using Logstash 5.4.0. Trying to pull all the new records that are in my Db after my last run. Below is my configuration. However, I am finding the content of the file mentioned in last\_run\_metadata\_path alwa… --- ## [Unable to push Logs on logstash to elasticsearch](https://discuss.elastic.co/t/unable-to-push-logs-on-logstash-to-elasticsearch/86409)
**Author:** [@gayathri](https://discuss.elastic.co/u/gayathri)\ **Replies:** 0\ **Last updated:** [May 19, 2017, 11:24am UTC](https://discuss.elastic.co/t/unable-to-push-logs-on-logstash-to-elasticsearch/86409 "2017-05-19T11:24:20Z")
hi Team, We are unable to push the data from logstash to Elasticsearch, getting below errors on debug: failed action with response of 404, dropping action: \["index", {:\_id=\>nil, :\_index=\>"logstash-2017.05.19", :\_type… --- ## [Handle multiple loglines as batch with condition](https://discuss.elastic.co/t/handle-multiple-loglines-as-batch-with-condition/86371)
**Author:** [@zorkin82](https://discuss.elastic.co/u/zorkin82)\ **Replies:** 1\ **Last updated:** [May 19, 2017, 10:32am UTC](https://discuss.elastic.co/t/handle-multiple-loglines-as-batch-with-condition/86371 "2017-05-19T10:32:04Z")
Hello together, I'm looking for a solution for the following problem: I have a logfile tat looks something like this: 2017-03-24T13:18:07,147 INFO \[topology-scan\_Worker-1\]: Executing \[PersistTopologyToZipCommand\] … --- ## [Logstash Kafka input, set interval/frequency](https://discuss.elastic.co/t/logstash-kafka-input-set-interval-frequency/86395)
**Author:** [@bkaathi](https://discuss.elastic.co/u/bkaathi)\ **Replies:** 1\ **Last updated:** [May 19, 2017, 10:10am UTC](https://discuss.elastic.co/t/logstash-kafka-input-set-interval-frequency/86395 "2017-05-19T10:10:13Z")
My setup is like FileBeat --\> Kafka --\> Logstash --\> Syslog Server Q: How to limit/interval frequency in input or output of Logstash. I'm using Logstash here to format the event logs to be sent to Syslog server. T… --- ## [File Discovery in Logstash](https://discuss.elastic.co/t/file-discovery-in-logstash/85747)
**Author:** [@thorsten1](https://discuss.elastic.co/u/thorsten1)\ **Replies:** 3\ **Last updated:** [May 19, 2017, 9:24am UTC](https://discuss.elastic.co/t/file-discovery-in-logstash/85747 "2017-05-19T09:24:02Z")
Hello, my Logstash Server has an Upload-Directory where other Users can upload their Log-Files. Logstash is configured to listen to all files in this directory. input { file { path =\> "/opt/logupload… --- ## [How to read topic from beginning in Kafka input](https://discuss.elastic.co/t/how-to-read-topic-from-beginning-in-kafka-input/86390)
**Author:** [@bkaathi](https://discuss.elastic.co/u/bkaathi)\ **Replies:** 0\ **Last updated:** [May 19, 2017, 9:16am UTC](https://discuss.elastic.co/t/how-to-read-topic-from-beginning-in-kafka-input/86390 "2017-05-19T09:16:26Z")
How can set Logstash Kafka input plugin to read topic data from the beginning? --- ## [ElasticSearch - Logstash multiple messages](https://discuss.elastic.co/t/elasticsearch-logstash-multiple-messages/36002)
**Author:** [@deseint](https://discuss.elastic.co/u/deseint)\ **Replies:** 1\ **Last updated:** [May 19, 2017, 7:26am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-multiple-messages/36002 "2017-05-19T07:26:42Z")
Hello, I'm new with logstash. I insert several files (doc and pdf format) into ES via logstash and it did well, but i'm desappointed by the field which name is message (fullfilled with the content of my file). In fact, e… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=979) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=981)