# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1129 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1130 --- ## [File output plugin permission denied](https://discuss.elastic.co/t/file-output-plugin-permission-denied/55211)
**Author:** [@asoong-94](https://discuss.elastic.co/u/asoong-94)\ **Replies:** 2\ **Last updated:** [July 11, 2016, 11:27pm UTC](https://discuss.elastic.co/t/file-output-plugin-permission-denied/55211 "2016-07-11T23:27:25Z")
This is my output configuration, right now i am just using grok to parse out IP and a WORD input {stdin {} } filter { grok { match =\> {"message" =\> "%{IP:user\_ip} %{WORD:computer\_name}"} } } output { file { … --- ## [Filters not working after split](https://discuss.elastic.co/t/filters-not-working-after-split/55242)
**Author:** [@Geezer](https://discuss.elastic.co/u/Geezer)\ **Replies:** 0\ **Last updated:** [July 11, 2016, 10:30pm UTC](https://discuss.elastic.co/t/filters-not-working-after-split/55242 "2016-07-11T22:30:06Z")
I have F5 appliances sending multiple events, so use a split filter to separate these before parsing them. The first grok filter I apply works, but any subsequent filters fail. This is the working filter, with the m… --- ## [How to split nested fields into separate events?](https://discuss.elastic.co/t/how-to-split-nested-fields-into-separate-events/55219)
**Author:** [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)\ **Replies:** 2\ **Last updated:** [July 11, 2016, 8:31pm UTC](https://discuss.elastic.co/t/how-to-split-nested-fields-into-separate-events/55219 "2016-07-11T20:31:59Z")
Example of incoming nested data below. Ultimately, I want logstash to submit 9 events for this single object. 1 of type storage 2 of type volume 3 of type share and 3 of type client I'm looking for something like the spl… --- ## [Use uuid or fingerprint for document\_id?](https://discuss.elastic.co/t/use-uuid-or-fingerprint-for-document-id/55225)
**Author:** [@djhart](https://discuss.elastic.co/u/djhart)\ **Replies:** 2\ **Last updated:** [July 11, 2016, 8:05pm UTC](https://discuss.elastic.co/t/use-uuid-or-fingerprint-for-document-id/55225 "2016-07-11T20:05:16Z")
Hello. I'm currently using a combination of a fingerprint field: fingerprint { key =\> “thisismykey“ method =\> "MD5" } And a timestamp to generate a unique id so that i can prevent duplicates if the sa… --- ## [Logstash 5.0.0-alpha4 - Can't install any plugins](https://discuss.elastic.co/t/logstash-5-0-0-alpha4-cant-install-any-plugins/54496)
**Author:** [@ceekay](https://discuss.elastic.co/u/ceekay)\ **Replies:** 3\ **Last updated:** [July 11, 2016, 6:05pm UTC](https://discuss.elastic.co/t/logstash-5-0-0-alpha4-cant-install-any-plugins/54496 "2016-07-11T18:05:38Z")
Having an issue with LS 5 alpha4: Trying to install any plugins throws an install error, resulting in a retry loop. I tried to do this first of all with a local .gem, however packaged plugins are also producing the sam… --- ## [Is logstash distributed real time computing system?](https://discuss.elastic.co/t/is-logstash-distributed-real-time-computing-system/55180)
**Author:** [@sen](https://discuss.elastic.co/u/sen)\ **Replies:** 6\ **Last updated:** [July 11, 2016, 2:50pm UTC](https://discuss.elastic.co/t/is-logstash-distributed-real-time-computing-system/55180 "2016-07-11T14:50:15Z")
Hi folks, The elastic website says that logstash can: Centralize data processing of all types Normalize varying schema and formats Quickly extend to custom log formats Easily add plugins for custom data sources … --- ## [Logstash 2.3.2 doesn't run with my Java?](https://discuss.elastic.co/t/logstash-2-3-2-doesnt-run-with-my-java/48566)
**Author:** [@Eric\_Amick](https://discuss.elastic.co/u/Eric_Amick)\ **Replies:** 8\ **Last updated:** [July 11, 2016, 1:27pm UTC](https://discuss.elastic.co/t/logstash-2-3-2-doesnt-run-with-my-java/48566 "2016-07-11T13:27:03Z")
When I try to run Logstash 2.3.2, I get the following message: \\Java\\jre1.8.0\_91 was unexpected at this time. 2.3.1 works without incident. --- ## [Logs filled with thousands of lines from PipelineReporter::Snapshot](https://discuss.elastic.co/t/logs-filled-with-thousands-of-lines-from-pipelinereporter-snapshot/49927)
**Author:** [@slackorama](https://discuss.elastic.co/u/slackorama)\ **Replies:** 1\ **Last updated:** [July 11, 2016, 11:51am UTC](https://discuss.elastic.co/t/logs-filled-with-thousands-of-lines-from-pipelinereporter-snapshot/49927 "2016-07-11T11:51:51Z")
Hi -- I recently upgraded to logstash 2.2.2 and at certain times my logs are getting filled up with lines like: {:timestamp=\>"2016-05-12T17:59:52.342000+0000", :message=\>#\ **Author:** [@gayathri](https://discuss.elastic.co/u/gayathri)\ **Replies:** 3\ **Last updated:** [July 11, 2016, 11:41am UTC](https://discuss.elastic.co/t/service-command-is-not-working-for-logstash/55183 "2016-07-11T11:41:19Z") Hi Team, "service logstash start" is not working for logstash, logstash.pid is creating after executing the command but logs are not pushing to elasticsearch On executing "/opt/logstash/bin/logstash agent -f /opt/lo… --- ## [Is it possible to remove/work http headers?](https://discuss.elastic.co/t/is-it-possible-to-remove-work-http-headers/55177)
**Author:** [@ruival](https://discuss.elastic.co/u/ruival)\ **Replies:** 0\ **Last updated:** [July 11, 2016, 10:43am UTC](https://discuss.elastic.co/t/is-it-possible-to-remove-work-http-headers/55177 "2016-07-11T10:43:56Z")
Hi all, we are investigating and building some proof of concept to validate the usage of ELK One use case is log events analyze and visualization. Each log event is one line in database table. For each event we sto… --- ## [Pipeline\_workers vs workers](https://discuss.elastic.co/t/pipeline-workers-vs-workers/55174)
**Author:** [@zpp](https://discuss.elastic.co/u/zpp)\ **Replies:** 0\ **Last updated:** [July 11, 2016, 10:04am UTC](https://discuss.elastic.co/t/pipeline-workers-vs-workers/55174 "2016-07-11T10:04:43Z")
Are pipeline\_workers and workers (elasticsearch output plugin) referring to the same setting? --- ## [Help me to convert String to TimeStamp](https://discuss.elastic.co/t/help-me-to-convert-string-to-timestamp/55155)
**Author:** [@Andrei\_Brajnicov](https://discuss.elastic.co/u/Andrei_Brajnicov)\ **Replies:** 1\ **Last updated:** [July 11, 2016, 6:58am UTC](https://discuss.elastic.co/t/help-me-to-convert-string-to-timestamp/55155 "2016-07-11T06:58:40Z")
Hello. I should convert my string field to the TImeStamp. These is the initial string field: "Jul 11, 2016 08:56:03.758207000" Now I'm converting it using the following rule: "date { match =\> \["Origin", "MMM dd, y… --- ## [Math operations with ruby](https://discuss.elastic.co/t/math-operations-with-ruby/55140)
**Author:** [@Mormaii](https://discuss.elastic.co/u/Mormaii)\ **Replies:** 2\ **Last updated:** [July 11, 2016, 1:12am UTC](https://discuss.elastic.co/t/math-operations-with-ruby/55140 "2016-07-11T01:12:00Z")
I want to multiply or divide two fields and create a new field based on that. I've got this ruby code: ruby { code =\> "event\['hit\_rate'\] = event\['MAIN\_client\_req'\] \* event\['MAIN\_cache\_hit'\]" } When I tr… --- ## [ES 2.2: index\_not\_found\_exception \[logstash-\*\]](https://discuss.elastic.co/t/es-2-2-index-not-found-exception-logstash/42929)
**Author:** [@alecswan](https://discuss.elastic.co/u/alecswan)\ **Replies:** 4\ **Last updated:** [July 10, 2016, 10:48pm UTC](https://discuss.elastic.co/t/es-2-2-index-not-found-exception-logstash/42929 "2016-07-10T22:48:10Z")
Hello, I am using ES 2.2/Shield 2.2 and for some reason it fails to acknowledge that logstash-\* indexes exist. Theses are the indexes I currently have: curl -u admin 'localhost:9200/\_cat/indices?v' Enter host passwo… --- ## [Error makes logstash to crash](https://discuss.elastic.co/t/error-makes-logstash-to-crash/55094)
**Author:** [@Adrian\_Gallego](https://discuss.elastic.co/u/Adrian_Gallego)\ **Replies:** 2\ **Last updated:** [July 10, 2016, 8:05pm UTC](https://discuss.elastic.co/t/error-makes-logstash-to-crash/55094 "2016-07-10T20:05:44Z")
Hi, Currently I'm using logstash with sns output plugin to send alerts to Amazon SNS service. I have found that logstash is crashing with the following error Seahorse::Client::NetworkingError: execution expired … --- ## [Index a text file into elasticsearch using logstash](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584)
**Author:** [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Replies:** 9\ **Last updated:** [July 10, 2016, 6:47pm UTC](https://discuss.elastic.co/t/index-a-text-file-into-elasticsearch-using-logstash/51584 "2016-07-10T18:47:32Z")
Is there a way to index a text file which has the data in the following format. Date Time : 23/05/2016-12.01.38.AM File : log Reason : parser is not working Stack : at java.util.package(Exception e) … --- ## [Input file section to read from latest folder](https://discuss.elastic.co/t/input-file-section-to-read-from-latest-folder/55131)
**Author:** [@Govind\_Raj](https://discuss.elastic.co/u/Govind_Raj)\ **Replies:** 2\ **Last updated:** [July 10, 2016, 4:52pm UTC](https://discuss.elastic.co/t/input-file-section-to-read-from-latest-folder/55131 "2016-07-10T16:52:22Z")
Hi we are looking for solution to read the file from the latest folder in the input path . our current configuration is as follows, input { file { path =\> "/var/log/orglogs/\*/G\*.log" start\_position =\> "end" … --- ## [Beat pipline blocked issues](https://discuss.elastic.co/t/beat-pipline-blocked-issues/55070)
**Author:** [@nanshan](https://discuss.elastic.co/u/nanshan)\ **Replies:** 2\ **Last updated:** [July 10, 2016, 3:52pm UTC](https://discuss.elastic.co/t/beat-pipline-blocked-issues/55070 "2016-07-10T15:52:22Z")
{:timestamp=\>"2016-07-08T16:42:35.906000+0000", :message=\>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect\_backoff\_sleep=\>0.5, :level=\>:warn} any configuration i need to make to so… --- ## [Applying grok on a field value after csv plugin](https://discuss.elastic.co/t/applying-grok-on-a-field-value-after-csv-plugin/55029)
**Author:** [@pvignesh92](https://discuss.elastic.co/u/pvignesh92)\ **Replies:** 5\ **Last updated:** [July 10, 2016, 8:31am UTC](https://discuss.elastic.co/t/applying-grok-on-a-field-value-after-csv-plugin/55029 "2016-07-10T08:31:34Z")
Hi team, I have a csv file which contains the output from Oracle tables. We are using csv filter to parse the data to appropriate fields. A field in them has a value which is a command getting executed. Ex. /home/run j… --- ## [Couchdb\_changes index only doc field from event](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223)
**Author:** [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)\ **Replies:** 31\ **Last updated:** [July 9, 2016, 6:41pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223 "2016-07-09T18:41:14Z")
Hello Logstash Community! I want to upgrade elasticsearch to newew version and replace couchdb\_river with couchdb\_changes plugin while working this input plugin create events with folowing fields \["doc","doc\_as\_upsert",… --- ## [Logstash filter failing to match all instances](https://discuss.elastic.co/t/logstash-filter-failing-to-match-all-instances/54824)
**Author:** [@fikse](https://discuss.elastic.co/u/fikse)\ **Replies:** 1\ **Last updated:** [July 8, 2016, 9:05pm UTC](https://discuss.elastic.co/t/logstash-filter-failing-to-match-all-instances/54824 "2016-07-08T21:05:08Z")
Some referrer fields are - and some are Direct Access. I've even tried deleting all the indexes and rerunning logstash, but I have the same problem. Any kind of help would be appreciated! Logstash conf file: input… --- ## [Default geoip bundled in logstash](https://discuss.elastic.co/t/default-geoip-bundled-in-logstash/55068)
**Author:** [@ctataryn](https://discuss.elastic.co/u/ctataryn)\ **Replies:** 1\ **Last updated:** [July 8, 2016, 8:53pm UTC](https://discuss.elastic.co/t/default-geoip-bundled-in-logstash/55068 "2016-07-08T20:53:03Z")
Hi, I installed logstash v2.3.4 on Ubuntu 14.04 via apt-get According to the documentation: Logstash releases ship with the GeoLiteCity database made available from Maxmind I do see a GeoLiteCity-2013-01-18.dat file… --- ## [Logs that are visible in stdout not showing up in es/kibana](https://discuss.elastic.co/t/logs-that-are-visible-in-stdout-not-showing-up-in-es-kibana/55090)
**Author:** [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\ **Replies:** 4\ **Last updated:** [July 8, 2016, 8:37pm UTC](https://discuss.elastic.co/t/logs-that-are-visible-in-stdout-not-showing-up-in-es-kibana/55090 "2016-07-08T20:37:40Z")
My output plugin is using BOTH stdout { codec =\> rubydebug } and an elasticsearch output connecting to our cluster, I do this sometimes to help me debug. When I run a test script to send a bunch of data to logstash, wi… --- ## [Multiline Codec stacking logs from different sources](https://discuss.elastic.co/t/multiline-codec-stacking-logs-from-different-sources/55062)
**Author:** [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\ **Replies:** 2\ **Last updated:** [July 8, 2016, 6:11pm UTC](https://discuss.elastic.co/t/multiline-codec-stacking-logs-from-different-sources/55062 "2016-07-08T18:11:59Z")
Hi there guys, We are using a logging framework in which a multitude of different servers write their logs to a kafka topic with two partitions, and then two instances of logstash on different VM's read from that topic… --- ## [\[indices:data/read/field\_stats\[s\]\]\]; nested: IllegalArgumentException\[field \[@timestamp\] doesn't exist\]](https://discuss.elastic.co/t/indices-data-read-field-stats-s-nested-illegalargumentexception-field-timestamp-doesnt-exist/52738)
**Author:** [@DreadPirateRob](https://discuss.elastic.co/u/DreadPirateRob)\ **Replies:** 14\ **Last updated:** [July 8, 2016, 4:04pm UTC](https://discuss.elastic.co/t/indices-data-read-field-stats-s-nested-illegalargumentexception-field-timestamp-doesnt-exist/52738 "2016-07-08T16:04:26Z")
First time user of ELK, originally I created an issue against elasticsearch on github, but it was suggested that I bring this issue to this forum, and so here we are. In my /var/log/elasticsearch/logstashTesting.log f… --- ## [Date filter not parsing this datetime](https://discuss.elastic.co/t/date-filter-not-parsing-this-datetime/55052)
**Author:** [@tmo](https://discuss.elastic.co/u/tmo)\ **Replies:** 2\ **Last updated:** [July 8, 2016, 2:10pm UTC](https://discuss.elastic.co/t/date-filter-not-parsing-this-datetime/55052 "2016-07-08T14:10:49Z")
It seems like this should work, but I'm getting a parsing error. Sample: 07/08/16-09:25:42.140797 date { match =\> \[ "datetime", "MM/dd/yy-HH:mm:ss.SSS" \] } What might I be doing wrong or how can I debug the e… --- ## [Logstash Input Plugin Java Implementation](https://discuss.elastic.co/t/logstash-input-plugin-java-implementation/55032)
**Author:** [@Ross\_Duncan](https://discuss.elastic.co/u/Ross_Duncan)\ **Replies:** 1\ **Last updated:** [July 8, 2016, 12:49pm UTC](https://discuss.elastic.co/t/logstash-input-plugin-java-implementation/55032 "2016-07-08T12:49:00Z")
Hello, I would like to write a custom Logstash Input plugin in java. I realise ruby/jruby is native in this instance, but Id rather not introduce new languages to my project. What is the simplest way to do this? --- ## [Keep only last record of each type](https://discuss.elastic.co/t/keep-only-last-record-of-each-type/54866)
**Author:** [@yodog](https://discuss.elastic.co/u/yodog)\ **Replies:** 4\ **Last updated:** [July 8, 2016, 11:48am UTC](https://discuss.elastic.co/t/keep-only-last-record-of-each-type/54866 "2016-07-08T11:48:45Z")
Hello all. I have to keep 2 types of indices on elasticsearch: 1- full, regular index holding all records sent to it (logstash-YYYY.MM.DD) 2- small, fast index holding only the last log message of each kind (lastacti… --- ## [Anonymize data based on output destination](https://discuss.elastic.co/t/anonymize-data-based-on-output-destination/54815)
**Author:** [@n3g](https://discuss.elastic.co/u/n3g)\ **Replies:** 2\ **Last updated:** [July 8, 2016, 7:14am UTC](https://discuss.elastic.co/t/anonymize-data-based-on-output-destination/54815 "2016-07-08T07:14:39Z")
Hi I currently have logstash outputting to 2 different ES clusters. Is it possible to anonymise some fields sent to one cluster, but not the other? (Using ES + LS 2.33) --- ## [File input fails after upgrading logstash from 2.3.3 to 5.0.0-alpha4](https://discuss.elastic.co/t/file-input-fails-after-upgrading-logstash-from-2-3-3-to-5-0-0-alpha4/55023)
**Author:** [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\ **Replies:** 0\ **Last updated:** [July 8, 2016, 7:05am UTC](https://discuss.elastic.co/t/file-input-fails-after-upgrading-logstash-from-2-3-3-to-5-0-0-alpha4/55023 "2016-07-08T07:05:01Z")
Hi! I would like ask question about the file input for ver 5.0.0-alpha4. Problem I just upgraded logstash from 2.3.3 to 5.0.0-alpha4 using yum update. However , file input for /var/log/message fails due to file perm… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1128) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1130)