# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1175 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1176 --- ## [Is redis really necessary in current version of ELK](https://discuss.elastic.co/t/is-redis-really-necessary-in-current-version-of-elk/45343)
**Author:** [@zhmq0124v](https://discuss.elastic.co/u/zhmq0124v)\ **Replies:** 2\ **Last updated:** [March 24, 2016, 6:25pm UTC](https://discuss.elastic.co/t/is-redis-really-necessary-in-current-version-of-elk/45343 "2016-03-24T18:25:49Z")
Dear ELK funs, I am new to ELK, had watched online videos from elastic.co and installed logstash 2.2.2 elasticsearch 2.2.1 locally on a standalone server. Learned from the videos, I know current version of logstash's … --- ## [Logstash mapping issue](https://discuss.elastic.co/t/logstash-mapping-issue/44598)
**Author:** [@VodoBaas1](https://discuss.elastic.co/u/VodoBaas1)\ **Replies:** 10\ **Last updated:** [March 24, 2016, 6:13pm UTC](https://discuss.elastic.co/t/logstash-mapping-issue/44598 "2016-03-24T18:13:34Z")
Hello, I hope i'm in the right category. I am using pf sense + ElasticSearch 2.2.0, Logstash 2.2.1, Kibana 4.4.1New to the site and new to ELK. I am having an a couple issues and I am not sure if this is a problem or… --- ## [How to handle errors with logstash?](https://discuss.elastic.co/t/how-to-handle-errors-with-logstash/44174)
**Author:** [@Mehmet\_Ozer](https://discuss.elastic.co/u/Mehmet_Ozer)\ **Replies:** 2\ **Last updated:** [March 24, 2016, 5:59pm UTC](https://discuss.elastic.co/t/how-to-handle-errors-with-logstash/44174 "2016-03-24T17:59:23Z")
Hello everyone, I just a have a question about error handling in Logstash. For example I have a cluster like; kafka -\> logstash -\> Elasticsearch and this logstash is also connected to a HDFS. so either kafk… --- ## [How to remove any field matching a string](https://discuss.elastic.co/t/how-to-remove-any-field-matching-a-string/45359)
**Author:** [@jclose](https://discuss.elastic.co/u/jclose)\ **Replies:** 1\ **Last updated:** [March 24, 2016, 5:02pm UTC](https://discuss.elastic.co/t/how-to-remove-any-field-matching-a-string/45359 "2016-03-24T17:02:30Z")
I have found the ruby fix for deleting events if they are 'nil', but I am wondering if there is a clean/easy way to delete any event if it matches one of a few strings. Essentially, I have some integer fields coming i… --- ## [Getting started with Logstash 2.2](https://discuss.elastic.co/t/getting-started-with-logstash-2-2/45350)
**Author:** [@chenbe2204](https://discuss.elastic.co/u/chenbe2204)\ **Replies:** 1\ **Last updated:** [March 24, 2016, 1:47pm UTC](https://discuss.elastic.co/t/getting-started-with-logstash-2-2/45350 "2016-03-24T13:47:54Z")
Hi , I worked with logstash-1.4.2 and now I'm upgrading to logstash 2.2. My conf file in version 1.4.2 is: input { file{ path =\> \["/tmp/31.json"\] type =\> "json" … --- ## [How to check logstash have listen snmptrap?](https://discuss.elastic.co/t/how-to-check-logstash-have-listen-snmptrap/44530)
**Author:** [@stefansaye](https://discuss.elastic.co/u/stefansaye)\ **Replies:** 4\ **Last updated:** [March 24, 2016, 1:04pm UTC](https://discuss.elastic.co/t/how-to-check-logstash-have-listen-snmptrap/44530 "2016-03-24T13:04:16Z")
I am a new to lostash and elastichsearch. I want to collection logs of network devices by snmptrap. I have a problem with logstash. +snmptrapd.conf authCommunity log,net public forward default localhost:1062 +logstas… --- ## [GREEDYDATA and newlines](https://discuss.elastic.co/t/greedydata-and-newlines/45268)
**Author:** [@CraigFoote](https://discuss.elastic.co/u/CraigFoote)\ **Replies:** 6\ **Last updated:** [March 24, 2016, 12:50pm UTC](https://discuss.elastic.co/t/greedydata-and-newlines/45268 "2016-03-24T12:50:40Z")
I have a log file from a java program coming from filebeat. Some of the events have stacktraces and so are multiline. I'm using the multiline option in filebeat and a grok filter in logstash to parse the event. Everythin… --- ## [Nested Json logs - how to filter correctly](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234)
**Author:** [@yuv14](https://discuss.elastic.co/u/yuv14)\ **Replies:** 11\ **Last updated:** [March 24, 2016, 12:23pm UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234 "2016-03-24T12:23:23Z")
Hi, I'm trying to input a complex Json log with nested fields. Here is my .conf file: input { file { path =\> /home/logstash-user/json-try.json" start\_position =\> "beginning" codec … --- ## [Relp zipped input](https://discuss.elastic.co/t/relp-zipped-input/45324)
**Author:** [@Luke3](https://discuss.elastic.co/u/Luke3)\ **Replies:** 0\ **Last updated:** [March 24, 2016, 9:40am UTC](https://discuss.elastic.co/t/relp-zipped-input/45324 "2016-03-24T09:40:31Z")
Hello everyone! I'm a newbie in logstash, for this reason I'm sorry if my question could be stupid or ridiculous. I'm using rsyslog for sending via relp my web server's logs to logstash ( it works! ) and also I was tryin… --- ## [Message\_format in otput "file"](https://discuss.elastic.co/t/message-format-in-otput-file/32006)
**Author:** [@neolithik](https://discuss.elastic.co/u/neolithik)\ **Replies:** 7\ **Last updated:** [March 24, 2016, 7:13am UTC](https://discuss.elastic.co/t/message-format-in-otput-file/32006 "2016-03-24T07:13:02Z")
hello, i have in input json file like this { "\_index": "null", "\_type": "logs", "\_id": "Wh7cLM6uT0mmSLnqOrWxQw", "\_score": null, "\_source": { "host": "null", "@timestamp": "2015-10-12T12:09:12.338Z", … --- ## [The configures is wrong?](https://discuss.elastic.co/t/the-configures-is-wrong/45213)
**Author:** [@thoma](https://discuss.elastic.co/u/thoma)\ **Replies:** 6\ **Last updated:** [March 24, 2016, 6:28am UTC](https://discuss.elastic.co/t/the-configures-is-wrong/45213 "2016-03-24T06:28:03Z")
my logstash-2.2.2 configure filter { if \[hostname\] == "192.168.1.107" and "ERROR" in \[message\] { mutate { add\_tag =\> \["error\_107"\] } metrics { meter =\> \["error\_107"\] add\_t… --- ## [Are there any ways to put eventlog plugin into Ubuntu](https://discuss.elastic.co/t/are-there-any-ways-to-put-eventlog-plugin-into-ubuntu/45299)
**Author:** [@Kennedy\_Kan](https://discuss.elastic.co/u/Kennedy_Kan)\ **Replies:** 1\ **Last updated:** [March 24, 2016, 4:36am UTC](https://discuss.elastic.co/t/are-there-any-ways-to-put-eventlog-plugin-into-ubuntu/45299 "2016-03-24T04:36:36Z")
I am new to EKL tools and I am installing it on Ubuntu 12.04. Are there any ways to put eventlog plugin into Ubuntu ? --- ## [Problem with received\_from not expanding](https://discuss.elastic.co/t/problem-with-received-from-not-expanding/45276)
**Author:** [@jcurbo](https://discuss.elastic.co/u/jcurbo)\ **Replies:** 3\ **Last updated:** [March 24, 2016, 4:32am UTC](https://discuss.elastic.co/t/problem-with-received-from-not-expanding/45276 "2016-03-24T04:32:49Z")
I just set up my first ELK cluster and I'm having a strange issue I can't track down. I have logstash-forwarder running on the ELK host itself (set it up before I knew filebeat was a thing), and I have an external host… --- ## [Logstash dies after about 8 hours](https://discuss.elastic.co/t/logstash-dies-after-about-8-hours/44931)
**Author:** [@snipervelli](https://discuss.elastic.co/u/snipervelli)\ **Replies:** 6\ **Last updated:** [March 24, 2016, 1:37am UTC](https://discuss.elastic.co/t/logstash-dies-after-about-8-hours/44931 "2016-03-24T01:37:21Z")
Hi everyone, I am new to ELK and i am running ELK on a centos 7 VM with about 16GB of Mem and 6virtual sockets with 2 per core. Logstash 2.2.2 Elasticsearch 2.2.1 and Kibana 4.4.2 are all on the latest versions. Everyt… --- ## [Create aggregated traffic stats at index time - metrics?](https://discuss.elastic.co/t/create-aggregated-traffic-stats-at-index-time-metrics/45291)
**Author:** [@tomr](https://discuss.elastic.co/u/tomr)\ **Replies:** 0\ **Last updated:** [March 24, 2016, 12:29am UTC](https://discuss.elastic.co/t/create-aggregated-traffic-stats-at-index-time-metrics/45291 "2016-03-24T00:29:22Z")
We have around 50m edge log events per day going into ES, with several billions of records stored historically. Events relate to any one of a few hundred different HTTP Host headers, and we routinely restrict searches b… --- ## [Turn messages into fields - querying in kibana](https://discuss.elastic.co/t/turn-messages-into-fields-querying-in-kibana/45259)
**Author:** [@bishaka](https://discuss.elastic.co/u/bishaka)\ **Replies:** 7\ **Last updated:** [March 23, 2016, 8:20pm UTC](https://discuss.elastic.co/t/turn-messages-into-fields-querying-in-kibana/45259 "2016-03-23T20:20:29Z")
Hello, I am trying to analyze some access and report logs in kibana..I want to be able to query on the message I receive on kibana...Basically I want to turn the messages onto fields to be able to query on them. How can… --- ## [Logstash JMS Input Plugin](https://discuss.elastic.co/t/logstash-jms-input-plugin/45035)
**Author:** [@parthu](https://discuss.elastic.co/u/parthu)\ **Replies:** 5\ **Last updated:** [March 23, 2016, 6:16pm UTC](https://discuss.elastic.co/t/logstash-jms-input-plugin/45035 "2016-03-23T18:16:00Z")
Does Logstash-Input-Jms plugin need a specific version of Logstash? I am using Logstash 1.4.2. And I'm struggling to get this plugin to work. Can anyone help me with the configuration? I copied jms.rb into /lib/logst… --- ## [Logstash.stdout](https://discuss.elastic.co/t/logstash-stdout/45152)
**Author:** [@pierrejnelson](https://discuss.elastic.co/u/pierrejnelson)\ **Replies:** 6\ **Last updated:** [March 23, 2016, 1:51pm UTC](https://discuss.elastic.co/t/logstash-stdout/45152 "2016-03-23T13:51:03Z")
Would anyone happen to know what makes this file. I need to move it to where I have all my space but I can't seem to find it. --- ## [Logstash - file input plugin with huge number of files](https://discuss.elastic.co/t/logstash-file-input-plugin-with-huge-number-of-files/45105)
**Author:** [@Chandan\_PR](https://discuss.elastic.co/u/Chandan_PR)\ **Replies:** 3\ **Last updated:** [March 23, 2016, 1:34pm UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-with-huge-number-of-files/45105 "2016-03-23T13:34:48Z")
Hi, I need to dump logs from one of our application to elastic search. The application generates a unique log file for each request. Once the request is completed, this log file will be never updated again. We generate… --- ## [Logstash Mutate replace field with value](https://discuss.elastic.co/t/logstash-mutate-replace-field-with-value/45086)
**Author:** [@manthan\_patel](https://discuss.elastic.co/u/manthan_patel)\ **Replies:** 7\ **Last updated:** [March 23, 2016, 11:48am UTC](https://discuss.elastic.co/t/logstash-mutate-replace-field-with-value/45086 "2016-03-23T11:48:37Z")
Hi I am trying to replace field "component " but not showing in output what exactly i am doing wrong . I am new to ELK. below is my logstash config. input { tcp { 'port' =\> '9563' } } filter { json { source =\> 'messa… --- ## [Flush\_Size & max\_retry\_item count metrics!](https://discuss.elastic.co/t/flush-size-max-retry-item-count-metrics/28093)
**Author:** [@sreeram](https://discuss.elastic.co/u/sreeram)\ **Replies:** 3\ **Last updated:** [March 23, 2016, 10:18am UTC](https://discuss.elastic.co/t/flush-size-max-retry-item-count-metrics/28093 "2016-03-23T10:18:25Z")
I see flush\_Size & max\_retry\_item default count as 5000. what this 5000 means? 5000 bytes? 5000 log count ? --- ## [Renaming fileds in nested json](https://discuss.elastic.co/t/renaming-fileds-in-nested-json/45189)
**Author:** [@bongo\_zg](https://discuss.elastic.co/u/bongo_zg)\ **Replies:** 1\ **Last updated:** [March 23, 2016, 10:09am UTC](https://discuss.elastic.co/t/renaming-fileds-in-nested-json/45189 "2016-03-23T10:09:02Z")
I have a json like: {"doc\_id": 1, "name": "John", "traveller": { "field\_1": "Johnny" }} and would like to have it loaded like this: doc\_id =\> doc\_id name - ignored traveller.field\_1 =\> name\_1 I wrote a config fil… --- ## [Trouble replacing a "||"](https://discuss.elastic.co/t/trouble-replacing-a/45202)
**Author:** [@noemie](https://discuss.elastic.co/u/noemie)\ **Replies:** 0\ **Last updated:** [March 23, 2016, 10:00am UTC](https://discuss.elastic.co/t/trouble-replacing-a/45202 "2016-03-23T10:00:55Z")
Hello, in some of my log lines, I have two pipes following each other but between is supposed to be a field. Processing this particularity with my grok pattern doesn't do the trick. I've tried the following: gsub =\> \["… --- ## [Logstash to ES with postgres](https://discuss.elastic.co/t/logstash-to-es-with-postgres/45080)
**Author:** [@Jpuch](https://discuss.elastic.co/u/Jpuch)\ **Replies:** 4\ **Last updated:** [March 23, 2016, 9:22am UTC](https://discuss.elastic.co/t/logstash-to-es-with-postgres/45080 "2016-03-23T09:22:46Z")
Hi there, I'm trying to set up an ELK system to have a better overview over some data. I have to flash code scanne which fill up a PostgreSQL database (Date - Hour - ID ect) So i installed ELK follywing a random tuto… --- ## [Joint database table and an array](https://discuss.elastic.co/t/joint-database-table-and-an-array/44839)
**Author:** [@Croos](https://discuss.elastic.co/u/Croos)\ **Replies:** 5\ **Last updated:** [March 23, 2016, 8:48am UTC](https://discuss.elastic.co/t/joint-database-table-and-an-array/44839 "2016-03-23T08:48:56Z")
I have a database table and a document array +---------+--------+ | m\_id | state | +---------+--------+ | m1 | s1 | | m2 | s2 | | m3 | s1 | +---------+--------+ state\_array = array(s1=\>'… --- ## [Sizing for udp/tcp (syslog type)](https://discuss.elastic.co/t/sizing-for-udp-tcp-syslog-type/44528)
**Author:** [@miko370](https://discuss.elastic.co/u/miko370)\ **Replies:** 10\ **Last updated:** [March 23, 2016, 7:47am UTC](https://discuss.elastic.co/t/sizing-for-udp-tcp-syslog-type/44528 "2016-03-23T07:47:28Z")
Hi. Does anyone have any sizing knowledge and experience using tcp/udp input plugin for syslog messages? --- ## [Data Recovery](https://discuss.elastic.co/t/data-recovery/45175)
**Author:** [@piyush](https://discuss.elastic.co/u/piyush)\ **Replies:** 3\ **Last updated:** [March 23, 2016, 5:18am UTC](https://discuss.elastic.co/t/data-recovery/45175 "2016-03-23T05:18:36Z")
Hi Team, My ES server got corrupted and i have to create a new one, no backup of previous ES server. But logstash server is as it is, can i recover/ship older logs to my new ES server, how? Regards... --- ## [Shipping logs to a remote server](https://discuss.elastic.co/t/shipping-logs-to-a-remote-server/44864)
**Author:** [@bishaka](https://discuss.elastic.co/u/bishaka)\ **Replies:** 7\ **Last updated:** [March 23, 2016, 1:23am UTC](https://discuss.elastic.co/t/shipping-logs-to-a-remote-server/44864 "2016-03-23T01:23:28Z")
How can I ship logs to another remote server and display on kibana dashboard? I tried to install logstash on the server i want to ship logs from and sent to the server I want to receive the logs on though redis queue..… --- ## [How to get around logstash s3 input prefix not able to do recursive paths?](https://discuss.elastic.co/t/how-to-get-around-logstash-s3-input-prefix-not-able-to-do-recursive-paths/45160)
**Author:** [@Matthew\_Prinvale](https://discuss.elastic.co/u/Matthew_Prinvale)\ **Replies:** 0\ **Last updated:** [March 22, 2016, 9:17pm UTC](https://discuss.elastic.co/t/how-to-get-around-logstash-s3-input-prefix-not-able-to-do-recursive-paths/45160 "2016-03-22T21:17:01Z")
edit: I had the wrong info. Not sure how to delete/close thread. --- ## [Remove raw fields from geoip?](https://discuss.elastic.co/t/remove-raw-fields-from-geoip/44705)
**Author:** [@AlexH](https://discuss.elastic.co/u/AlexH)\ **Replies:** 4\ **Last updated:** [March 22, 2016, 11:29pm UTC](https://discuss.elastic.co/t/remove-raw-fields-from-geoip/44705 "2016-03-22T23:29:56Z")
I am using Elasticsearch for log analysis, and I am trying to increase performance, reduce storage as much as possible. I have turned off dynamic mapping, added the not\_analyzed strings explicitly where needed, as well a… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1174) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1176)