# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1242 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1243 --- ## [Upgrade procedure from 1.4.2.to 1.5](https://discuss.elastic.co/t/upgrade-procedure-from-1-4-2-to-1-5/745)
**Author:** [@Chris\_Adams](https://discuss.elastic.co/u/Chris_Adams)\ **Replies:** 12\ **Last updated:** [August 24, 2015, 5:52pm UTC](https://discuss.elastic.co/t/upgrade-procedure-from-1-4-2-to-1-5/745 "2015-08-24T17:52:29Z")
I am currently testing an ELK stack based on 1.4.2 - what steps should I take to upgrade. Isn't this a great FAQ question? \[smile\] --- ## [Timestamp and @timestamp problem with a big gap](https://discuss.elastic.co/t/timestamp-and-timestamp-problem-with-a-big-gap/27948)
**Author:** [@michaelzoet](https://discuss.elastic.co/u/michaelzoet)\ **Replies:** 2\ **Last updated:** [August 24, 2015, 2:46pm UTC](https://discuss.elastic.co/t/timestamp-and-timestamp-problem-with-a-big-gap/27948 "2015-08-24T14:46:36Z")
Hello all, I have a Logstash, Elasticsearch, Kibana, Logstash-forwarder setup running. Now I found a problem with the timestamp and @timestamp mismatch on some logfile types. I have no issue with syslog, Apache and Pos… --- ## [\[Resolved\] Grok filter discard entities](https://discuss.elastic.co/t/resolved-grok-filter-discard-entities/27862)
**Author:** [@rishi](https://discuss.elastic.co/u/rishi)\ **Replies:** 2\ **Last updated:** [August 24, 2015, 1:36pm UTC](https://discuss.elastic.co/t/resolved-grok-filter-discard-entities/27862 "2015-08-24T13:36:07Z")
How can I exclude / discard those events which the grok filter fails to parse? I would like to just discard such events instead of crashing the service. --- ## [ELK stack and additional CloudWatch output](https://discuss.elastic.co/t/elk-stack-and-additional-cloudwatch-output/26892)
**Author:** [@sebastianhoitz](https://discuss.elastic.co/u/sebastianhoitz)\ **Replies:** 4\ **Last updated:** [August 24, 2015, 1:28pm UTC](https://discuss.elastic.co/t/elk-stack-and-additional-cloudwatch-output/26892 "2015-08-24T13:28:53Z")
I would like to output all my logs to Elastic Search and when certain conditions are met, send them as metrics to Cloud Watch, which should take care of alerting me. They way I understand it is, that I have to add a CW… --- ## [Require-Gem in Ruby Filter fails](https://discuss.elastic.co/t/require-gem-in-ruby-filter-fails/27942)
**Author:** [@chlehmann](https://discuss.elastic.co/u/chlehmann)\ **Replies:** 0\ **Last updated:** [August 24, 2015, 11:46am UTC](https://discuss.elastic.co/t/require-gem-in-ruby-filter-fails/27942 "2015-08-24T11:46:41Z")
Hi there I have set-up my ELK-Stack on my Macbook and its working good. I am now trying to improve the pre-processing of my logs in Logstash; I try to count Emoji's with the ruby-gem "emoji\_data". If i create a ruby-filt… --- ## [Receive logs from Pfsense but noting show in charts](https://discuss.elastic.co/t/receive-logs-from-pfsense-but-noting-show-in-charts/27939)
**Author:** [@mega\_robo](https://discuss.elastic.co/u/mega_robo)\ **Replies:** 2\ **Last updated:** [August 24, 2015, 11:43am UTC](https://discuss.elastic.co/t/receive-logs-from-pfsense-but-noting-show-in-charts/27939 "2015-08-24T11:43:08Z")
hi every one i install ELK with elasticsearch and logstash 1.5 and kibana 3 and use pfsense 2.2.3 its work correctly and send logst to logstash and i can see logs in kibana and in /var/log/logstash/logstash.stutd but i… --- ## [How to use logstash-input-plugin for sequel unsupported DB](https://discuss.elastic.co/t/how-to-use-logstash-input-plugin-for-sequel-unsupported-db/27931)
**Author:** [@kikuchan](https://discuss.elastic.co/u/kikuchan)\ **Replies:** 0\ **Last updated:** [August 24, 2015, 9:20am UTC](https://discuss.elastic.co/t/how-to-use-logstash-input-plugin-for-sequel-unsupported-db/27931 "2015-08-24T09:20:22Z")
hi. I have to use HiRDB(RDB),but I could not connect HiRDB by logstash-input-plugin. When I tried to connect HiRDB by logstash-input-plugin, I used the following config file. ・config file input { jdbc { … --- ## [Problem with send log from pfsense 2.2 to ELK](https://discuss.elastic.co/t/problem-with-send-log-from-pfsense-2-2-to-elk/27892)
**Author:** [@mega\_robo](https://discuss.elastic.co/u/mega_robo)\ **Replies:** 4\ **Last updated:** [August 24, 2015, 7:34am UTC](https://discuss.elastic.co/t/problem-with-send-log-from-pfsense-2-2-to-elk/27892 "2015-08-24T07:34:18Z")
hi i install ELK with elasticsearch 1.7.1 and logstash 1.5.4 and kibana 3 and try to send my firewal logs to ELK i use pfsense 2.2.3 and i config all but have difrent error in start logstash and i got this damn error man… --- ## [Twitter input for mentioned Hashtag](https://discuss.elastic.co/t/twitter-input-for-mentioned-hashtag/27825)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 6\ **Last updated:** [August 24, 2015, 6:02am UTC](https://discuss.elastic.co/t/twitter-input-for-mentioned-hashtag/27825 "2015-08-24T06:02:37Z")
HI All, i am using twitter lagstash input configuration to pull data from twitter. i used below configuration file to pull data. input { twitter { consumer\_key =\> "XXXXXXXXXXXXXXX" consume… --- ## [Logstash\[ version-1.5.4\] stopped if I insert lumberjack input in configuration](https://discuss.elastic.co/t/logstash-version-1-5-4-stopped-if-i-insert-lumberjack-input-in-configuration/27872)
**Author:** [@gugansankar](https://discuss.elastic.co/u/gugansankar)\ **Replies:** 1\ **Last updated:** [August 24, 2015, 2:20am UTC](https://discuss.elastic.co/t/logstash-version-1-5-4-stopped-if-i-insert-lumberjack-input-in-configuration/27872 "2015-08-24T02:20:22Z")
Not sure whether I can post this issue here. as I have posted the same on github. Problem : When I insert lumberjack configuration in logstash-1.5.4 , its getting stopped with this error message " The error reported i… --- ## [Any gotchas upgrading from 1.4.\* to 1.5.\*?](https://discuss.elastic.co/t/any-gotchas-upgrading-from-1-4-to-1-5/27898)
**Author:** [@DigiAngel](https://discuss.elastic.co/u/DigiAngel)\ **Replies:** 2\ **Last updated:** [August 24, 2015, 12:12am UTC](https://discuss.elastic.co/t/any-gotchas-upgrading-from-1-4-to-1-5/27898 "2015-08-24T00:12:16Z")
Not expecting any, but you never know....currently running 1.4.5. Thank you. --- ## [Using KV with JSON](https://discuss.elastic.co/t/using-kv-with-json/27907)
**Author:** [@coderlol](https://discuss.elastic.co/u/coderlol)\ **Replies:** 6\ **Last updated:** [August 23, 2015, 10:11pm UTC](https://discuss.elastic.co/t/using-kv-with-json/27907 "2015-08-23T22:11:12Z")
My app generates the log entries in the following format. Logstash Forwarder ships the logs to Logstash Indexer which in turn pushes the logs into Elasticsearch. I'm using LSF 0.4.0, Logstash 1.5.3, ES 1.7.1 {"@vers… --- ## [Optimize Transport App to Logstash Forwarder to LogStash Indexer to Elasticsearch](https://discuss.elastic.co/t/optimize-transport-app-to-logstash-forwarder-to-logstash-indexer-to-elasticsearch/27906)
**Author:** [@coderlol](https://discuss.elastic.co/u/coderlol)\ **Replies:** 1\ **Last updated:** [August 23, 2015, 6:04pm UTC](https://discuss.elastic.co/t/optimize-transport-app-to-logstash-forwarder-to-logstash-indexer-to-elasticsearch/27906 "2015-08-23T18:04:20Z")
My app generates JSON the following in log files: {"@version":"1","@timestamp":"2015-08-23T17:37:33.235Z","thread":"098658","level":"INFO","marker":{"name":"LOGGABLE"},"message":"someid=PMCtuW9.6sKDC,result=blabla"} T… --- ## [Csvfile Plugin](https://discuss.elastic.co/t/csvfile-plugin/27761)
**Author:** [@jweite](https://discuss.elastic.co/u/jweite)\ **Replies:** 2\ **Last updated:** [August 22, 2015, 1:30pm UTC](https://discuss.elastic.co/t/csvfile-plugin/27761 "2015-08-22T13:30:11Z")
I need to process CSV files with first line schemas (that can vary from file-to-file) and after reading a bit see that improved CSV handling is on the "to do" list. I got to work and coded up a subclass of logstash-input… --- ## [Config file for importing data from multiple couch databases](https://discuss.elastic.co/t/config-file-for-importing-data-from-multiple-couch-databases/27846)
**Author:** [@imad](https://discuss.elastic.co/u/imad)\ **Replies:** 1\ **Last updated:** [August 22, 2015, 8:32am UTC](https://discuss.elastic.co/t/config-file-for-importing-data-from-multiple-couch-databases/27846 "2015-08-22T08:32:04Z")
We are storing our data types in each of its database in couchdb. What sort of format will the config file have to import data from multiple databases? Or do I need to have multiple config files for importing data from e… --- ## [Logstash Output Not Working For File Input](https://discuss.elastic.co/t/logstash-output-not-working-for-file-input/27618)
**Author:** [@rholloway](https://discuss.elastic.co/u/rholloway)\ **Replies:** 5\ **Last updated:** [August 21, 2015, 10:40pm UTC](https://discuss.elastic.co/t/logstash-output-not-working-for-file-input/27618 "2015-08-21T22:40:02Z")
Been poking around for the past few hours trying to get this working but no luck. I have a simple file input and simple stdout output (alongside a redis output, but let's ignore for now). I'll toss out that I've alread… --- ## [How can I check if the "tags" array is empty?](https://discuss.elastic.co/t/how-can-i-check-if-the-tags-array-is-empty/27856)
**Author:** [@spynxo](https://discuss.elastic.co/u/spynxo)\ **Replies:** 2\ **Last updated:** [August 21, 2015, 8:09pm UTC](https://discuss.elastic.co/t/how-can-i-check-if-the-tags-array-is-empty/27856 "2015-08-21T20:09:39Z")
Hi All, How can I check if the "tags" array has any elements? I am adding and removing tags in my pipeline. At the end there are some messages that end up with an empty tags array ("tags" =\> \[\]). I will like to check i… --- ## [REDIS input stops working?](https://discuss.elastic.co/t/redis-input-stops-working/27834)
**Author:** [@slee](https://discuss.elastic.co/u/slee)\ **Replies:** 3\ **Last updated:** [August 21, 2015, 7:54pm UTC](https://discuss.elastic.co/t/redis-input-stops-working/27834 "2015-08-21T19:54:36Z")
Hello, We currently have a server with redis installed, and several logstash instances. Half of these instances listen on a port and output to redis, and half of them input from redis, filter, and output to elasticsearc… --- ## [Error running logstash-input-couchdb\_changes plugin](https://discuss.elastic.co/t/error-running-logstash-input-couchdb-changes-plugin/27794)
**Author:** [@imad](https://discuss.elastic.co/u/imad)\ **Replies:** 2\ **Last updated:** [August 21, 2015, 7:06pm UTC](https://discuss.elastic.co/t/error-running-logstash-input-couchdb-changes-plugin/27794 "2015-08-21T19:06:01Z")
Hi, I am having problem running this plugin. I downloaded logstash and logstash-input-couchdb\_changes plugin, unzipped those and then followed 1st two steps in the sections below. I am running windows 7. I didn't insta… --- ## [Help with parsing XML content](https://discuss.elastic.co/t/help-with-parsing-xml-content/27445)
**Author:** [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\ **Replies:** 15\ **Last updated:** [August 21, 2015, 7:04pm UTC](https://discuss.elastic.co/t/help-with-parsing-xml-content/27445 "2015-08-21T19:04:12Z")
I have a log that has well-formed XML in it. I would like to parse out and extract several of the elements from that XML and publish them as fields in the /sprint/slic, my index/type pair. Here is the data (two separ… --- ## [Picking data out of logs with Logstash](https://discuss.elastic.co/t/picking-data-out-of-logs-with-logstash/27804)
**Author:** [@meggiel](https://discuss.elastic.co/u/meggiel)\ **Replies:** 2\ **Last updated:** [August 21, 2015, 5:57pm UTC](https://discuss.elastic.co/t/picking-data-out-of-logs-with-logstash/27804 "2015-08-21T17:57:14Z")
Hi, I'm pretty new to Logstash, and I'm trying to evaluate it to see if it will do what I'd like to do. I'm starting with logs that are partially log4j, but also include lines that aren't log4j, so a straight log4j f… --- ## [Two Logstash instances on same Docker container](https://discuss.elastic.co/t/two-logstash-instances-on-same-docker-container/27852)
**Author:** [@Piyush\_Mattoo](https://discuss.elastic.co/u/Piyush_Mattoo)\ **Replies:** 1\ **Last updated:** [August 21, 2015, 5:21pm UTC](https://discuss.elastic.co/t/two-logstash-instances-on-same-docker-container/27852 "2015-08-21T17:21:18Z")
Am wondering if there is a way two logstash processes with separate configurations can be run on a single Docker container. My setup has a Logstash process using file as input sending events to Redis and from there to se… --- ## [Faster speed when indexing log files](https://discuss.elastic.co/t/faster-speed-when-indexing-log-files/27827)
**Author:** [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Replies:** 10\ **Last updated:** [August 21, 2015, 2:40pm UTC](https://discuss.elastic.co/t/faster-speed-when-indexing-log-files/27827 "2015-08-21T14:40:25Z")
Hi! Right now I'm indexing about 15 log files with a total of 2 million log events all together. This takes a very long time and I am wondering if there is a way to speed up the process? Best regards Simon --- ## [Getting Json from a CSV](https://discuss.elastic.co/t/getting-json-from-a-csv/27800)
**Author:** [@rnigam](https://discuss.elastic.co/u/rnigam)\ **Replies:** 4\ **Last updated:** [August 21, 2015, 10:48am UTC](https://discuss.elastic.co/t/getting-json-from-a-csv/27800 "2015-08-21T10:48:43Z")
Hi, I have a usecase where I have CSV files with a few columns with JSON values. How can I take the get the JSON column and store it in elasticsearch? Thanks a lot! Much appreciate everyone's help. Regards, Raja… --- ## [How make multi shipping without duplicates](https://discuss.elastic.co/t/how-make-multi-shipping-without-duplicates/27583)
**Author:** [@inceptor](https://discuss.elastic.co/u/inceptor)\ **Replies:** 3\ **Last updated:** [August 21, 2015, 9:39am UTC](https://discuss.elastic.co/t/how-make-multi-shipping-without-duplicates/27583 "2015-08-21T09:39:02Z")
Hello everybody, I was try it some examples (architectural) from the documentation : https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html#deploying-scaling But I have now a problem : The multi … --- ## [Aggregates all lines messages between 2 date using multiline filter](https://discuss.elastic.co/t/aggregates-all-lines-messages-between-2-date-using-multiline-filter/27823)
**Author:** [@MichaelScofield](https://discuss.elastic.co/u/MichaelScofield)\ **Replies:** 1\ **Last updated:** [August 21, 2015, 9:38am UTC](https://discuss.elastic.co/t/aggregates-all-lines-messages-between-2-date-using-multiline-filter/27823 "2015-08-21T09:38:35Z")
I have a log like this: 2015-06-26 23:00:48,935 foo as "date1" whatever lines not start with date ... 2015-06-16 18:46:16,136 bar as "date2" I'm trying to get all lines between date… --- ## [Moving from single redis instances to clustered caching pool](https://discuss.elastic.co/t/moving-from-single-redis-instances-to-clustered-caching-pool/27713)
**Author:** [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Replies:** 5\ **Last updated:** [August 21, 2015, 7:36am UTC](https://discuss.elastic.co/t/moving-from-single-redis-instances-to-clustered-caching-pool/27713 "2015-08-21T07:36:24Z")
We currently run single instance of redis to cache logs before sending to Elasticsearch. We want to move to a cluster to avoid single point of failure. At this point it feels like we need to move away from Redis due to t… --- ## [Sprintf format doesn't work in clone filter](https://discuss.elastic.co/t/sprintf-format-doesnt-work-in-clone-filter/27815)
**Author:** [@dtrv](https://discuss.elastic.co/u/dtrv)\ **Replies:** 0\ **Last updated:** [August 21, 2015, 6:51am UTC](https://discuss.elastic.co/t/sprintf-format-doesnt-work-in-clone-filter/27815 "2015-08-21T06:51:55Z")
Hi. To keep the type of an event when it is cloned, I wantet to use clone { clones =\> \[ "%{type}\_cloned" \] } and end up with an type %{type}\_cloned instead of my\_type\_cloned. Is this a bug or a feature? You ca… --- ## [Emails from logstash are sometimes delayed](https://discuss.elastic.co/t/emails-from-logstash-are-sometimes-delayed/27751)
**Author:** [@angelsmile](https://discuss.elastic.co/u/angelsmile)\ **Replies:** 8\ **Last updated:** [August 21, 2015, 1:37am UTC](https://discuss.elastic.co/t/emails-from-logstash-are-sometimes-delayed/27751 "2015-08-21T01:37:46Z")
Hi, I'm a new user of ELK. I have set up logstash to send out emails upon some syslog messages received from remote node. However, the emails sent from logstash are sometimes delayed for a long time (minutes to hours).… --- ## [Logstash-input-jdbc install issues](https://discuss.elastic.co/t/logstash-input-jdbc-install-issues/27787)
**Author:** [@jonlieb](https://discuss.elastic.co/u/jonlieb)\ **Replies:** 1\ **Last updated:** [August 20, 2015, 8:50pm UTC](https://discuss.elastic.co/t/logstash-input-jdbc-install-issues/27787 "2015-08-20T20:50:54Z")
I am trying to configure logstash with jdbc input for oracle db in my lab environment; I am not permitted to access internet directly. I downloaded logstash-input-jdbc-master.zip and scp to my rhel logstash host. I unzip… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1241) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1243)