# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1205 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1206 --- ## [PV vs Metrics for benchmarking](https://discuss.elastic.co/t/pv-vs-metrics-for-benchmarking/38085)
**Author:** [@Johntdyer](https://discuss.elastic.co/u/Johntdyer)\ **Replies:** 3\ **Last updated:** [January 4, 2016, 5:21pm UTC](https://discuss.elastic.co/t/pv-vs-metrics-for-benchmarking/38085 "2016-01-04T17:21:24Z")
Hello, I am trying to load test my grok patterns and I am confused on the output of metrics{} vs using codec=\> dots along with pv -War \> /dev/null. I am consuming the same 400,000 log lines as my input so that shou… --- ## [Multiple logstash instances reading the same files](https://discuss.elastic.co/t/multiple-logstash-instances-reading-the-same-files/38220)
**Author:** [@asphaltsurfer](https://discuss.elastic.co/u/asphaltsurfer)\ **Replies:** 2\ **Last updated:** [January 4, 2016, 4:16pm UTC](https://discuss.elastic.co/t/multiple-logstash-instances-reading-the-same-files/38220 "2016-01-04T16:16:00Z")
Hi guys! My problem: I have single NAS logs share connected to my application nodes, and logstash node. One logstash instance is reading logfiles, parsing and shipping to ES. Now, I want to add failover to logstash by … --- ## [How to EXPIRE a key in logstash?](https://discuss.elastic.co/t/how-to-expire-a-key-in-logstash/38320)
**Author:** [@N\_M\_10](https://discuss.elastic.co/u/N_M_10)\ **Replies:** 2\ **Last updated:** [January 4, 2016, 3:02pm UTC](https://discuss.elastic.co/t/how-to-expire-a-key-in-logstash/38320 "2016-01-04T15:02:50Z")
correct way of expiring a key in logstash input { http { host =\> "127.0.0.1" # default: 0.0.0.0 port =\> 31311 # default: 8080 } } output { redis { host =\> 'localhost' data\_type =\> 'list' key… --- ## [Logstash information on csv loaded](https://discuss.elastic.co/t/logstash-information-on-csv-loaded/37847)
**Author:** [@chicco\_95](https://discuss.elastic.co/u/chicco_95)\ **Replies:** 2\ **Last updated:** [January 4, 2016, 11:05am UTC](https://discuss.elastic.co/t/logstash-information-on-csv-loaded/37847 "2016-01-04T11:05:35Z")
hi, How can I config Logstash filter to know: 1) how many records in a csv have been uploaded 2) how many records in a csv have been updated 3) how many records in a csv have NOT been uploaded thank you --- ## [How to configure geoip Accurate Database (GeoipcCity.Dat) in Logstash](https://discuss.elastic.co/t/how-to-configure-geoip-accurate-database-geoipccity-dat-in-logstash/38223)
**Author:** [@aniamin](https://discuss.elastic.co/u/aniamin)\ **Replies:** 1\ **Last updated:** [January 4, 2016, 10:55am UTC](https://discuss.elastic.co/t/how-to-configure-geoip-accurate-database-geoipccity-dat-in-logstash/38223 "2016-01-04T10:55:26Z")
Currently I am using GeoLiteCity.Dat in Logstash. But this Database is not correct. I want to use more accurate database (GeoIPCity.dat). How to configure this? --- ## [To give input to ELasticsearch via logstash](https://discuss.elastic.co/t/to-give-input-to-elasticsearch-via-logstash/38240)
**Author:** [@trushad](https://discuss.elastic.co/u/trushad)\ **Replies:** 4\ **Last updated:** [January 4, 2016, 10:33am UTC](https://discuss.elastic.co/t/to-give-input-to-elasticsearch-via-logstash/38240 "2016-01-04T10:33:32Z")
I m having txt file (size 4 GB ) with json string as below , eg: {"index":{"\_id": "178790318"}} {"Id":"178790318","Description":"Kynoch Rd, Durban City, Durban, KwaZulu-Natal, South Africa","Odometer":"578676","Ignitio… --- ## [Logstash: Custom filter reading CSV](https://discuss.elastic.co/t/logstash-custom-filter-reading-csv/38302)
**Author:** [@krushnat\_khawale](https://discuss.elastic.co/u/krushnat_khawale)\ **Replies:** 1\ **Last updated:** [January 4, 2016, 10:14am UTC](https://discuss.elastic.co/t/logstash-custom-filter-reading-csv/38302 "2016-01-04T10:14:01Z")
In my use case, I need to upload data into the logstash and simultaniously need to check the data based on another file. For this reason, I need to read mapping file at once and to validate against data. I am writing m… --- ## [Logstash can't start up,how i can solve this error](https://discuss.elastic.co/t/logstash-cant-start-up-how-i-can-solve-this-error/38297)
**Author:** [@rousth](https://discuss.elastic.co/u/rousth)\ **Replies:** 0\ **Last updated:** [January 4, 2016, 8:19am UTC](https://discuss.elastic.co/t/logstash-cant-start-up-how-i-can-solve-this-error/38297 "2016-01-04T08:19:45Z")
LoadError: no such file to load -- bundler require at org/jruby/RubyKernel.java:940 require at C:/jruby-9.0.0.0/lib/ruby/stdlib/rubygems/core\_ext/kernel\_require.rb:54 setup! at C:/Users/ryan.dai/Desktop/logstash-1… --- ## [Cassandra -logstash JDBC connectivity example](https://discuss.elastic.co/t/cassandra-logstash-jdbc-connectivity-example/38290)
**Author:** [@Vijay\_Dodla](https://discuss.elastic.co/u/Vijay_Dodla)\ **Replies:** 0\ **Last updated:** [January 4, 2016, 6:04am UTC](https://discuss.elastic.co/t/cassandra-logstash-jdbc-connectivity-example/38290 "2016-01-04T06:04:20Z")
Can you anyone please throw some light incase you got succeeded in integrating logstash with CassandraDB using Cassandra JDBC drivers ? --- ## [How to include %{} variable in the gsub](https://discuss.elastic.co/t/how-to-include-variable-in-the-gsub/37482)
**Author:** [@darshan](https://discuss.elastic.co/u/darshan)\ **Replies:** 7\ **Last updated:** [January 4, 2016, 4:57am UTC](https://discuss.elastic.co/t/how-to-include-variable-in-the-gsub/37482 "2016-01-04T04:57:46Z")
Hi, I am new to logstash, and i stuck in parsin some antivirus log . Can any one please tell me how to use dynamic variable like %{} in the gsub to replace. Thank you -Regards darshan --- ## [Kafka plugin makes logstash go into 100% CPU infinite loop](https://discuss.elastic.co/t/kafka-plugin-makes-logstash-go-into-100-cpu-infinite-loop/38259)
**Author:** [@Omar\_Al\_Zabir](https://discuss.elastic.co/u/Omar_Al_Zabir)\ **Replies:** 1\ **Last updated:** [January 2, 2016, 8:08pm UTC](https://discuss.elastic.co/t/kafka-plugin-makes-logstash-go-into-100-cpu-infinite-loop/38259 "2016-01-02T20:08:55Z")
I am using Kafka plugin in logstash to read items and push to ES. After processing 3M messages, I can see the logstash process remains at 100% CPU continuously, although it is doing nothing. Here's my config: input … --- ## [Logstash Shipper to Redis](https://discuss.elastic.co/t/logstash-shipper-to-redis/29948)
**Author:** [@ariceELK](https://discuss.elastic.co/u/ariceELK)\ **Replies:** 15\ **Last updated:** [January 2, 2016, 7:31pm UTC](https://discuss.elastic.co/t/logstash-shipper-to-redis/29948 "2016-01-02T19:31:52Z")
Hi, Im using LS Shipper to send to Redis queue. The conf input is reading from different log files but I only see logs from type syslog being sent. Application and Web logs are not getting sent. I've run LS shipper… --- ## [Logstash: Custom filter, Can we access members/columns/fields of another filter in a custom filter?](https://discuss.elastic.co/t/logstash-custom-filter-can-we-access-members-columns-fields-of-another-filter-in-a-custom-filter/38156)
**Author:** [@krushnat\_khawale](https://discuss.elastic.co/u/krushnat_khawale)\ **Replies:** 1\ **Last updated:** [January 2, 2016, 8:17am UTC](https://discuss.elastic.co/t/logstash-custom-filter-can-we-access-members-columns-fields-of-another-filter-in-a-custom-filter/38156 "2016-01-02T08:17:45Z")
I am creating my own filter, as I am uploading a CSV file and I need to access fields in CSV in my custom filter. Is it possible to access those fields in my filter? --- ## [Logstash error to recreate Index for Elasticsearch](https://discuss.elastic.co/t/logstash-error-to-recreate-index-for-elasticsearch/38173)
**Author:** [@carmelom](https://discuss.elastic.co/u/carmelom)\ **Replies:** 1\ **Last updated:** [January 2, 2016, 8:15am UTC](https://discuss.elastic.co/t/logstash-error-to-recreate-index-for-elasticsearch/38173 "2016-01-02T08:15:49Z")
I am using : - elasticsearch-2.1.1 - kibana-4.3.1-linux-x64 - logstash-2.1.1 I followed this tutorial: https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html Then logstash was able to create the index… --- ## [Logstash Custom Filter: How to access, fields of csv being uploaded in a custom filter](https://discuss.elastic.co/t/logstash-custom-filter-how-to-access-fields-of-csv-being-uploaded-in-a-custom-filter/38098)
**Author:** [@krushnat\_khawale](https://discuss.elastic.co/u/krushnat_khawale)\ **Replies:** 4\ **Last updated:** [January 1, 2016, 9:47am UTC](https://discuss.elastic.co/t/logstash-custom-filter-how-to-access-fields-of-csv-being-uploaded-in-a-custom-filter/38098 "2016-01-01T09:47:53Z")
Logstash Custom Filter: How to access, fields of csv being uploaded in a custom filter I am uploading a csv and I'm creating a custom filter to perform the operations on fields in CSV. I have no idea of accessing field… --- ## [Selective fields per output](https://discuss.elastic.co/t/selective-fields-per-output/2046)
**Author:** [@Akumar](https://discuss.elastic.co/u/Akumar)\ **Replies:** 1\ **Last updated:** [December 31, 2015, 5:34pm UTC](https://discuss.elastic.co/t/selective-fields-per-output/2046 "2015-12-31T17:34:27Z")
I would like to drop certain fields from elasticsearch indexing but retain them for output to another destination. I have two use cases 1. Some fields contain detail that I only want for an investigation and do not wan… --- ## [Pulling logs stored in a database into Logstash](https://discuss.elastic.co/t/pulling-logs-stored-in-a-database-into-logstash/26509)
**Author:** [@tz1413](https://discuss.elastic.co/u/tz1413)\ **Replies:** 4\ **Last updated:** [December 31, 2015, 4:00pm UTC](https://discuss.elastic.co/t/pulling-logs-stored-in-a-database-into-logstash/26509 "2015-12-31T16:00:19Z")
I have a Cassandra database in which we are storing information from Jenkins from build jobs. One of these fields contains the complete log file. I would like to be able to use Logstash to read these logs and search th… --- ## [Not getting grok pattern for 12/11/2013 8:30:48 AM](https://discuss.elastic.co/t/not-getting-grok-pattern-for-12-11-2013-8-30-48-am/38209)
**Author:** [@sree1](https://discuss.elastic.co/u/sree1)\ **Replies:** 1\ **Last updated:** [December 31, 2015, 3:42pm UTC](https://discuss.elastic.co/t/not-getting-grok-pattern-for-12-11-2013-8-30-48-am/38209 "2015-12-31T15:42:52Z")
Hi All, can any one help grok pattern for following logs.i tried for the below logs but i am getting difficulty during am/pm.please help me. My logs: Started : 12/11/2013 8:30:48 AM … --- ## [Duplicate Data in two indexes](https://discuss.elastic.co/t/duplicate-data-in-two-indexes/36869)
**Author:** [@Gurveer\_Singh](https://discuss.elastic.co/u/Gurveer_Singh)\ **Replies:** 4\ **Last updated:** [December 31, 2015, 12:37pm UTC](https://discuss.elastic.co/t/duplicate-data-in-two-indexes/36869 "2015-12-31T12:37:36Z")
Hi, I am using logstash version 1.4.2 I configure logstash to store data in elasticsearch in a manual index name "xyz" and I am using following template for this index. { "template" : "xyz-", "settings" : { "index… --- ## [New fields not being created and grok help](https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895)
**Author:** [@tweetybird](https://discuss.elastic.co/u/tweetybird)\ **Replies:** 5\ **Last updated:** [December 30, 2015, 7:58pm UTC](https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895 "2015-12-30T19:58:47Z")
I'm very new to all this ELK stuff and I'm having a lot of trouble getting logstash to parse my file. I followed a tutorial and managed to get filebeat to send my logs to logstash and I can see them in kibana. Now I'm tr… --- ## [ERROR: Couldn't find any input plugin named 'date'](https://discuss.elastic.co/t/error-couldnt-find-any-input-plugin-named-date/38148)
**Author:** [@mchen](https://discuss.elastic.co/u/mchen)\ **Replies:** 4\ **Last updated:** [December 30, 2015, 3:10pm UTC](https://discuss.elastic.co/t/error-couldnt-find-any-input-plugin-named-date/38148 "2015-12-30T15:10:53Z")
I'm working with a config file to import a csv file into logstash sample below: filter { csv { columns =\> \[Column names here\] } date { match =\> \["donation\_timestamp", "yyyy-MM-dd HH:mm:ss.SSS", "yyy… --- ## [Removing fields for certain outputs](https://discuss.elastic.co/t/removing-fields-for-certain-outputs/38171)
**Author:** [@kerneldemon](https://discuss.elastic.co/u/kerneldemon)\ **Replies:** 2\ **Last updated:** [December 30, 2015, 12:39pm UTC](https://discuss.elastic.co/t/removing-fields-for-certain-outputs/38171 "2015-12-30T12:39:30Z")
Hi there I want to process certain log data using logstash and then pipe it to multiple outputs. The data I wish to process always contains sensitive fields that I wish to send to some of the outputs and remove for oth… --- ## [Display an xml attribute value using xml filter](https://discuss.elastic.co/t/display-an-xml-attribute-value-using-xml-filter/38104)
**Author:** [@Sudeesh](https://discuss.elastic.co/u/Sudeesh)\ **Replies:** 3\ **Last updated:** [December 30, 2015, 8:40am UTC](https://discuss.elastic.co/t/display-an-xml-attribute-value-using-xml-filter/38104 "2015-12-30T08:40:42Z")
Here is the xml file, i want to print the attribute value name from class tag. And my conf file is below input { file { path=\>\["C:/logstash-2.1.1/Data/Te… --- ## [:message=\>"Failed action. ", :status=\>404, :action=\>\["index",](https://discuss.elastic.co/t/message-failed-action-status-404-action-index/36780)
**Author:** [@thomas.dotreppe](https://discuss.elastic.co/u/thomas.dotreppe)\ **Replies:** 12\ **Last updated:** [December 30, 2015, 12:28am UTC](https://discuss.elastic.co/t/message-failed-action-status-404-action-index/36780 "2015-12-30T00:28:16Z")
I receive very long log entries with issues such as these (this one is truncated) every few seconds in the logs: {:timestamp=\>"2015-12-09T10:51:52.061000-0700", :message=\>"Failed action. ", :status=\>404, :action=\>\["i… --- ## [Rsyslog to Redis to logstash to elasticsearch without logstash forwarder?](https://discuss.elastic.co/t/rsyslog-to-redis-to-logstash-to-elasticsearch-without-logstash-forwarder/26435)
**Author:** [@kurtcoke](https://discuss.elastic.co/u/kurtcoke)\ **Replies:** 5\ **Last updated:** [December 29, 2015, 10:52pm UTC](https://discuss.elastic.co/t/rsyslog-to-redis-to-logstash-to-elasticsearch-without-logstash-forwarder/26435 "2015-12-29T22:52:50Z")
I see lots of tutorials like this one: http://www.everybodyhertz.co.uk/setting-up-a-relk-stack-a-how-to/ suggesting that you use logstash forwarder to send logs to redis before sending it to logstash and then to elastics… --- ## [How to set RequestDate as the @timestamp so that I can use it in Kibana?](https://discuss.elastic.co/t/how-to-set-requestdate-as-the-timestamp-so-that-i-can-use-it-in-kibana/38107)
**Author:** [@Rajeev\_Gaurav](https://discuss.elastic.co/u/Rajeev_Gaurav)\ **Replies:** 2\ **Last updated:** [December 29, 2015, 9:10pm UTC](https://discuss.elastic.co/t/how-to-set-requestdate-as-the-timestamp-so-that-i-can-use-it-in-kibana/38107 "2015-12-29T21:10:22Z")
Hi , My logstash config file has filter as : date { match =\> \["RequestDate","MM/dd/yyyy: HH:mm:ss" \] timezone =\> "UTC" } in My input file RequestDate is having value like :12/23/2015 3:32:00 AM ,12/25/201… --- ## [Watching a Logstash JVM via Datadog](https://discuss.elastic.co/t/watching-a-logstash-jvm-via-datadog/38118)
**Author:** [@ecweaver](https://discuss.elastic.co/u/ecweaver)\ **Replies:** 0\ **Last updated:** [December 29, 2015, 6:45pm UTC](https://discuss.elastic.co/t/watching-a-logstash-jvm-via-datadog/38118 "2015-12-29T18:45:26Z")
I would very much like to be able to watch the Logstash hosts' JVMs on Datadog, the way we can with Elasticsearch. Does anyone have a datadog config file for this? Is anything special needed to enable say JMX to all… --- ## [Logstash startup completed and then the system hangs logs are not passed into elastic search how can i overcome this?](https://discuss.elastic.co/t/logstash-startup-completed-and-then-the-system-hangs-logs-are-not-passed-into-elastic-search-how-can-i-overcome-this/38021)
**Author:** [@Akash\_Katakam](https://discuss.elastic.co/u/Akash_Katakam)\ **Replies:** 5\ **Last updated:** [December 29, 2015, 11:49am UTC](https://discuss.elastic.co/t/logstash-startup-completed-and-then-the-system-hangs-logs-are-not-passed-into-elastic-search-how-can-i-overcome-this/38021 "2015-12-29T11:49:04Z")
My command prompt shows logstash startup completed i changed the filename and tried pushing the log again but that doesnt help how can i overcome this problem --- ## [Logstash 2.1 and kafka 0.9](https://discuss.elastic.co/t/logstash-2-1-and-kafka-0-9/35666)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 3\ **Last updated:** [December 29, 2015, 11:38am UTC](https://discuss.elastic.co/t/logstash-2-1-and-kafka-0-9/35666 "2015-12-29T11:38:26Z")
Hi, I'm looking at using kafka as a message queue in our elk environment. I have two logstash instances running, the first takes inbound json and outputs to kafka while the second reads from kafka and outputs to elastic… --- ## [Logstash elasticsearch filter plugin - set index](https://discuss.elastic.co/t/logstash-elasticsearch-filter-plugin-set-index/37583)
**Author:** [@Julia\_Murinova](https://discuss.elastic.co/u/Julia_Murinova)\ **Replies:** 1\ **Last updated:** [December 29, 2015, 9:34am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-filter-plugin-set-index/37583 "2015-12-29T09:34:20Z")
Hi, I am trying to use elasticsearch filter plugin in logstash to query previous messages for the same session. Code: elasticsearch { hosts =\> \["localhost"\] query =\> "tags:Connection\_new AND session\_id:%… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1204) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1206)