# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1115 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1116 --- ## [Logstash file input](https://discuss.elastic.co/t/logstash-file-input/58076)
**Author:** [@datadude](https://discuss.elastic.co/u/datadude)\ **Replies:** 3\ **Last updated:** [August 16, 2016, 3:15pm UTC](https://discuss.elastic.co/t/logstash-file-input/58076 "2016-08-16T15:15:01Z")
Trying to figure out an issue with file input It just doesn't seem to be reading the file. It is a suricata eve.json file - permissions look fine (world readable) - verbose output says it registers the file - … --- ## [Logstash and Kafka input issue](https://discuss.elastic.co/t/logstash-and-kafka-input-issue/44848)
**Author:** [@HelpComputer](https://discuss.elastic.co/u/HelpComputer)\ **Replies:** 7\ **Last updated:** [August 16, 2016, 1:35pm UTC](https://discuss.elastic.co/t/logstash-and-kafka-input-issue/44848 "2016-08-16T13:35:55Z")
Hello, I am trying to setup the Kafka input in Logstash but something is not working right. Any help would be appreciated. Thanks in advance. Error Message on Server 2 in logstash.log {:timestamp=\>"2016-03-18T10:… --- ## [Kafka input and output plugin for logstash is not working](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271)
**Author:** [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Replies:** 11\ **Last updated:** [August 16, 2016, 1:35pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271 "2016-08-16T13:35:27Z")
Hello All, I am using Logstash 2.1.1 and kafka version 2.10-0.8.2.1. I am getting following syntax error. What are the minumum required settings for kafka plugin. Input.conf input { tcp { port =\> 5010 #typ… --- ## [Logstash kafka input issues in reconnecting to zookeeper](https://discuss.elastic.co/t/logstash-kafka-input-issues-in-reconnecting-to-zookeeper/57741)
**Author:** [@prabhakar349](https://discuss.elastic.co/u/prabhakar349)\ **Replies:** 5\ **Last updated:** [August 16, 2016, 1:29pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-issues-in-reconnecting-to-zookeeper/57741 "2016-08-16T13:29:17Z")
We are using logstash version : logstash-2.3.3 Right now we are using kafka input to read messages from broker which is version 0.9 { kafka { topic\_id =\> "{{kafka\_log\_consumer\_topic\_id}}" group\_id =\>… --- ## [Two difference group\_id get a same consumer of kafka](https://discuss.elastic.co/t/two-difference-group-id-get-a-same-consumer-of-kafka/58121)
**Author:** [@da\_Teng](https://discuss.elastic.co/u/da_Teng)\ **Replies:** 2\ **Last updated:** [August 16, 2016, 1:16pm UTC](https://discuss.elastic.co/t/two-difference-group-id-get-a-same-consumer-of-kafka/58121 "2016-08-16T13:16:40Z")
I use two logstash server with two difference group\_id in input.kafka to get log data from kafka-input and ,but they become a same consumer and average the log data. What the error I am not discover? This is logstash-1… --- ## [How to let logstash splits event field values and assign it to @metadata field](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833)
**Author:** [@daiyue](https://discuss.elastic.co/u/daiyue)\ **Replies:** 12\ **Last updated:** [August 16, 2016, 11:42am UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833 "2016-08-16T11:42:52Z")
Hi, I have a logstash event, which has the following field, { "\_index": "logstash-2016.08.09", "\_type": "log", "\_id": "AVZvz2ix", "\_score": null, "\_source": { "message": "function\_name~execute||line\_no~12… --- ## [Date filter not working in Logstash](https://discuss.elastic.co/t/date-filter-not-working-in-logstash/58073)
**Author:** [@RileyShu](https://discuss.elastic.co/u/RileyShu)\ **Replies:** 2\ **Last updated:** [August 16, 2016, 8:33am UTC](https://discuss.elastic.co/t/date-filter-not-working-in-logstash/58073 "2016-08-16T08:33:06Z")
input { file { path =\> "C:\\SystemErrors.log" start\_position =\> beginning ignore\_older =\> 0 type =\> "SystemErrors" } } filter { date { locale =\> "en" timezone =\> "America/Los\_Angeles" match =\> \["\\… --- ## [Keep raw syslog event untampered](https://discuss.elastic.co/t/keep-raw-syslog-event-untampered/58053)
**Author:** [@jaykumar](https://discuss.elastic.co/u/jaykumar)\ **Replies:** 3\ **Last updated:** [August 16, 2016, 6:39am UTC](https://discuss.elastic.co/t/keep-raw-syslog-event-untampered/58053 "2016-08-16T06:39:23Z")
Hi There, I have Logstash config with few groks to rename fields, make values human readable, geoip, and few more stuffs. I would like to make two outputs: Raw compressed file with untampered, this means it must not… --- ## [Logstash exception/error](https://discuss.elastic.co/t/logstash-exception-error/58068)
**Author:** [@bishaka](https://discuss.elastic.co/u/bishaka)\ **Replies:** 1\ **Last updated:** [August 16, 2016, 12:14am UTC](https://discuss.elastic.co/t/logstash-exception-error/58068 "2016-08-16T00:14:50Z")
Everything I run a logstash file.. for example, ./logstash agent -f twitter.conf..I get the follwoing error message: {:exception=\>Twitter::Error::Unauthorized, :backtrace=\>\["/opt/Oracle/relk/logstash/vendor/bundle/jrub… --- ## [\[Calling stored procedure in the output \]](https://discuss.elastic.co/t/calling-stored-procedure-in-the-output/58070)
**Author:** [@varun\_kumar1](https://discuss.elastic.co/u/varun_kumar1)\ **Replies:** 1\ **Last updated:** [August 15, 2016, 10:06pm UTC](https://discuss.elastic.co/t/calling-stored-procedure-in-the-output/58070 "2016-08-15T22:06:44Z")
Hello , Is Logstash supports calling sql stored procedure as an output in the config file .Can somebody show me with simple example . --- ## [Error to run logstash](https://discuss.elastic.co/t/error-to-run-logstash/57957)
**Author:** [@rai](https://discuss.elastic.co/u/rai)\ **Replies:** 7\ **Last updated:** [August 15, 2016, 9:45pm UTC](https://discuss.elastic.co/t/error-to-run-logstash/57957 "2016-08-15T21:45:59Z")
indent preformatted text by 4 spacesI got following error on console when tried to start the logstash. root@111:/opt/logstash/bin# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/10-syslog-filter.conf --configtest E… --- ## [Adding time dimension attributes to date fields](https://discuss.elastic.co/t/adding-time-dimension-attributes-to-date-fields/58037)
**Author:** [@sagizvi](https://discuss.elastic.co/u/sagizvi)\ **Replies:** 1\ **Last updated:** [August 15, 2016, 8:29pm UTC](https://discuss.elastic.co/t/adding-time-dimension-attributes-to-date-fields/58037 "2016-08-15T20:29:49Z")
Hi All :slight\_smile: I was wondering what is the current best method to add more attribute to date fields, for example : holiday flag, month number, quarter number, last day in month, is it a working day or not .… --- ## [Translate Issue](https://discuss.elastic.co/t/translate-issue/58052)
**Author:** [@jaykumar](https://discuss.elastic.co/u/jaykumar)\ **Replies:** 1\ **Last updated:** [August 15, 2016, 8:22pm UTC](https://discuss.elastic.co/t/translate-issue/58052 "2016-08-15T20:22:23Z")
Hi All, I have tested Logstash translate feature, which works perfectly fine. But it can understand only two columns one as source and another as destination field. I am looking at a solution to match key-value with … --- ## [Logstash Index creation and parsing custom app log](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711)
**Author:** [@jukkendar](https://discuss.elastic.co/u/jukkendar)\ **Replies:** 14\ **Last updated:** [August 15, 2016, 7:41pm UTC](https://discuss.elastic.co/t/logstash-index-creation-and-parsing-custom-app-log/57711 "2016-08-15T19:41:27Z")
I have installed Logstash 2.3.1, elastic search 2.3.1 and Kibana 4.5.0 in windows 2008 server as services. I can launch Kibana but unable to create Index in Kibana , Refer the log below. I have verified the below config… --- ## [Tile map "no results found"](https://discuss.elastic.co/t/tile-map-no-results-found/57862)
**Author:** [@bishaka](https://discuss.elastic.co/u/bishaka)\ **Replies:** 1\ **Last updated:** [August 15, 2016, 4:56pm UTC](https://discuss.elastic.co/t/tile-map-no-results-found/57862 "2016-08-15T16:56:27Z")
Hi, I am trying to show the location information using a tile map in Kibana..However, when i try to visualize the tile map, i get "No Results Founds" on kibana I have the following logs "path" =\> "/logs/access.log… --- ## [Logstash with multiple kafka inputs](https://discuss.elastic.co/t/logstash-with-multiple-kafka-inputs/57646)
**Author:** [@abhi\_chinchole](https://discuss.elastic.co/u/abhi_chinchole)\ **Replies:** 4\ **Last updated:** [August 15, 2016, 4:10pm UTC](https://discuss.elastic.co/t/logstash-with-multiple-kafka-inputs/57646 "2016-08-15T16:10:18Z")
I am trying to filter kafka events from multiple topics, but once all events from one topic has been filtered logstash is not able to fetch events from the other kafka topic. I am using topics with 3 partitions and 2 rep… --- ## [Conditional being ignored for grok match](https://discuss.elastic.co/t/conditional-being-ignored-for-grok-match/58010)
**Author:** [@harrytewkesbury](https://discuss.elastic.co/u/harrytewkesbury)\ **Replies:** 2\ **Last updated:** [August 15, 2016, 11:08am UTC](https://discuss.elastic.co/t/conditional-being-ignored-for-grok-match/58010 "2016-08-15T11:08:59Z")
Hi all, I have a working filter for a specific log, and I am happy with that. However, one other log is being matched on it, even though the conditional I am using should not do so. I am matching on tags. if "access"… --- ## [Using redis, get error like "Failed to send event to Redis"](https://discuss.elastic.co/t/using-redis-get-error-like-failed-to-send-event-to-redis/58006)
**Author:** [@kk\_op](https://discuss.elastic.co/u/kk_op)\ **Replies:** 2\ **Last updated:** [August 15, 2016, 7:24am UTC](https://discuss.elastic.co/t/using-redis-get-error-like-failed-to-send-event-to-redis/58006 "2016-08-15T07:24:40Z")
Just installed logstash 2.3.4 -redis-elasticsearch 2.3.4 when test, error repeated in file logstash.log: {:timestamp=\>"2016-08-15T14:31:35.306000+0800", :message=\>"Failed to send event to Redis", :event=\>#\ **Author:** [@111127](https://discuss.elastic.co/u/111127)\ **Replies:** 2\ **Last updated:** [August 15, 2016, 4:24am UTC](https://discuss.elastic.co/t/data-of-logstash-isnt-preserved-by-elastisearch/57995 "2016-08-15T04:24:21Z") Hi all I'm using ・filebeat-1.2.3-1.x86\_64(centos6.4) ・logstash-2.3.2-1.noarch.rpm(amazon linux) ・elasticsearch-2.3.5-1.noarch(amazon linux) ・kibana-4.5.4-1.x86\_64(amazon linux) Access log of nginx, it's set as LTS… --- ## [Logstash s3 plugin poll interval?](https://discuss.elastic.co/t/logstash-s3-plugin-poll-interval/57960)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 1\ **Last updated:** [August 14, 2016, 10:18pm UTC](https://discuss.elastic.co/t/logstash-s3-plugin-poll-interval/57960 "2016-08-14T22:18:58Z")
How often the s3 input plugin polls for data from s3? Is it real time or there is an interval for the same. -- Niraj --- ## [Adding duration field from start date and end date](https://discuss.elastic.co/t/adding-duration-field-from-start-date-and-end-date/51857)
**Author:** [@Hans](https://discuss.elastic.co/u/Hans)\ **Replies:** 7\ **Last updated:** [August 14, 2016, 6:32pm UTC](https://discuss.elastic.co/t/adding-duration-field-from-start-date-and-end-date/51857 "2016-08-14T18:32:50Z")
Hi. I am struggling to get the duration for a start and end date in a log entry. I have the following fields: TIME=2016/05/29 13:23:04.286 TIME\_END=2016/05/29 13:23:04.493 I would like to get the duration between the t… --- ## [How to forward logs from an activeMQ queue to logstash server?](https://discuss.elastic.co/t/how-to-forward-logs-from-an-activemq-queue-to-logstash-server/57987)
**Author:** [@Vinay\_Reddy](https://discuss.elastic.co/u/Vinay_Reddy)\ **Replies:** 2\ **Last updated:** [August 14, 2016, 3:11pm UTC](https://discuss.elastic.co/t/how-to-forward-logs-from-an-activemq-queue-to-logstash-server/57987 "2016-08-14T15:11:39Z")
I'm new to ELK stack and have the following use case to achieve.. I have an existing environment where some 100 odd servers sends an application log to a ActiveMQ queue on a Central Server. I need to send data from t… --- ## [Logstash - Pipeline main started not executing further](https://discuss.elastic.co/t/logstash-pipeline-main-started-not-executing-further/57954)
**Author:** [@kmk3173](https://discuss.elastic.co/u/kmk3173)\ **Replies:** 2\ **Last updated:** [August 14, 2016, 2:43pm UTC](https://discuss.elastic.co/t/logstash-pipeline-main-started-not-executing-further/57954 "2016-08-14T14:43:35Z")
Hi, I created a config file that runs but stalls after pipeline main started. I'm not sure if the issue relates to the fact that I have two config files in the config folder to read 2 different csv files and output it … --- ## [Indexing many xml files](https://discuss.elastic.co/t/indexing-many-xml-files/56420)
**Author:** [@mark.demichele](https://discuss.elastic.co/u/mark.demichele)\ **Replies:** 3\ **Last updated:** [August 14, 2016, 1:54pm UTC](https://discuss.elastic.co/t/indexing-many-xml-files/56420 "2016-08-14T13:54:07Z")
I have an application that maintains many xml files under a folder and set of sub-folders. The xml files are added and removed from these folders. Is there a way to use logstash to send these files to Elastic Search al… --- ## [Logstash input Cisco SDEE plugin](https://discuss.elastic.co/t/logstash-input-cisco-sdee-plugin/29653)
**Author:** [@rootik](https://discuss.elastic.co/u/rootik)\ **Replies:** 1\ **Last updated:** [August 14, 2016, 8:44am UTC](https://discuss.elastic.co/t/logstash-input-cisco-sdee-plugin/29653 "2016-08-14T08:44:02Z")
Hello I'm currently working on logstash Cisco SDEE/CIDEE input plugin. Fill free to contribute. https://github.com/rootik/logstash-input-sdee --- ## [Logstash-plugin update](https://discuss.elastic.co/t/logstash-plugin-update/57952)
**Author:** [@Greppy](https://discuss.elastic.co/u/Greppy)\ **Replies:** 1\ **Last updated:** [August 12, 2016, 8:17pm UTC](https://discuss.elastic.co/t/logstash-plugin-update/57952 "2016-08-12T20:17:37Z")
Logstash 2.3.4 and Throttle Filter Checking on Github and it appears that the throttle filter plugin has been updated so that it is threadsafe (4.0.0). When I try to update , however, this fails with little information: … --- ## [Stumped on TIMESTAMP\_ISO8601 in grok](https://discuss.elastic.co/t/stumped-on-timestamp-iso8601-in-grok/57733)
**Author:** [@Ron\_Pritchett](https://discuss.elastic.co/u/Ron_Pritchett)\ **Replies:** 3\ **Last updated:** [August 12, 2016, 4:17pm UTC](https://discuss.elastic.co/t/stumped-on-timestamp-iso8601-in-grok/57733 "2016-08-12T16:17:40Z")
Hello, I'm not sure why this is throwing an error. the input file: \<133\>2016-08-10T16:33:39.827713+00:00 localhost opt-log-nginx-access the configfile: input { stdin { type =\> syslog } } filter { … --- ## [Logstash doesn't catch severity/loglevel from Jboss log using rsyslog](https://discuss.elastic.co/t/logstash-doesnt-catch-severity-loglevel-from-jboss-log-using-rsyslog/57847)
**Author:** [@lplinux](https://discuss.elastic.co/u/lplinux)\ **Replies:** 5\ **Last updated:** [August 12, 2016, 3:58pm UTC](https://discuss.elastic.co/t/logstash-doesnt-catch-severity-loglevel-from-jboss-log-using-rsyslog/57847 "2016-08-12T15:58:59Z")
Hello everybody. I'm starting with ELK and I was able to configure a lot of filters and other stuff. Right now I'm trying to configure JBoss log, but I couldn't configure logstash to match the correct severity. It's alw… --- ## [JSON filter do not encode part of JSON](https://discuss.elastic.co/t/json-filter-do-not-encode-part-of-json/57935)
**Author:** [@DK\_3](https://discuss.elastic.co/u/DK_3)\ **Replies:** 0\ **Last updated:** [August 12, 2016, 2:32pm UTC](https://discuss.elastic.co/t/json-filter-do-not-encode-part-of-json/57935 "2016-08-12T14:32:43Z")
I'm using the json filter plugin and want to exclude parsing part of the message into JSON https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html filter { json { source =\> "messa… --- ## [Cannot install latest metrics filter plugin](https://discuss.elastic.co/t/cannot-install-latest-metrics-filter-plugin/57861)
**Author:** [@devkmsg](https://discuss.elastic.co/u/devkmsg)\ **Replies:** 0\ **Last updated:** [August 11, 2016, 9:28pm UTC](https://discuss.elastic.co/t/cannot-install-latest-metrics-filter-plugin/57861 "2016-08-11T21:28:38Z")
I am getting a Gem conflict trying to install the latest version of the plugin. I am running Logstash version 2.3.4. The following is the command I am trying to run. /opt/logstash/bin/plugin install --version 4.0.2… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1114) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1116)