# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1056 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1057 --- ## [Kafka Input performance / config tuning](https://discuss.elastic.co/t/kafka-input-performance-config-tuning/70923)
**Author:** [@imehl](https://discuss.elastic.co/u/imehl)\ **Replies:** 3\ **Last updated:** [January 9, 2017, 9:07pm UTC](https://discuss.elastic.co/t/kafka-input-performance-config-tuning/70923 "2017-01-09T21:07:35Z")
Hi, I have a problem with the kafka input plugin. I get the following log message after some minutes and then it stops indexing. WARN org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto offset com… --- ## [How do I parse inner fields from json?](https://discuss.elastic.co/t/how-do-i-parse-inner-fields-from-json/70988)
**Author:** [@Ben\_Davis](https://discuss.elastic.co/u/Ben_Davis)\ **Replies:** 2\ **Last updated:** [January 9, 2017, 8:34pm UTC](https://discuss.elastic.co/t/how-do-i-parse-inner-fields-from-json/70988 "2017-01-09T20:34:53Z")
My log is full of nested json like so... { "foo": 1, "result":{ "time":"2017-01-09T02.01:50.000+0000", "product":"blahblah", "quantity… --- ## [Highly available logstash with Redis](https://discuss.elastic.co/t/highly-available-logstash-with-redis/70622)
**Author:** [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\ **Replies:** 3\ **Last updated:** [January 9, 2017, 8:25pm UTC](https://discuss.elastic.co/t/highly-available-logstash-with-redis/70622 "2017-01-09T20:25:30Z")
I would like to setup highly available logstash with highly available Redis queue. My idea is to have Logstash shipping instances (configured on app servers) send logs to Redis and from their, Logstash indexers work on t… --- ## [Couchdb changes, looses sync after long period of inactivity](https://discuss.elastic.co/t/couchdb-changes-looses-sync-after-long-period-of-inactivity/70907)
**Author:** [@teebu](https://discuss.elastic.co/u/teebu)\ **Replies:** 1\ **Last updated:** [January 9, 2017, 8:00pm UTC](https://discuss.elastic.co/t/couchdb-changes-looses-sync-after-long-period-of-inactivity/70907 "2017-01-09T20:00:48Z")
I cant tell what is happening, but I have many couchdb inputs, and they have different frequency of changes. It seems to me that if an index doesnt get \_changes after a long period of time (many hours, 1h+), that part… --- ## [Logstash errors out-Couldn't find any input plugin named 'beats'](https://discuss.elastic.co/t/logstash-errors-out-couldnt-find-any-input-plugin-named-beats/70987)
**Author:** [@dubul](https://discuss.elastic.co/u/dubul)\ **Replies:** 1\ **Last updated:** [January 9, 2017, 7:28pm UTC](https://discuss.elastic.co/t/logstash-errors-out-couldnt-find-any-input-plugin-named-beats/70987 "2017-01-09T19:28:09Z")
here is the logstash conf file input { beats { port =\> 5044 } } filter { csv { columns =\> \[ "date", "time", "s-sitename", "s-ip", "cs-method", "cs-uri-stem", "cs-uri-query", "s-port", "cs-username", "c-ip… --- ## [Kafka Input Tutorial for Logstash 5.1.1](https://discuss.elastic.co/t/kafka-input-tutorial-for-logstash-5-1-1/70762)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 5\ **Last updated:** [January 9, 2017, 5:30pm UTC](https://discuss.elastic.co/t/kafka-input-tutorial-for-logstash-5-1-1/70762 "2017-01-09T17:30:21Z")
Is there any third party documentation on how to use the Kafka plugin. Below is my logstash script input { kafka { bootstrap\_servers =\> \["192.168.31.55:9092,192.168.31.54:9092"\] topics =\> "test-beat33" } } fil… --- ## [Upgrade Logstash 5 -Kalfka plugin issue](https://discuss.elastic.co/t/upgrade-logstash-5-kalfka-plugin-issue/70225)
**Author:** [@sreejithps303](https://discuss.elastic.co/u/sreejithps303)\ **Replies:** 3\ **Last updated:** [January 9, 2017, 5:18pm UTC](https://discuss.elastic.co/t/upgrade-logstash-5-kalfka-plugin-issue/70225 "2017-01-09T17:18:33Z")
I am upgrading logstash from 2.3 to 5.1 . I am facing issue with kalfka input plugin. I have tried to remove the 5.1 plugin and install lower version but it is failing due to other dependencies.Also i have tried to chang… --- ## [Problems logstash directory - link with FBeat](https://discuss.elastic.co/t/problems-logstash-directory-link-with-fbeat/70967)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 0\ **Last updated:** [January 9, 2017, 4:17pm UTC](https://discuss.elastic.co/t/problems-logstash-directory-link-with-fbeat/70967 "2017-01-09T16:17:42Z")
Hi THANK YOU FOR YOUR HELP ! My message is to long but this really short in reality :slight\_smile: (I'm french who write you !) I have a problems with installation and configuration to Logstash and Fbeat. I have fo… --- ## [@timestamp in microseconds](https://discuss.elastic.co/t/timestamp-in-microseconds/70369)
**Author:** [@aharon.haber](https://discuss.elastic.co/u/aharon.haber)\ **Replies:** 4\ **Last updated:** [January 9, 2017, 1:22pm UTC](https://discuss.elastic.co/t/timestamp-in-microseconds/70369 "2017-01-09T13:22:35Z")
Are there any plans to support @timestamps in microseconds (or even nanoseconds)? I am trying to measure elapsed time between two events and I cannot see an easy way to do this in logstash (I am new BTW). I could of… --- ## [Jdbc plugin: how to make a date convertion for a column?](https://discuss.elastic.co/t/jdbc-plugin-how-to-make-a-date-convertion-for-a-column/70904)
**Author:** [@Lemec\_Cinq](https://discuss.elastic.co/u/Lemec_Cinq)\ **Replies:** 4\ **Last updated:** [January 9, 2017, 12:58pm UTC](https://discuss.elastic.co/t/jdbc-plugin-how-to-make-a-date-convertion-for-a-column/70904 "2017-01-09T12:58:48Z")
In my MS SQL table, i have a column with date stored as string on format "dd-mm-yyyy" During my import, i want to convert this column's values into date type. I saw a feature called filter which could do this type of t… --- ## [Read JSON with Logstash](https://discuss.elastic.co/t/read-json-with-logstash/70931)
**Author:** [@srneodani](https://discuss.elastic.co/u/srneodani)\ **Replies:** 5\ **Last updated:** [January 9, 2017, 11:38am UTC](https://discuss.elastic.co/t/read-json-with-logstash/70931 "2017-01-09T11:38:50Z")
Hi people! I want parser a json file and put output in a Elastic every minute, but I don't want add the same json each time. I would like if json have 10 element, have only this 10 elements in my index of Elastic but u… --- ## [Logstash config file execution error?](https://discuss.elastic.co/t/logstash-config-file-execution-error/70869)
**Author:** [@Yaswanth](https://discuss.elastic.co/u/Yaswanth)\ **Replies:** 4\ **Last updated:** [January 9, 2017, 9:10am UTC](https://discuss.elastic.co/t/logstash-config-file-execution-error/70869 "2017-01-09T09:10:10Z")
Hi, I have one config file which is used to analyse the slowlogs and keep it in file.The code for that is input { file { path =\> "C:\\Users\\571952\\Downloads\\elasticsearch-5.1.1\\elasticsearch-5.1.1\\logs\\elasticsea… --- ## [Jmx input: undefined method 'event'](https://discuss.elastic.co/t/jmx-input-undefined-method-event/70442)
**Author:** [@frankkoornstra](https://discuss.elastic.co/u/frankkoornstra)\ **Replies:** 4\ **Last updated:** [January 9, 2017, 10:55am UTC](https://discuss.elastic.co/t/jmx-input-undefined-method-event/70442 "2017-01-09T10:55:33Z")
I'm trying to use the jmx input for Logstash (2.4.0) but get the following error: {:timestamp=\>"2017-01-03T14:28:13.344000+0000", :message=\>"undefined method 'event' for #\", :level=\>:war… --- ## [Missing authentication token MetricBeat](https://discuss.elastic.co/t/missing-authentication-token-metricbeat/70912)
**Author:** [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\ **Replies:** 10\ **Last updated:** [January 9, 2017, 10:27am UTC](https://discuss.elastic.co/t/missing-authentication-token-metricbeat/70912 "2017-01-09T10:27:41Z")
helo, i have an issue with metricbeat. before i install xpack, metricbeat running very well. but suddenly after i install xpack, i cant access it from my kibana. i run this command in my terminal : curl -XGET 'http:/… --- ## [How to validate the data after parsing with KV Filter](https://discuss.elastic.co/t/how-to-validate-the-data-after-parsing-with-kv-filter/70811)
**Author:** [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Replies:** 3\ **Last updated:** [January 9, 2017, 10:26am UTC](https://discuss.elastic.co/t/how-to-validate-the-data-after-parsing-with-kv-filter/70811 "2017-01-09T10:26:19Z")
I have log messages like: 2017-01-06 19:27:53,893 INFO \[\[ACTIVE\] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'\] com.symantec.olp.sas.log.SummaryLogAspect - {country=JP, svid=182, typ=sassum, vi… --- ## [Config file won't stash a log](https://discuss.elastic.co/t/config-file-wont-stash-a-log/70876)
**Author:** [@HeatherB](https://discuss.elastic.co/u/HeatherB)\ **Replies:** 8\ **Last updated:** [January 9, 2017, 8:29am UTC](https://discuss.elastic.co/t/config-file-wont-stash-a-log/70876 "2017-01-09T08:29:31Z")
Hi:) I'm very new to logstash and elasticsearch, I am trying to stash my first log to log stash in a way that I can (correct me if it is not the purpose) search it using elasticsearch.... I have a log that looks like t… --- ## [Logstash - How to use the node.name as a field value?](https://discuss.elastic.co/t/logstash-how-to-use-the-node-name-as-a-field-value/70753)
**Author:** [@hk-awil](https://discuss.elastic.co/u/hk-awil)\ **Replies:** 2\ **Last updated:** [January 9, 2017, 8:16am UTC](https://discuss.elastic.co/t/logstash-how-to-use-the-node-name-as-a-field-value/70753 "2017-01-09T08:16:46Z")
Hello I would like to add the node.name value to an extra message field. Is this value readable from inside of logstash? I tried to use the node.name like the host field: "fqdn" =\> "%{host}" without success. Dose an… --- ## [Logstash and codec =\> "Windows-1251" bug?](https://discuss.elastic.co/t/logstash-and-codec-windows-1251-bug/69798)
**Author:** [@LiVong](https://discuss.elastic.co/u/LiVong)\ **Replies:** 1\ **Last updated:** [December 23, 2016, 12:15pm UTC](https://discuss.elastic.co/t/logstash-and-codec-windows-1251-bug/69798 "2016-12-23T12:15:52Z")
Good time of the day. Try to configure ELK for some system, where parsing logs throw filebeat and recive some problem. I am new in ELK, so may be I am just make some stupid mistake. I have some logs in Windows-1251 en… --- ## [COMBINEDAPACHELOG grok definition creating all fields as type string](https://discuss.elastic.co/t/combinedapachelog-grok-definition-creating-all-fields-as-type-string/70630)
**Author:** [@hp-adl](https://discuss.elastic.co/u/hp-adl)\ **Replies:** 2\ **Last updated:** [January 9, 2017, 3:49am UTC](https://discuss.elastic.co/t/combinedapachelog-grok-definition-creating-all-fields-as-type-string/70630 "2017-01-09T03:49:43Z")
Hi all, I'm using the %{COMBINEDAPACHELOG} grok pattern for my Apache logs and parsing to Elasticsearch service (v2.3) in AWS and all the fields are created as string type. I have tried a number of custom patterns and … --- ## [New to ELK - looking for Grok filter docs and help](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480)
**Author:** [@dotson83](https://discuss.elastic.co/u/dotson83)\ **Replies:** 13\ **Last updated:** [January 8, 2017, 6:09pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480 "2017-01-08T18:09:09Z")
Hello, I am new to ELK and I'm having issues with Grok. I'm trying to parse logs that look similar to this: "message" =\> "\<188\>362233: Jan 3 15:17:27: %IGMP\_QUERIER-4-ZERO\_SRC\_IP\_ADDR: An IGMP General Query packet w… --- ## [Importing datetime2 column from sql is giving error](https://discuss.elastic.co/t/importing-datetime2-column-from-sql-is-giving-error/70889)
**Author:** [@saisn](https://discuss.elastic.co/u/saisn)\ **Replies:** 1\ **Last updated:** [January 8, 2017, 5:37pm UTC](https://discuss.elastic.co/t/importing-datetime2-column-from-sql-is-giving-error/70889 "2017-01-08T17:37:56Z")
Hi, I'm importing a table from SQL using logstash and one of the fields in table has a 'datetime2' field which gives an error as below Exception when executing JDBC query {:exception=\>#, :level=\>:warn} can someone … --- ## [Not able to load full data into elastic search by using logstash](https://discuss.elastic.co/t/not-able-to-load-full-data-into-elastic-search-by-using-logstash/47467)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 3\ **Last updated:** [January 8, 2017, 4:57pm UTC](https://discuss.elastic.co/t/not-able-to-load-full-data-into-elastic-search-by-using-logstash/47467 "2017-01-08T16:57:11Z")
Case1: I am using log-stash jdbc input to load data from oracle db to elastic search. But its loading some amount of records only.Its not loading all records. Example: when i run query in sql developer i am getting … --- ## [Tomcat/catalina.out logs are getting parsed by grok debugger but not by logstash](https://discuss.elastic.co/t/tomcat-catalina-out-logs-are-getting-parsed-by-grok-debugger-but-not-by-logstash/70005)
**Author:** [@NITIN-BHAISARE](https://discuss.elastic.co/u/NITIN-BHAISARE)\ **Replies:** 3\ **Last updated:** [January 8, 2017, 4:55pm UTC](https://discuss.elastic.co/t/tomcat-catalina-out-logs-are-getting-parsed-by-grok-debugger-but-not-by-logstash/70005 "2017-01-08T16:55:49Z")
Hi, We have tomcat/catalina.out log files which are getting parsed by grok debugger properly but when we provide same grok pattern in logstash indexer, it never parses it. In almost all logs it gives grokparsefailure. I… --- ## [Fluentd to logstash](https://discuss.elastic.co/t/fluentd-to-logstash/70887)
**Author:** [@Nir\_Dothan](https://discuss.elastic.co/u/Nir_Dothan)\ **Replies:** 0\ **Last updated:** [January 8, 2017, 4:43pm UTC](https://discuss.elastic.co/t/fluentd-to-logstash/70887 "2017-01-08T16:43:01Z")
Hi, what is the recommended way to ship from fluentd to logstash? We need to run some post processing in a central location after logs are shipped by fluentd from remote servers. I've tried using out\_forward plugin on t… --- ## [Remove "tags" field not possible?](https://discuss.elastic.co/t/remove-tags-field-not-possible/70882)
**Author:** [@imehl](https://discuss.elastic.co/u/imehl)\ **Replies:** 0\ **Last updated:** [January 8, 2017, 3:46pm UTC](https://discuss.elastic.co/t/remove-tags-field-not-possible/70882 "2017-01-08T15:46:23Z")
hi, i try to remove the "tags" field and other fields. remove\_field =\> \["@version", "@timestamp", "tags"\] @version and @timestamp field will be removed, but not the "tags" field. is there a way to also remove this f… --- ## [\[Solved\] Weekly indexes instead of daily](https://discuss.elastic.co/t/solved-weekly-indexes-instead-of-daily/56871)
**Author:** [@brayndasilva](https://discuss.elastic.co/u/brayndasilva)\ **Replies:** 2\ **Last updated:** [January 8, 2017, 1:43pm UTC](https://discuss.elastic.co/t/solved-weekly-indexes-instead-of-daily/56871 "2017-01-08T13:43:19Z")
Right now I am working with indexes that are created daily using the following output in my config file: output { elasticsearch { hosts =\> \["localhost:9200"\] index =\> "logstash-events-%{+YYYY.MM.dd}" } } T… --- ## [Using Regex in Yaml for Logstash Translate Filter](https://discuss.elastic.co/t/using-regex-in-yaml-for-logstash-translate-filter/70855)
**Author:** [@Kyle\_Belitz](https://discuss.elastic.co/u/Kyle_Belitz)\ **Replies:** 2\ **Last updated:** [January 8, 2017, 1:46am UTC](https://discuss.elastic.co/t/using-regex-in-yaml-for-logstash-translate-filter/70855 "2017-01-08T01:46:51Z")
Good day all, I need some help. I am trying to use the Logstash Translate Filter to enrich network data that bro is generating and that I'm ingesting into my ELK stack. For example, here is how i was enriching data man… --- ## [Logstash Not parsing the file when running the again on the same file](https://discuss.elastic.co/t/logstash-not-parsing-the-file-when-running-the-again-on-the-same-file/70864)
**Author:** [@Akul\_Kapoor](https://discuss.elastic.co/u/Akul_Kapoor)\ **Replies:** 2\ **Last updated:** [January 7, 2017, 8:54pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-the-file-when-running-the-again-on-the-same-file/70864 "2017-01-07T20:54:17Z")
I ran logstash successfully using my log and config file. After that I stopped it and ran again using the same config file and log file, the logstash does not give any output. Can someone explain why logstash is not runn… --- ## [Add 1 filed is sum of other field in logstash](https://discuss.elastic.co/t/add-1-filed-is-sum-of-other-field-in-logstash/70832)
**Author:** [@Tr\_ng\_Trang](https://discuss.elastic.co/u/Tr_ng_Trang)\ **Replies:** 1\ **Last updated:** [January 7, 2017, 6:17am UTC](https://discuss.elastic.co/t/add-1-filed-is-sum-of-other-field-in-logstash/70832 "2017-01-07T06:17:47Z")
i have 2 filed is number is upbyte and downbyte, and i want to add 1 filed is bandwith (sum of there) in logstash, how can i do this? --- ## [Logstash 5.1.1 plugin install not working](https://discuss.elastic.co/t/logstash-5-1-1-plugin-install-not-working/69200)
**Author:** [@B200](https://discuss.elastic.co/u/B200)\ **Replies:** 3\ **Last updated:** [January 7, 2017, 5:24am UTC](https://discuss.elastic.co/t/logstash-5-1-1-plugin-install-not-working/69200 "2017-01-07T05:24:20Z")
I am new to elastic and doing a PoC project for logstash , elastic search and kibana with JVM monitoring with JMX. I am trying to install logstash-input-jmx and getting the below error. I tried with another plugin logsta… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1055) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1057)