# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1073 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1074 --- ## [How to setup a logstash cluster of 3 nodes?](https://discuss.elastic.co/t/how-to-setup-a-logstash-cluster-of-3-nodes/67108)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 0\ **Last updated:** [November 24, 2016, 12:42pm UTC](https://discuss.elastic.co/t/how-to-setup-a-logstash-cluster-of-3-nodes/67108 "2016-11-24T12:42:30Z")
Hi, I have a Logstash configuration like below; input { kafka{ group\_id =\> "mygroup" topics =\> \["mytopic"\] bootstrap\_servers =\> "192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092" codec =\> json add\_field … --- ## [Difficulty with zeromq input plugin](https://discuss.elastic.co/t/difficulty-with-zeromq-input-plugin/54146)
**Author:** [@stash](https://discuss.elastic.co/u/stash)\ **Replies:** 1\ **Last updated:** [November 24, 2016, 11:28am UTC](https://discuss.elastic.co/t/difficulty-with-zeromq-input-plugin/54146 "2016-11-24T11:28:44Z")
I am trying to use logstash with the zeromq input plugin. Whenever I start logstash with my configuration, I get the following error: Unable to load this gem. The libzmq library (or DLL) could not be found. If this is… --- ## [Sniffing paramter from elasticsearch plugin does not work with ES 5.0](https://discuss.elastic.co/t/sniffing-paramter-from-elasticsearch-plugin-does-not-work-with-es-5-0/67022)
**Author:** [@Musashisan](https://discuss.elastic.co/u/Musashisan)\ **Replies:** 0\ **Last updated:** [November 23, 2016, 5:42pm UTC](https://discuss.elastic.co/t/sniffing-paramter-from-elasticsearch-plugin-does-not-work-with-es-5-0/67022 "2016-11-23T17:42:12Z")
Sniffing is not working becose the plugin is looking for a field http\_address from \_nodes that does not exists anymore in ES 5.0.0 Related issue in ES 5: According to the code file (https://github.com/logstash-p… --- ## [Grok filter behaviour different for some messages](https://discuss.elastic.co/t/grok-filter-behaviour-different-for-some-messages/66078)
**Author:** [@Arthur\_Francis](https://discuss.elastic.co/u/Arthur_Francis)\ **Replies:** 1\ **Last updated:** [November 24, 2016, 9:39am UTC](https://discuss.elastic.co/t/grok-filter-behaviour-different-for-some-messages/66078 "2016-11-24T09:39:09Z")
Hi All, I am currently trying to setup some data collections for our app using the full elk stack (Beats - Logstash - ElasticSearch-Kibana). So far everything is working as it should but I have a requirement to capture… --- ## [NoMethodError in logstash-input-github plugin](https://discuss.elastic.co/t/nomethoderror-in-logstash-input-github-plugin/67063)
**Author:** [@fiskeben](https://discuss.elastic.co/u/fiskeben)\ **Replies:** 1\ **Last updated:** [November 24, 2016, 8:48am UTC](https://discuss.elastic.co/t/nomethoderror-in-logstash-input-github-plugin/67063 "2016-11-24T08:48:57Z")
Hi, I'm testing out Logstash and the logstash-input-github plugin but I'm getting an error on incoming requests: #\\> /usr/share/logstash/vendor… --- ## [Limit Elasticsearch index size](https://discuss.elastic.co/t/limit-elasticsearch-index-size/67059)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 1\ **Last updated:** [November 24, 2016, 8:41am UTC](https://discuss.elastic.co/t/limit-elasticsearch-index-size/67059 "2016-11-24T08:41:56Z")
I have a data stream that have around 100 - 200 GB of data coming into my logstash. How do I configure logstash or elasticsearch so that the output indexes each has a maximum size of 20 GB? E.g. 200 GB = 10 (20 GB ind… --- ## [3x higher load with Logstash 5.x than with LS 2.4](https://discuss.elastic.co/t/3x-higher-load-with-logstash-5-x-than-with-ls-2-4/67028)
**Author:** [@skrueger8](https://discuss.elastic.co/u/skrueger8)\ **Replies:** 0\ **Last updated:** [November 23, 2016, 6:45pm UTC](https://discuss.elastic.co/t/3x-higher-load-with-logstash-5-x-than-with-ls-2-4/67028 "2016-11-23T18:45:17Z")
Hi, we're trying to upgrade LS 2.4 to LS 5.0 (Solaris SPARC, both 2.4 and 5.x running java8) but we're seeing a 3x higher load with 5.x... Here's a Grafana Graph showing an idle test server's load while running Logst… --- ## [Rename ORA- field after match](https://discuss.elastic.co/t/rename-ora-field-after-match/66918)
**Author:** [@FrankC](https://discuss.elastic.co/u/FrankC)\ **Replies:** 1\ **Last updated:** [November 23, 2016, 4:38pm UTC](https://discuss.elastic.co/t/rename-ora-field-after-match/66918 "2016-11-23T16:38:33Z")
Hello, I am pretty close with extracting the ORA error out of the message field. However the field created has the dash ORA-. I want to name the field something else. For example ORA\_err. Grok snippet: grok { m… --- ## [Logstash failed to open file](https://discuss.elastic.co/t/logstash-failed-to-open-file/65019)
**Author:** [@Salafer](https://discuss.elastic.co/u/Salafer)\ **Replies:** 1\ **Last updated:** [November 23, 2016, 4:24pm UTC](https://discuss.elastic.co/t/logstash-failed-to-open-file/65019 "2016-11-23T16:24:14Z")
Hi i have a simple problem. Logstash 5.0 can't open my log file. It simply say: \[2016-11-04T16:11:39,838\]\[WARN \]\[logstash.inputs.file \] failed to open /logs/icp2/localhost\_access\_log.2016-10-31.txt: \\logs\\icp2\\loca… --- ## [Tomcat catalina.out in json format](https://discuss.elastic.co/t/tomcat-catalina-out-in-json-format/67009)
**Author:** [@nick76](https://discuss.elastic.co/u/nick76)\ **Replies:** 0\ **Last updated:** [November 23, 2016, 4:23pm UTC](https://discuss.elastic.co/t/tomcat-catalina-out-in-json-format/67009 "2016-11-23T16:23:59Z")
hello all, I'm trying to manage all the different stack trace we have in our tomcat catalina.out log. I've implemented for the tomcat.log file the log4j-\>json formatter (https://github.com/logstash/log4j-jsonevent-layou… --- ## [Grok rewrite field value if matches string](https://discuss.elastic.co/t/grok-rewrite-field-value-if-matches-string/66958)
**Author:** [@Arthur\_Francis](https://discuss.elastic.co/u/Arthur_Francis)\ **Replies:** 1\ **Last updated:** [November 23, 2016, 4:13pm UTC](https://discuss.elastic.co/t/grok-rewrite-field-value-if-matches-string/66958 "2016-11-23T16:13:45Z")
Hello, I am trying to extract a value from a log message that can take various forms, I have the 2 different regex patters which work individually. if the first regex produces a certain field value I want to rewrite … --- ## [Logstash - multiple indices for one input file](https://discuss.elastic.co/t/logstash-multiple-indices-for-one-input-file/66927)
**Author:** [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\ **Replies:** 2\ **Last updated:** [November 23, 2016, 1:31pm UTC](https://discuss.elastic.co/t/logstash-multiple-indices-for-one-input-file/66927 "2016-11-23T13:31:59Z")
I want to create multiple indices using Logstash configuration into ES. From my understanding, it is easy to use type in file-input if we have multiple inputs. Then, create one index per one type. But, I do not have m… --- ## [Logstash basic setup not working](https://discuss.elastic.co/t/logstash-basic-setup-not-working/66949)
**Author:** [@rocky4bmw](https://discuss.elastic.co/u/rocky4bmw)\ **Replies:** 3\ **Last updated:** [November 23, 2016, 8:34am UTC](https://discuss.elastic.co/t/logstash-basic-setup-not-working/66949 "2016-11-23T08:34:37Z")
Hi Team , I have installed logstash and tried to use filters to get the logs which are present locally to print. But it is not working. I get below error can you please check and help me. Please find below logstash con… --- ## [How to create map based on longitude, latitude fileds?](https://discuss.elastic.co/t/how-to-create-map-based-on-longitude-latitude-fileds/66664)
**Author:** [@namcx](https://discuss.elastic.co/u/namcx)\ **Replies:** 3\ **Last updated:** [November 23, 2016, 6:42am UTC](https://discuss.elastic.co/t/how-to-create-map-based-on-longitude-latitude-fileds/66664 "2016-11-23T06:42:18Z")
Hi Experts, I had add 2 field with name lat,lon and value 21.033333, 105.849998 to index by logstash filter plugin "mutate" if \[fields\]\[Contract\] =~ "HNH" { mutate { add\_field =\> \[ "city", "Hanoi" \] … --- ## [The output cann't be seen in real time](https://discuss.elastic.co/t/the-output-cannt-be-seen-in-real-time/66942)
**Author:** [@ChrisChen](https://discuss.elastic.co/u/ChrisChen)\ **Replies:** 0\ **Last updated:** [November 23, 2016, 5:24am UTC](https://discuss.elastic.co/t/the-output-cannt-be-seen-in-real-time/66942 "2016-11-23T05:24:06Z")
Hi everyone, I am a fresh to Logstash, Now i have a issue as following: I want to send data from one logstash on A machine to other logstash on B machine. the configuration file: B machine: input { tcp{ port … --- ## [Fail to get varnishlog to work](https://discuss.elastic.co/t/fail-to-get-varnishlog-to-work/66941)
**Author:** [@lhwong](https://discuss.elastic.co/u/lhwong)\ **Replies:** 0\ **Last updated:** [November 23, 2016, 4:59am UTC](https://discuss.elastic.co/t/fail-to-get-varnishlog-to-work/66941 "2016-11-23T04:59:48Z")
logstash keeps restarting with the following error when using varnishlog (logstash-input-varnishlog) with the following configuration. Any idea how to get it to work? input { varnishlog {} } output { elasticse… --- ## [TCP output to TCP input](https://discuss.elastic.co/t/tcp-output-to-tcp-input/66532)
**Author:** [@luka](https://discuss.elastic.co/u/luka)\ **Replies:** 11\ **Last updated:** [November 23, 2016, 3:26am UTC](https://discuss.elastic.co/t/tcp-output-to-tcp-input/66532 "2016-11-23T03:26:39Z")
Hi everyone, I'm learning Logstash and I've got myself into this trouble. I want to send data from a Logstash with tcp output to another one with tcp input. I guess that should work, but even though I can see tcp pac… --- ## [Remove part of message string](https://discuss.elastic.co/t/remove-part-of-message-string/66809)
**Author:** [@Elias\_Lim](https://discuss.elastic.co/u/Elias_Lim)\ **Replies:** 6\ **Last updated:** [November 23, 2016, 2:46am UTC](https://discuss.elastic.co/t/remove-part-of-message-string/66809 "2016-11-23T02:46:47Z")
Hi there, In the message field in kibana i am getting 2016-11-22 11:37:50, 943 app INFO: error\_test - 82 test message 2016-11-22 11:37:50, 943 app ERROR: error\_test - 83 test error log As I have a separate @times… --- ## [Bar graph of values in one event](https://discuss.elastic.co/t/bar-graph-of-values-in-one-event/66663)
**Author:** [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\ **Replies:** 2\ **Last updated:** [November 23, 2016, 1:44am UTC](https://discuss.elastic.co/t/bar-graph-of-values-in-one-event/66663 "2016-11-23T01:44:58Z")
I am testing out ELK for specific use case. I have several JSON objects like this: { "field1": "uniquevalue1", "field2": \[ { "field2a": "value2a", "time": \ }, { "fi… --- ## [How to split a JSON array inside an object](https://discuss.elastic.co/t/how-to-split-a-json-array-inside-an-object/25511)
**Author:** [@joerno](https://discuss.elastic.co/u/joerno)\ **Replies:** 1\ **Last updated:** [November 23, 2016, 1:05am UTC](https://discuss.elastic.co/t/how-to-split-a-json-array-inside-an-object/25511 "2016-11-23T01:05:13Z")
I have a JSON-Message with an array in an array. I want to split that into multiple events (because of kibana): { "type": "monitor", "server": "10.111.222.333", "host": "abc.de", "bean": \[{ "name": "beanName1", … --- ## [Why always need to run first logstash conf file then put log file on specific folder](https://discuss.elastic.co/t/why-always-need-to-run-first-logstash-conf-file-then-put-log-file-on-specific-folder/66852)
**Author:** [@Roshan](https://discuss.elastic.co/u/Roshan)\ **Replies:** 1\ **Last updated:** [November 22, 2016, 9:22pm UTC](https://discuss.elastic.co/t/why-always-need-to-run-first-logstash-conf-file-then-put-log-file-on-specific-folder/66852 "2016-11-22T21:22:52Z")
Hi, i am running logstash through pragmatically when first i putted file on specific folder then started running longstash file through command . then need to open that log file from specific folder do some operation and… --- ## [Logstash S3 Bucket filename in input](https://discuss.elastic.co/t/logstash-s3-bucket-filename-in-input/66916)
**Author:** [@JMeter\_Dude](https://discuss.elastic.co/u/JMeter_Dude)\ **Replies:** 0\ **Last updated:** [November 22, 2016, 9:08pm UTC](https://discuss.elastic.co/t/logstash-s3-bucket-filename-in-input/66916 "2016-11-22T21:08:05Z")
Hi. I want to send the log file name stored inside my S3 bucket as an input to the logstash. If I try to give the path, it fails with error. If I just give the bucket name it works. If I try to give it as a path it… --- ## [XML attributes and arrays](https://discuss.elastic.co/t/xml-attributes-and-arrays/66522)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 2\ **Last updated:** [November 22, 2016, 9:06pm UTC](https://discuss.elastic.co/t/xml-attributes-and-arrays/66522 "2016-11-22T21:06:52Z")
Hi, what about parsing this XML with Logstash \ \ \ \ \ \ \ \ I'm using the… --- ## [Why do I can select two Timestamp field in Configure an index pattern](https://discuss.elastic.co/t/why-do-i-can-select-two-timestamp-field-in-configure-an-index-pattern/66546)
**Author:** [@zoram](https://discuss.elastic.co/u/zoram)\ **Replies:** 1\ **Last updated:** [November 22, 2016, 9:03pm UTC](https://discuss.elastic.co/t/why-do-i-can-select-two-timestamp-field-in-configure-an-index-pattern/66546 "2016-11-22T21:03:38Z")
Dear All, Why do I can select two Timestamp field in "Configure an index pattern". @timestamp timeststamp What is different between this two time-field name? Many thanks in Advance. Regards, TK --- ## [Filebeat To Logstash caused circuit break errors](https://discuss.elastic.co/t/filebeat-to-logstash-caused-circuit-break-errors/66494)
**Author:** [@locolbd](https://discuss.elastic.co/u/locolbd)\ **Replies:** 1\ **Last updated:** [November 22, 2016, 9:02pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-caused-circuit-break-errors/66494 "2016-11-22T21:02:14Z")
First overview of the environment showing the flow of logs Filebeat -\> logstash(ingest) -\> Redis(Queue Messaging) -\> Logstash Filter Nodes (3) -\> Elastic Search... We currently have syslog sending logs and it showing u… --- ## [Reindexing with logstash](https://discuss.elastic.co/t/reindexing-with-logstash/66464)
**Author:** [@nin](https://discuss.elastic.co/u/nin)\ **Replies:** 1\ **Last updated:** [November 22, 2016, 8:59pm UTC](https://discuss.elastic.co/t/reindexing-with-logstash/66464 "2016-11-22T20:59:44Z")
I have a an old elastic cluster running 1.7.5 that is underperforming. I've setup a new cluster on 2.4.6 and want to use logstash to reindex data from old to new. In the process I am consolidating daily indexes to month… --- ## [ELK-5.0 Logstash geoip filter target "geoip", unable to remove fields](https://discuss.elastic.co/t/elk-5-0-logstash-geoip-filter-target-geoip-unable-to-remove-fields/66465)
**Author:** [@sps90](https://discuss.elastic.co/u/sps90)\ **Replies:** 1\ **Last updated:** [November 22, 2016, 8:58pm UTC](https://discuss.elastic.co/t/elk-5-0-logstash-geoip-filter-target-geoip-unable-to-remove-fields/66465 "2016-11-22T20:58:03Z")
I am currently using ELK 5.0 version I need only few of the tags from the default "geoip" target. But i'm unable to delete the unwanted fields. I have tried the following settings, but none worked. I still see all the… --- ## [Logstash 2.4.1 pipeline is not getting started](https://discuss.elastic.co/t/logstash-2-4-1-pipeline-is-not-getting-started/66417)
**Author:** [@PraveenRLV](https://discuss.elastic.co/u/PraveenRLV)\ **Replies:** 1\ **Last updated:** [November 22, 2016, 8:56pm UTC](https://discuss.elastic.co/t/logstash-2-4-1-pipeline-is-not-getting-started/66417 "2016-11-22T20:56:22Z")
Hi all, i am newbie to logstash,, i am using logstash 2.4.1 and when i process the log it is getting processed first time but from second time the pipe line it not getting started. could any one help why this is occurr… --- ## [Limiting the maximum number of exec input commands](https://discuss.elastic.co/t/limiting-the-maximum-number-of-exec-input-commands/66415)
**Author:** [@valentinul](https://discuss.elastic.co/u/valentinul)\ **Replies:** 1\ **Last updated:** [November 22, 2016, 8:55pm UTC](https://discuss.elastic.co/t/limiting-the-maximum-number-of-exec-input-commands/66415 "2016-11-22T20:55:33Z")
Hi there, I'm using the exec input to run some perl scrips that connects to some db and retrieves some data. The issue is that I have like 30 exec statements and they all get executed when I start LS for the first time… --- ## [COMBINEDAPACHELOG did not add any matched fileds?](https://discuss.elastic.co/t/combinedapachelog-did-not-add-any-matched-fileds/66791)
**Author:** [@robertchen117](https://discuss.elastic.co/u/robertchen117)\ **Replies:** 5\ **Last updated:** [November 22, 2016, 8:37pm UTC](https://discuss.elastic.co/t/combinedapachelog-did-not-add-any-matched-fileds/66791 "2016-11-22T20:37:00Z")
my logstash filter rule is like: if \[type\] == "apache" { grok { match =\> \["message", "%{COMBINEDAPACHELOG}"\] match =\> \["message", "%{HTTPD20\_ERRORLOG}"\] } } COMBINEDAPACHELOG did not add any matched fileds, or it … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1072) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1074)