# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1203 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1204 --- ## [Not getting all Records imported from Logstash to Elasticsearch](https://discuss.elastic.co/t/not-getting-all-records-imported-from-logstash-to-elasticsearch/38697)
**Author:** [@trushad](https://discuss.elastic.co/u/trushad)\ **Replies:** 4\ **Last updated:** [January 11, 2016, 9:20am UTC](https://discuss.elastic.co/t/not-getting-all-records-imported-from-logstash-to-elasticsearch/38697 "2016-01-11T09:20:40Z")
My config file is as below: input { jdbc { jdbc\_driver\_library =\> "D:\\Microsoft JDBC Driver 6.0 for SQL Server\\sqljdbc\_6.0\\enu\\sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc… --- ## [Logstash: ArgumentError: malformed format string - %a](https://discuss.elastic.co/t/logstash-argumenterror-malformed-format-string-a/38837)
**Author:** [@krushnat\_khawale](https://discuss.elastic.co/u/krushnat_khawale)\ **Replies:** 1\ **Last updated:** [January 11, 2016, 7:19am UTC](https://discuss.elastic.co/t/logstash-argumenterror-malformed-format-string-a/38837 "2016-01-11T07:19:23Z")
Custom filter fails with field containing '%' sign. I have created a custom filter where I am doing comparison operations on the one of the field, while comparing the values I am using 'include' and 'empty' method of st… --- ## [Enriching new documents with fields from other ES indexes](https://discuss.elastic.co/t/enriching-new-documents-with-fields-from-other-es-indexes/38827)
**Author:** [@treehouse](https://discuss.elastic.co/u/treehouse)\ **Replies:** 0\ **Last updated:** [January 11, 2016, 1:06am UTC](https://discuss.elastic.co/t/enriching-new-documents-with-fields-from-other-es-indexes/38827 "2016-01-11T01:06:49Z")
Hey guys I'm looking for a very performant way of enriching new log data with data already in ES. For a potential scenario assume that we have a (scrubbed) copy of our MySQL users table in ES. This logstash instance… --- ## [Logstash with Kafka](https://discuss.elastic.co/t/logstash-with-kafka/37776)
**Author:** [@Deb](https://discuss.elastic.co/u/Deb)\ **Replies:** 5\ **Last updated:** [January 10, 2016, 9:55pm UTC](https://discuss.elastic.co/t/logstash-with-kafka/37776 "2016-01-10T21:55:28Z")
Does logstash support kafka 0.9? I am not seeing anything about the Kafka version in the logstash-kafka doc. --- ## [Output to unicast ES Cluster, how?](https://discuss.elastic.co/t/output-to-unicast-es-cluster-how/38796)
**Author:** [@jnpetty](https://discuss.elastic.co/u/jnpetty)\ **Replies:** 3\ **Last updated:** [January 10, 2016, 8:59am UTC](https://discuss.elastic.co/t/output-to-unicast-es-cluster-how/38796 "2016-01-10T08:59:55Z")
I have a 3 node ES cluster setup and im trying to figure out how to make Logstash output to a cluster. I know that I can add something similar to the following: elasticsearch { cluster =\> "logstash-clus… --- ## [Logstash "The error reported is: \\n Permission denied - bind(2)"](https://discuss.elastic.co/t/logstash-the-error-reported-is-n-permission-denied-bind-2/38770)
**Author:** [@shihab\_rabee](https://discuss.elastic.co/u/shihab_rabee)\ **Replies:** 3\ **Last updated:** [January 10, 2016, 4:56am UTC](https://discuss.elastic.co/t/logstash-the-error-reported-is-n-permission-denied-bind-2/38770 "2016-01-10T04:56:41Z")
Logstash fails to start when listining on port 80. although i am starting it with root user logstash "The error reported is: \\n Permission denied - bind(2)" --- ## [Odd indices created automatically - what's happening?](https://discuss.elastic.co/t/odd-indices-created-automatically-whats-happening/38709)
**Author:** [@hpniemitube](https://discuss.elastic.co/u/hpniemitube)\ **Replies:** 1\ **Last updated:** [January 10, 2016, 12:46am UTC](https://discuss.elastic.co/t/odd-indices-created-automatically-whats-happening/38709 "2016-01-10T00:46:53Z")
Hi, Just started to use ES 2.1.1. I have several servers with a logstash 1.5.6-1. Now I noticed, that in addition to the correct daily index, that is created automatically: ylefi\_drupalsv-2016.01.08 5 2 576213 0 … --- ## [Sending logs to logstash securely](https://discuss.elastic.co/t/sending-logs-to-logstash-securely/38784)
**Author:** [@pritchardjonathan](https://discuss.elastic.co/u/pritchardjonathan)\ **Replies:** 3\ **Last updated:** [January 9, 2016, 6:30pm UTC](https://discuss.elastic.co/t/sending-logs-to-logstash-securely/38784 "2016-01-09T18:30:47Z")
Hi, First, apologies for the noob question. I have my ELK server and application servers in different locations. I'm unclear about the best route to take to securely send my logs over the web. I know that lumberjack an… --- ## [Logstash is complaining about the path of the certificates](https://discuss.elastic.co/t/logstash-is-complaining-about-the-path-of-the-certificates/38788)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 2\ **Last updated:** [January 9, 2016, 3:43pm UTC](https://discuss.elastic.co/t/logstash-is-complaining-about-the-path-of-the-certificates/38788 "2016-01-09T15:43:41Z")
Hello, I am running the latest version of logstash 2.1 and I have configured the certificates as it: beats { port =\> 5044 type =\> "logs" ssl =\> "true" ssl\_certificate =\> "/etc/letsencrypt/live/core.z0z0… --- ## [Parsing Websphere SysErr log file using logstash](https://discuss.elastic.co/t/parsing-websphere-syserr-log-file-using-logstash/38677)
**Author:** [@praveerg](https://discuss.elastic.co/u/praveerg)\ **Replies:** 5\ **Last updated:** [January 9, 2016, 3:31pm UTC](https://discuss.elastic.co/t/parsing-websphere-syserr-log-file-using-logstash/38677 "2016-01-09T15:31:38Z")
I need help to create a pipeline file for IBM websphere systemerr log file which has combination of single line and multiline events. Can some help here ? --- ## [Unique identified per line during logstash](https://discuss.elastic.co/t/unique-identified-per-line-during-logstash/38705)
**Author:** [@subbu.nv](https://discuss.elastic.co/u/subbu.nv)\ **Replies:** 3\ **Last updated:** [January 9, 2016, 12:33am UTC](https://discuss.elastic.co/t/unique-identified-per-line-during-logstash/38705 "2016-01-09T00:33:58Z")
Hi All, is there a way to add a uniquer identified per line when we index log files to elastic search. initially i thought of considering time stamp as the unique identifier per line if we are running a single thread, … --- ## [Logstash shipper reports hostname as localhost](https://discuss.elastic.co/t/logstash-shipper-reports-hostname-as-localhost/38648)
**Author:** [@mburlington](https://discuss.elastic.co/u/mburlington)\ **Replies:** 3\ **Last updated:** [January 8, 2016, 6:06pm UTC](https://discuss.elastic.co/t/logstash-shipper-reports-hostname-as-localhost/38648 "2016-01-08T18:06:15Z")
Hi, I have a set of configuration files I'm using to monitor Tomcat. I use these on various servers but when I recently deployed these to a new server I noticed that the "host" field appears as "localhost" when I view… --- ## [Logstash grok pattern not working even though testers say it should](https://discuss.elastic.co/t/logstash-grok-pattern-not-working-even-though-testers-say-it-should/38739)
**Author:** [@OneTrueBob74](https://discuss.elastic.co/u/OneTrueBob74)\ **Replies:** 4\ **Last updated:** [January 8, 2016, 4:05pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-not-working-even-though-testers-say-it-should/38739 "2016-01-08T16:05:18Z")
I've been building some grok patterns to parse the /var/log/secure log file and everything has been working fine. I create the grok patterns at http://grokconstructor.appspot.com/ and even then test them at http://grokd… --- ## [Logstash grok filter thinks we are still in 2015](https://discuss.elastic.co/t/logstash-grok-filter-thinks-we-are-still-in-2015/38731)
**Author:** [@achechen](https://discuss.elastic.co/u/achechen)\ **Replies:** 4\ **Last updated:** [January 8, 2016, 2:51pm UTC](https://discuss.elastic.co/t/logstash-grok-filter-thinks-we-are-still-in-2015/38731 "2016-01-08T14:51:57Z")
Hello there, weird problem. I have the following config, which worked in 2015: input { tcp { port =\> 5000 type =\> syslog } udp { port =\> 5000 type =\> syslog } udp { port =\> 122… --- ## [Logstash udp input queue\_size](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253)
**Author:** [@achechen](https://discuss.elastic.co/u/achechen)\ **Replies:** 9\ **Last updated:** [January 8, 2016, 2:02pm UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253 "2016-01-08T14:02:41Z")
Hello There, what exactly does queue\_size parameter in UDP input do? I have a logstash server which is not able to consume all UDP events that are sent to it therefore a lot of UDP events are being dropped. Documen… --- ## [Odd indices created automatically - what is happening?](https://discuss.elastic.co/t/odd-indices-created-automatically-what-is-happening/38711)
**Author:** [@hpniemitube](https://discuss.elastic.co/u/hpniemitube)\ **Replies:** 4\ **Last updated:** [January 8, 2016, 11:45am UTC](https://discuss.elastic.co/t/odd-indices-created-automatically-what-is-happening/38711 "2016-01-08T11:45:59Z")
Hi, Just started to use ES 2.1.1. I have several servers with a logstash 1.5.6-1. Now I noticed, that in addition to the correct daily index, that is created automatically: ylefi\_drupalsv-2016.01.08 5 2 576213 0 1.1g… --- ## [Server workflow direction](https://discuss.elastic.co/t/server-workflow-direction/38640)
**Author:** [@nick76](https://discuss.elastic.co/u/nick76)\ **Replies:** 2\ **Last updated:** [January 8, 2016, 9:43am UTC](https://discuss.elastic.co/t/server-workflow-direction/38640 "2016-01-08T09:43:03Z")
Hello all, I'm absolutely new to Logstash, Elasticsearch and kibana. My boss asked me to put in place a cluster of logstash and elastic search servers for collecting all the logs we have in our DMZ server. What I wish t… --- ## [LSF has been deprecated in favour of Filebeat](https://discuss.elastic.co/t/lsf-has-been-deprecated-in-favour-of-filebeat/38696)
**Author:** [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Replies:** 2\ **Last updated:** [January 8, 2016, 7:39am UTC](https://discuss.elastic.co/t/lsf-has-been-deprecated-in-favour-of-filebeat/38696 "2016-01-08T07:39:33Z")
The Logstash Forwarder is being retired and replaced with Filebeat. Filebeat has improved functionality, including the ability to ship to multiple different outputs such as directly to Elasticsearch or to Logstash, b… --- ## [Insert into logstash select data from database](https://discuss.elastic.co/t/insert-into-logstash-select-data-from-database/38600)
**Author:** [@Architha](https://discuss.elastic.co/u/Architha)\ **Replies:** 1\ **Last updated:** [January 8, 2016, 5:53am UTC](https://discuss.elastic.co/t/insert-into-logstash-select-data-from-database/38600 "2016-01-08T05:53:55Z")
Hi All, I have been following the article on inserting select data from database to Logstash by Tal Levy. (URL: https://www.elastic.co/blog/logstash-jdbc-input-plugin ) My query is can we include multiple query stateme… --- ## [Need advice on AWS logs indexing using logstash](https://discuss.elastic.co/t/need-advice-on-aws-logs-indexing-using-logstash/38667)
**Author:** [@subbu.nv](https://discuss.elastic.co/u/subbu.nv)\ **Replies:** 6\ **Last updated:** [January 8, 2016, 1:35am UTC](https://discuss.elastic.co/t/need-advice-on-aws-logs-indexing-using-logstash/38667 "2016-01-08T01:35:44Z")
i have elastic search + logstash installed on one server. i am trying to index the logs from AWS and push into elastic search. can any one help with the format i have to use in logstash conf file input. --- ## [Jdbc input plugin and rebuilding elasticsearch index?](https://discuss.elastic.co/t/jdbc-input-plugin-and-rebuilding-elasticsearch-index/38674)
**Author:** [@Joshua\_P](https://discuss.elastic.co/u/Joshua_P)\ **Replies:** 2\ **Last updated:** [January 7, 2016, 10:52pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-and-rebuilding-elasticsearch-index/38674 "2016-01-07T22:52:57Z")
Is there a way to use the clean\_run parameter in the jdbc input plugin to rebuild an index once a day while still doing periodic updates throughout the day to the same index? Using clean\_run doesn't seem to work the way … --- ## [Use threshold to trigger email sending](https://discuss.elastic.co/t/use-threshold-to-trigger-email-sending/35549)
**Author:** [@marcmga](https://discuss.elastic.co/u/marcmga)\ **Replies:** 5\ **Last updated:** [January 7, 2016, 10:21pm UTC](https://discuss.elastic.co/t/use-threshold-to-trigger-email-sending/35549 "2016-01-07T22:21:08Z")
Hi, I'm using logstash + elasticsearch + kibana to monitor my logfiles. What i'm doing is sending e-mail each time a field contains a pattern. But it is not usefull as i get a lot of entries with that pattern. I want t… --- ## [Filters in a Distributed Logstash Setup](https://discuss.elastic.co/t/filters-in-a-distributed-logstash-setup/38663)
**Author:** [@lethalMango](https://discuss.elastic.co/u/lethalMango)\ **Replies:** 2\ **Last updated:** [January 7, 2016, 8:20pm UTC](https://discuss.elastic.co/t/filters-in-a-distributed-logstash-setup/38663 "2016-01-07T20:20:38Z")
I've recently inherited the following setup to manage: Logstash (Collector) -\> Redis -\> Logstash Indexer -\> Elasticsearch I can see a number of filters on the collector instances as well as the indexers. Am I correct… --- ## [LSF to send multiple log files with one config](https://discuss.elastic.co/t/lsf-to-send-multiple-log-files-with-one-config/38669)
**Author:** [@Kaufusihm](https://discuss.elastic.co/u/Kaufusihm)\ **Replies:** 0\ **Last updated:** [January 7, 2016, 7:20pm UTC](https://discuss.elastic.co/t/lsf-to-send-multiple-log-files-with-one-config/38669 "2016-01-07T19:20:45Z")
I'm trying to send multiple log files using one LSF.conf to a logstash collector. I seem to only get one file sending. Or send a directory? { "network": { "servers": \[ "ls-ip:5000" \], "timeout": 15, "ssl ca"… --- ## [WARN: org.elasticsearch.transport.netty:](https://discuss.elastic.co/t/warn-org-elasticsearch-transport-netty/38653)
**Author:** [@constantino](https://discuss.elastic.co/u/constantino)\ **Replies:** 4\ **Last updated:** [January 7, 2016, 7:14pm UTC](https://discuss.elastic.co/t/warn-org-elasticsearch-transport-netty/38653 "2016-01-07T19:14:33Z")
Hi, I'm working with ElasticSearch 1.5.1. and LogsTash1.4.2 if send message as follows: logstash -e 'input { stdin { } } output { elasticsearch { host =\> \["192.168.32.124"\] } }' everything works well, but whe… --- ## [Filter mutate replace field value with value of another field](https://discuss.elastic.co/t/filter-mutate-replace-field-value-with-value-of-another-field/38660)
**Author:** [@Chris\_Clifton](https://discuss.elastic.co/u/Chris_Clifton)\ **Replies:** 2\ **Last updated:** [January 7, 2016, 5:49pm UTC](https://discuss.elastic.co/t/filter-mutate-replace-field-value-with-value-of-another-field/38660 "2016-01-07T17:49:58Z")
Hoping this is a simple syntax issue, I'm adding a tag to events from filebeat on the client shipper, fields: tag\_hostname: "Dev Server" host value is already present in LS, I want to replace the value of the… --- ## [Logstash how to handle exceptions](https://discuss.elastic.co/t/logstash-how-to-handle-exceptions/37843)
**Author:** [@chicco\_95](https://discuss.elastic.co/u/chicco_95)\ **Replies:** 27\ **Last updated:** [January 7, 2016, 1:53pm UTC](https://discuss.elastic.co/t/logstash-how-to-handle-exceptions/37843 "2016-01-07T13:53:28Z")
Hi, i have a problem in managing exceptions in filter logstash. I have to define some dates which don't always show the same format and this causes the following error: Ruby exception occurred: invalid strptime format… --- ## [Parsing multiline logs : line + xml](https://discuss.elastic.co/t/parsing-multiline-logs-line-xml/38417)
**Author:** [@carmelom](https://discuss.elastic.co/u/carmelom)\ **Replies:** 9\ **Last updated:** [January 7, 2016, 12:11pm UTC](https://discuss.elastic.co/t/parsing-multiline-logs-line-xml/38417 "2016-01-07T12:11:46Z")
Below an example of my logs indent preformatted text by 4 spaces INFO 05-01-16 08:06:01 \[http-nio-8080-exec-8\] (AbstractServer.java:454) - \ \localhost\ \311\ … --- ## [Need to add timezone field to my output](https://discuss.elastic.co/t/need-to-add-timezone-field-to-my-output/38628)
**Author:** [@miko370](https://discuss.elastic.co/u/miko370)\ **Replies:** 1\ **Last updated:** [January 7, 2016, 3:09pm UTC](https://discuss.elastic.co/t/need-to-add-timezone-field-to-my-output/38628 "2016-01-07T15:09:38Z")
Hi. I need to add an extra field that will include the timezone of the local (LogStash agent) machine. I have tried using add\_field =\> {"TZ" =\> "%{+z}"} but it always give the UTC time. Any way I can do that withou… --- ## [Drop entire log if condition is met](https://discuss.elastic.co/t/drop-entire-log-if-condition-is-met/38604)
**Author:** [@lethalMango](https://discuss.elastic.co/u/lethalMango)\ **Replies:** 2\ **Last updated:** [January 7, 2016, 1:43pm UTC](https://discuss.elastic.co/t/drop-entire-log-if-condition-is-met/38604 "2016-01-07T13:43:51Z")
Forgive me if I've missed this in the docs as I couldn't see it. Is it possible to drop an entire log based on a condition? i.e. eventlog\_id 5156 is never needed and is disabled on the majority of servers we have. Ra… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1202) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1204)