# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1088 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1089 --- ## [KAFKA server keeping too much TCP ESTABLISHED connections with Logstash Windows Client](https://discuss.elastic.co/t/kafka-server-keeping-too-much-tcp-established-connections-with-logstash-windows-client/63825)
**Author:** [@zhoufwind](https://discuss.elastic.co/u/zhoufwind)\ **Replies:** 0\ **Last updated:** [October 25, 2016, 6:04am UTC](https://discuss.elastic.co/t/kafka-server-keeping-too-much-tcp-established-connections-with-logstash-windows-client/63825 "2016-10-25T06:04:30Z")
Hi folks, My kafka server connect a bunch of logstash shippers, which running on Windows platform. We have detected Kafka TCP established connections over 30k alarms these days, it seems like Kafka server keeping too m… --- ## [Cant disable debug](https://discuss.elastic.co/t/cant-disable-debug/63662)
**Author:** [@DocMAX1](https://discuss.elastic.co/u/DocMAX1)\ **Replies:** 1\ **Last updated:** [October 25, 2016, 5:47am UTC](https://discuss.elastic.co/t/cant-disable-debug/63662 "2016-10-25T05:47:53Z")
I added switch --quiet but still i get these strange lines...: Okt 22 15:23:53 server logstash\[28752\]: D, \[2016-10-22T15:23:53.734000 #28752\] DEBUG -- : SENDING: Okt 22 15:23:53 server logstash\[28752\]: Okt 22 15:24… --- ## [Grok will not change field type to Int or IP](https://discuss.elastic.co/t/grok-will-not-change-field-type-to-int-or-ip/63756)
**Author:** [@sancroth](https://discuss.elastic.co/u/sancroth)\ **Replies:** 5\ **Last updated:** [October 25, 2016, 5:37am UTC](https://discuss.elastic.co/t/grok-will-not-change-field-type-to-int-or-ip/63756 "2016-10-25T05:37:16Z")
While trying to match my VPN connections with the match below : match =\> \[ "message", "%{WORD:openvpn\_user}/%{IP:openvpn\_scr\_ip}:%{INT:openvpn\_scr\_port} MULTI\_sva: pool returned IPv4=%{IP:openvpn\_ip}" \] I get my log… --- ## [Using Multiple IF conditionals in Filter](https://discuss.elastic.co/t/using-multiple-if-conditionals-in-filter/63714)
**Author:** [@setlem](https://discuss.elastic.co/u/setlem)\ **Replies:** 4\ **Last updated:** [October 25, 2016, 5:36am UTC](https://discuss.elastic.co/t/using-multiple-if-conditionals-in-filter/63714 "2016-10-25T05:36:21Z")
am getting output if i have all three (CREATION\_DATE,SUBMITTED\_DATE,LAST\_MODIFIED\_DATE) in date format.If any is STRING am not getting that log file in my output. for ex: my input is 12-JUL-13 11.33.56.259 AM,12-… --- ## [Logstash sqlite plugin can't output in elasticsearch](https://discuss.elastic.co/t/logstash-sqlite-plugin-cant-output-in-elasticsearch/63701)
**Author:** [@Rem](https://discuss.elastic.co/u/Rem)\ **Replies:** 3\ **Last updated:** [October 25, 2016, 3:43am UTC](https://discuss.elastic.co/t/logstash-sqlite-plugin-cant-output-in-elasticsearch/63701 "2016-10-25T03:43:28Z")
Hello, I really need to store my logs in a database because I will modify the data so I tried using the sqlite pluging of logstash. Here's my config file : input { sqlite { path =\> "C:/Users/Library/Documen… --- ## [Logstash Date Filter, UNIX date format parse error, date is malformed](https://discuss.elastic.co/t/logstash-date-filter-unix-date-format-parse-error-date-is-malformed/63647)
**Author:** [@Dark\_Light](https://discuss.elastic.co/u/Dark_Light)\ **Replies:** 2\ **Last updated:** [October 24, 2016, 4:09pm UTC](https://discuss.elastic.co/t/logstash-date-filter-unix-date-format-parse-error-date-is-malformed/63647 "2016-10-24T16:09:32Z")
I am trying to parse the Unix date from Squid logs with the Date plugin using UNIX date format. The date does not convert and there are errors in the Logstash log stating the date is malformed at the last number in the … --- ## [Aggregate filter not run at all](https://discuss.elastic.co/t/aggregate-filter-not-run-at-all/39602)
**Author:** [@CamJN](https://discuss.elastic.co/u/CamJN)\ **Replies:** 3\ **Last updated:** [October 24, 2016, 3:20pm UTC](https://discuss.elastic.co/t/aggregate-filter-not-run-at-all/39602 "2016-10-24T15:20:15Z")
I'm trying to carry forward a year that is specified infrequently in my logs, and it seemed like the aggregate filter would work beautifully for this, however the filter doesn't seem to actually do anything (I've added t… --- ## [Grokprasefailure Issue with hyphen in the log](https://discuss.elastic.co/t/grokprasefailure-issue-with-hyphen-in-the-log/63332)
**Author:** [@13koushik](https://discuss.elastic.co/u/13koushik)\ **Replies:** 6\ **Last updated:** [October 24, 2016, 2:28pm UTC](https://discuss.elastic.co/t/grokprasefailure-issue-with-hyphen-in-the-log/63332 "2016-10-24T14:28:18Z")
I am getting a grokparsefailure message for the below stated conf. I'm not getting the SeverityLevel. The problem for with the - ( hyphen ) in the log. NOTSPACE isn't working for me. I would appreciate any help. Input … --- ## [Logstash Heroku plugin credentials](https://discuss.elastic.co/t/logstash-heroku-plugin-credentials/63759)
**Author:** [@prash101\_2000](https://discuss.elastic.co/u/prash101_2000)\ **Replies:** 0\ **Last updated:** [October 24, 2016, 1:55pm UTC](https://discuss.elastic.co/t/logstash-heroku-plugin-credentials/63759 "2016-10-24T13:55:34Z")
I am trying to fetch the Heroku app logs into elastic search using the Heroku input plugin.I am able to successfully get the logs into elastic search if I run the logstash from the bin by passing the config. But when I t… --- ## [Is the jdbc fetch\_size the same like Logstash batch\_size?](https://discuss.elastic.co/t/is-the-jdbc-fetch-size-the-same-like-logstash-batch-size/63752)
**Author:** [@TWalter](https://discuss.elastic.co/u/TWalter)\ **Replies:** 0\ **Last updated:** [October 24, 2016, 1:27pm UTC](https://discuss.elastic.co/t/is-the-jdbc-fetch-size-the-same-like-logstash-batch-size/63752 "2016-10-24T13:27:54Z")
Hello there, i have a Question about Logstash an the JDBC-Input Plugin. If i query a database and get 2000 results. Then i set the jdbc\_fetch\_size to 10 and the Logstash batch\_size to 1. Am i wright if i'm saying… --- ## [Scaling Logstash: Redis queue - channels or list?](https://discuss.elastic.co/t/scaling-logstash-redis-queue-channels-or-list/63741)
**Author:** [@kariboe](https://discuss.elastic.co/u/kariboe)\ **Replies:** 0\ **Last updated:** [October 24, 2016, 11:02am UTC](https://discuss.elastic.co/t/scaling-logstash-redis-queue-channels-or-list/63741 "2016-10-24T11:02:26Z")
We are in the process of setting up new ELK stack. The idea is to have: N nodes --\> 2Logstash Shippers --\> Redis (Master/Slave) --\> 2 Logstash Indexers --\> ES cluster --\> Kibana Only problem we have right now is … --- ## [Help with Multiline](https://discuss.elastic.co/t/help-with-multiline/63479)
**Author:** [@debarun](https://discuss.elastic.co/u/debarun)\ **Replies:** 4\ **Last updated:** [October 24, 2016, 10:41am UTC](https://discuss.elastic.co/t/help-with-multiline/63479 "2016-10-24T10:41:40Z")
Hi , I just came across an event where the log lines are of the format: Date:2016-10-18 Time:14:31:05 Event Code:0x60a Description:Internal information only. A logical unit has been enabled Subsystem:CKM0012480… --- ## [Issues with Logstash and Elasticsearch Input plugin (Out of Memory)](https://discuss.elastic.co/t/issues-with-logstash-and-elasticsearch-input-plugin-out-of-memory/63717)
**Author:** [@fxiger](https://discuss.elastic.co/u/fxiger)\ **Replies:** 0\ **Last updated:** [October 24, 2016, 8:37am UTC](https://discuss.elastic.co/t/issues-with-logstash-and-elasticsearch-input-plugin-out-of-memory/63717 "2016-10-24T08:37:12Z")
Hi, ubuntu 16.04 lts logstash 2.4.0 elasticsearch 2.4.1, Build: c67dc32/2016-09-27T18:57:55Z JVM: 1.8.0\_91 let's assume, i have 2 systems (A and B). System A is running somewhere and has an ELK stack on top with an … --- ## [Documents not in elasticsearch](https://discuss.elastic.co/t/documents-not-in-elasticsearch/62979)
**Author:** [@meaglin](https://discuss.elastic.co/u/meaglin)\ **Replies:** 3\ **Last updated:** [October 24, 2016, 8:32am UTC](https://discuss.elastic.co/t/documents-not-in-elasticsearch/62979 "2016-10-24T08:32:52Z")
Hello Logstash and Elasticsearch users, Once again i did something without fully knowing how it works and what I did. I have a topic in the logstash part where i got some help with a problem of not getting events in … --- ## [Logstash 5.0 :Ruby exception occurred](https://discuss.elastic.co/t/logstash-5-0-ruby-exception-occurred/63562)
**Author:** [@suntuo](https://discuss.elastic.co/u/suntuo)\ **Replies:** 2\ **Last updated:** [October 24, 2016, 7:55am UTC](https://discuss.elastic.co/t/logstash-5-0-ruby-exception-occurred/63562 "2016-10-24T07:55:21Z")
The filter works fine with logstash 2.X, when it upgrade to 5.0 Ruby exception occurred: undefined method \`\[\]' ruby { init =\>"@kname =\['time\_local','jsoncontent'\]" code =\>"event.append(H… --- ## [Is it possible to use dynamic string in "users" field in output-xmpp plugin?](https://discuss.elastic.co/t/is-it-possible-to-use-dynamic-string-in-users-field-in-output-xmpp-plugin/63593)
**Author:** [@lshekhar](https://discuss.elastic.co/u/lshekhar)\ **Replies:** 2\ **Last updated:** [October 24, 2016, 4:26am UTC](https://discuss.elastic.co/t/is-it-possible-to-use-dynamic-string-in-users-field-in-output-xmpp-plugin/63593 "2016-10-24T04:26:14Z")
I need to filter sender id from log and place that in "users" field. output { xmpp { message =\> " Message from %{job\_submitter}" users =\> \["%{job\_submitter}"\] Can someone suggest me the way? … --- ## [Logstash can't talk to elasticsearch after several hours' running](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198)
**Author:** [@caoping](https://discuss.elastic.co/u/caoping)\ **Replies:** 6\ **Last updated:** [October 24, 2016, 3:03am UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198 "2016-10-24T03:03:38Z")
I found that after several hours running, my logstash can't connect to elasticsearch any more. From logstash log, it said: {:timestamp=\>"2016-10-17T04:39:40.312000+0200", :message=\>"Starting pipeline", :id=\>"main", :pi… --- ## [Environment variables in IF-statement](https://discuss.elastic.co/t/environment-variables-in-if-statement/63613)
**Author:** [@krzysztof\_pl](https://discuss.elastic.co/u/krzysztof_pl)\ **Replies:** 2\ **Last updated:** [October 23, 2016, 7:53pm UTC](https://discuss.elastic.co/t/environment-variables-in-if-statement/63613 "2016-10-23T19:53:08Z")
hi Everyone! My goal is to 'implement' filter to process only traces with specific levels. Perfect solution: levels should be defined in separated file or environment variable. So, I would like to achive it in this w… --- ## [Logstash output for multiple elasticsearch instance](https://discuss.elastic.co/t/logstash-output-for-multiple-elasticsearch-instance/2194)
**Author:** [@saif](https://discuss.elastic.co/u/saif)\ **Replies:** 8\ **Last updated:** [October 23, 2016, 5:09pm UTC](https://discuss.elastic.co/t/logstash-output-for-multiple-elasticsearch-instance/2194 "2016-10-23T17:09:56Z")
hello i m working to collect logs from 3000 server I installed 2 servers with ElasticSearch and logstash indexer 2 with redis , logstash shipper ans elasticsearch So i have 4 instance for ElasticSearch I tried to find e… --- ## [Gzipped input for logstash](https://discuss.elastic.co/t/gzipped-input-for-logstash/63448)
**Author:** [@soudipta](https://discuss.elastic.co/u/soudipta)\ **Replies:** 1\ **Last updated:** [October 23, 2016, 4:49pm UTC](https://discuss.elastic.co/t/gzipped-input-for-logstash/63448 "2016-10-23T16:49:57Z")
Hi, Is it possible to ingest tar.gz files into logstash. I tried the gzip-lines codec, but it did not work. can someone please suggest if there is alternatives to get zipped files ingested into logstash? Thanks, Soud… --- ## [Pull logs via ssh - push not possible](https://discuss.elastic.co/t/pull-logs-via-ssh-push-not-possible/48102)
**Author:** [@Thomas-Guettler](https://discuss.elastic.co/u/Thomas-Guettler)\ **Replies:** 4\ **Last updated:** [October 23, 2016, 9:03am UTC](https://discuss.elastic.co/t/pull-logs-via-ssh-push-not-possible/48102 "2016-10-23T09:03:10Z")
We have some servers in remote data centers where we can open a ssh from our central server to the remote-server. But the remote-server (where the logs get created) can't push the logs to our central server. What is th… --- ## [Input file log rotation](https://discuss.elastic.co/t/input-file-log-rotation/26742)
**Author:** [@mparthas](https://discuss.elastic.co/u/mparthas)\ **Replies:** 13\ **Last updated:** [October 23, 2016, 2:06am UTC](https://discuss.elastic.co/t/input-file-log-rotation/26742 "2016-10-23T02:06:11Z")
Hi, I am new to this group. I have been playing around with Logstash for about a couple of weeks. I was surprised to see that Logstash does not provide the feature where in, it reads a log file once and forgets about i… --- ## [Problem with aggregate filter while using the push\_map\_as\_event\_on\_timeout feature](https://discuss.elastic.co/t/problem-with-aggregate-filter-while-using-the-push-map-as-event-on-timeout-feature/63670)
**Author:** [@Ola-Vish](https://discuss.elastic.co/u/Ola-Vish)\ **Replies:** 0\ **Last updated:** [October 22, 2016, 8:22pm UTC](https://discuss.elastic.co/t/problem-with-aggregate-filter-while-using-the-push-map-as-event-on-timeout-feature/63670 "2016-10-22T20:22:33Z")
Hi, I am having issues with the aggregate filter. I have version 2.4 of this filter installed. I am running logstash 2.4 as a service on Ubuntu 16.04. In filter's GitHub readme file I found a nice feature that should… --- ## [Aggregate filter timeout by event timestamp rather than by system time](https://discuss.elastic.co/t/aggregate-filter-timeout-by-event-timestamp-rather-than-by-system-time/63671)
**Author:** [@Ola-Vish](https://discuss.elastic.co/u/Ola-Vish)\ **Replies:** 0\ **Last updated:** [October 22, 2016, 8:49pm UTC](https://discuss.elastic.co/t/aggregate-filter-timeout-by-event-timestamp-rather-than-by-system-time/63671 "2016-10-22T20:49:42Z")
Hi, Is it at all possible for the aggregate filter to have the timeout according to a timestamp field in the event itself rather than the system time? This is important for me because I want to index some old logs an… --- ## [Issue with monitoring Oracle Databases with Logstash](https://discuss.elastic.co/t/issue-with-monitoring-oracle-databases-with-logstash/49368)
**Author:** [@r4z0rw1r3666](https://discuss.elastic.co/u/r4z0rw1r3666)\ **Replies:** 8\ **Last updated:** [October 21, 2016, 10:31pm UTC](https://discuss.elastic.co/t/issue-with-monitoring-oracle-databases-with-logstash/49368 "2016-10-21T22:31:07Z")
Hello, I'm having some issues with Logstash and monitoring a single Oracle database. Logstash takes 30 seconds to "warm up" before generating data, for a further 45 seconds before promptly imploding. Error that I've c… --- ## [Json codec: blank lines make logstash throw errors](https://discuss.elastic.co/t/json-codec-blank-lines-make-logstash-throw-errors/62718)
**Author:** [@Suny](https://discuss.elastic.co/u/Suny)\ **Replies:** 1\ **Last updated:** [October 21, 2016, 4:04pm UTC](https://discuss.elastic.co/t/json-codec-blank-lines-make-logstash-throw-errors/62718 "2016-10-21T16:04:18Z")
A blank line in a json log makes logstash throw an error: :message=\>"JSON codec is expecting array or object/map", :data=\>"", :level=\>:error since upgrade to logstash 2.3.4 Is this a bug or a feature? I'd prefer if bl… --- ## [Underscore preventing us from upgrading from 1.7.1 to 2.x](https://discuss.elastic.co/t/underscore-preventing-us-from-upgrading-from-1-7-1-to-2-x/63634)
**Author:** [@genesis](https://discuss.elastic.co/u/genesis)\ **Replies:** 0\ **Last updated:** [October 21, 2016, 3:18pm UTC](https://discuss.elastic.co/t/underscore-preventing-us-from-upgrading-from-1-7-1-to-2-x/63634 "2016-10-21T15:18:36Z")
I have been trying to resolve this issue which is preventing me from upgrading from 1.7.1 to 2.x due to this error: The uid, \_id, \_type, \_source, \_all, \_parent, \_fieldnames, \_routing, \_index, \_size, \_timestamp, and \_ttl … --- ## [How to catch errors in output plugin?](https://discuss.elastic.co/t/how-to-catch-errors-in-output-plugin/60297)
**Author:** [@tiwilliam](https://discuss.elastic.co/u/tiwilliam)\ **Replies:** 1\ **Last updated:** [October 21, 2016, 1:40pm UTC](https://discuss.elastic.co/t/how-to-catch-errors-in-output-plugin/60297 "2016-10-21T13:40:25Z")
Hi, I'm trying to write a tag or a field in @metadata in an output plugin but can't match that value in a if statement. Here's the output block: output { something { ...setting the tag/field inside this plu… --- ## [Help - can't run logstash with java 1.8.0\_77? (logstash 2.3.4 and 2.3.2)](https://discuss.elastic.co/t/help-cant-run-logstash-with-java-1-8-0-77-logstash-2-3-4-and-2-3-2/63545)
**Author:** [@arp2500](https://discuss.elastic.co/u/arp2500)\ **Replies:** 1\ **Last updated:** [October 21, 2016, 12:45pm UTC](https://discuss.elastic.co/t/help-cant-run-logstash-with-java-1-8-0-77-logstash-2-3-4-and-2-3-2/63545 "2016-10-21T12:45:10Z")
(background: I'm new to logstash and have just installed on CentOS 6.7 with java 1.8.0\_77 installed) when I attempt to run 'logstash' or even run 'plugin list' all I am seeing is... WARNING: Default JAVAOPTS will be … --- ## [Indexing through logstash](https://discuss.elastic.co/t/indexing-through-logstash/63353)
**Author:** [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Replies:** 2\ **Last updated:** [October 21, 2016, 10:30am UTC](https://discuss.elastic.co/t/indexing-through-logstash/63353 "2016-10-21T10:30:44Z")
Hi, I'm using the following config file to index the data. input { jdbc { jdbc\_driver\_library =\> "E:/softwares/sqljdbc-1.2.0.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc\_connection\_st… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1087) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1089)