# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1183 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1184 --- ## [Logstash lose connection to ElasticSearch](https://discuss.elastic.co/t/logstash-lose-connection-to-elasticsearch/37071)
**Author:** [@ofer\_peretz](https://discuss.elastic.co/u/ofer_peretz)\ **Replies:** 2\ **Last updated:** [March 3, 2016, 2:49pm UTC](https://discuss.elastic.co/t/logstash-lose-connection-to-elasticsearch/37071 "2016-03-03T14:49:06Z")
Hi Guys, Recently ive installed new cluster ES-2.1.0 with LS-2.1.1 & RabbitMQ in the middle. Occasionally LS loses connection to ES and post this error: Log from Logstash in DEBUG: Attempted to send a bulk request t… --- ## [Logstash - input/process/output XMLs to ElasticSearch - Java OutputStream](https://discuss.elastic.co/t/logstash-input-process-output-xmls-to-elasticsearch-java-outputstream/43086)
**Author:** [@Paco\_B](https://discuss.elastic.co/u/Paco_B)\ **Replies:** 4\ **Last updated:** [March 3, 2016, 2:24pm UTC](https://discuss.elastic.co/t/logstash-input-process-output-xmls-to-elasticsearch-java-outputstream/43086 "2016-03-03T14:24:06Z")
Hello, I was wondering if there is a way to configure logstash as following: Java application sends a longer XML file by OutputStream to Logstash. Logstash "listens" over this outStream to receive, process and store… --- ## [\[Solved\] \[Logstash\] Elasticsearch output plugin question](https://discuss.elastic.co/t/solved-logstash-elasticsearch-output-plugin-question/43390)
**Author:** [@axelfelix](https://discuss.elastic.co/u/axelfelix)\ **Replies:** 2\ **Last updated:** [March 3, 2016, 2:07pm UTC](https://discuss.elastic.co/t/solved-logstash-elasticsearch-output-plugin-question/43390 "2016-03-03T14:07:17Z")
Hi all, I'm new here, so hello to everybody !!! I have a question regarding the elasticsearch output plugin. When I set the index, what is the base of the macro %{+YYYY.MM.dd} ? index =\> "logstash-%{+YYYY.MM.dd}" … --- ## [LogStash agent on Windows machine](https://discuss.elastic.co/t/logstash-agent-on-windows-machine/43392)
**Author:** [@miko370](https://discuss.elastic.co/u/miko370)\ **Replies:** 0\ **Last updated:** [March 3, 2016, 1:41pm UTC](https://discuss.elastic.co/t/logstash-agent-on-windows-machine/43392 "2016-03-03T13:41:45Z")
Hi. I want to test the eventlog plugin and the wmi plugin. For that, I need to install logstash locally. Is there a way to install only the agent part without the whole Logstash installation (150MB). BTW, I am aware … --- ## [Haproxy 1.6 + logstash 2.2; grok vs. online validators](https://discuss.elastic.co/t/haproxy-1-6-logstash-2-2-grok-vs-online-validators/43318)
**Author:** [@rmrf](https://discuss.elastic.co/u/rmrf)\ **Replies:** 2\ **Last updated:** [March 3, 2016, 1:25pm UTC](https://discuss.elastic.co/t/haproxy-1-6-logstash-2-2-grok-vs-online-validators/43318 "2016-03-03T13:25:47Z")
I'm having a bit of a head banging against the wall issue... I have haproxy (v. 1.6) config'ed to syslog to logstash (v. 2.2.2). Grok's packaged haproxy filters are for v. 1.4- so i've had to hack them up a bit, plus add… --- ## [Logstash filter user\_agent or geoip very slow](https://discuss.elastic.co/t/logstash-filter-user-agent-or-geoip-very-slow/43283)
**Author:** [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Replies:** 4\ **Last updated:** [March 3, 2016, 12:58pm UTC](https://discuss.elastic.co/t/logstash-filter-user-agent-or-geoip-very-slow/43283 "2016-03-03T12:58:59Z")
I am using logstash with RabbitMQ as input, elasticsearch as output, and user\_agent/geoip filter. Configuration is pretty simple: input { rabbitmq { host =\> "queue" exchange =\> "bench" queue… --- ## [Logstash- transfrom multiple log lines to one line](https://discuss.elastic.co/t/logstash-transfrom-multiple-log-lines-to-one-line/43376)
**Author:** [@Atul\_Patel](https://discuss.elastic.co/u/Atul_Patel)\ **Replies:** 5\ **Last updated:** [March 3, 2016, 12:46pm UTC](https://discuss.elastic.co/t/logstash-transfrom-multiple-log-lines-to-one-line/43376 "2016-03-03T12:46:16Z")
I have a situation where I am getting logs from forwarder in the below format: timestamp \ entry: ProcessAPI{AddEmployee} :hostname timestamp \ send: UserAPI{ValidateUser} :hostname timestamp \ **Author:** [@mehrentraut](https://discuss.elastic.co/u/mehrentraut)\ **Replies:** 3\ **Last updated:** [March 3, 2016, 12:14pm UTC](https://discuss.elastic.co/t/multiple-logfiles-from-diffrent-clusternodes-to-one-merged-logfile/43344 "2016-03-03T12:14:56Z") Hi, i hope someone can help me. I want merge/aggregate logfiles from diffrent Clusternodes to one Logfile on logfile Storage. Currently I use log / slog, but this is no longer being developed. I evaluate the in… --- ## [Ruby Filter Question](https://discuss.elastic.co/t/ruby-filter-question/43371)
**Author:** [@Kryten](https://discuss.elastic.co/u/Kryten)\ **Replies:** 0\ **Last updated:** [March 3, 2016, 11:14am UTC](https://discuss.elastic.co/t/ruby-filter-question/43371 "2016-03-03T11:14:12Z")
Hi, I'm experimenting with the ruby filter to execute arbitrary ruby code on events and so far, its been very successful, but I have a couple of questions please. \[1\] Is it possible to persist data between events? I … --- ## [\[Solved\] List of logstash grok data types](https://discuss.elastic.co/t/solved-list-of-logstash-grok-data-types/43355)
**Author:** [@mikula](https://discuss.elastic.co/u/mikula)\ **Replies:** 2\ **Last updated:** [March 3, 2016, 9:47am UTC](https://discuss.elastic.co/t/solved-list-of-logstash-grok-data-types/43355 "2016-03-03T09:47:55Z")
Hello, I there anywhere some list of basic types usable by grok like %{NUMBER:number:int} I should be part of grok documentation but is not there! With thanks and wishes of great day AM --- ## [How to check if a field exists in the Ruby plugin?](https://discuss.elastic.co/t/how-to-check-if-a-field-exists-in-the-ruby-plugin/43354)
**Author:** [@gunay](https://discuss.elastic.co/u/gunay)\ **Replies:** 0\ **Last updated:** [March 3, 2016, 9:03am UTC](https://discuss.elastic.co/t/how-to-check-if-a-field-exists-in-the-ruby-plugin/43354 "2016-03-03T09:03:33Z")
Hi, I am trying to find how to check if a field exist in the ruby filter. To check the existence of a field in Ruby, I tried the following three, seems that none of them work if event\['duration'\]\['time'\] =~ /.+/ … --- ## [Logstash installation on a linux server error](https://discuss.elastic.co/t/logstash-installation-on-a-linux-server-error/42962)
**Author:** [@bishaka](https://discuss.elastic.co/u/bishaka)\ **Replies:** 3\ **Last updated:** [March 3, 2016, 9:26am UTC](https://discuss.elastic.co/t/logstash-installation-on-a-linux-server-error/42962 "2016-03-03T09:26:25Z")
I was trying to install logstash on a linux server usinf the command yum install logstash-2.2.2-1.noarch.rpm and i get the following error Loaded plugins: product-id, security, subscription-manager This system is no… --- ## [Can't parse Linux OS generated log files in windows environment with file input plugin](https://discuss.elastic.co/t/cant-parse-linux-os-generated-log-files-in-windows-environment-with-file-input-plugin/43209)
**Author:** [@guojingjing](https://discuss.elastic.co/u/guojingjing)\ **Replies:** 3\ **Last updated:** [March 3, 2016, 8:56am UTC](https://discuss.elastic.co/t/cant-parse-linux-os-generated-log-files-in-windows-environment-with-file-input-plugin/43209 "2016-03-03T08:56:33Z")
Any one ran into this problem: log files generated by a Linux system which is stored as \*.log, can not be parsed in windows environment. Specifically, I have my logstash conf to read \*.log files in a folder: the Linux… --- ## [How to use "op\_type" with logstash to prevent duplicate entries?](https://discuss.elastic.co/t/how-to-use-op-type-with-logstash-to-prevent-duplicate-entries/43101)
**Author:** [@gunay](https://discuss.elastic.co/u/gunay)\ **Replies:** 2\ **Last updated:** [March 3, 2016, 8:49am UTC](https://discuss.elastic.co/t/how-to-use-op-type-with-logstash-to-prevent-duplicate-entries/43101 "2016-03-03T08:49:02Z")
Hi, I would like to prevent the same data log indexed twice. I found out from the documentation that it can be achieved using "op\_type" parameter. $ curl -XPUT 'http://localhost:9200/twitter/tweet/1?op\_type=cre… --- ## [Logstash reads log file one step behind](https://discuss.elastic.co/t/logstash-reads-log-file-one-step-behind/43306)
**Author:** [@bilguuna](https://discuss.elastic.co/u/bilguuna)\ **Replies:** 3\ **Last updated:** [March 3, 2016, 6:32am UTC](https://discuss.elastic.co/t/logstash-reads-log-file-one-step-behind/43306 "2016-03-03T06:32:18Z")
Hi, I've been trying to create index on json log file that are generated by Log4j2 and viewed by Kibana. Everything is working fine except LogStash skips last object on the log file, next time when a log file is changed … --- ## [Logstash hangs after some time](https://discuss.elastic.co/t/logstash-hangs-after-some-time/43321)
**Author:** [@abordia](https://discuss.elastic.co/u/abordia)\ **Replies:** 0\ **Last updated:** [March 3, 2016, 12:09am UTC](https://discuss.elastic.co/t/logstash-hangs-after-some-time/43321 "2016-03-03T00:09:21Z")
Logstash 1.4.2 hangs after some time. This used to work perfectly earlier but last few weeks been giving issues. Appreciate quick help. "LogStash::Runner" prio=10 tid=0x00007f7b4c009000 nid=0x5daa in Object.wait() \[0… --- ## [Snmptrap input adding dots to field names](https://discuss.elastic.co/t/snmptrap-input-adding-dots-to-field-names/43277)
**Author:** [@arizonawayfarer](https://discuss.elastic.co/u/arizonawayfarer)\ **Replies:** 0\ **Last updated:** [March 2, 2016, 4:59pm UTC](https://discuss.elastic.co/t/snmptrap-input-adding-dots-to-field-names/43277 "2016-03-02T16:59:08Z")
The snmptrap input plugin for logstash is adding dots to field names. This is preventing entries from being indexed in elastic.. I understand I can use the de\_dot filter, but since that's expensive - I was hoping there … --- ## [Pulling logs from remote machine](https://discuss.elastic.co/t/pulling-logs-from-remote-machine/43309)
**Author:** [@bishaka](https://discuss.elastic.co/u/bishaka)\ **Replies:** 6\ **Last updated:** [March 2, 2016, 11:06pm UTC](https://discuss.elastic.co/t/pulling-logs-from-remote-machine/43309 "2016-03-02T23:06:57Z")
How can I configure logstash to read from a particular log file on a remote location of a server? In other words, how can I pull logs from remote machines? Please provide me with couple of options.. --- ## [Linux logstash using input-jdbc and active directory issue](https://discuss.elastic.co/t/linux-logstash-using-input-jdbc-and-active-directory-issue/43298)
**Author:** [@aaroy428](https://discuss.elastic.co/u/aaroy428)\ **Replies:** 2\ **Last updated:** [March 2, 2016, 7:22pm UTC](https://discuss.elastic.co/t/linux-logstash-using-input-jdbc-and-active-directory-issue/43298 "2016-03-02T19:22:49Z")
Can someone help me with an input-jdbc on a Linux logstash trying to connect to a MS sqlserver database using only active directory as an authentifcation mechanism. --- ## [Grok parse failure with java logs](https://discuss.elastic.co/t/grok-parse-failure-with-java-logs/43159)
**Author:** [@daniel1](https://discuss.elastic.co/u/daniel1)\ **Replies:** 2\ **Last updated:** [March 2, 2016, 5:29pm UTC](https://discuss.elastic.co/t/grok-parse-failure-with-java-logs/43159 "2016-03-02T17:29:25Z")
I'm having quite a bit of trouble with parsing some java logs. The log output format is the standard java logging format, matching the pattern defined in %{JAVALOGMESSAGE}. The application is running on jboss inside a d… --- ## [Ruby filter - How to add fields?](https://discuss.elastic.co/t/ruby-filter-how-to-add-fields/43272)
**Author:** [@Kryten](https://discuss.elastic.co/u/Kryten)\ **Replies:** 0\ **Last updated:** [March 2, 2016, 4:34pm UTC](https://discuss.elastic.co/t/ruby-filter-how-to-add-fields/43272 "2016-03-02T16:34:18Z")
Hi I'm trying to do this: if \[directive\] { ruby { code =\> "if event\['directive'\] \<=30 event\[scenario\] = 'scenario\_one' end " } } Basically if the directive field exists (which I h… --- ## [What is the better way to take full advantage of Logstash (MEM and CPU)?](https://discuss.elastic.co/t/what-is-the-better-way-to-take-full-advantage-of-logstash-mem-and-cpu/43009)
**Author:** [@syunusic](https://discuss.elastic.co/u/syunusic)\ **Replies:** 2\ **Last updated:** [March 2, 2016, 4:44pm UTC](https://discuss.elastic.co/t/what-is-the-better-way-to-take-full-advantage-of-logstash-mem-and-cpu/43009 "2016-03-02T16:44:26Z")
Hi, I'm running Logstash 2.2.2 in Ubuntu as a service (sudo service logstash start) and I want to have as much speed (in shipping) as possible. Is it better to run several instances in the same box or using as much memo… --- ## [Logstash UDP output](https://discuss.elastic.co/t/logstash-udp-output/43219)
**Author:** [@apoc](https://discuss.elastic.co/u/apoc)\ **Replies:** 2\ **Last updated:** [March 2, 2016, 3:48pm UTC](https://discuss.elastic.co/t/logstash-udp-output/43219 "2016-03-02T15:48:05Z")
Hello, Is it possible to send only the message by UDP (output). If i remove the host field for example, i get %{host} instead :disappointed: Here is my conf : input { udp { port =\> 3000 codec =\> plain } } … --- ## [Problem with beats/logstash parsing](https://discuss.elastic.co/t/problem-with-beats-logstash-parsing/43168)
**Author:** [@corey\_jones](https://discuss.elastic.co/u/corey_jones)\ **Replies:** 2\ **Last updated:** [March 2, 2016, 1:33pm UTC](https://discuss.elastic.co/t/problem-with-beats-logstash-parsing/43168 "2016-03-02T13:33:46Z")
Hi guys. I have a server running beats and another server that has a docker container running logstash. Both are at aws. It seems that the connection is being made. From my debug.log from logstash I see this. {:timesta… --- ## [Define upload fields from json file](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938)
**Author:** [@chenbe2204](https://discuss.elastic.co/u/chenbe2204)\ **Replies:** 6\ **Last updated:** [March 2, 2016, 2:30pm UTC](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938 "2016-03-02T14:30:11Z")
Hi , In the conf file , how can I define which fields I want to upload to the ElasticSearch ? My Json file contain one row that need to be uploaded to the ElasticSearch , So after the upload I would like to see I row… --- ## [Run Logstash in Java using JRuby](https://discuss.elastic.co/t/run-logstash-in-java-using-jruby/43257)
**Author:** [@roman.k](https://discuss.elastic.co/u/roman.k)\ **Replies:** 0\ **Last updated:** [March 2, 2016, 2:23pm UTC](https://discuss.elastic.co/t/run-logstash-in-java-using-jruby/43257 "2016-03-02T14:23:00Z")
Hi, My final goal is to run Logstash within Tomcat (hence, packaged in a WAR file). Has anybody some experience with that? To gain some experience in running logstash within Java, I'm trying to start it in a small Java… --- ## [Concatenate field and String](https://discuss.elastic.co/t/concatenate-field-and-string/43239)
**Author:** [@jobobabas](https://discuss.elastic.co/u/jobobabas)\ **Replies:** 1\ **Last updated:** [March 2, 2016, 1:25pm UTC](https://discuss.elastic.co/t/concatenate-field-and-string/43239 "2016-03-02T13:25:39Z")
First Sry for my bad englisch I am using logstash since 2 days I want to concatenate a field and a String in the field Like the field is "Test" and i want to add a "\_NR1" to the end Thanks for help --- ## [What is the way to copy data to hdfs using logstash without using webhdfs](https://discuss.elastic.co/t/what-is-the-way-to-copy-data-to-hdfs-using-logstash-without-using-webhdfs/43247)
**Author:** [@sdaruna](https://discuss.elastic.co/u/sdaruna)\ **Replies:** 0\ **Last updated:** [March 2, 2016, 12:46pm UTC](https://discuss.elastic.co/t/what-is-the-way-to-copy-data-to-hdfs-using-logstash-without-using-webhdfs/43247 "2016-03-02T12:46:52Z")
Hi, We want to move the data to hdfs using logstash. However, the webhdfs way is not accepted by our client, so we want to check other way around to move the data to hdfs. What is the way forward with logstash to move … --- ## [Problem of using input{filebeat} on logstash](https://discuss.elastic.co/t/problem-of-using-input-filebeat-on-logstash/37815)
**Author:** [@bruclan](https://discuss.elastic.co/u/bruclan)\ **Replies:** 4\ **Last updated:** [March 2, 2016, 9:27am UTC](https://discuss.elastic.co/t/problem-of-using-input-filebeat-on-logstash/37815 "2016-03-02T09:27:57Z")
I used input{filebeat} on logstash to receive events sent by filebeat, but I have found a problem. When I used multiple filebeat processes to send events to multiple logstash process, some logstash would issue following … --- ## [Log4j input host without port](https://discuss.elastic.co/t/log4j-input-host-without-port/43105)
**Author:** [@David\_Pocivalnik](https://discuss.elastic.co/u/David_Pocivalnik)\ **Replies:** 1\ **Last updated:** [March 2, 2016, 8:12am UTC](https://discuss.elastic.co/t/log4j-input-host-without-port/43105 "2016-03-02T08:12:55Z")
I'm using log4j input to ES with a SocketAppender. I'm looking for a way to remove the port-portion of the host-field, or add another field with only containing the host/IP of that host field. Is there an easy way … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1182) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1184)