# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1258 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1259 --- ## [Unusual number of HEAD requests being made by S3 input plugin](https://discuss.elastic.co/t/unusual-number-of-head-requests-being-made-by-s3-input-plugin/23894)
**Author:** [@vangap](https://discuss.elastic.co/u/vangap)\ **Replies:** 0\ **Last updated:** [June 18, 2015, 9:02am UTC](https://discuss.elastic.co/t/unusual-number-of-head-requests-being-made-by-s3-input-plugin/23894 "2015-06-18T09:02:24Z")
We have S3 access logs being collected in a bucket. We are using S3 input plugin to index these files into ELK. After a couple of months usage we noticed unusual no of requests made to S3 (~1 Billion/Month) which cos… --- ## [DNS filter performance issue?](https://discuss.elastic.co/t/dns-filter-performance-issue/2544)
**Author:** [@yuphing](https://discuss.elastic.co/u/yuphing)\ **Replies:** 5\ **Last updated:** [June 18, 2015, 8:35am UTC](https://discuss.elastic.co/t/dns-filter-performance-issue/2544 "2015-06-18T08:35:52Z")
I'm trying to use the DNS filter to do reverse DNS lookups for various IPs in netflow data, like so: dns { reverse =\> \[ "ipv4\_src\_hostname", "ipv4\_dst\_hostname" \] action =\> "replace" } The filter is working i.e. i… --- ## [Is it possible to compress output from the S3 plugin?](https://discuss.elastic.co/t/is-it-possible-to-compress-output-from-the-s3-plugin/23883)
**Author:** [@Sam\_Barham](https://discuss.elastic.co/u/Sam_Barham)\ **Replies:** 1\ **Last updated:** [June 18, 2015, 2:37am UTC](https://discuss.elastic.co/t/is-it-possible-to-compress-output-from-the-s3-plugin/23883 "2015-06-18T02:37:48Z")
I'm looking at using the S3 output plugin to archive all our logs to an S3 bucket. My current system is using the file output plugin to write gzipped files to disk, which are then uploaded to S3 via a custom script. Th… --- ## [Logstash import of JSON-format file fails](https://discuss.elastic.co/t/logstash-import-of-json-format-file-fails/23773)
**Author:** [@johnwlambert](https://discuss.elastic.co/u/johnwlambert)\ **Replies:** 3\ **Last updated:** [June 18, 2015, 1:30am UTC](https://discuss.elastic.co/t/logstash-import-of-json-format-file-fails/23773 "2015-06-18T01:30:46Z")
I am trying to input logs from my server log file. This is my config file. input { file { path =\> "/oncology-server/log/service.log" start\_position =\> "beginning" } } filter { json { source =\> "me… --- ## [Logstash silently fails on trying to import csv](https://discuss.elastic.co/t/logstash-silently-fails-on-trying-to-import-csv/2651)
**Author:** [@juju](https://discuss.elastic.co/u/juju)\ **Replies:** 8\ **Last updated:** [June 18, 2015, 12:33am UTC](https://discuss.elastic.co/t/logstash-silently-fails-on-trying-to-import-csv/2651 "2015-06-18T00:33:50Z")
Hello, I'm trying to import a csv file into ELK and it fails silently even with output set to stdout/debug. $ cat /vagrant/sinkhole.csv | /opt/logstash/bin/logstash -f /vagrant/logstash-sinkhole.conf | tee log J… --- ## [Difficulties to parse/filter on xml file](https://discuss.elastic.co/t/difficulties-to-parse-filter-on-xml-file/2649)
**Author:** [@juju](https://discuss.elastic.co/u/juju)\ **Replies:** 2\ **Last updated:** [June 18, 2015, 12:32am UTC](https://discuss.elastic.co/t/difficulties-to-parse-filter-on-xml-file/2649 "2015-06-18T00:32:45Z")
Hello, I'm trying to parse a xml file (Export of Arbor Alerts. example here http://pastebin.com/jpQWMbjz) but it currently fails. Either Most lines are going unparsed into kibana, either nothing get inserted in elastic… --- ## [Questions regarding upgrade to Logstash 1.5](https://discuss.elastic.co/t/questions-regarding-upgrade-to-logstash-1-5/23864)
**Author:** [@skebo5150](https://discuss.elastic.co/u/skebo5150)\ **Replies:** 1\ **Last updated:** [June 17, 2015, 6:18pm UTC](https://discuss.elastic.co/t/questions-regarding-upgrade-to-logstash-1-5/23864 "2015-06-17T18:18:54Z")
I've upgraded ELK to the versions (ES 1.5.2 LS 1.5.0 K 4) from LS 1.4.2 and the environment is mostly up and running, but I was wondering how to bring over my old dashboards/saved searches? Is there any documentation av… --- ## [Calculation in mutate field](https://discuss.elastic.co/t/calculation-in-mutate-field/2703)
**Author:** [@Ankit\_Pradhan](https://discuss.elastic.co/u/Ankit_Pradhan)\ **Replies:** 2\ **Last updated:** [June 17, 2015, 2:25pm UTC](https://discuss.elastic.co/t/calculation-in-mutate-field/2703 "2015-06-17T14:25:09Z")
I have filed TIME field and I would like to convert HHmmss to only Seconds. e.g. "duration": \[ \[ "0:00:00" \] \], "HOUR": \[ \[ "0" \] \], "MINUTE": \[ \[ "00" \] \], "SECON… --- ## [Logstash elastic search on ec2 (cloud plugin)](https://discuss.elastic.co/t/logstash-elastic-search-on-ec2-cloud-plugin/1771)
**Author:** [@mans4singh](https://discuss.elastic.co/u/mans4singh)\ **Replies:** 2\ **Last updated:** [June 17, 2015, 11:53am UTC](https://discuss.elastic.co/t/logstash-elastic-search-on-ec2-cloud-plugin/1771 "2015-06-17T11:53:06Z")
Hi: I was just wondering if logstash can auto discover elastic search nodes using ec2 discovery on aws. For elastic search we need to use elastic search cloud aws plugin (https://github.com/elastic/elasticsearch-cloud… --- ## [Multiple line with extreme long log "line"](https://discuss.elastic.co/t/multiple-line-with-extreme-long-log-line/23806)
**Author:** [@loc](https://discuss.elastic.co/u/loc)\ **Replies:** 1\ **Last updated:** [June 17, 2015, 7:03am UTC](https://discuss.elastic.co/t/multiple-line-with-extreme-long-log-line/23806 "2015-06-17T07:03:08Z")
I am using multiple-line in my logstash. Logstash is receiving log from logstash-forwarder. It works fine. However, when there is extreme long multiple line log (some dump information), logstash stop processing. I can i… --- ## [Output Exec Script](https://discuss.elastic.co/t/output-exec-script/2750)
**Author:** [@neoform](https://discuss.elastic.co/u/neoform)\ **Replies:** 10\ **Last updated:** [June 17, 2015, 3:25am UTC](https://discuss.elastic.co/t/output-exec-script/2750 "2015-06-17T03:25:58Z")
Hello, I’m attempting to read a log file and run a script as output. The script accepts STDIN data, but I don’t think I’ve set up the log stash configs properly since the script never gets executed the way it’s curren… --- ## [Bad Outputs moving from RabbitMQ to Rackspace Queueing](https://discuss.elastic.co/t/bad-outputs-moving-from-rabbitmq-to-rackspace-queueing/2796)
**Author:** [@DKlotz](https://discuss.elastic.co/u/DKlotz)\ **Replies:** 2\ **Last updated:** [June 17, 2015, 1:33am UTC](https://discuss.elastic.co/t/bad-outputs-moving-from-rabbitmq-to-rackspace-queueing/2796 "2015-06-17T01:33:10Z")
After running into some frustrations with RabbitMQ's stability, I am attempting to move to Rackspace's Queuing plugin. I have 2 servers, one which populates the queue, one which pulls from the queue. I have replaced… --- ## [Go to the latest documentation topics for 1.5.0 result in 404](https://discuss.elastic.co/t/go-to-the-latest-documentation-topics-for-1-5-0-result-in-404/23763)
**Author:** [@brunomiranda](https://discuss.elastic.co/u/brunomiranda)\ **Replies:** 2\ **Last updated:** [June 16, 2015, 10:31pm UTC](https://discuss.elastic.co/t/go-to-the-latest-documentation-topics-for-1-5-0-result-in-404/23763 "2015-06-16T22:31:31Z")
Browsing to http://www.logstash.net/docs/1.1.10/extending/example-add-a-new-filter or pretty much and input/output has a big red box at the top advising people where to find the 1.5.0 docs. However those links all point … --- ## [UDP/Netflow Performance](https://discuss.elastic.co/t/udp-netflow-performance/2737)
**Author:** [@andsch](https://discuss.elastic.co/u/andsch)\ **Replies:** 3\ **Last updated:** [June 16, 2015, 9:18pm UTC](https://discuss.elastic.co/t/udp-netflow-performance/2737 "2015-06-16T21:18:06Z")
Hi there, Can anyone share experience about server scaling and UDP performance. my Logstash setup is quite simple: UDP input with Netflow Codec -\> no Filters -\> Elasticsearch output via HTTP via bulk API My current… --- ## [Unable to Read from File in Windows](https://discuss.elastic.co/t/unable-to-read-from-file-in-windows/23784)
**Author:** [@JamesRenaud](https://discuss.elastic.co/u/JamesRenaud)\ **Replies:** 2\ **Last updated:** [June 16, 2015, 8:27pm UTC](https://discuss.elastic.co/t/unable-to-read-from-file-in-windows/23784 "2015-06-16T20:27:01Z")
Hey all, I'm at a bit of a loss on this one after spending a couple hours on Google and checking through about a dozen articles I'm still not getting this resolved. I have a flat \*.log file that I'm trying to read fr… --- ## [How to use multiline to break up messages delimited with - (hyphen)](https://discuss.elastic.co/t/how-to-use-multiline-to-break-up-messages-delimited-with-hyphen/2378)
**Author:** [@michaellizhou](https://discuss.elastic.co/u/michaellizhou)\ **Replies:** 2\ **Last updated:** [June 16, 2015, 7:09pm UTC](https://discuss.elastic.co/t/how-to-use-multiline-to-break-up-messages-delimited-with-hyphen/2378 "2015-06-16T19:09:19Z")
So I am going through this one type of log where some of the messages are broken up by hyphen (-----) for example: 2015-05-01 10:08:08 \[http-request\] Level Outbound Message ------------------------ ID: 2 … --- ## [Logstash skipping some lines](https://discuss.elastic.co/t/logstash-skipping-some-lines/23770)
**Author:** [@chaitanyavvs](https://discuss.elastic.co/u/chaitanyavvs)\ **Replies:** 0\ **Last updated:** [June 16, 2015, 5:34pm UTC](https://discuss.elastic.co/t/logstash-skipping-some-lines/23770 "2015-06-16T17:34:32Z")
Hi, I am using logstash 1.5.0 on a windows platform. I am new to logstash. I have written logstash configuration for parsing my log file. The configuration is as follows input{ file { path =\> "C:/dummy.log" start\_p… --- ## [Groking syslog, seem to be getting a match but still getting \_grokparsefailure](https://discuss.elastic.co/t/groking-syslog-seem-to-be-getting-a-match-but-still-getting--grokparsefailure/2685)
**Author:** [@bobthomson](https://discuss.elastic.co/u/bobthomson)\ **Replies:** 7\ **Last updated:** [June 16, 2015, 12:34pm UTC](https://discuss.elastic.co/t/groking-syslog-seem-to-be-getting-a-match-but-still-getting--grokparsefailure/2685 "2015-06-16T12:34:37Z")
I am trying to parse and store a lot of syslog data, currently RHEL syslog + log4j output from Spring apps (Constraint is it has to come via rsyslog for now). My input is : input { udp { port =\> 514 … --- ## [Plugins via RPM](https://discuss.elastic.co/t/plugins-via-rpm/1999)
**Author:** [@Duncan\_Innes](https://discuss.elastic.co/u/Duncan_Innes)\ **Replies:** 2\ **Last updated:** [June 16, 2015, 8:52am UTC](https://discuss.elastic.co/t/plugins-via-rpm/1999 "2015-06-16T08:52:51Z")
Would it be possible to look at distributing Logstash plugins via RPM? The majority of corporate RHEL estates, in my experience, don't permit external access unless a server is required to face the internet. The genera… --- ## [INFO: I/O exception](https://discuss.elastic.co/t/info-i-o-exception/2761)
**Author:** [@Silver](https://discuss.elastic.co/u/Silver)\ **Replies:** 2\ **Last updated:** [June 16, 2015, 7:54am UTC](https://discuss.elastic.co/t/info-i-o-exception/2761 "2015-06-16T07:54:22Z")
This is my setup IIS Server --\> logstash-forwarder --\> 2 logstash instances (fqdn certs) --\> haproxy --\> 2 elasticsearch node The logstash-forwarder is working and my cluster is also mostly in green.. but out of now… --- ## [Convert site name to IP address in Logstash](https://discuss.elastic.co/t/convert-site-name-to-ip-address-in-logstash/2753)
**Author:** [@Mark\_Nuttall](https://discuss.elastic.co/u/Mark_Nuttall)\ **Replies:** 4\ **Last updated:** [June 16, 2015, 1:28am UTC](https://discuss.elastic.co/t/convert-site-name-to-ip-address-in-logstash/2753 "2015-06-16T01:28:03Z")
I am parsing Apache log files and, instead of IP addressess , it mostly contains things like ec2-54-200-236-252.us-west-2.compute.amazonaws.com . I have tried the DNS filter to no avail. I'd like to have the IP address … --- ## [Migrating Filter/Plugin/Pattern to Logstash 1.5](https://discuss.elastic.co/t/migrating-filter-plugin-pattern-to-logstash-1-5/2216)
**Author:** [@xiro](https://discuss.elastic.co/u/xiro)\ **Replies:** 3\ **Last updated:** [June 16, 2015, 12:43am UTC](https://discuss.elastic.co/t/migrating-filter-plugin-pattern-to-logstash-1-5/2216 "2015-06-16T00:43:29Z")
Hi, I've just set up a new ELK-stack for monitoring of Cisco ASA Log-files. We have already one running with kibana 3 and logstash 1.4. Unfortunately, my filter is not working with the new version, since it contains "g… --- ## [Removing fields from log](https://discuss.elastic.co/t/removing-fields-from-log/2701)
**Author:** [@Paul\_Letski](https://discuss.elastic.co/u/Paul_Letski)\ **Replies:** 1\ **Last updated:** [June 15, 2015, 11:05pm UTC](https://discuss.elastic.co/t/removing-fields-from-log/2701 "2015-06-15T23:05:10Z")
Hello! I'm trying to setup ELK stack. I'm importing logs from remote servers to ELK-server. Logs in JSON format with some additions from rsyslog (timestamp, remote server name, app name). if I understand correctly,… --- ## [Batch process at logstash](https://discuss.elastic.co/t/batch-process-at-logstash/2707)
**Author:** [@Ducheol\_Kim](https://discuss.elastic.co/u/Ducheol_Kim)\ **Replies:** 1\ **Last updated:** [June 15, 2015, 10:57pm UTC](https://discuss.elastic.co/t/batch-process-at-logstash/2707 "2015-06-15T22:57:56Z")
Hi All. I got a lot of help from here and Thanks for whole help. I have another dump question. Is it possible to do batch process at logstash ? Below is my situation. We are researching replace river to logstash… --- ## [SQL Server input for Logstash](https://discuss.elastic.co/t/sql-server-input-for-logstash/2725)
**Author:** [@Win\_Wang](https://discuss.elastic.co/u/Win_Wang)\ **Replies:** 1\ **Last updated:** [June 15, 2015, 10:52pm UTC](https://discuss.elastic.co/t/sql-server-input-for-logstash/2725 "2015-06-15T22:52:35Z")
I didn't see an implementation of an SQL Server plugin for Logstash, and a quick StackOverflow reseult used a file as an intermediary. Does anyone know of a better solution? For example, if I could use the included sqlit… --- ## [Logstash input speed VERY slow on AWS](https://discuss.elastic.co/t/logstash-input-speed-very-slow-on-aws/2715)
**Author:** [@nathaniel](https://discuss.elastic.co/u/nathaniel)\ **Replies:** 0\ **Last updated:** [June 15, 2015, 4:36pm UTC](https://discuss.elastic.co/t/logstash-input-speed-very-slow-on-aws/2715 "2015-06-15T16:36:42Z")
Hi guys, thanks for noticing this question. I posted a question about logstash input speed a couple of days ago, and it turns out the problem is because of I'm using aws. I realized that if I use Amazon's service, the lo… --- ## [Reading multiline data from third line](https://discuss.elastic.co/t/reading-multiline-data-from-third-line/2510)
**Author:** [@falafel](https://discuss.elastic.co/u/falafel)\ **Replies:** 4\ **Last updated:** [June 15, 2015, 2:13pm UTC](https://discuss.elastic.co/t/reading-multiline-data-from-third-line/2510 "2015-06-15T14:13:55Z")
I have a log file which contains lines which begin with a timestamp. Optional lines (an uncertain number of them) might follow each such timestamped line: SOMETIMESTAMP some data optional more data 1 2 optional other … --- ## [Parsing nested logs](https://discuss.elastic.co/t/parsing-nested-logs/2688)
**Author:** [@lladelfa](https://discuss.elastic.co/u/lladelfa)\ **Replies:** 0\ **Last updated:** [June 15, 2015, 11:55am UTC](https://discuss.elastic.co/t/parsing-nested-logs/2688 "2015-06-15T11:55:16Z")
Hello, I've been testing the logstash (1.5.0) in order to process a log generated from a fortianalyzer (by fortinet) without being successful. I have to say it's my first try with logstash so I kindly appreciate some he… --- ## [Can't add a new key into hash during iteration](https://discuss.elastic.co/t/cant-add-a-new-key-into-hash-during-iteration/2678)
**Author:** [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Replies:** 0\ **Last updated:** [June 15, 2015, 9:04am UTC](https://discuss.elastic.co/t/cant-add-a-new-key-into-hash-during-iteration/2678 "2015-06-15T09:04:40Z")
Getting exception while executing below ruby block: ruby { code =\> " event.to\_hash.each\_pair do |k, v| if k == 'Time\_FV' event\['TimeFV'\]= event\[k\] elsif k== 'Progess\_FV' event\['ProgessFV'\]= e… --- ## [GeoIP output IPs to file](https://discuss.elastic.co/t/geoip-output-ips-to-file/2534)
**Author:** [@gromit](https://discuss.elastic.co/u/gromit)\ **Replies:** 5\ **Last updated:** [June 13, 2015, 8:51am UTC](https://discuss.elastic.co/t/geoip-output-ips-to-file/2534 "2015-06-13T08:51:32Z")
I would like to output just the geoip.ip field to file (just the IP address). I have tried the following but just get errors. I think the ip is a nested field of geoip, but not sure how to extract. file { message\_f… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1257) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1259)