# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1085 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1086 --- ## [No raw fields, not using template](https://discuss.elastic.co/t/no-raw-fields-not-using-template/64551)
**Author:** [@frankkoornstra](https://discuss.elastic.co/u/frankkoornstra)\ **Replies:** 2\ **Last updated:** [November 1, 2016, 1:59pm UTC](https://discuss.elastic.co/t/no-raw-fields-not-using-template/64551 "2016-11-01T13:59:36Z")
I saw the question come by several times but here I am, same problem, everything seems to be in order but I'm not getting my raw fields :frowning: I haven't changed anything about the index naming in the Elasticsearch … --- ## [Using Elasticsearch as an input Errors](https://discuss.elastic.co/t/using-elasticsearch-as-an-input-errors/34400)
**Author:** [@andy.barton](https://discuss.elastic.co/u/andy.barton)\ **Replies:** 2\ **Last updated:** [November 1, 2016, 1:44pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-an-input-errors/34400 "2016-11-01T13:44:20Z")
Hopefully someone will be able to help with this as it is driving me mad at the moment! I have successfully managed to load 2.8m records into an elasticsearch cluster using sql server as the source. I am now trying t… --- ## [Beats plugin: "Connection closed by foreign host."](https://discuss.elastic.co/t/beats-plugin-connection-closed-by-foreign-host/60597)
**Author:** [@DrPhlux](https://discuss.elastic.co/u/DrPhlux)\ **Replies:** 0\ **Last updated:** [September 15, 2016, 11:01am UTC](https://discuss.elastic.co/t/beats-plugin-connection-closed-by-foreign-host/60597 "2016-09-15T11:01:28Z")
Hello. I'm using an Winlogbeat/filebeat -\> logstash(linux) -\> ES(linux) environment where my logstash process reaches a specific number of connected beats clients and starts to drop new connections. Example from actual l… --- ## [Multiline issue with matching repeated lines](https://discuss.elastic.co/t/multiline-issue-with-matching-repeated-lines/64545)
**Author:** [@Mark\_Dolan](https://discuss.elastic.co/u/Mark_Dolan)\ **Replies:** 0\ **Last updated:** [November 1, 2016, 11:30am UTC](https://discuss.elastic.co/t/multiline-issue-with-matching-repeated-lines/64545 "2016-11-01T11:30:11Z")
I have some data from the IBM nmon Linux/AIX monitoring tool. It's an awkward format with several different delimited and headerless fields. I've managed to create all of its grok patterns and process it (mostly) using t… --- ## [LS error while reindexing data](https://discuss.elastic.co/t/ls-error-while-reindexing-data/40218)
**Author:** [@vikas\_gopal](https://discuss.elastic.co/u/vikas_gopal)\ **Replies:** 3\ **Last updated:** [November 1, 2016, 10:08am UTC](https://discuss.elastic.co/t/ls-error-while-reindexing-data/40218 "2016-11-01T10:08:36Z")
Hi Experts , I am using logstash1.5.4 with ES1.7.1 to reindex old data to a new index . My LS conf file is like input { elasticsearch { hosts =\> "localhost" index =\> "based1" query =\> "\*" size =\> 5… --- ## [Adding more logs to filebeat to export and on elk server to logstash to receive](https://discuss.elastic.co/t/adding-more-logs-to-filebeat-to-export-and-on-elk-server-to-logstash-to-receive/64527)
**Author:** [@pkhodaveissi](https://discuss.elastic.co/u/pkhodaveissi)\ **Replies:** 6\ **Last updated:** [November 1, 2016, 9:46am UTC](https://discuss.elastic.co/t/adding-more-logs-to-filebeat-to-export-and-on-elk-server-to-logstash-to-receive/64527 "2016-11-01T09:46:15Z")
Hello I have this starter config which derives from DigitalOcean and looks like this: input { beats { port =\> 5044 ssl =\> true ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.c… --- ## [Logstash output configuration for beat](https://discuss.elastic.co/t/logstash-output-configuration-for-beat/64217)
**Author:** [@piyush](https://discuss.elastic.co/u/piyush)\ **Replies:** 2\ **Last updated:** [October 31, 2016, 9:19pm UTC](https://discuss.elastic.co/t/logstash-output-configuration-for-beat/64217 "2016-10-31T21:19:47Z")
Hi Team, My logstash is sending Json log data to below two indexes project-%{environment}-%{type}-%{+YYYY.MM.dd} filebeat-%{+YYYY.MM.dd} PFB my logstash.conf, what condition i should put to stop sending Json log dat… --- ## [Performance issue in ELK stack](https://discuss.elastic.co/t/performance-issue-in-elk-stack/64478)
**Author:** [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Replies:** 1\ **Last updated:** [October 31, 2016, 8:38pm UTC](https://discuss.elastic.co/t/performance-issue-in-elk-stack/64478 "2016-10-31T20:38:47Z")
Hi, I have established an ELK stack with 3 shipper logstash (for Linux, Windows, Network Devices data each) , with kafka cluster (3 topics each for linux, windows and network devices) , with 3 logstash indexer (each … --- ## [How to parse logs produced by logstash](https://discuss.elastic.co/t/how-to-parse-logs-produced-by-logstash/64197)
**Author:** [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\ **Replies:** 1\ **Last updated:** [October 31, 2016, 8:22pm UTC](https://discuss.elastic.co/t/how-to-parse-logs-produced-by-logstash/64197 "2016-10-31T20:22:29Z")
I'm ironically having trouble parsing logstash logs using logstash. It is not json, it looks more like a ruby hash output. Is there an existing filter that can effectively parse logstash's logs for me?? Thanks! --- ## [Log4net parsing in cent os 7](https://discuss.elastic.co/t/log4net-parsing-in-cent-os-7/64178)
**Author:** [@mmehraban](https://discuss.elastic.co/u/mmehraban)\ **Replies:** 1\ **Last updated:** [October 31, 2016, 8:21pm UTC](https://discuss.elastic.co/t/log4net-parsing-in-cent-os-7/64178 "2016-10-31T20:21:21Z")
Hey guys,I have a log file like this: 2016-10-17 03:20:37,690 \[319299\] ERROR Global - Unhandled application error System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. Parameter n… --- ## [Logstash-filter-prune gem dependencies](https://discuss.elastic.co/t/logstash-filter-prune-gem-dependencies/64136)
**Author:** [@lanejlanej](https://discuss.elastic.co/u/lanejlanej)\ **Replies:** 1\ **Last updated:** [October 31, 2016, 8:19pm UTC](https://discuss.elastic.co/t/logstash-filter-prune-gem-dependencies/64136 "2016-10-31T20:19:19Z")
The logstash-filter-prune gem version 2.0.5 on rubygems.org appears to have dependency set as logstash-core-plugin-api ~\> 1.0. Is that right? --- ## [Plugin input twitter option proxy](https://discuss.elastic.co/t/plugin-input-twitter-option-proxy/64115)
**Author:** [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Replies:** 1\ **Last updated:** [October 31, 2016, 8:10pm UTC](https://discuss.elastic.co/t/plugin-input-twitter-option-proxy/64115 "2016-10-31T20:10:24Z")
Hello in the previous version of twitter plugin could put a proxy username and password config :proxy\_user, :validate =\> :string, :default =\> "username" config :proxy\_password, :validate =\> :string, :default =\> "passwo… --- ## [Connection refused error from logstash.log](https://discuss.elastic.co/t/connection-refused-error-from-logstash-log/64330)
**Author:** [@rai](https://discuss.elastic.co/u/rai)\ **Replies:** 6\ **Last updated:** [October 31, 2016, 6:08pm UTC](https://discuss.elastic.co/t/connection-refused-error-from-logstash-log/64330 "2016-10-31T18:08:13Z")
I am getting following error on elk server and not able to see the log on Kibana. {:timestamp=\>"2016-10-28T21:52:02.243000+0000", :message=\>"Connection refused", :class=\>"Manticore::SocketException", {:timestamp=\>"201… --- ## [Elasticsearch remove docs of Suricata](https://discuss.elastic.co/t/elasticsearch-remove-docs-of-suricata/64284)
**Author:** [@brunopsitech](https://discuss.elastic.co/u/brunopsitech)\ **Replies:** 3\ **Last updated:** [October 31, 2016, 4:06pm UTC](https://discuss.elastic.co/t/elasticsearch-remove-docs-of-suricata/64284 "2016-10-31T16:06:02Z")
Hi, i am using Logstash to generate logs of IPS. But elasticsearch delete all docs except last one, i dont know why. My Logstash config: input { file { path =\> \["/var/log/suricata/eve.json"\] … --- ## [Which is the proper way to check Logstash activity?](https://discuss.elastic.co/t/which-is-the-proper-way-to-check-logstash-activity/63057)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 2\ **Last updated:** [October 31, 2016, 3:40pm UTC](https://discuss.elastic.co/t/which-is-the-proper-way-to-check-logstash-activity/63057 "2016-10-31T15:40:30Z")
Hello, I'm using Logstash on a docker container (see \[https://hub.docker.com/\_/logstash/\]) which I'm starting with the detach (-d) option. The thing is working fine, however now that I'm parsing a lot of logs I would lik… --- ## [Difference between File Beats and Logstash file input](https://discuss.elastic.co/t/difference-between-file-beats-and-logstash-file-input/64435)
**Author:** [@sjivan](https://discuss.elastic.co/u/sjivan)\ **Replies:** 2\ **Last updated:** [October 31, 2016, 12:09pm UTC](https://discuss.elastic.co/t/difference-between-file-beats-and-logstash-file-input/64435 "2016-10-31T12:09:35Z")
Hi, I have all my log files accessible on the machine where Logstash is running and have been using the file input configuration of Logstash. Is there any functional difference is using this approach vs running File Beat… --- ## [Ruby string test](https://discuss.elastic.co/t/ruby-string-test/64299)
**Author:** [@Gregs](https://discuss.elastic.co/u/Gregs)\ **Replies:** 3\ **Last updated:** [October 31, 2016, 9:04am UTC](https://discuss.elastic.co/t/ruby-string-test/64299 "2016-10-31T09:04:33Z")
Logstash 2.3.4: Here the case infos "clientip" =\> \[ \[0\] "151.38.152.143", \[1\] "-" \], ruby { code =\> 'event\["clientip"\].each { |k,v| if v == "-" event\["clientip2"\]\[k\] = v end } ' } Bu… --- ## [:sql\_last\_value doesn't reiterate with multiple jdbc inputs](https://discuss.elastic.co/t/sql-last-value-doesnt-reiterate-with-multiple-jdbc-inputs/63523)
**Author:** [@wmedlen](https://discuss.elastic.co/u/wmedlen)\ **Replies:** 7\ **Last updated:** [October 30, 2016, 10:28pm UTC](https://discuss.elastic.co/t/sql-last-value-doesnt-reiterate-with-multiple-jdbc-inputs/63523 "2016-10-30T22:28:44Z")
I hope my title wasn't confusing - we have 2 jdbc inputs in our logstash config, one that pulls a commentid and one that pulls a messageid from our application. the messageid does not get added to logstash unless I … --- ## [Logstash Grok unable to create new fields for apache-access log](https://discuss.elastic.co/t/logstash-grok-unable-to-create-new-fields-for-apache-access-log/64371)
**Author:** [@kishorramanan](https://discuss.elastic.co/u/kishorramanan)\ **Replies:** 3\ **Last updated:** [October 30, 2016, 7:07pm UTC](https://discuss.elastic.co/t/logstash-grok-unable-to-create-new-fields-for-apache-access-log/64371 "2016-10-30T19:07:54Z")
Hi, New to ELK. I am trying to build a simple dashboard for apache access log. I need to extract the second IP from the log Sample log 127.0.0.1 172.0.0.10, 123.56.98.356 - - \[date\] "GET /loca/lib/images.img HTTP/1.… --- ## [Logstash doesn't reload config if ip is wrong 5.0.0](https://discuss.elastic.co/t/logstash-doesnt-reload-config-if-ip-is-wrong-5-0-0/64376)
**Author:** [@glagidse](https://discuss.elastic.co/u/glagidse)\ **Replies:** 0\ **Last updated:** [October 30, 2016, 3:10pm UTC](https://discuss.elastic.co/t/logstash-doesnt-reload-config-if-ip-is-wrong-5-0-0/64376 "2016-10-30T15:10:35Z")
My config file: # The # character at the beginning of a line indicates a comment. Use # comments to describe your configuration. input { beats { host =\> "localhost" port =\> "5044" } } # The filt… --- ## [Filebeats sometimes not sending?](https://discuss.elastic.co/t/filebeats-sometimes-not-sending/64374)
**Author:** [@glagidse](https://discuss.elastic.co/u/glagidse)\ **Replies:** 0\ **Last updated:** [October 30, 2016, 2:30pm UTC](https://discuss.elastic.co/t/filebeats-sometimes-not-sending/64374 "2016-10-30T14:30:58Z")
I'm currently following this tutorial: https://www.elastic.co/guide/en/logstash/master/advanced-pipeline.html after 2 hours of searching I discovered that beats only sends files if they're changed, if I stop and rest… --- ## [The performance of using ruby code in config file](https://discuss.elastic.co/t/the-performance-of-using-ruby-code-in-config-file/64366)
**Author:** [@WangXiangUSTC](https://discuss.elastic.co/u/WangXiangUSTC)\ **Replies:** 2\ **Last updated:** [October 30, 2016, 10:32am UTC](https://discuss.elastic.co/t/the-performance-of-using-ruby-code-in-config-file/64366 "2016-10-30T10:32:39Z")
I use logstash to solve logs, and include some ruby code in logstash's config file. I have total four server to run logstash, and every cpu has 32 cores. I found they can only solve about 10000 logs every second, and th… --- ## [2.3.4 - Adding Nested Fields to Events](https://discuss.elastic.co/t/2-3-4-adding-nested-fields-to-events/64252)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 3\ **Last updated:** [October 30, 2016, 10:04am UTC](https://discuss.elastic.co/t/2-3-4-adding-nested-fields-to-events/64252 "2016-10-30T10:04:43Z")
Hi, I'm trying to add a k/v pair of type: type.field.size to events using the ruby filter and it is failing. My latest attempt looks like this: ruby { code =\> " event.append({'document' =\>{'message' =\> {'siz… --- ## [Logstash to Kafka](https://discuss.elastic.co/t/logstash-to-kafka/64361)
**Author:** [@kalapakim](https://discuss.elastic.co/u/kalapakim)\ **Replies:** 1\ **Last updated:** [October 30, 2016, 9:12am UTC](https://discuss.elastic.co/t/logstash-to-kafka/64361 "2016-10-30T09:12:58Z")
I am trying to set up logstash, client side, so that logstash can consume logs locally on an application server and send it to kafka. I am pretty new to the ELK stack and could use some guidance on how to do this correct… --- ## [Get Foldername from file sent by filebeat](https://discuss.elastic.co/t/get-foldername-from-file-sent-by-filebeat/64280)
**Author:** [@txisme](https://discuss.elastic.co/u/txisme)\ **Replies:** 1\ **Last updated:** [October 29, 2016, 8:00pm UTC](https://discuss.elastic.co/t/get-foldername-from-file-sent-by-filebeat/64280 "2016-10-29T20:00:37Z")
How can i get the folder name of a file sent by filebeat?? i tried this: grok { path =\> "/path/to/folder/%{WORD:foldername}/%{GREEDYDATA}" } but it s not working :confused: --- ## [Log stash how to pattern match a phrase? Custom Regex?](https://discuss.elastic.co/t/log-stash-how-to-pattern-match-a-phrase-custom-regex/64290)
**Author:** [@shield](https://discuss.elastic.co/u/shield)\ **Replies:** 2\ **Last updated:** [October 29, 2016, 7:57pm UTC](https://discuss.elastic.co/t/log-stash-how-to-pattern-match-a-phrase-custom-regex/64290 "2016-10-29T19:57:47Z")
New to Logstash and I have seen many items that discuss using defined patterns to parse logs but I have some ugly logs that don't fit a defined format. I need to do accomplish two things: Parse an ugly time stamp int… --- ## [Got strange problem on logstash: configuration file reset automatically](https://discuss.elastic.co/t/got-strange-problem-on-logstash-configuration-file-reset-automatically/63970)
**Author:** [@Hao\_Zhang](https://discuss.elastic.co/u/Hao_Zhang)\ **Replies:** 1\ **Last updated:** [October 29, 2016, 7:51pm UTC](https://discuss.elastic.co/t/got-strange-problem-on-logstash-configuration-file-reset-automatically/63970 "2016-10-29T19:51:47Z")
Hi, I am a totally new beginner of ELK. At work I was asked to build a ELK monitoring system on company's servers. Here I got a very strange question. I have three configure files corresponding to input, filter and outpu… --- ## [Logstash-cloudwatch-input plugin is not sending data to Elasticsearch](https://discuss.elastic.co/t/logstash-cloudwatch-input-plugin-is-not-sending-data-to-elasticsearch/46867)
**Author:** [@Rajat\_Toshniwal](https://discuss.elastic.co/u/Rajat_Toshniwal)\ **Replies:** 4\ **Last updated:** [October 28, 2016, 7:45pm UTC](https://discuss.elastic.co/t/logstash-cloudwatch-input-plugin-is-not-sending-data-to-elasticsearch/46867 "2016-10-28T19:45:33Z")
Hi All, I am little new in this world and really struggling in setting up a centralized log management server for AWS cloud servers. I am using logstash-cloudwatch-input plugin to do the same. Below is my Logstash conf… --- ## [Elasticsearch output plugin throws error on sniffing =\> true as it makes the pool to empty](https://discuss.elastic.co/t/elasticsearch-output-plugin-throws-error-on-sniffing-true-as-it-makes-the-pool-to-empty/62205)
**Author:** [@Tarini\_Dash](https://discuss.elastic.co/u/Tarini_Dash)\ **Replies:** 1\ **Last updated:** [October 28, 2016, 6:16pm UTC](https://discuss.elastic.co/t/elasticsearch-output-plugin-throws-error-on-sniffing-true-as-it-makes-the-pool-to-empty/62205 "2016-10-28T18:16:28Z")
Product : logstash-5.0.0-beta1 With the folowing output configuration I get error when start a pipeline output { elasticsearch { hosts =\> \["x.x.x.x:9200","y.y.y.y:9200"\] sniffing =\> true } } when run as b… --- ## [Aggregate plugin perfomance](https://discuss.elastic.co/t/aggregate-plugin-perfomance/63833)
**Author:** [@111126](https://discuss.elastic.co/u/111126)\ **Replies:** 4\ **Last updated:** [October 28, 2016, 4:59pm UTC](https://discuss.elastic.co/t/aggregate-plugin-perfomance/63833 "2016-10-28T16:59:51Z")
Hello. I forward my logs to redis and I have logstash redis input. Logstash filter use aggregate plugin. I have real events per second rate 20k, but logstash can only 1.5 k events per sec. I use one logstash worker. My … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1084) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1086)