# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1097 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1098 --- ## [Read slow for large files in S3?](https://discuss.elastic.co/t/read-slow-for-large-files-in-s3/61551)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 4\ **Last updated:** [September 29, 2016, 4:59pm UTC](https://discuss.elastic.co/t/read-slow-for-large-files-in-s3/61551 "2016-09-29T16:59:06Z")
I have two files in a folder, one small 300kb, and one large, 2G. Logstash only reads 300kb one. Is it right? Shall i wait more time? --- ## [Need help with Logstash](https://discuss.elastic.co/t/need-help-with-logstash/61761)
**Author:** [@nareshsife](https://discuss.elastic.co/u/nareshsife)\ **Replies:** 3\ **Last updated:** [September 29, 2016, 12:18pm UTC](https://discuss.elastic.co/t/need-help-with-logstash/61761 "2016-09-29T12:18:22Z")
I am new to Logstash, i dont have a background of building a logstash before. I built Suricata (IDS) and on the same machine i built logstash. I was successful in installing it and bringing up the webpage. However, i am … --- ## [Issue with Re-Indexing](https://discuss.elastic.co/t/issue-with-re-indexing/61598)
**Author:** [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Replies:** 3\ **Last updated:** [September 29, 2016, 10:53am UTC](https://discuss.elastic.co/t/issue-with-re-indexing/61598 "2016-09-29T10:53:53Z")
HI, I have tried Re-Index one of the index. While Re-Indexing, it is taking the data from other indexes as well. Have given the config file below. input { elasticsearch { hosts =\> \[ "localhost" \] index =\> "s… --- ## [Logstash error while starting!](https://discuss.elastic.co/t/logstash-error-while-starting/61792)
**Author:** [@amits](https://discuss.elastic.co/u/amits)\ **Replies:** 1\ **Last updated:** [September 29, 2016, 9:05am UTC](https://discuss.elastic.co/t/logstash-error-while-starting/61792 "2016-09-29T09:05:43Z")
Hi, I am facing a logstash issue on starting it. \[amits@localhost bin\]$ ./logstash -e 'input { stdin { } } output { stdout {} }' LoadError: no such file to load -- logstash/util/wrapped\_synchronous\_queue require a… --- ## [Elasticsearch get huge amount of duplicated Data from redis](https://discuss.elastic.co/t/elasticsearch-get-huge-amount-of-duplicated-data-from-redis/61477)
**Author:** [@Tal\_Bar\_Or](https://discuss.elastic.co/u/Tal_Bar_Or)\ **Replies:** 4\ **Last updated:** [September 29, 2016, 8:46am UTC](https://discuss.elastic.co/t/elasticsearch-get-huge-amount-of-duplicated-data-from-redis/61477 "2016-09-29T08:46:26Z")
Hello, I data feed from Snort under Pfsense configuration getting from syslog\>logstash \>\> redis and redis\>\>elasticsearch as follows below The issue i have is that in the orginal source (snort logs)i have very few mess… --- ## [Logstash failing to start or configtest](https://discuss.elastic.co/t/logstash-failing-to-start-or-configtest/54934)
**Author:** [@Cylindric](https://discuss.elastic.co/u/Cylindric)\ **Replies:** 1\ **Last updated:** [September 29, 2016, 8:36am UTC](https://discuss.elastic.co/t/logstash-failing-to-start-or-configtest/54934 "2016-09-29T08:36:52Z")
Hi folks. I'm not sure what happened, but yesterday my ELK was logging, and today it isn't. I tried to check the configs on one of the Logstash nodes, and I get an error about loading logstash/runner: sudo /opt/logstas… --- ## [Logstash beta - 1 does not include logrotate file](https://discuss.elastic.co/t/logstash-beta-1-does-not-include-logrotate-file/61680)
**Author:** [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\ **Replies:** 7\ **Last updated:** [September 29, 2016, 7:37am UTC](https://discuss.elastic.co/t/logstash-beta-1-does-not-include-logrotate-file/61680 "2016-09-29T07:37:48Z")
I just installed brandnew logstash 5.0 beta1 but it seem that package does not include the logrotate script. \[root@BS-POC-Test-Env logstash\]# yum install logstash Loaded plugins: fastestmirror, langpacks Loading mirror… --- ## [Length of an array in logstash](https://discuss.elastic.co/t/length-of-an-array-in-logstash/61750)
**Author:** [@saisn](https://discuss.elastic.co/u/saisn)\ **Replies:** 1\ **Last updated:** [September 29, 2016, 5:38am UTC](https://discuss.elastic.co/t/length-of-an-array-in-logstash/61750 "2016-09-29T05:38:08Z")
I am trying to find length of an array for a field using ruby filter. But, I'm getting below error even though nul case is handled. Please let me know if there is any error in the filter. Thanks in advance filter { ru… --- ## [Logstash HTTP output plugin logging vague "Manticore::ClientProtocolException" errors](https://discuss.elastic.co/t/logstash-http-output-plugin-logging-vague-manticore-clientprotocolexception-errors/61730)
**Author:** [@Yunchi\_Luo](https://discuss.elastic.co/u/Yunchi_Luo)\ **Replies:** 0\ **Last updated:** [September 28, 2016, 5:44pm UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-logging-vague-manticore-clientprotocolexception-errors/61730 "2016-09-28T17:44:12Z")
For all general issues, please provide the following details for fast resolution: Version: logstash 2.4.0 Operating System: Ubuntu 14.04.5 LTS Config File: output { http { http\_method =\> "post" url =\> "… --- ## [Remove part of field name from json inputted fields](https://discuss.elastic.co/t/remove-part-of-field-name-from-json-inputted-fields/61412)
**Author:** [@sirmodok](https://discuss.elastic.co/u/sirmodok)\ **Replies:** 8\ **Last updated:** [September 28, 2016, 3:45pm UTC](https://discuss.elastic.co/t/remove-part-of-field-name-from-json-inputted-fields/61412 "2016-09-28T15:45:38Z")
I have an input into elasticsearch from a large amount of json formatted data. The input looks like this file{ codec =\> json path =\> "/opt/apps/logs/health/\*.log" type =\> "health" } } Very simple. … --- ## [Mysql JDBC Extremely Slow](https://discuss.elastic.co/t/mysql-jdbc-extremely-slow/61709)
**Author:** [@bv3435](https://discuss.elastic.co/u/bv3435)\ **Replies:** 0\ **Last updated:** [September 28, 2016, 2:24pm UTC](https://discuss.elastic.co/t/mysql-jdbc-extremely-slow/61709 "2016-09-28T14:24:10Z")
I am trying to move to logstash from the elasticsearch-jdbc river to run a query on a table with 100M+ rows but it is extremely slow with logstash even after tuning the parameters and running it on 32 threads. I had to u… --- ## [Extracting contents of the filename (from the source) and creating fields](https://discuss.elastic.co/t/extracting-contents-of-the-filename-from-the-source-and-creating-fields/57199)
**Author:** [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Replies:** 6\ **Last updated:** [September 28, 2016, 8:54am UTC](https://discuss.elastic.co/t/extracting-contents-of-the-filename-from-the-source-and-creating-fields/57199 "2016-09-28T08:54:39Z")
Hello I am trying to extract contents of the filename (made available by the sourcefield) and would like to add tags/create new fields based on the criterie. For example if this was the data in the source field: /app… --- ## [Best secure way of getting remote logs?](https://discuss.elastic.co/t/best-secure-way-of-getting-remote-logs/61570)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 2\ **Last updated:** [September 28, 2016, 9:02am UTC](https://discuss.elastic.co/t/best-secure-way-of-getting-remote-logs/61570 "2016-09-28T09:02:41Z")
Hi there, following scenery: We have a log producer running on a machine which which has internet access. We have a logstash instance which is connected to elasticsearch in a different which has currently no internet… --- ## [Adding a date filter to CSV](https://discuss.elastic.co/t/adding-a-date-filter-to-csv/61663)
**Author:** [@setlem](https://discuss.elastic.co/u/setlem)\ **Replies:** 4\ **Last updated:** [September 28, 2016, 7:37am UTC](https://discuss.elastic.co/t/adding-a-date-filter-to-csv/61663 "2016-09-28T07:37:47Z")
hi magnus, I am new to ELk.i have used the same date filter but index file is not generating in elastic search and am unable to see data in kibana. here is my config file and my input is date, count sep 03 2016,55 --- ## [Csv and date filter](https://discuss.elastic.co/t/csv-and-date-filter/26033)
**Author:** [@zpp](https://discuss.elastic.co/u/zpp)\ **Replies:** 6\ **Last updated:** [September 28, 2016, 7:00am UTC](https://discuss.elastic.co/t/csv-and-date-filter/26033 "2016-09-28T07:00:56Z")
Hi, i have a logstash agent setup on a window box with below configuration. input { file { type =\> "autolog" path =\> "E:/elkcluster/log/\*.log" } file { type =\> "testscv" path =\> "e:/elkcluster/test/\*.\*" … --- ## [Selectively read certain files from S3?](https://discuss.elastic.co/t/selectively-read-certain-files-from-s3/61548)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 1\ **Last updated:** [September 28, 2016, 5:37am UTC](https://discuss.elastic.co/t/selectively-read-certain-files-from-s3/61548 "2016-09-28T05:37:58Z")
how can i selectively only read certain files?: The prefix setting below doesn't work.. What can i do?? My config below (i used CLI credential to terminal). input { s3 { bucket =\> "fotolia-prod-logs" prefix =\> "/\*\*/… --- ## [Setting java.library.path or windows authentication](https://discuss.elastic.co/t/setting-java-library-path-or-windows-authentication/61635)
**Author:** [@net](https://discuss.elastic.co/u/net)\ **Replies:** 0\ **Last updated:** [September 27, 2016, 4:53pm UTC](https://discuss.elastic.co/t/setting-java-library-path-or-windows-authentication/61635 "2016-09-27T16:53:49Z")
Hello, I'm using jdbc and I have the following filled out: input { jdbc { jdbc\_driver\_library =\> "the/path/to/sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDrive… --- ## [Sorting with elasticsearch input](https://discuss.elastic.co/t/sorting-with-elasticsearch-input/61614)
**Author:** [@aviv.ratzon](https://discuss.elastic.co/u/aviv.ratzon)\ **Replies:** 0\ **Last updated:** [September 27, 2016, 1:58pm UTC](https://discuss.elastic.co/t/sorting-with-elasticsearch-input/61614 "2016-09-27T13:58:48Z")
Hi everyone, I'm trying to use the elasticsearch input and the results are not as I excpect. I want the data to be inputed ordered by @timestamp. This is my configuration: input { Read all documents from Elasticsearc… --- ## [Error input jdbc](https://discuss.elastic.co/t/error-input-jdbc/61603)
**Author:** [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Replies:** 3\ **Last updated:** [September 27, 2016, 1:32pm UTC](https://discuss.elastic.co/t/error-input-jdbc/61603 "2016-09-27T13:32:00Z")
Hello, I configured as input logstash jdbc and when do I start logstash I have an error message and I can not find the error my logstash config input { jdbc { jdbc\_driver\_library =\> "/oracle/client/11.2.0… --- ## [Logstash alpha4/alpha5/beta1 - Error while installing any plugin on CentOS #5966](https://discuss.elastic.co/t/logstash-alpha4-alpha5-beta1-error-while-installing-any-plugin-on-centos-5966/61607)
**Author:** [@shog](https://discuss.elastic.co/u/shog)\ **Replies:** 0\ **Last updated:** [September 27, 2016, 1:26pm UTC](https://discuss.elastic.co/t/logstash-alpha4-alpha5-beta1-error-while-installing-any-plugin-on-centos-5966/61607 "2016-09-27T13:26:07Z")
Hi, I've already created this as a bug report here, but maybe someone from community was able to address this problem faster: https://github.com/elastic/logstash/issues/5966 TLDR; Installation of any plugin on Cent… --- ## [Logstash not recieving data from kafka](https://discuss.elastic.co/t/logstash-not-recieving-data-from-kafka/61606)
**Author:** [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Replies:** 0\ **Last updated:** [September 27, 2016, 1:12pm UTC](https://discuss.elastic.co/t/logstash-not-recieving-data-from-kafka/61606 "2016-09-27T13:12:29Z")
Hi, I am sending data from kafka cluster to logstash(indexer). I have 3 kafka servers of which one acts as the leader and logstash sends the data to the leader. Now I changed the leader (by shuting down the kafka… --- ## [My logstash got a hashed value hostname](https://discuss.elastic.co/t/my-logstash-got-a-hashed-value-hostname/61588)
**Author:** [@wangshuogr](https://discuss.elastic.co/u/wangshuogr)\ **Replies:** 2\ **Last updated:** [September 27, 2016, 10:57am UTC](https://discuss.elastic.co/t/my-logstash-got-a-hashed-value-hostname/61588 "2016-09-27T10:57:52Z")
The feedback message is { "message":"xxx", "host":"da30faafb1s4e2d2" } the host is a hashed value Is there any way that I can get the ip address or domain name in the host field? How can I configure it? --- ## [How to logstash using csv file upload](https://discuss.elastic.co/t/how-to-logstash-using-csv-file-upload/61573)
**Author:** [@sathish](https://discuss.elastic.co/u/sathish)\ **Replies:** 4\ **Last updated:** [September 27, 2016, 9:40am UTC](https://discuss.elastic.co/t/how-to-logstash-using-csv-file-upload/61573 "2016-09-27T09:40:25Z")
how to logstash using upload csv file any one help me --- ## [Converting XML key value pairs](https://discuss.elastic.co/t/converting-xml-key-value-pairs/61582)
**Author:** [@ydkadri](https://discuss.elastic.co/u/ydkadri)\ **Replies:** 1\ **Last updated:** [September 27, 2016, 8:57am UTC](https://discuss.elastic.co/t/converting-xml-key-value-pairs/61582 "2016-09-27T08:57:37Z")
I have used Tika to convert a number of documents into XML. These documents contain metadata in the format \ \ \ **Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 0\ **Last updated:** [September 27, 2016, 7:56am UTC](https://discuss.elastic.co/t/gracefully-stop-logstash/61569 "2016-09-27T07:56:38Z") I am currently running logstash script in the command prompt. ..\\logstash\\bin\\logstash -f myconfig.conf I have an output to two databases. What happens here is that If I were to run it with a static file as input the … --- ## [How to load csv file into elasticsearch for visualizaion in kibana](https://discuss.elastic.co/t/how-to-load-csv-file-into-elasticsearch-for-visualizaion-in-kibana/61501)
**Author:** [@Jaishil\_Dhal](https://discuss.elastic.co/u/Jaishil_Dhal)\ **Replies:** 7\ **Last updated:** [September 27, 2016, 6:01am UTC](https://discuss.elastic.co/t/how-to-load-csv-file-into-elasticsearch-for-visualizaion-in-kibana/61501 "2016-09-27T06:01:04Z")
I have csv file that i attached with this. I have already try with logstash, with this i can able to open that file but can not do the visualization in kibana. its shows some error. I m using the following code for logs… --- ## [IP Address and Dictionary | New Field as String](https://discuss.elastic.co/t/ip-address-and-dictionary-new-field-as-string/61544)
**Author:** [@dneto](https://discuss.elastic.co/u/dneto)\ **Replies:** 1\ **Last updated:** [September 27, 2016, 5:53am UTC](https://discuss.elastic.co/t/ip-address-and-dictionary-new-field-as-string/61544 "2016-09-27T05:53:36Z")
Hi Friends, I'm using logstash as syslog server to my firewall traffic. I have two IP fields: ip\_dst and ip\_src. I need to translate them to HOSTNAMES or some ALIAS against a dictionary, so need create 2 new fields an… --- ## [Invalid Config with S3](https://discuss.elastic.co/t/invalid-config-with-s3/61531)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 1\ **Last updated:** [September 26, 2016, 8:05pm UTC](https://discuss.elastic.co/t/invalid-config-with-s3/61531 "2016-09-26T20:05:47Z")
I am configuring to read from s3 like below: input { s3 { access\_key\_id =\> "\*\*\*\*\*\*\*\*\*\*\*\*" secret\_access\_key =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*" bucket =\> "fotolia-prod-logs" region =\> "eu-west-1" type =\> "sy… --- ## [Pipline aborted on Logstash 2.4 reading pfSense 2.3 logs](https://discuss.elastic.co/t/pipline-aborted-on-logstash-2-4-reading-pfsense-2-3-logs/61152)
**Author:** [@Soren](https://discuss.elastic.co/u/Soren)\ **Replies:** 2\ **Last updated:** [September 26, 2016, 8:23pm UTC](https://discuss.elastic.co/t/pipline-aborted-on-logstash-2-4-reading-pfsense-2-3-logs/61152 "2016-09-26T20:23:37Z")
Hi, The following code in my 11-pfsense-filter.conf file causes Logstash to fail with a pipeline error: if \[prog\] =~ /^filterlog$/ { mutate { remove\_field =\> \[ "msg", "datetime" \] } grok { … --- ## [Multiple Issues Parsing a CSV file with Logstash such as Multiline and Orphan Quotes](https://discuss.elastic.co/t/multiple-issues-parsing-a-csv-file-with-logstash-such-as-multiline-and-orphan-quotes/61443)
**Author:** [@Logstash-lover](https://discuss.elastic.co/u/Logstash-lover)\ **Replies:** 1\ **Last updated:** [September 26, 2016, 8:06pm UTC](https://discuss.elastic.co/t/multiple-issues-parsing-a-csv-file-with-logstash-such-as-multiline-and-orphan-quotes/61443 "2016-09-26T20:06:56Z")
Using a Windows implementation of the most current versiom of Logstash, i am importing a CSV file with log events, originally from a windows .evtx file. I noticed there appear to be several issues with the .csv file th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1096) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1098)