# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1018 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1019 --- ## [How can i schedule logstash every second for jdbc input plugin?](https://discuss.elastic.co/t/how-can-i-schedule-logstash-every-second-for-jdbc-input-plugin/27393)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 10\ **Last updated:** [March 15, 2017, 2:26pm UTC](https://discuss.elastic.co/t/how-can-i-schedule-logstash-every-second-for-jdbc-input-plugin/27393 "2017-03-15T14:26:33Z")
Hi All I am using this configuration for jdbc input plugin schedule =\> "\* \* \* \* \*" How I can schedule logstash for every 1 second or 2 second? Regards Ritesh --- ## ["document\_type" on filebeat & "if" and "type" on logstash are not working](https://discuss.elastic.co/t/document-type-on-filebeat-if-and-type-on-logstash-are-not-working/78636)
**Author:** [@yoshifuji\_tokyo](https://discuss.elastic.co/u/yoshifuji_tokyo)\ **Replies:** 2\ **Last updated:** [March 15, 2017, 2:09pm UTC](https://discuss.elastic.co/t/document-type-on-filebeat-if-and-type-on-logstash-are-not-working/78636 "2017-03-15T14:09:51Z")
Trouble I set "document\_type" on filebeat, and set "if" and "type" on logstash conf. I've confirmed these logs are on AWS S3 bucket, however all logs are combined into one bucket "web-app" without separation. It seem… --- ## [Converting a json field to a list json](https://discuss.elastic.co/t/converting-a-json-field-to-a-list-json/78683)
**Author:** [@E.Eric](https://discuss.elastic.co/u/E.Eric)\ **Replies:** 6\ **Last updated:** [March 15, 2017, 2:08pm UTC](https://discuss.elastic.co/t/converting-a-json-field-to-a-list-json/78683 "2017-03-15T14:08:44Z")
Hi, having a field: "aa": { "b":"123", "c":"123" } I want to convert that field to a list JSON with one object: "aa": \[ { "b":"123", "c":"123" } \] Thanks in advance! --- ## [Can't load an invalid configuration (:reason=\>"Expected one of #](https://discuss.elastic.co/t/cant-load-an-invalid-configuration-reason-expected-one-of/77422)
**Author:** [@hispeed](https://discuss.elastic.co/u/hispeed)\ **Replies:** 16\ **Last updated:** [March 15, 2017, 1:44pm UTC](https://discuss.elastic.co/t/cant-load-an-invalid-configuration-reason-expected-one-of/77422 "2017-03-15T13:44:34Z")
Hi, I have now figured out a little bit more about my problems. I get the following error in: /var/log/logstash: Error message: \[2017-03-05T00:01:22,530\]\[ERROR\]\[logstash.agent \] Cannot load an invalid con… --- ## [Unable to install beats plugin for logstash](https://discuss.elastic.co/t/unable-to-install-beats-plugin-for-logstash/50021)
**Author:** [@Srinivasan](https://discuss.elastic.co/u/Srinivasan)\ **Replies:** 5\ **Last updated:** [March 15, 2017, 1:03pm UTC](https://discuss.elastic.co/t/unable-to-install-beats-plugin-for-logstash/50021 "2017-03-15T13:03:24Z")
Hi, I am trying to install ELK module on a test environment. Windows Server 2008 R2 -x64 box I have successfully installed Elasticsearch, kibana and Logstash along with filebeat and winlogbeat as well.. Everything … --- ## [Windows: reload filter during runtime?](https://discuss.elastic.co/t/windows-reload-filter-during-runtime/78490)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 2\ **Last updated:** [March 15, 2017, 11:58am UTC](https://discuss.elastic.co/t/windows-reload-filter-during-runtime/78490 "2017-03-15T11:58:21Z")
Hi, I am starting logstash.bat with -f parameter on commandline. During development of new filters I often need to restart logstash because I changed filter condiguration. Is there a way to reload the configuration o… --- ## [How to read binary files using logstash](https://discuss.elastic.co/t/how-to-read-binary-files-using-logstash/78667)
**Author:** [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Replies:** 3\ **Last updated:** [March 15, 2017, 11:08am UTC](https://discuss.elastic.co/t/how-to-read-binary-files-using-logstash/78667 "2017-03-15T11:08:05Z")
Hi, Is there any way to read binary files and parse it to text using ELK stack? br, Sunil --- ## [Logstash Date Type](https://discuss.elastic.co/t/logstash-date-type/78637)
**Author:** [@Shubham\_Joshi](https://discuss.elastic.co/u/Shubham_Joshi)\ **Replies:** 19\ **Last updated:** [March 15, 2017, 11:06am UTC](https://discuss.elastic.co/t/logstash-date-type/78637 "2017-03-15T11:06:57Z")
I am using logstash to parse my logs on S3(gz). My log has data like: 172.31.0.14 - - \[15/May/2016:06:49:02 +0000\] "GET /ottsale/youthstars.html?color=112&manufacturer=116&ram=384&utm\_campaign=mi\_2704&utm\_medium=post&utm… --- ## [Invalid malformaed date](https://discuss.elastic.co/t/invalid-malformaed-date/78459)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 10\ **Last updated:** [March 15, 2017, 9:38am UTC](https://discuss.elastic.co/t/invalid-malformaed-date/78459 "2017-03-15T09:38:50Z")
Hi, Im facing below error while processing log kindly help me to resolve the issue, Error: Failed parsing date from field {:field=\>"iis\_eventDate", :value=\>"2017-03-12 02:01:43", :exception=\>"Cannot parse \\"2017-03-1… --- ## [Getting Flush downstream error](https://discuss.elastic.co/t/getting-flush-downstream-error/57906)
**Author:** [@kandito](https://discuss.elastic.co/u/kandito)\ **Replies:** 6\ **Last updated:** [March 15, 2017, 9:23am UTC](https://discuss.elastic.co/t/getting-flush-downstream-error/57906 "2017-03-15T09:23:44Z")
Hi, i have currently setup ELK stack with filebeat as log shipper. Everything works good. But sometimes, logstash not passing the logs to elasticsearch. I'm getting error as {:timestamp=\>"2016-08-12T08:58:54.7240… --- ## [Kafka (Key-Value) \> Logstash \> S3](https://discuss.elastic.co/t/kafka-key-value-logstash-s3/78664)
**Author:** [@mav3n](https://discuss.elastic.co/u/mav3n)\ **Replies:** 0\ **Last updated:** [March 15, 2017, 9:21am UTC](https://discuss.elastic.co/t/kafka-key-value-logstash-s3/78664 "2017-03-15T09:21:52Z")
Hi, I am new to Logstash. I have requirement to store streaming logs in S3 bucket based on the OrderId. One OrderId will have separate log file for itself. To reduce the load from server we thought of using Kafka and Log… --- ## [How map datatype dynamically for fields as per input logs in elasticsearch](https://discuss.elastic.co/t/how-map-datatype-dynamically-for-fields-as-per-input-logs-in-elasticsearch/78546)
**Author:** [@Teja](https://discuss.elastic.co/u/Teja)\ **Replies:** 1\ **Last updated:** [March 15, 2017, 8:37am UTC](https://discuss.elastic.co/t/how-map-datatype-dynamically-for-fields-as-per-input-logs-in-elasticsearch/78546 "2017-03-15T08:37:57Z")
Hi Folks, I might be asking asking silly question, Please help me on this. I have configured log-stash to listen on specific port and codec as "JSON" because my system will logs in JSON format. I am getting JSON a… --- ## [Json parse exception on \\r in logstash 5.2. for nested json on windows](https://discuss.elastic.co/t/json-parse-exception-on-r-in-logstash-5-2-for-nested-json-on-windows/78104)
**Author:** [@Divya\_Bhardwaj](https://discuss.elastic.co/u/Divya_Bhardwaj)\ **Replies:** 4\ **Last updated:** [March 15, 2017, 8:14am UTC](https://discuss.elastic.co/t/json-parse-exception-on-r-in-logstash-5-2-for-nested-json-on-windows/78104 "2017-03-15T08:14:22Z")
Hi All, I have a nested json as a file input as below: \[ { "status": "OK", "total":2, "doc\_value": { "state": "1", "value": "0.203222", "type": "new" }, "language": "english", "doc\_st… --- ## [Help to parse JSON key value object](https://discuss.elastic.co/t/help-to-parse-json-key-value-object/78319)
**Author:** [@Hunk\_Huang](https://discuss.elastic.co/u/Hunk_Huang)\ **Replies:** 1\ **Last updated:** [March 15, 2017, 7:09am UTC](https://discuss.elastic.co/t/help-to-parse-json-key-value-object/78319 "2017-03-15T07:09:35Z")
Hi Logstash Team. I'd like to load data struct as below. INPUT: { "vmdata":{ "vm1": "UP", "vm2": "Down", "vm3": "UP" ..... } … --- ## [Geopoint using Latitude and Longitude using Logstash](https://discuss.elastic.co/t/geopoint-using-latitude-and-longitude-using-logstash/43857)
**Author:** [@Meenal.Luktuke](https://discuss.elastic.co/u/Meenal.Luktuke)\ **Replies:** 2\ **Last updated:** [March 15, 2017, 7:05am UTC](https://discuss.elastic.co/t/geopoint-using-latitude-and-longitude-using-logstash/43857 "2017-03-15T07:05:33Z")
Hi, I am trying to plot data on Tile Map using geo-point coordinates from ES. Logstash data will have Latitude and Longitude. I also followed http://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/ but… --- ## [\[Solved\] Logstash pattern issue](https://discuss.elastic.co/t/solved-logstash-pattern-issue/77772)
**Author:** [@raamee](https://discuss.elastic.co/u/raamee)\ **Replies:** 3\ **Last updated:** [March 15, 2017, 6:51am UTC](https://discuss.elastic.co/t/solved-logstash-pattern-issue/77772 "2017-03-15T06:51:07Z")
Hi, I came across a weird issue. I made a grok pattern that works well and matches my log in http://grokdebug.herokuapp.com, but when I use the same pattern in my logstash, its not working properly. I get a field misma… --- ## [XML Confusion](https://discuss.elastic.co/t/xml-confusion/78558)
**Author:** [@Kvetch](https://discuss.elastic.co/u/Kvetch)\ **Replies:** 1\ **Last updated:** [March 15, 2017, 6:35am UTC](https://discuss.elastic.co/t/xml-confusion/78558 "2017-03-15T06:35:49Z")
Hello, I am trying to parse an xml file but am confused on the best way to tackle it. I see some folks using split to process and xml file and others using xpath. My xml file looks like the following: \ \ **Author:** [@orlevy08](https://discuss.elastic.co/u/orlevy08)\ **Replies:** 4\ **Last updated:** [March 15, 2017, 6:27am UTC](https://discuss.elastic.co/t/cant-remove-fields-in-logstash-5-2-2/78571 "2017-03-15T06:27:01Z") Hi all, I'm trying to remove the "remove" field and the default fields (host, @version, etc...) but can't seems to achieve this goal. Here is my conf file: input { file { path =\> \["E:/TestLog.txt"\] … --- ## [Conditional check on json response](https://discuss.elastic.co/t/conditional-check-on-json-response/77738)
**Author:** [@timmy8ken](https://discuss.elastic.co/u/timmy8ken)\ **Replies:** 2\ **Last updated:** [March 15, 2017, 6:18am UTC](https://discuss.elastic.co/t/conditional-check-on-json-response/77738 "2017-03-15T06:18:20Z")
Hello Elastic World, I have got logs from our web application shipping into Logstash then onto Elasticsearch. It's all working very well, however I would like to filter some entries in our logs to trigger an alert in Za… --- ## [How to run multiple logstash instances for s3 input](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624)
**Author:** [@HugoBang](https://discuss.elastic.co/u/HugoBang)\ **Replies:** 6\ **Last updated:** [March 15, 2017, 5:59am UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624 "2017-03-15T05:59:57Z")
Hello. I am trying to install multiple logstash instances for s3 input but it seems to be impossible because each logstash saves a sincedb file locally and even if the sincedb file is shared between the logstash instan… --- ## [Data Not Being Fed Into Kibana](https://discuss.elastic.co/t/data-not-being-fed-into-kibana/78509)
**Author:** [@marcjbaker](https://discuss.elastic.co/u/marcjbaker)\ **Replies:** 1\ **Last updated:** [March 14, 2017, 10:02pm UTC](https://discuss.elastic.co/t/data-not-being-fed-into-kibana/78509 "2017-03-14T22:02:18Z")
We are attempting to deploy the ELK stack for the first time but data is not being received by Kibana. The file path for alerts file is: /var/ossec/logs/alerts/alerts.json. A check of permissions indicates Logstash can … --- ## [Tail database table entries](https://discuss.elastic.co/t/tail-database-table-entries/78375)
**Author:** [@ggajanan](https://discuss.elastic.co/u/ggajanan)\ **Replies:** 2\ **Last updated:** [March 14, 2017, 9:25pm UTC](https://discuss.elastic.co/t/tail-database-table-entries/78375 "2017-03-14T21:25:02Z")
Hi, I am using JDBC input in the logstash. The query used in it selects data from a couple of tables. With the initial run, data is extracted using the query and stored in elasticsearch. But if new records are being in… --- ## [Persistent Queues: logstash could not start after forced shutdown](https://discuss.elastic.co/t/persistent-queues-logstash-could-not-start-after-forced-shutdown/78592)
**Author:** [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Replies:** 0\ **Last updated:** [March 14, 2017, 8:13pm UTC](https://discuss.elastic.co/t/persistent-queues-logstash-could-not-start-after-forced-shutdown/78592 "2017-03-14T20:13:56Z")
Hi All, the logstash process was stopped by ^C and could not be started until the queue files have been removed. The following errors were thrown: C:\\programms\\logstash\\bin\>logstash -f C:\\projects\\logstash.conf --pat… --- ## [Add json fields to webhdfs and ES output](https://discuss.elastic.co/t/add-json-fields-to-webhdfs-and-es-output/78409)
**Author:** [@dixitnikhil2004](https://discuss.elastic.co/u/dixitnikhil2004)\ **Replies:** 0\ **Last updated:** [March 13, 2017, 7:45pm UTC](https://discuss.elastic.co/t/add-json-fields-to-webhdfs-and-es-output/78409 "2017-03-13T19:45:40Z")
got it working.. removing the post --- ## [Address already in use - Logstash tried to bind to port range 9600..9700, but all the ports are already in use](https://discuss.elastic.co/t/address-already-in-use-logstash-tried-to-bind-to-port-range-9600-9700-but-all-the-ports-are-already-in-use/78310)
**Author:** [@tonyjames3212](https://discuss.elastic.co/u/tonyjames3212)\ **Replies:** 2\ **Last updated:** [March 14, 2017, 6:40pm UTC](https://discuss.elastic.co/t/address-already-in-use-logstash-tried-to-bind-to-port-range-9600-9700-but-all-the-ports-are-already-in-use/78310 "2017-03-14T18:40:12Z")
Version:5.2.0 OS:CentOS6.5 The OS restarts every day so I have to restart the logstash every day.The logstash was running normally, but it was suddenly unable to start today. It cannot work after I change the http.port… --- ## [Conditional filtering by source with IP addresses](https://discuss.elastic.co/t/conditional-filtering-by-source-with-ip-addresses/78351)
**Author:** [@rayl7786](https://discuss.elastic.co/u/rayl7786)\ **Replies:** 6\ **Last updated:** [March 14, 2017, 5:36pm UTC](https://discuss.elastic.co/t/conditional-filtering-by-source-with-ip-addresses/78351 "2017-03-14T17:36:11Z")
Hello, First, I would like to apologize if I do not add enough data but can add more if needed. I have created logstash filters to check if the source has the source IP in it. (ex. /var/log/XX.XX.XX.XX/\*.log) but I a… --- ## [How to get ES aggregation data as Logstash input?](https://discuss.elastic.co/t/how-to-get-es-aggregation-data-as-logstash-input/64776)
**Author:** [@KIVagant](https://discuss.elastic.co/u/KIVagant)\ **Replies:** 4\ **Last updated:** [March 14, 2017, 4:46pm UTC](https://discuss.elastic.co/t/how-to-get-es-aggregation-data-as-logstash-input/64776 "2017-03-14T16:46:16Z")
I have a Logstash configuration with the 'elasticsearch' input. input { elasticsearch { hosts =\> "localhost" index =\> "logstash-\*" type =\> 'aggregation\_metric' size =\> 0 query =\> ' { … --- ## [Transform a field](https://discuss.elastic.co/t/transform-a-field/78504)
**Author:** [@Mohamed](https://discuss.elastic.co/u/Mohamed)\ **Replies:** 1\ **Last updated:** [March 14, 2017, 2:30pm UTC](https://discuss.elastic.co/t/transform-a-field/78504 "2017-03-14T14:30:32Z")
Hello, I have a field that is created from xml file with xpath formula : "time": "2017-03-03T16:48:28.933+01:00" (\2017-03-03T16:48:28.933+01:00\<'/time\> in xml input). I want to create a nested field like… --- ## [Is it possible to parse the whole data to a filter?](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475)
**Author:** [@Vittorio](https://discuss.elastic.co/u/Vittorio)\ **Replies:** 13\ **Last updated:** [March 14, 2017, 2:24pm UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475 "2017-03-14T14:24:29Z")
Hi all, Is it possible to give the whole data I'm reading from rabbitmq and give it to a filter(for example as source for cipher filter so that it will encrypt all of it) ?? Thanks!!! --- ## [Cabin::Channel#error(message, data={}) - second argument you gave me wasa Array, I require a Hash](https://discuss.elastic.co/t/cabin-channel-error-message-data-second-argument-you-gave-me-wasa-array-i-require-a-hash/58306)
**Author:** [@da\_Teng](https://discuss.elastic.co/u/da_Teng)\ **Replies:** 12\ **Last updated:** [March 14, 2017, 2:18pm UTC](https://discuss.elastic.co/t/cabin-channel-error-message-data-second-argument-you-gave-me-wasa-array-i-require-a-hash/58306 "2017-03-14T14:18:37Z")
{:timestamp=\>"2016-08-18T02:38:07.483000+0000", :message=\>"Encountered an unexpected error submitting a bulk request! Will retry.", :error\_message=\>"Cabin::Channel#error(message, data={}) - second argument you gave me w… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1017) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1019)