# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=737 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 738 --- ## [How to run Logstash configuration file Background?](https://discuss.elastic.co/t/how-to-run-logstash-configuration-file-background/136679)
**Author:** [@kartheek91](https://discuss.elastic.co/u/kartheek91)\ **Replies:** 7\ **Last updated:** [June 21, 2018, 10:57am UTC](https://discuss.elastic.co/t/how-to-run-logstash-configuration-file-background/136679 "2018-06-21T10:57:57Z")
Hi All... I have conf file and I need to run for every 5 minutes automatically --- ## [Date filter to parse custom date](https://discuss.elastic.co/t/date-filter-to-parse-custom-date/136814)
**Author:** [@punit\_j](https://discuss.elastic.co/u/punit_j)\ **Replies:** 3\ **Last updated:** [June 21, 2018, 10:36am UTC](https://discuss.elastic.co/t/date-filter-to-parse-custom-date/136814 "2018-06-21T10:36:16Z")
Hi, I have a storage device which prints date as below: Date: Wed Oct 26 16:13:33 JST 2016 How do I parse this date using date filter ? --- ## [File - failed to open fichier1.log: Unhandled IOException: java.io.IOException: unhandled errno: Too many open files](https://discuss.elastic.co/t/file-failed-to-open-fichier1-log-unhandled-ioexception-java-io-ioexception-unhandled-errno-too-many-open-files/136825)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 0\ **Last updated:** [June 21, 2018, 10:26am UTC](https://discuss.elastic.co/t/file-failed-to-open-fichier1-log-unhandled-ioexception-java-io-ioexception-unhandled-errno-too-many-open-files/136825 "2018-06-21T10:26:43Z")
Hey, I use logstash to analyse files and drop those which not interest me. I currently listen : /data/mypath/\*.log But this path contains nearly 15.000 files and I need to analyse them. So that's why when I launch my… --- ## [Logstash suddenly doesn't process all logs](https://discuss.elastic.co/t/logstash-suddenly-doesnt-process-all-logs/136808)
**Author:** [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)\ **Replies:** 0\ **Last updated:** [June 21, 2018, 9:00am UTC](https://discuss.elastic.co/t/logstash-suddenly-doesnt-process-all-logs/136808 "2018-06-21T09:00:28Z")
Hello In my company I've set up our ELK stack and this worked the way it should but since yesterday it isn't able to process all logs in the morning. Yesterday I could fix this with a simple restart but today that doesn… --- ## [Logstash Dead Letter Messages back to RabbitMQ](https://discuss.elastic.co/t/logstash-dead-letter-messages-back-to-rabbitmq/136270)
**Author:** [@davidbugeja](https://discuss.elastic.co/u/davidbugeja)\ **Replies:** 11\ **Last updated:** [June 21, 2018, 7:20am UTC](https://discuss.elastic.co/t/logstash-dead-letter-messages-back-to-rabbitmq/136270 "2018-06-21T07:20:58Z")
Logstash and Elastic versions: 5.6.9 We're currently taking messages from RabbitMQ using logstash and outputting them to Elasticsearch. We noticed some scenarios of 400 responses and have enabled Logstash dead letter to… --- ## [Logstash Grok Filter patteren](https://discuss.elastic.co/t/logstash-grok-filter-patteren/136663)
**Author:** [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Replies:** 6\ **Last updated:** [June 21, 2018, 5:26am UTC](https://discuss.elastic.co/t/logstash-grok-filter-patteren/136663 "2018-06-21T05:26:38Z")
Hi All, I want to write Grok Filter for 100 Unique type of logs. Currently i have written for syslog and one log of our application. So i need to write 99 more grok filter. Can you please let me know how i can make it e… --- ## [Rabbitmq connection error](https://discuss.elastic.co/t/rabbitmq-connection-error/135052)
**Author:** [@JeMunos](https://discuss.elastic.co/u/JeMunos)\ **Replies:** 5\ **Last updated:** [June 21, 2018, 4:50am UTC](https://discuss.elastic.co/t/rabbitmq-connection-error/135052 "2018-06-21T04:50:18Z")
I'm trying to connect logstash to my AMP for Endpoints streaming event server hosted by Cisco. I had to do a little juggling to get the proper user/pw and the correct URL. But I have them now. I have tested them using a… --- ## [How to parse json values from http poller into event fields?](https://discuss.elastic.co/t/how-to-parse-json-values-from-http-poller-into-event-fields/136597)
**Author:** [@wailoont](https://discuss.elastic.co/u/wailoont)\ **Replies:** 2\ **Last updated:** [June 21, 2018, 4:30am UTC](https://discuss.elastic.co/t/how-to-parse-json-values-from-http-poller-into-event-fields/136597 "2018-06-21T04:30:00Z")
I am using logstash http poller plugin to call an API to retrieve information. Below is my logstash conf. input { http\_poller { urls =\> { test2 =\> { method =\> get url =\> "http://localhost:90… --- ## [JMX plugin guidance](https://discuss.elastic.co/t/jmx-plugin-guidance/136233)
**Author:** [@hburnswell](https://discuss.elastic.co/u/hburnswell)\ **Replies:** 8\ **Last updated:** [June 20, 2018, 11:20pm UTC](https://discuss.elastic.co/t/jmx-plugin-guidance/136233 "2018-06-20T23:20:32Z")
All, I'm looking for some guidance on using the JMX input plugin to gather multiple JMX streams. Ideally I was thinking to create unique indexes for each JMX stream by using: if "spefic tag" in \[tags\] {..} However,… --- ## [Jvm.security property overwrite in Logstash 5.5.0](https://discuss.elastic.co/t/jvm-security-property-overwrite-in-logstash-5-5-0/136745)
**Author:** [@Mrigank\_Mehta](https://discuss.elastic.co/u/Mrigank_Mehta)\ **Replies:** 0\ **Last updated:** [June 20, 2018, 5:49pm UTC](https://discuss.elastic.co/t/jvm-security-property-overwrite-in-logstash-5-5-0/136745 "2018-06-20T17:49:31Z")
Hi, How do we properly overwrite a property within a JDK's java.security file in logstash 5.5.0 Thanks --- ## [Extracting data from message](https://discuss.elastic.co/t/extracting-data-from-message/136602)
**Author:** [@srimaheswari\_chowdar](https://discuss.elastic.co/u/srimaheswari_chowdar)\ **Replies:** 1\ **Last updated:** [June 20, 2018, 5:15pm UTC](https://discuss.elastic.co/t/extracting-data-from-message/136602 "2018-06-20T17:15:00Z")
Hi, I used grok and geo IP filter to extract the data fields from the message but I am unable to do. So, I even used disscent filter as all my logs have the same pattern I found no luck. How can I achieve this task? Ple… --- ## [Logstash nil:NilClass error](https://discuss.elastic.co/t/logstash-nil-nilclass-error/136741)
**Author:** [@aayush](https://discuss.elastic.co/u/aayush)\ **Replies:** 0\ **Last updated:** [June 20, 2018, 5:05pm UTC](https://discuss.elastic.co/t/logstash-nil-nilclass-error/136741 "2018-06-20T17:05:15Z")
Hi, I've been using ELK for the past couple of weeks with success but recently logstash has been giving me some problems. I've looked at other posts addressing similar issues, but I'm still having trouble understanding … --- ## [Kafka input plugin - decorate\_events does not take effect](https://discuss.elastic.co/t/kafka-input-plugin-decorate-events-does-not-take-effect/136712)
**Author:** [@ArunANayagam](https://discuss.elastic.co/u/ArunANayagam)\ **Replies:** 11\ **Last updated:** [June 20, 2018, 3:24pm UTC](https://discuss.elastic.co/t/kafka-input-plugin-decorate-events-does-not-take-effect/136712 "2018-06-20T15:24:03Z")
Hi, I have a Kafka logstash input plugin that's reading messages from a Kafka 1.0.0 fine. But the "decorate\_events" property does not seem to take effect, i.e, I receive none of the kafka topic/partition information, h… --- ## [Date in logstash input](https://discuss.elastic.co/t/date-in-logstash-input/136699)
**Author:** [@kpover](https://discuss.elastic.co/u/kpover)\ **Replies:** 3\ **Last updated:** [June 20, 2018, 2:08pm UTC](https://discuss.elastic.co/t/date-in-logstash-input/136699 "2018-06-20T14:08:19Z")
Hi, apologies if this has been asked before. I have logstash pulling the full contents of an index from elastic on a daily basis. In elastic a new index is created every day that contains that days data, it's called ind… --- ## [I'm getting error while running logstash conf in ubuntu](https://discuss.elastic.co/t/im-getting-error-while-running-logstash-conf-in-ubuntu/136694)
**Author:** [@kartheek91](https://discuss.elastic.co/u/kartheek91)\ **Replies:** 1\ **Last updated:** [June 20, 2018, 1:55pm UTC](https://discuss.elastic.co/t/im-getting-error-while-running-logstash-conf-in-ubuntu/136694 "2018-06-20T13:55:01Z")
\[image\] This is the error I'm getting. Any body can please help. Thanks in advance Kartheek --- ## [Logstash is taking long time to load config and run?](https://discuss.elastic.co/t/logstash-is-taking-long-time-to-load-config-and-run/134736)
**Author:** [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\ **Replies:** 7\ **Last updated:** [June 20, 2018, 12:42pm UTC](https://discuss.elastic.co/t/logstash-is-taking-long-time-to-load-config-and-run/134736 "2018-06-20T12:42:29Z")
Hi, My Logstash is taking 15 minutes to start if I have increased the size of config file .Althougth i have increasedsize of jvm heap not getting why this is happening. --- ## [Logstash fails to run if data.path is a symlink](https://discuss.elastic.co/t/logstash-fails-to-run-if-data-path-is-a-symlink/134674)
**Author:** [@altmind](https://discuss.elastic.co/u/altmind)\ **Replies:** 1\ **Last updated:** [June 20, 2018, 12:16pm UTC](https://discuss.elastic.co/t/logstash-fails-to-run-if-data-path-is-a-symlink/134674 "2018-06-20T12:16:48Z")
I host /var/lib/logstash on a separate partition, this directory is a symlink elsewhere. Logstash fails to start with the following error message in logs: \[2018-06-05T13:12:24,295\]\[INFO \]\[logstash.modules.scaffold\] In… --- ## [Not enough free disk space available to allocate persisted queue](https://discuss.elastic.co/t/not-enough-free-disk-space-available-to-allocate-persisted-queue/136149)
**Author:** [@panaman](https://discuss.elastic.co/u/panaman)\ **Replies:** 7\ **Last updated:** [June 20, 2018, 12:10pm UTC](https://discuss.elastic.co/t/not-enough-free-disk-space-available-to-allocate-persisted-queue/136149 "2018-06-20T12:10:18Z")
I am testing out the new logstash 6.3.0 and all it does is error out and crash saying i don't have enough disk space if my queue.max\_bytes: is set to anything higher than about 9400mb. My data dir has 190GB free and I h… --- ## [How to save array of nested type using Logstash?](https://discuss.elastic.co/t/how-to-save-array-of-nested-type-using-logstash/136677)
**Author:** [@stevefai](https://discuss.elastic.co/u/stevefai)\ **Replies:** 0\ **Last updated:** [June 20, 2018, 11:35am UTC](https://discuss.elastic.co/t/how-to-save-array-of-nested-type-using-logstash/136677 "2018-06-20T11:35:06Z")
I store an array of stacktrace rows containing multiple fields, but can't seem to make the objects as nested type. The array looks like this and is returned by a custom ruby filter that I made: \[ { "name": "name1", … --- ## [Solved - Duplicate field data problem and timestamp question](https://discuss.elastic.co/t/solved-duplicate-field-data-problem-and-timestamp-question/136590)
**Author:** [@mikewitters](https://discuss.elastic.co/u/mikewitters)\ **Replies:** 2\ **Last updated:** [June 20, 2018, 11:06am UTC](https://discuss.elastic.co/t/solved-duplicate-field-data-problem-and-timestamp-question/136590 "2018-06-20T11:06:02Z")
I had an invalid date format so it was unable to parse the date given the format I was applying. Timestamp Question: Given the configs and log messages below, how would I go about making the timestamp field in my log m… --- ## [Problem regarding input from a file](https://discuss.elastic.co/t/problem-regarding-input-from-a-file/136619)
**Author:** [@JAYESH\_PAUNIKAR](https://discuss.elastic.co/u/JAYESH_PAUNIKAR)\ **Replies:** 5\ **Last updated:** [June 20, 2018, 9:57am UTC](https://discuss.elastic.co/t/problem-regarding-input-from-a-file/136619 "2018-06-20T09:57:52Z")
i have a file which has a single line of 53000 length and i am trying to take input from this file in logstash, but it is not happening. What may be the issue? --- ## [Logstash - Error](https://discuss.elastic.co/t/logstash-error/136495)
**Author:** [@Aby](https://discuss.elastic.co/u/Aby)\ **Replies:** 6\ **Last updated:** [June 20, 2018, 8:10am UTC](https://discuss.elastic.co/t/logstash-error/136495 "2018-06-20T08:10:24Z")
Hi I am running a logstash program to read the log file into the elastic db. Below is the sample file: \[2018-05-29 08:04:23,687\] DEBUG - http-outgoing-2845 HTTP/1.1 500 Internal Server Error {org.apache.synapse.trans… --- ## [Include conf files](https://discuss.elastic.co/t/include-conf-files/135480)
**Author:** [@saramali](https://discuss.elastic.co/u/saramali)\ **Replies:** 9\ **Last updated:** [June 20, 2018, 7:18am UTC](https://discuss.elastic.co/t/include-conf-files/135480 "2018-06-20T07:18:06Z")
filter { .... # do stuff A } B.conf : filter { .... # do stuff B } Can this be achieved somehow? run.conf : input { ... } include('B') include('C') output { ... } --- ## [Sql Server to Logstash](https://discuss.elastic.co/t/sql-server-to-logstash/136608)
**Author:** [@kartheek91](https://discuss.elastic.co/u/kartheek91)\ **Replies:** 15\ **Last updated:** [June 20, 2018, 6:43am UTC](https://discuss.elastic.co/t/sql-server-to-logstash/136608 "2018-06-20T06:43:56Z")
I'm trying to index records that are fetched from sql server and inserted into elasticsearch.Actually now I have 25k records and while I'm running logstash.conf file it is indexing upto 2000 records and logstash is termi… --- ## [Processing jdbc input formatted date field using date filter is giving extra slashes](https://discuss.elastic.co/t/processing-jdbc-input-formatted-date-field-using-date-filter-is-giving-extra-slashes/136607)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 0\ **Last updated:** [June 20, 2018, 5:03am UTC](https://discuss.elastic.co/t/processing-jdbc-input-formatted-date-field-using-date-filter-is-giving-extra-slashes/136607 "2018-06-20T05:03:18Z")
Hi, I am reading from oracle table using jdbc plugin and I have a date field, which I am querying as formatted string using the statement; SELECT field\_a,field\_b,field\_c,TO\_CHAR(date\_field,'YYYY-MM-DDTHH24:MI:SS.ff3Z')… --- ## [Logstash http output plugin with msgpack](https://discuss.elastic.co/t/logstash-http-output-plugin-with-msgpack/136259)
**Author:** [@kishoreck](https://discuss.elastic.co/u/kishoreck)\ **Replies:** 4\ **Last updated:** [June 20, 2018, 4:25am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-with-msgpack/136259 "2018-06-20T04:25:28Z")
Hi, We have been working with "json\_batch" feature that was enabled recently in logstash-output-http plugin. We are also exploring the usage of "msgpack" codec along with this plugin. I couldn't find any documentation … --- ## [Aggregate Index Data Daily to another Index](https://discuss.elastic.co/t/aggregate-index-data-daily-to-another-index/136035)
**Author:** [@cawoodm](https://discuss.elastic.co/u/cawoodm)\ **Replies:** 3\ **Last updated:** [June 19, 2018, 7:44pm UTC](https://discuss.elastic.co/t/aggregate-index-data-daily-to-another-index/136035 "2018-06-19T19:44:14Z")
We have tomcat access logs in one index called "access" (with fields like timeTaken and bytesSent) and we'd like to have this data aggregated into another index "stats" with, for example, the average timeTaken for certai… --- ## [Setting up IIS log collection with filebeat](https://discuss.elastic.co/t/setting-up-iis-log-collection-with-filebeat/136361)
**Author:** [@alexserd](https://discuss.elastic.co/u/alexserd)\ **Replies:** 6\ **Last updated:** [June 19, 2018, 4:16pm UTC](https://discuss.elastic.co/t/setting-up-iis-log-collection-with-filebeat/136361 "2018-06-19T16:16:44Z")
I'm trying to collect IIS logs with filebeat. Here is my filebeat yml file How do I configure logstash yml and do I need to create new index in kibana? I'm new to ELK stack any help is appreciated filebeat.prospectors… --- ## [Parse json message](https://discuss.elastic.co/t/parse-json-message/136549)
**Author:** [@Roque\_Moyano](https://discuss.elastic.co/u/Roque_Moyano)\ **Replies:** 1\ **Last updated:** [June 19, 2018, 6:45pm UTC](https://discuss.elastic.co/t/parse-json-message/136549 "2018-06-19T18:45:28Z")
Hi , I have a json message log like: { 'date': "xx" , 'env': "xx" } I want to create those fields in elasticsearch just if those fields are present and anything else , my filter is: filter { json { source =\>"message" … --- ## [Logstash not showing service method after installing with RPM methos](https://discuss.elastic.co/t/logstash-not-showing-service-method-after-installing-with-rpm-methos/136547)
**Author:** [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Replies:** 0\ **Last updated:** [June 19, 2018, 5:46pm UTC](https://discuss.elastic.co/t/logstash-not-showing-service-method-after-installing-with-rpm-methos/136547 "2018-06-19T17:46:18Z")
Hi Guys, I have installed logstash 6.2.4 with rpm way but it did not linked to /etc/init.d or neither works as as a service method in Centos 6.8.. However for logstash everything runs fine but while checking for config … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=736) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=738)