# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=287 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 288 --- ## [Cron job for CSV filter](https://discuss.elastic.co/t/cron-job-for-csv-filter/253744)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 3\ **Last updated:** [October 30, 2020, 2:41pm UTC](https://discuss.elastic.co/t/cron-job-for-csv-filter/253744 "2020-10-30T14:41:04Z")
Hi All, I am using logstash to upload data into Elastic, we have been using csv filter to upload the data, i need to run this csv file on a every 15 minute frequency. csv filter doesnot allow "schedule" , how do i set… --- ## [ES 6.5.3 : having two separate logstash configuration files (and separate indexes) is causing fields from one index to bleed into the other](https://discuss.elastic.co/t/es-6-5-3-having-two-separate-logstash-configuration-files-and-separate-indexes-is-causing-fields-from-one-index-to-bleed-into-the-other/253736)
**Author:** [@terry433iid](https://discuss.elastic.co/u/terry433iid)\ **Replies:** 3\ **Last updated:** [October 30, 2020, 1:23pm UTC](https://discuss.elastic.co/t/es-6-5-3-having-two-separate-logstash-configuration-files-and-separate-indexes-is-causing-fields-from-one-index-to-bleed-into-the-other/253736 "2020-10-30T13:23:41Z")
I have two logstash configuration files - when I run logstash it is creating a bizarre situation where one of the indexes from config1 has fields from the other index in config2 See below for steps :- Create an index … --- ## [Create ILM Policy from Logstash when does not exist](https://discuss.elastic.co/t/create-ilm-policy-from-logstash-when-does-not-exist/253798)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 0\ **Last updated:** [October 30, 2020, 10:47am UTC](https://discuss.elastic.co/t/create-ilm-policy-from-logstash-when-does-not-exist/253798 "2020-10-30T10:47:48Z")
Hi there, I was wondering if it is possible to create an ilm policy automatically in a logstash pipeline. My use case: I collect logs from an kubernetes cluster and create indices based on the applications for example: … --- ## [Filtering- Delete a field to save storage space](https://discuss.elastic.co/t/filtering-delete-a-field-to-save-storage-space/253593)
**Author:** [@ASRLO](https://discuss.elastic.co/u/ASRLO)\ **Replies:** 2\ **Last updated:** [October 30, 2020, 10:29am UTC](https://discuss.elastic.co/t/filtering-delete-a-field-to-save-storage-space/253593 "2020-10-30T10:29:36Z")
Hello , I am a beginner on the ELK solution, I would like to know if by deleting fields and their contents from my logs I can gain storage space. If so, how to do this task? This is what I tried to do: ' if \[event\]… --- ## [Logstash converts integer value in scientific notation to float, causing mapping error in Elasticsearch](https://discuss.elastic.co/t/logstash-converts-integer-value-in-scientific-notation-to-float-causing-mapping-error-in-elasticsearch/253396)
**Author:** [@GrahamHannington](https://discuss.elastic.co/u/GrahamHannington)\ **Replies:** 7\ **Last updated:** [October 30, 2020, 3:40am UTC](https://discuss.elastic.co/t/logstash-converts-integer-value-in-scientific-notation-to-float-causing-mapping-error-in-elasticsearch/253396 "2020-10-30T03:40:22Z")
Background Given the following input (these are snippets from the input JSON Lines): "MEMLIMIT Size":0 ... "MEMLIMIT Size":0.8590E+10 Logstash (I'm using 7.9.3) outputs: "MEMLIMIT Size" =\> 0 "MEMLIMIT Size" =\> 859000… --- ## [How to use logstash with docker?](https://discuss.elastic.co/t/how-to-use-logstash-with-docker/253638)
**Author:** [@mohhef](https://discuss.elastic.co/u/mohhef)\ **Replies:** 3\ **Last updated:** [October 29, 2020, 9:20pm UTC](https://discuss.elastic.co/t/how-to-use-logstash-with-docker/253638 "2020-10-29T21:20:49Z")
Hello, My enviroment: Host: windows 10 Containers: linux I am building a spring boot application and I am logging my logs to "C:/elk/spring-boot-elk.log" I want to have a docker setup such that logstash will be abl… --- ## [Date parse error on only one record](https://discuss.elastic.co/t/date-parse-error-on-only-one-record/253653)
**Author:** [@Evan\_Vujcec](https://discuss.elastic.co/u/Evan_Vujcec)\ **Replies:** 7\ **Last updated:** [October 29, 2020, 5:19pm UTC](https://discuss.elastic.co/t/date-parse-error-on-only-one-record/253653 "2020-10-29T17:19:25Z")
I'm trying to import data from a csv into an ES index using logstash. In the CSV I have seperate date and time columns that I'm combining and then parsing it with the date filter plugin so it can use it as the @timestamp.… --- ## [New event not inserted](https://discuss.elastic.co/t/new-event-not-inserted/253411)
**Author:** [@micobarac](https://discuss.elastic.co/u/micobarac)\ **Replies:** 7\ **Last updated:** [October 29, 2020, 3:24pm UTC](https://discuss.elastic.co/t/new-event-not-inserted/253411 "2020-10-29T15:24:49Z")
I have a problem with an event not getting inserted on timeout. This is my configuration: filter { if !\[messageid\] { drop {} } aggregate { task\_id =\> "%{messageid}" code =\> " map\['avamis\_status… --- ## [Readfile - buffer\_extract error: a delimiter can't be found](https://discuss.elastic.co/t/readfile-buffer-extract-error-a-delimiter-cant-be-found/253709)
**Author:** [@eddy](https://discuss.elastic.co/u/eddy)\ **Replies:** 0\ **Last updated:** [October 29, 2020, 2:18pm UTC](https://discuss.elastic.co/t/readfile-buffer-extract-error-a-delimiter-cant-be-found/253709 "2020-10-29T14:18:05Z")
Hi everyone, I'm currently dealing with an error in logstash. Sometimes when I launch Logstash , It doesn't process some data and this error appears : \[INFO \] 2020-10-25 13:48:52.328 \[\[main\]\ **Author:** [@dfoot](https://discuss.elastic.co/u/dfoot)\ **Replies:** 3\ **Last updated:** [October 29, 2020, 2:09pm UTC](https://discuss.elastic.co/t/logstash-poll-messages/253382 "2020-10-29T14:09:22Z") I am getting below logstash log message that stopping data flow for some reason I had to restart logstash and then re-occur. Please advise. \[org.apache.kafka.clients.consumer.internals.AbstractCoordinator\] \[Consumer cli… --- ## [Spliting and increasing timestamp](https://discuss.elastic.co/t/spliting-and-increasing-timestamp/253094)
**Author:** [@iEMH](https://discuss.elastic.co/u/iEMH)\ **Replies:** 2\ **Last updated:** [October 29, 2020, 1:02pm UTC](https://discuss.elastic.co/t/spliting-and-increasing-timestamp/253094 "2020-10-29T13:02:12Z")
Hello, I am trying to do the following: I have a json message with a consolidated array of values from a sensor (every x minutes). {"id":"C2","timestamp":"2020-10-20T17:07:57.260Z","gap\_in\_ms":20,"array\_of\_values":{"x… --- ## [XML multiline files](https://discuss.elastic.co/t/xml-multiline-files/253577)
**Author:** [@Matias\_Aguero\_Escoba](https://discuss.elastic.co/u/Matias_Aguero_Escoba)\ **Replies:** 6\ **Last updated:** [October 29, 2020, 12:43pm UTC](https://discuss.elastic.co/t/xml-multiline-files/253577 "2020-10-29T12:43:49Z")
Hi everyone! my name is Matias Aguero. and i need ingest a lot XML files in elasticsearch, but i can't do it. My XMLs files then this structure: \ \ **Author:** [@sjne](https://discuss.elastic.co/u/sjne)\ **Replies:** 5\ **Last updated:** [October 29, 2020, 12:35pm UTC](https://discuss.elastic.co/t/grok-pattern-to-match-filebeat-multiline-input-up-to-the-first-new-line-character/253575 "2020-10-29T12:35:57Z") Logstash 6.8.10 Filebeat 6.8.9 Filebeat config: filebeat.inputs: - type: log enabled: true paths: - '/path/to/file' tags: - 'app' fields: app\_id: some\_app multiline.pattern: '^\\\[\[0-9\]{4}\\-\[0-9\]{2}-\[… --- ## [java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit](https://discuss.elastic.co/t/java-lang-illegalstateexception-logstash-stopped-processing-because-of-an-error-systemexit-exit/253614)
**Author:** [@Michael25](https://discuss.elastic.co/u/Michael25)\ **Replies:** 2\ **Last updated:** [October 29, 2020, 12:20pm UTC](https://discuss.elastic.co/t/java-lang-illegalstateexception-logstash-stopped-processing-because-of-an-error-systemexit-exit/253614 "2020-10-29T12:20:17Z")
Hi there, I know this error may have been solved before but tried implementing the solutions that I've seen but nothing seems to be working for me. I've been stuck on this problem for 2 days now. I was wondering if you g… --- ## [Use an array as parameter](https://discuss.elastic.co/t/use-an-array-as-parameter/253686)
**Author:** [@tallavi](https://discuss.elastic.co/u/tallavi)\ **Replies:** 0\ **Last updated:** [October 29, 2020, 12:00pm UTC](https://discuss.elastic.co/t/use-an-array-as-parameter/253686 "2020-10-29T12:00:53Z")
Hi all, I'm using Datadog Metrics Plugin. I want to use tags. I have an array of strings in my event called "datadog.tags" which I want to pass to dd\_tags, which gets an array of strings: datadog\_metrics { api\_key =\>… --- ## [Shipping logs using websocket instead filebeat to logstash](https://discuss.elastic.co/t/shipping-logs-using-websocket-instead-filebeat-to-logstash/253677)
**Author:** [@sudhakar](https://discuss.elastic.co/u/sudhakar)\ **Replies:** 1\ **Last updated:** [October 29, 2020, 11:35am UTC](https://discuss.elastic.co/t/shipping-logs-using-websocket-instead-filebeat-to-logstash/253677 "2020-10-29T11:35:49Z")
Hi, I am trying to use websocket instead of filebeat to ship the logs from a log file to logstash. I can see that the configuration for the logstash config for input is something like below: input { websocket { … --- ## [Default Cisco ASA dashboard not working via logstash](https://discuss.elastic.co/t/default-cisco-asa-dashboard-not-working-via-logstash/253660)
**Author:** [@asaravanan](https://discuss.elastic.co/u/asaravanan)\ **Replies:** 0\ **Last updated:** [October 29, 2020, 9:12am UTC](https://discuss.elastic.co/t/default-cisco-asa-dashboard-not-working-via-logstash/253660 "2020-10-29T09:12:14Z")
I am filebeat to send logs to ELK via logstash but I am not able to view default dashboard or timestamp logs. At the same time, if I send all logs from filebeat to Elastic directly i can see dashboard without any issues.… --- ## [Can't connect to Logstash port 5044 from Filebeat](https://discuss.elastic.co/t/cant-connect-to-logstash-port-5044-from-filebeat/253639)
**Author:** [@aloalo2242](https://discuss.elastic.co/u/aloalo2242)\ **Replies:** 0\ **Last updated:** [October 29, 2020, 3:15am UTC](https://discuss.elastic.co/t/cant-connect-to-logstash-port-5044-from-filebeat/253639 "2020-10-29T03:15:27Z")
Hi all, This is my config in /etc/logstash/conf.d/01-logstash.conf from ELK Server. input { beats { port =\> 5044 } } filter { grok { match =\> { "message" =\> \[ "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST… --- ## [Add field from ECS field (winlogbeats)](https://discuss.elastic.co/t/add-field-from-ecs-field-winlogbeats/253507)
**Author:** [@Camilo\_Diaz](https://discuss.elastic.co/u/Camilo_Diaz)\ **Replies:** 2\ **Last updated:** [October 28, 2020, 8:25pm UTC](https://discuss.elastic.co/t/add-field-from-ecs-field-winlogbeats/253507 "2020-10-28T20:25:45Z")
I am trying to add some fields to the logs coming from wingbeat 7.9 to Logstash. I need to extract the value from either user.name or related.user and add a new 'username' field. I was doing something like this but th… --- ## [No Implicit conversion of Nil](https://discuss.elastic.co/t/no-implicit-conversion-of-nil/253594)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 1\ **Last updated:** [October 28, 2020, 7:54pm UTC](https://discuss.elastic.co/t/no-implicit-conversion-of-nil/253594 "2020-10-28T19:54:16Z")
I am getting following error. but do not where in code it is getting this error from. \[ERROR\] 2020-10-28 11:41:31.825 \[\[main\]\>worker7\] WorkerLoop - Exception in pipelineworker, the pipeline stopped processing new events… --- ## [S3 logstash conf grok filter](https://discuss.elastic.co/t/s3-logstash-conf-grok-filter/253357)
**Author:** [@deep9300](https://discuss.elastic.co/u/deep9300)\ **Replies:** 5\ **Last updated:** [October 28, 2020, 6:43pm UTC](https://discuss.elastic.co/t/s3-logstash-conf-grok-filter/253357 "2020-10-28T18:43:03Z")
Hello. I want to collect s3 access logs from an s3 bucket and process them to logstash and elasticsearch. I have it working properly but the filter in the logstash conf is not working properly. Currently its giving me t… --- ## [Logstash not working with ActiveMQ](https://discuss.elastic.co/t/logstash-not-working-with-activemq/252022)
**Author:** [@kaushik.vankayala](https://discuss.elastic.co/u/kaushik.vankayala)\ **Replies:** 4\ **Last updated:** [October 28, 2020, 5:22pm UTC](https://discuss.elastic.co/t/logstash-not-working-with-activemq/252022 "2020-10-28T17:22:53Z")
I am trying to consume a message from an ActiveMQ with the following configuration; input { jms { broker\_url =\> 'failover:(ssl://b-d5bc797b-sjd8-41d2-dfjh7-127ba48c6238-1.mq.eu-west-1.amazonaws.com:61617)?init… --- ## [Logstash http output plugin documentation outdated](https://discuss.elastic.co/t/logstash-http-output-plugin-documentation-outdated/253595)
**Author:** [@bosand](https://discuss.elastic.co/u/bosand)\ **Replies:** 0\ **Last updated:** [October 28, 2020, 5:06pm UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-documentation-outdated/253595 "2020-10-28T17:06:12Z")
I did not see a feedback option on the website other than a contact form for sales purposes, so I thought I might as well post it here... The documentation on Elastic's website is not in sync with the Github version. D… --- ## [Which (json or xml) input is better for parsing in logstash in terms of scalibility , performance and nested structure?](https://discuss.elastic.co/t/which-json-or-xml-input-is-better-for-parsing-in-logstash-in-terms-of-scalibility-performance-and-nested-structure/253555)
**Author:** [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\ **Replies:** 2\ **Last updated:** [October 28, 2020, 2:21pm UTC](https://discuss.elastic.co/t/which-json-or-xml-input-is-better-for-parsing-in-logstash-in-terms-of-scalibility-performance-and-nested-structure/253555 "2020-10-28T14:21:46Z")
hi everyone , i need to decide which input type to parse in logstash for performance, scalibility and nested structure ? the input file could be nearly 100k records in size the data source is informatica. please let … --- ## [Process multiple Dead Letter Queues with one pipeline](https://discuss.elastic.co/t/process-multiple-dead-letter-queues-with-one-pipeline/253430)
**Author:** [@michielw](https://discuss.elastic.co/u/michielw)\ **Replies:** 2\ **Last updated:** [October 28, 2020, 2:11pm UTC](https://discuss.elastic.co/t/process-multiple-dead-letter-queues-with-one-pipeline/253430 "2020-10-28T14:11:38Z")
In our Elastic environment we use multiple pipelines on a single Logstash instance. Also, we enabled the Dead Letter Queue on all these pipelines. On creating the pipeline to process events in the DLQs I noticed I need t… --- ## [Passing only loglevel ERROR FROM LOG USING GROK FILTRATION](https://discuss.elastic.co/t/passing-only-loglevel-error-from-log-using-grok-filtration/253369)
**Author:** [@noobman2logstash](https://discuss.elastic.co/u/noobman2logstash)\ **Replies:** 6\ **Last updated:** [October 28, 2020, 12:58pm UTC](https://discuss.elastic.co/t/passing-only-loglevel-error-from-log-using-grok-filtration/253369 "2020-10-28T12:58:03Z")
Hi I'm new to logstash and i need help parsing only error logs when loglevel == error i have tried many filtrations on here and im not getting the result i need. here is a sample of my error log. i just want to pick an… --- ## [Logstash read pdf by ruby-Filter](https://discuss.elastic.co/t/logstash-read-pdf-by-ruby-filter/253542)
**Author:** [@xiaoping1993](https://discuss.elastic.co/u/xiaoping1993)\ **Replies:** 2\ **Last updated:** [October 28, 2020, 12:24pm UTC](https://discuss.elastic.co/t/logstash-read-pdf-by-ruby-filter/253542 "2020-10-28T12:24:35Z")
I want use logstash to read the pdf、docx .... I think the ruby-Filter can solve the problem so I do some try: 1）create a function pdf\_to\_text(pdf\_filename) require 'docsplit' def pdf\_to\_text(pdf\_filename) Docsplit… --- ## [JDBC Oracle table to Elastic through logstash](https://discuss.elastic.co/t/jdbc-oracle-table-to-elastic-through-logstash/253338)
**Author:** [@Gopal\_Chauhan](https://discuss.elastic.co/u/Gopal_Chauhan)\ **Replies:** 2\ **Last updated:** [October 28, 2020, 12:24pm UTC](https://discuss.elastic.co/t/jdbc-oracle-table-to-elastic-through-logstash/253338 "2020-10-28T12:24:30Z")
Hi , I am trying to get oracle table data into Elastic through logstash (version 6.2, windows). I have created a pipe line. Added pipeline id into logstash.yml file and restarting logstash service. Pipeline is starting… --- ## [ILM Logstash and Elastic (again)](https://discuss.elastic.co/t/ilm-logstash-and-elastic-again/253541)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 1\ **Last updated:** [October 28, 2020, 10:47am UTC](https://discuss.elastic.co/t/ilm-logstash-and-elastic-again/253541 "2020-10-28T10:47:39Z")
Hi, I have following configurations: logstash output: output { elasticsearch { hosts =\> \["elastic.server.com"\] ilm\_rollover\_alias =\> "filebeat-vm-linux" ilm\_pattern =\> "{now/d}-000001" ilm\_policy =\> "test\_… --- ## [Possible memory leak in Logstash 6.8.10](https://discuss.elastic.co/t/possible-memory-leak-in-logstash-6-8-10/253526)
**Author:** [@shreyas29](https://discuss.elastic.co/u/shreyas29)\ **Replies:** 0\ **Last updated:** [October 28, 2020, 8:04am UTC](https://discuss.elastic.co/t/possible-memory-leak-in-logstash-6-8-10/253526 "2020-10-28T08:04:20Z")
Version: Logstash 6.8.10 Operating System: Ubuntu 16.04 Config File (if you have sensitive info, please remove it): Not possible to share Sample Data: NA Steps to Reproduce: NA We are using Logstash in one of our micro… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=286) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=288)