# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=396 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 397 --- ## [Need Help Groking](https://discuss.elastic.co/t/need-help-groking/221772)
**Author:** [@uklipse](https://discuss.elastic.co/u/uklipse)\ **Replies:** 2\ **Last updated:** [March 3, 2020, 9:11pm UTC](https://discuss.elastic.co/t/need-help-groking/221772 "2020-03-03T21:11:58Z")
I am attempting to write my first grok filter but I'm not getting any logs parsed. Here is my sample log. \<30\>2020:02:26-12:41:48 aua\[18909\]: id="3005" severity="warn" sys="System" sub="auth" name="Authentication faile… --- ## [How to apply json filter to every field that ends with "json"?](https://discuss.elastic.co/t/how-to-apply-json-filter-to-every-field-that-ends-with-json/221652)
**Author:** [@sahar](https://discuss.elastic.co/u/sahar)\ **Replies:** 2\ **Last updated:** [March 3, 2020, 8:16pm UTC](https://discuss.elastic.co/t/how-to-apply-json-filter-to-every-field-that-ends-with-json/221652 "2020-03-03T20:16:52Z")
Hello, I'm extracting fields from the message using KV filter, some of these fields might be holding a json value - I would like to iterate all fields and check if the field key ends with "json", to apply json filter on… --- ## [KV how to extract data between square brackets](https://discuss.elastic.co/t/kv-how-to-extract-data-between-square-brackets/221678)
**Author:** [@sahar](https://discuss.elastic.co/u/sahar)\ **Replies:** 7\ **Last updated:** [March 3, 2020, 8:14pm UTC](https://discuss.elastic.co/t/kv-how-to-extract-data-between-square-brackets/221678 "2020-03-03T20:14:29Z")
Hello, Assume I have the following message: "Some message \[key1=val1\] more text \[key2=some \[special\] value\], more text." From the above message I would like to extract key1 and key2 as follows: key1 = val1 key2 = so… --- ## [Date, change day and month position](https://discuss.elastic.co/t/date-change-day-and-month-position/221606)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 10\ **Last updated:** [March 3, 2020, 6:28pm UTC](https://discuss.elastic.co/t/date-change-day-and-month-position/221606 "2020-03-03T18:28:39Z")
I have a field called "timeanddate" and I use it as timestamp. date { match =\> \["timeanddate", "HH:mm:ss MM/dd/yyyy"\] target =\> "@timestamp" } But I want to change the order of the date and put the day f… --- ## [Existing mapping for \[user\] must be of type object but found \[text\]](https://discuss.elastic.co/t/existing-mapping-for-user-must-be-of-type-object-but-found-text/221929)
**Author:** [@boyhittscar](https://discuss.elastic.co/u/boyhittscar)\ **Replies:** 0\ **Last updated:** [March 3, 2020, 5:36pm UTC](https://discuss.elastic.co/t/existing-mapping-for-user-must-be-of-type-object-but-found-text/221929 "2020-03-03T17:36:57Z")
Hello! I'm getting the following error out of logstash: "reason"=\>"Could not dynamically add mapping for field \[user.name\]. Existing mapping for \[user\] must be of type object but found \[text\]."}}}} I've found another … --- ## [Invalid date 'February 29' as '1970' is not a leap year](https://discuss.elastic.co/t/invalid-date-february-29-as-1970-is-not-a-leap-year/221693)
**Author:** [@aviationfan](https://discuss.elastic.co/u/aviationfan)\ **Replies:** 5\ **Last updated:** [March 3, 2020, 3:03pm UTC](https://discuss.elastic.co/t/invalid-date-february-29-as-1970-is-not-a-leap-year/221693 "2020-03-03T15:03:04Z")
Posting here again as this has caused some real pain trying to figure out how to get this data into Elastic. Ultimately we will reconfigure this log file for the 1 day of the year to be a different date format that inclu… --- ## [Multiple pipelines - logstash cannot start](https://discuss.elastic.co/t/multiple-pipelines-logstash-cannot-start/221892)
**Author:** [@manosmustang](https://discuss.elastic.co/u/manosmustang)\ **Replies:** 4\ **Last updated:** [March 3, 2020, 1:56pm UTC](https://discuss.elastic.co/t/multiple-pipelines-logstash-cannot-start/221892 "2020-03-03T13:56:12Z")
Hello, I have logstash 7.6.0 and I am trying to have multiple pipelines the configuration is the below \[root@elastic1 logstash\]# vi pipelines.yml # This file is where you define your pipelines. You can def… --- ## [Unable to configure dead\_letter\_queue input for logstash](https://discuss.elastic.co/t/unable-to-configure-dead-letter-queue-input-for-logstash/221837)
**Author:** [@Darshan\_Parab](https://discuss.elastic.co/u/Darshan_Parab)\ **Replies:** 1\ **Last updated:** [March 3, 2020, 1:51pm UTC](https://discuss.elastic.co/t/unable-to-configure-dead-letter-queue-input-for-logstash/221837 "2020-03-03T13:51:39Z")
I have configure a path to dead\_letter\_queue in logstash.yml like this path.dead\_letter\_queue: "/var/log/logstash/dead\_letter\_queue" And below is the input configuration to read DLQ. input { dead\_letter\_queue { … --- ## [I have tried to ingest 8 records data from database..but after some times the rows gets replicated.it is duplicated like 16,24,32...perioodically..how to clear this?](https://discuss.elastic.co/t/i-have-tried-to-ingest-8-records-data-from-database-but-after-some-times-the-rows-gets-replicated-it-is-duplicated-like-16-24-32-perioodically-how-to-clear-this/221811)
**Author:** [@chelseia](https://discuss.elastic.co/u/chelseia)\ **Replies:** 1\ **Last updated:** [March 3, 2020, 1:49pm UTC](https://discuss.elastic.co/t/i-have-tried-to-ingest-8-records-data-from-database-but-after-some-times-the-rows-gets-replicated-it-is-duplicated-like-16-24-32-perioodically-how-to-clear-this/221811 "2020-03-03T13:49:55Z")
input{ jdbc{ clean\_run =\> true jdbc\_driver\_library =\> "C:/Program Files/SQL Developer 4/jdbc/lib/ojdbc8.jar" jdbc\_driver\_class =\> "Java::oracle.jdbc.driver.OracleDriver" jdbc\_connection\_string =\> "some connection st… --- ## [Unnable to start Logstash correctly](https://discuss.elastic.co/t/unnable-to-start-logstash-correctly/221424)
**Author:** [@Pablo95](https://discuss.elastic.co/u/Pablo95)\ **Replies:** 13\ **Last updated:** [March 3, 2020, 1:09pm UTC](https://discuss.elastic.co/t/unnable-to-start-logstash-correctly/221424 "2020-03-03T13:09:28Z")
I´m trying to connect Logstash with Filebeat in Centos 7 and this is the error that occurs: This is the command I´m using: bin/logstash -f logstash-sample.conf --config.reload.automatic OpenJDK 64-Bit Server VM warnin… --- ## [Getting error in logstash when we run in platform ibm aix 7.1?](https://discuss.elastic.co/t/getting-error-in-logstash-when-we-run-in-platform-ibm-aix-7-1/221885)
**Author:** [@talbehat](https://discuss.elastic.co/u/talbehat)\ **Replies:** 0\ **Last updated:** [March 3, 2020, 12:56pm UTC](https://discuss.elastic.co/t/getting-error-in-logstash-when-we-run-in-platform-ibm-aix-7-1/221885 "2020-03-03T12:56:07Z")
stty: tcgetattr: Not a typewriter io/console on JRuby shell out to study for most operations in logstash logstash 5.0 ibm aix 7.1 --- ## [How to create a Logstash service startup Log](https://discuss.elastic.co/t/how-to-create-a-logstash-service-startup-log/221858)
**Author:** [@ckough](https://discuss.elastic.co/u/ckough)\ **Replies:** 0\ **Last updated:** [March 3, 2020, 10:47am UTC](https://discuss.elastic.co/t/how-to-create-a-logstash-service-startup-log/221858 "2020-03-03T10:47:15Z")
Hi There, I can start logstash if I run a cmd like: sudo .../logstash -f /etc/logstash/conf.d/myconfig.conf --path.settings /etc/logstash/ --debug However I can't start up the logstash service to run the same config. … --- ## [Multiple pipelines syntax error](https://discuss.elastic.co/t/multiple-pipelines-syntax-error/221699)
**Author:** [@Devopsio](https://discuss.elastic.co/u/Devopsio)\ **Replies:** 4\ **Last updated:** [March 3, 2020, 9:44am UTC](https://discuss.elastic.co/t/multiple-pipelines-syntax-error/221699 "2020-03-03T09:44:32Z")
Hello. I'm trying to make multipipeline in Logstash (in a container) but it says there a syntax error. What's wrong? Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>… --- ## [Logstash Stomp-Plugin can't connect to ActiveMQ](https://discuss.elastic.co/t/logstash-stomp-plugin-cant-connect-to-activemq/221844)
**Author:** [@Roma\_Kap](https://discuss.elastic.co/u/Roma_Kap)\ **Replies:** 0\ **Last updated:** [March 3, 2020, 9:33am UTC](https://discuss.elastic.co/t/logstash-stomp-plugin-cant-connect-to-activemq/221844 "2020-03-03T09:33:40Z")
Hello, i'am new in Logstash and i try to connect from WindowsVM with stomp output-plugin to activeMQ with SSL. I've checked a connection to activeMQ like this: Test-NetConnection -ComputerName myActiveMq -Port 50614 … --- ## [How to Control the Ingestion of data using logstash to elasticsearch](https://discuss.elastic.co/t/how-to-control-the-ingestion-of-data-using-logstash-to-elasticsearch/221822)
**Author:** [@Ozil](https://discuss.elastic.co/u/Ozil)\ **Replies:** 0\ **Last updated:** [March 3, 2020, 7:27am UTC](https://discuss.elastic.co/t/how-to-control-the-ingestion-of-data-using-logstash-to-elasticsearch/221822 "2020-03-03T07:27:53Z")
Hi I am facing some issues due to the high rate of EPS on the data node. How can I control the ingestion of data (based on the EPS) using the Logstash output Any settings will help to reduse the indexing EPS ratio --- ## [Unable to run mulitiple logstash](https://discuss.elastic.co/t/unable-to-run-mulitiple-logstash/221821)
**Author:** [@kopilim96](https://discuss.elastic.co/u/kopilim96)\ **Replies:** 0\ **Last updated:** [March 3, 2020, 7:25am UTC](https://discuss.elastic.co/t/unable-to-run-mulitiple-logstash/221821 "2020-03-03T07:25:53Z")
I would like to work with multi logstash but I failed after gone through the steps and instruction from the internet. Firstly, I did not install Logstash somehow I unzip the Logstash with tar.gz format. So the question… --- ## [Sharepoint List to Logstash](https://discuss.elastic.co/t/sharepoint-list-to-logstash/219114)
**Author:** [@yummycake](https://discuss.elastic.co/u/yummycake)\ **Replies:** 2\ **Last updated:** [March 3, 2020, 6:37am UTC](https://discuss.elastic.co/t/sharepoint-list-to-logstash/219114 "2020-03-03T06:37:34Z")
We have stored our data in Sharepoint list. I was wondering if there is any way or any plugins that I can call our the data from the SharePoint list to Logstash and parsed the data to Elasticsearch so that we can use Kib… --- ## [I cant load my 15Gb csv data by logstash](https://discuss.elastic.co/t/i-cant-load-my-15gb-csv-data-by-logstash/221809)
**Author:** [@jake75](https://discuss.elastic.co/u/jake75)\ **Replies:** 1\ **Last updated:** [March 3, 2020, 6:12am UTC](https://discuss.elastic.co/t/i-cant-load-my-15gb-csv-data-by-logstash/221809 "2020-03-03T06:12:58Z")
im using docker based elk stack(sebp/elk). i trying to load 15GB csv datas to elasticsearch. first i tried migrate from mariadb. input { jdbc { jdbc\_driver\_library =\> "/var/lib/elasticsearch/mariadb-java-clien… --- ## [Send Parameter With Reset Plugin](https://discuss.elastic.co/t/send-parameter-with-reset-plugin/221803)
**Author:** [@shakesperoo](https://discuss.elastic.co/u/shakesperoo)\ **Replies:** 1\ **Last updated:** [March 3, 2020, 4:01am UTC](https://discuss.elastic.co/t/send-parameter-with-reset-plugin/221803 "2020-03-03T04:01:25Z")
I am using reset with twitter API and i need to pass conditonal paramter like tweet\_id" =\>"tweet" =\>(truncated) ? :"%{extend\_full\_text}%" :"%{text}" below my code . rest { request =\> { cacert =\> 'E:/Momp\_Social\_Me… --- ## [Nothing in filter block is being parsed. Why is only the filter not working in logstash?](https://discuss.elastic.co/t/nothing-in-filter-block-is-being-parsed-why-is-only-the-filter-not-working-in-logstash/221796)
**Author:** [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Replies:** 5\ **Last updated:** [March 3, 2020, 3:41am UTC](https://discuss.elastic.co/t/nothing-in-filter-block-is-being-parsed-why-is-only-the-filter-not-working-in-logstash/221796 "2020-03-03T03:41:39Z")
I tried adding mutate and add\_filed along with grok filter but nothing is working. The logs are only parsed as default method. #listening on this port input { beats { port =\> 5044 } } filter { if\[fields\]\[tags… --- ## [Logstash not sending to Elasticsearch if it receives input from Filebeat](https://discuss.elastic.co/t/logstash-not-sending-to-elasticsearch-if-it-receives-input-from-filebeat/221735)
**Author:** [@zebulyon](https://discuss.elastic.co/u/zebulyon)\ **Replies:** 2\ **Last updated:** [March 3, 2020, 3:12am UTC](https://discuss.elastic.co/t/logstash-not-sending-to-elasticsearch-if-it-receives-input-from-filebeat/221735 "2020-03-03T03:12:12Z")
Hello, I am creating the Elastic stack using the official docker images. My problem is that logstash will not send any info to elasticsearch when I start in in a docker-compose file This is my docker-compose file: vers… --- ## [Logstash new codec plugin fails while using](https://discuss.elastic.co/t/logstash-new-codec-plugin-fails-while-using/221756)
**Author:** [@i-mohit-a](https://discuss.elastic.co/u/i-mohit-a)\ **Replies:** 1\ **Last updated:** [March 3, 2020, 12:59am UTC](https://discuss.elastic.co/t/logstash-new-codec-plugin-fails-while-using/221756 "2020-03-03T00:59:20Z")
While developing a new logstash codec plugin, the plugin gets installed successfully. But error occurs while using it. Steps to replicate: Install logstash 7.6 or pull the code from git and cd to the folder where log… --- ## [Ingesting to Logstash Beat through Istio Gateway Ingress](https://discuss.elastic.co/t/ingesting-to-logstash-beat-through-istio-gateway-ingress/221774)
**Author:** [@DougR](https://discuss.elastic.co/u/DougR)\ **Replies:** 0\ **Last updated:** [March 2, 2020, 9:22pm UTC](https://discuss.elastic.co/t/ingesting-to-logstash-beat-through-istio-gateway-ingress/221774 "2020-03-02T21:22:52Z")
I have exposed port 5044 through an Istio gateway/virtual service for Logstash beats ingestion. All other ports defined in this gateway/VS are accessible through Istio, but 5044 is not. However, the service it points to … --- ## [Aggregate filter in Logstash not behaving correctly](https://discuss.elastic.co/t/aggregate-filter-in-logstash-not-behaving-correctly/221764)
**Author:** [@avaishnav](https://discuss.elastic.co/u/avaishnav)\ **Replies:** 0\ **Last updated:** [March 2, 2020, 7:51pm UTC](https://discuss.elastic.co/t/aggregate-filter-in-logstash-not-behaving-correctly/221764 "2020-03-02T19:51:51Z")
I have a use case where application is writing data into log file. Filebeat sends log to Logstash, Logstash aggregates data and sends aggregated data to Elasticsearch to store it. Logstash drops the original data and sen… --- ## [Possible to change order of key fields with S3 Output plugin?](https://discuss.elastic.co/t/possible-to-change-order-of-key-fields-with-s3-output-plugin/221740)
**Author:** [@joshuarobinson](https://discuss.elastic.co/u/joshuarobinson)\ **Replies:** 0\ **Last updated:** [March 2, 2020, 4:52pm UTC](https://discuss.elastic.co/t/possible-to-change-order-of-key-fields-with-s3-output-plugin/221740 "2020-03-02T16:52:33Z")
Hi, According to https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html, the objects created have a key with uuid before the date field. This makes listing objects by date range far less efficient, for… --- ## [Comparing number in conditionnal event treatement](https://discuss.elastic.co/t/comparing-number-in-conditionnal-event-treatement/217043)
**Author:** [@toms130](https://discuss.elastic.co/u/toms130)\ **Replies:** 5\ **Last updated:** [March 2, 2020, 3:43pm UTC](https://discuss.elastic.co/t/comparing-number-in-conditionnal-event-treatement/217043 "2020-03-02T15:43:03Z")
Hi there, I have a problem with conditionnals events, when I use some of equalities (\< ; \> ; \<= ; \>= ) on a number field, I got a java error and logstash stop Tried this : input { beats { port =\> 5045 } } … --- ## [Ilm rollover](https://discuss.elastic.co/t/ilm-rollover/221713)
**Author:** [@juuuhuuu](https://discuss.elastic.co/u/juuuhuuu)\ **Replies:** 0\ **Last updated:** [March 2, 2020, 3:28pm UTC](https://discuss.elastic.co/t/ilm-rollover/221713 "2020-03-02T15:28:09Z")
Hello :slight\_smile: Could you please help me to understand better rollover function ? I set up police in Kibana like hot 7 days, warm .. that should be 27 days and after delete Logstash output output { … --- ## [Logstash 7.6.0 regression](https://discuss.elastic.co/t/logstash-7-6-0-regression/221593)
**Author:** [@toog](https://discuss.elastic.co/u/toog)\ **Replies:** 3\ **Last updated:** [March 2, 2020, 1:59pm UTC](https://discuss.elastic.co/t/logstash-7-6-0-regression/221593 "2020-03-02T13:59:00Z")
With the same filter, logstash 7.6 have not the same result than previous version of logstash 6.5.4. I have include on the filter part some ruby counter and for one log in the 6.5.4 version all the counters have the sam… --- ## [Grok filter on logstash](https://discuss.elastic.co/t/grok-filter-on-logstash/221410)
**Author:** [@carmine.fabrizio](https://discuss.elastic.co/u/carmine.fabrizio)\ **Replies:** 4\ **Last updated:** [March 2, 2020, 1:05pm UTC](https://discuss.elastic.co/t/grok-filter-on-logstash/221410 "2020-03-02T13:05:39Z")
Hi there, I am trying to create some filter on logstash based on fields name example filebeat.yml filebeat.inputs: type: log paths: /var/log/httpd/\*log fields: app\_id: apache\_httpd fields\_under\_root: true enab… --- ## [Jdbc connection to mariadb connection issue](https://discuss.elastic.co/t/jdbc-connection-to-mariadb-connection-issue/221404)
**Author:** [@annk](https://discuss.elastic.co/u/annk)\ **Replies:** 2\ **Last updated:** [March 2, 2020, 11:57am UTC](https://discuss.elastic.co/t/jdbc-connection-to-mariadb-connection-issue/221404 "2020-03-02T11:57:13Z")
Hi, I am using jdbc\_streaming filter plugin to connect to mariadb and pull data . I am getting the below error in logstash logs \[WARN \]\[logstash.filters.jdbcstreaming\] Exception when executing JDBC query {:exception=\>#\<… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=395) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=397)