# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=208 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 209 --- ## [Current date jdbc plugin](https://discuss.elastic.co/t/current-date-jdbc-plugin/279154)
**Author:** [@Jakub\_Kaczmarek](https://discuss.elastic.co/u/Jakub_Kaczmarek)\ **Replies:** 9\ **Last updated:** [July 27, 2021, 4:12pm UTC](https://discuss.elastic.co/t/current-date-jdbc-plugin/279154 "2021-07-27T16:12:38Z")
Hi, is it possible to filter query inside jdbc input by CURRENT\_DATE? I have the following config: input { jdbc { statement =\> "SELECT userId, timestamp FROM users;" schedule =\> "\* \* \* \* \*" tags … --- ## [Getting this error while ingesting data from Mongodb into Elasticsearch](https://discuss.elastic.co/t/getting-this-error-while-ingesting-data-from-mongodb-into-elasticsearch/279651)
**Author:** [@Husnain](https://discuss.elastic.co/u/Husnain)\ **Replies:** 4\ **Last updated:** [July 27, 2021, 2:01pm UTC](https://discuss.elastic.co/t/getting-this-error-while-ingesting-data-from-mongodb-into-elasticsearch/279651 "2021-07-27T14:01:26Z")
Hii.I am trying to ingest data from Mongodb into Elasticsearch using logstash pipeline.I am using "logstash-input-mongodb" plugin to ingest data from Mongodb into Elasticsearch.This is my configuration pipeline. input {… --- ## [Logstash can not connect Elasticsearch URL](https://discuss.elastic.co/t/logstash-can-not-connect-elasticsearch-url/279257)
**Author:** [@Aboli\_77](https://discuss.elastic.co/u/Aboli_77)\ **Replies:** 2\ **Last updated:** [July 27, 2021, 11:48am UTC](https://discuss.elastic.co/t/logstash-can-not-connect-elasticsearch-url/279257 "2021-07-27T11:48:09Z")
Hello Team, When I run the following command: ./logstash -f logstash-conf-name (from /opt/logstash/bin directory) I get the error as follows: \[2020-09-23T22:08:16,324\]\[INFO \]\[logstash.runner \] Starting Logst… --- ## [Intermittent fault](https://discuss.elastic.co/t/intermittent-fault/279710)
**Author:** [@JesusDavid](https://discuss.elastic.co/u/JesusDavid)\ **Replies:** 0\ **Last updated:** [July 27, 2021, 9:21am UTC](https://discuss.elastic.co/t/intermittent-fault/279710 "2021-07-27T09:21:59Z")
We have the following error intermittently. It occurs between the communication between a SpringBoot service and the elastic. Caused by: java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPE… --- ## [Split string by newline to Logstash](https://discuss.elastic.co/t/split-string-by-newline-to-logstash/279620)
**Author:** [@esther90](https://discuss.elastic.co/u/esther90)\ **Replies:** 2\ **Last updated:** [July 27, 2021, 1:23am UTC](https://discuss.elastic.co/t/split-string-by-newline-to-logstash/279620 "2021-07-27T01:23:27Z")
\[2021-07-26 12:09:19.877 +08\] \[pgAdmin 4 - DB:testdatabase\] \[postgres\]-\[testdatabase\]-\[::1(57223)\] \[\] \[00000\]LOG: disconnection: session time: 0:00:00.218 user=postgres database=testdatabase host=::1 port=57223 \[2021-07… --- ## [Matching yyyy-MM-dd HH:mm:ss.SSSZ in a logstash filter](https://discuss.elastic.co/t/matching-yyyy-mm-dd-hhss-sssz-in-a-logstash-filter/279643)
**Author:** [@mshwf](https://discuss.elastic.co/u/mshwf)\ **Replies:** 1\ **Last updated:** [July 26, 2021, 3:10pm UTC](https://discuss.elastic.co/t/matching-yyyy-mm-dd-hhss-sssz-in-a-logstash-filter/279643 "2021-07-26T15:10:26Z")
I'm trying to extract some fields from this log entry: 2021-07-26 16:45:59.4640|0|WARN|LoggerTestASP.Controllers.WeatherForecastController|NEW\_FROM\_NLOG: 67b9de16-47a1-4308-a163-263f5f06e841, Hola Pola, 11/26/2017 16:45… --- ## [Kafka Input - avro https schema\_registry\_url: unable to find valid certification path to requested target](https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438)
**Author:** [@Malec](https://discuss.elastic.co/u/Malec)\ **Replies:** 5\ **Last updated:** [July 26, 2021, 3:09pm UTC](https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438 "2021-07-26T15:09:48Z")
Hi friends, Logstash version is 7.11.1 I am having trouble consuming messages from a kafka that uses confluent avro. I can't seem to connect to an https schema registry. Here is my config : input { kafka { … --- ## [Multi if condition on filter aggregation](https://discuss.elastic.co/t/multi-if-condition-on-filter-aggregation/279495)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 10\ **Last updated:** [July 26, 2021, 1:56pm UTC](https://discuss.elastic.co/t/multi-if-condition-on-filter-aggregation/279495 "2021-07-26T13:56:03Z")
Hi guys. I hope somebody can help to understand this problem I am having while trying to aggregate a field based on multi if condition. I have a field named message in which I have multiple events coming to. for exampl… --- ## [Rufus-scheduler error: TZInfo::PeriodNotFound](https://discuss.elastic.co/t/rufus-scheduler-error-tzinfo-periodnotfound/278811)
**Author:** [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\ **Replies:** 3\ **Last updated:** [July 26, 2021, 12:12pm UTC](https://discuss.elastic.co/t/rufus-scheduler-error-tzinfo-periodnotfound/278811 "2021-07-26T12:12:58Z")
Hi all, I recently posted a question about my logstash pipeline at some point (seemingly) randomly not updating the jdbc metadata for sql\_last\_value substitution. For completeness, I'll include the input plugin of my co… --- ## [Pipelines limits](https://discuss.elastic.co/t/pipelines-limits/279563)
**Author:** [@Sunflower](https://discuss.elastic.co/u/Sunflower)\ **Replies:** 2\ **Last updated:** [July 26, 2021, 8:20am UTC](https://discuss.elastic.co/t/pipelines-limits/279563 "2021-07-26T08:20:21Z")
Hi, I'm using logstash for multiple data sources(around 10-12). I'm using different configuration files for each data source, each configuration file has a dedicated pipeline in the pipelines.yml file. I wanted to und… --- ## [Logstash filter json warning](https://discuss.elastic.co/t/logstash-filter-json-warning/279352)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 6\ **Last updated:** [July 25, 2021, 3:16pm UTC](https://discuss.elastic.co/t/logstash-filter-json-warning/279352 "2021-07-25T15:16:56Z")
Using ELK 7.13.2 The logstash is consuming logs from kafka in json format earlier it was parsing the data but now it is not working \[2021-07-22T12:18:44,401\]\[WARN \]\[logstash.filters.json \]\[ise\]\[d51b308934c5e59604230… --- ## [Logstash mixing up log entries despite multip. pipelines](https://discuss.elastic.co/t/logstash-mixing-up-log-entries-despite-multip-pipelines/279557)
**Author:** [@pdanjou](https://discuss.elastic.co/u/pdanjou)\ **Replies:** 1\ **Last updated:** [July 25, 2021, 12:56pm UTC](https://discuss.elastic.co/t/logstash-mixing-up-log-entries-despite-multip-pipelines/279557 "2021-07-25T12:56:39Z")
So I have the following. 2 conf files with different ports, machine A sends to port1 and machine B sends to port2. I have pipelines.yml: pipeline.id: nginx path.config: "/etc/logstash/conf.d/logstash-nginx-es.conf" p… --- ## [How to remove %3A and such hash in text while parsing?](https://discuss.elastic.co/t/how-to-remove-3a-and-such-hash-in-text-while-parsing/279421)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 5\ **Last updated:** [July 24, 2021, 5:54am UTC](https://discuss.elastic.co/t/how-to-remove-3a-and-such-hash-in-text-while-parsing/279421 "2021-07-24T05:54:44Z")
Hello, I have a field in my logs: "name+Of+the%3A%C2%AE+field" which sometimes can be like this: "%D9+%81+%D8+%B1+%D8+%A7+%D8" How can I remove extra expressions in logstash filters? The desired output for the first … --- ## [Logstash ConcurrentModificationException: null](https://discuss.elastic.co/t/logstash-concurrentmodificationexception-null/278759)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 10\ **Last updated:** [July 24, 2021, 5:53am UTC](https://discuss.elastic.co/t/logstash-concurrentmodificationexception-null/278759 "2021-07-24T05:53:22Z")
Hi, has anyone encountered this issue? Logstash instances keep restarting after a while with this error: \[2021-07-14T14:35:23,972\]\[FATAL\]\[logstash.runner \] An unexpected error occurred! {:error=\>java.lang.Illeg… --- ## [Show custom geoip fields](https://discuss.elastic.co/t/show-custom-geoip-fields/279266)
**Author:** [@Velon](https://discuss.elastic.co/u/Velon)\ **Replies:** 4\ **Last updated:** [July 24, 2021, 2:19am UTC](https://discuss.elastic.co/t/show-custom-geoip-fields/279266 "2021-07-24T02:19:48Z")
I have add custom fields AcmeCorp.DeptName and AcmeCorp.DeptName whith maxmind\_mmdb, but I can't find it in the last result with logstash geoip plugins, so I'm asking some help ~ Thanks you~ --- ## [How to set auto-generated \_id to json field(csv column) in logstash file](https://discuss.elastic.co/t/how-to-set-auto-generated-id-to-json-field-csv-column-in-logstash-file/279482)
**Author:** [@Ashutosh2808](https://discuss.elastic.co/u/Ashutosh2808)\ **Replies:** 3\ **Last updated:** [July 23, 2021, 10:19pm UTC](https://discuss.elastic.co/t/how-to-set-auto-generated-id-to-json-field-csv-column-in-logstash-file/279482 "2021-07-23T22:19:51Z")
I've csv columns as below in logstash conf file: filter{ csv { columns =\> \["ResolutionId","IssueId"\] separator =\> "," skip\_header =\> "true" } mu… --- ## [Logstash Grok - Extract field if field exists & deleting character](https://discuss.elastic.co/t/logstash-grok-extract-field-if-field-exists-deleting-character/279492)
**Author:** [@gringorion](https://discuss.elastic.co/u/gringorion)\ **Replies:** 2\ **Last updated:** [July 23, 2021, 7:13pm UTC](https://discuss.elastic.co/t/logstash-grok-extract-field-if-field-exists-deleting-character/279492 "2021-07-23T19:13:38Z")
Hello Guys, My case is simple, but can't find the perfect response. I need to extract the field: user@domain into 2 fields. This step is OK: %{WORD:user}%{NOTSPACE:domain}? gives: { "user": "user", "domain": "@… --- ## [How to load json data into another json data structure](https://discuss.elastic.co/t/how-to-load-json-data-into-another-json-data-structure/279512)
**Author:** [@venkataraman](https://discuss.elastic.co/u/venkataraman)\ **Replies:** 1\ **Last updated:** [July 23, 2021, 6:33pm UTC](https://discuss.elastic.co/t/how-to-load-json-data-into-another-json-data-structure/279512 "2021-07-23T18:33:00Z")
Hello Everyone, I just need a help in loading data in a nested json format. I have a output something like this in the below format { "product" : "IPS" "Product\_Country" : "EMEA" "Tracking\_ID": "2335454523" "Start… --- ## [Logstash aggregate field and increase count](https://discuss.elastic.co/t/logstash-aggregate-field-and-increase-count/279454)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 3\ **Last updated:** [July 23, 2021, 6:10pm UTC](https://discuss.elastic.co/t/logstash-aggregate-field-and-increase-count/279454 "2021-07-23T18:10:49Z")
Hello everyone. I am again here. I would like to start thanking every single one one of you for this amazing community. Those days I was testing elasticsearch and logstash and I am just falling in love with how many th… --- ## [Change Timezone for a particular field "calldate"](https://discuss.elastic.co/t/change-timezone-for-a-particular-field-calldate/279493)
**Author:** [@rmartinez.rv](https://discuss.elastic.co/u/rmartinez.rv)\ **Replies:** 3\ **Last updated:** [July 23, 2021, 5:58pm UTC](https://discuss.elastic.co/t/change-timezone-for-a-particular-field-calldate/279493 "2021-07-23T17:58:38Z")
Hi. I'm using logstash jdbc to pull data from a mysql database. So far so good. This is part of the configuration file: input { jdbc { jdbc\_driver\_library =\> "/usr/share/java/mysql-connector-j… --- ## [Logstash URL parameter as tags](https://discuss.elastic.co/t/logstash-url-parameter-as-tags/279285)
**Author:** [@juanmav](https://discuss.elastic.co/u/juanmav)\ **Replies:** 7\ **Last updated:** [July 23, 2021, 9:22am UTC](https://discuss.elastic.co/t/logstash-url-parameter-as-tags/279285 "2021-07-23T09:22:28Z")
Hi all, I have this working configuration that receives logs from heroku and process then using grok and add a couple of tags from enviroments vars. input { http { port =\> "${PORT}" tags =\> \["${TAG}… --- ## [LOGSTASH - GSUB - DOESN'T MATCH](https://discuss.elastic.co/t/logstash-gsub-doesnt-match/279227)
**Author:** [@Daniel\_Lopez](https://discuss.elastic.co/u/Daniel_Lopez)\ **Replies:** 13\ **Last updated:** [July 23, 2021, 7:18am UTC](https://discuss.elastic.co/t/logstash-gsub-doesnt-match/279227 "2021-07-23T07:18:27Z")
Hi I tried to change a value in a field for use it like id but i couldn't modify the field value with gsub The data i have in the field is like this: rId = ABC0\_L123456\_789012 and only data i want is 123456 from rId, t… --- ## [Parse springboot logs using logstash](https://discuss.elastic.co/t/parse-springboot-logs-using-logstash/279422)
**Author:** [@Nikparab](https://discuss.elastic.co/u/Nikparab)\ **Replies:** 8\ **Last updated:** [July 23, 2021, 5:44am UTC](https://discuss.elastic.co/t/parse-springboot-logs-using-logstash/279422 "2021-07-23T05:44:25Z")
Hi Team, I need to parse below logs using logstash. I used grok but it is giving parsingerror. Grok- filter { grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{JAVACLASS:logclass} … --- ## [Loosing logs with logstash-trasnfer protocol](https://discuss.elastic.co/t/loosing-logs-with-logstash-trasnfer-protocol/279439)
**Author:** [@Francisco\_Yanez](https://discuss.elastic.co/u/Francisco_Yanez)\ **Replies:** 0\ **Last updated:** [July 23, 2021, 3:09am UTC](https://discuss.elastic.co/t/loosing-logs-with-logstash-trasnfer-protocol/279439 "2021-07-23T03:09:34Z")
I recently realized that our replication between our clusters is broken. we loose logs every minute (not a lot but we are loosing data). The way we replicate from master to slave is with Logstash-transfer. I tried to imp… --- ## [How to disable JSON parser](https://discuss.elastic.co/t/how-to-disable-json-parser/279336)
**Author:** [@maxim.mokhov](https://discuss.elastic.co/u/maxim.mokhov)\ **Replies:** 2\ **Last updated:** [July 22, 2021, 11:22pm UTC](https://discuss.elastic.co/t/how-to-disable-json-parser/279336 "2021-07-22T23:22:37Z")
I have different logs from applications, and I want sent it to elasticsearch. In some cases in logs appear JSON. I want to send it in elasticsearch as plaintext. But in log logstash I see "JSON parse failure": \[2021-07… --- ## [How to use kafka output plugin with OAUTH authentication](https://discuss.elastic.co/t/how-to-use-kafka-output-plugin-with-oauth-authentication/279427)
**Author:** [@burkeg](https://discuss.elastic.co/u/burkeg)\ **Replies:** 0\ **Last updated:** [July 22, 2021, 9:26pm UTC](https://discuss.elastic.co/t/how-to-use-kafka-output-plugin-with-oauth-authentication/279427 "2021-07-22T21:26:44Z")
How do i configure custom login call back handler with kafka output plugin? I'm using oauth authentication to my broker and not sure how to configure. Any leads/suggestions will be helpful --- ## [Same config file for test and development](https://discuss.elastic.co/t/same-config-file-for-test-and-development/279263)
**Author:** [@Loulou](https://discuss.elastic.co/u/Loulou)\ **Replies:** 6\ **Last updated:** [July 22, 2021, 8:15pm UTC](https://discuss.elastic.co/t/same-config-file-for-test-and-development/279263 "2021-07-22T20:15:50Z")
Hi, What I am trying to do is having the same logstash.conf file for my test and deployment/shipping. However for testing I want the output to go to my local elastic search, and for deployment to a http address. I trie… --- ## [How can I pass field reference in ruby filter while constructing a big string](https://discuss.elastic.co/t/how-can-i-pass-field-reference-in-ruby-filter-while-constructing-a-big-string/279419)
**Author:** [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\ **Replies:** 1\ **Last updated:** [July 22, 2021, 7:16pm UTC](https://discuss.elastic.co/t/how-can-i-pass-field-reference-in-ruby-filter-while-constructing-a-big-string/279419 "2021-07-22T19:16:10Z")
I would want to construct a sentence having multiple dynamic values, for example ruby { code =\> ' my\_object = { "reason" =\> "I received event.get("baseValueUnitAmount") event.get("currency") having di… --- ## [Logstash mutate add field not showing the added field](https://discuss.elastic.co/t/logstash-mutate-add-field-not-showing-the-added-field/279385)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 3\ **Last updated:** [July 22, 2021, 3:55pm UTC](https://discuss.elastic.co/t/logstash-mutate-add-field-not-showing-the-added-field/279385 "2021-07-22T15:55:51Z")
Hello guys. I am testing some functionalities of logstash but I am a bit confused about the process of implementing a filter mutate. Just for testing purpose I have a kiwi syslog generating random logs. Those logs have… --- ## [Lumberjack vs TCP for encrypted and compressed communication between logstash instances](https://discuss.elastic.co/t/lumberjack-vs-tcp-for-encrypted-and-compressed-communication-between-logstash-instances/279400)
**Author:** [@Akshay\_Sarvankar](https://discuss.elastic.co/u/Akshay_Sarvankar)\ **Replies:** 0\ **Last updated:** [July 22, 2021, 2:54pm UTC](https://discuss.elastic.co/t/lumberjack-vs-tcp-for-encrypted-and-compressed-communication-between-logstash-instances/279400 "2021-07-22T14:54:06Z")
Hi All, We have implemented Logstash to Logstash communication using TCP output and input in source and destination logstash instances. This communication is encrypted with SSL certificate. As per elastic documention, … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=207) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=209)