# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=376 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 377 --- ## [Does logstash queue data until crashed elasticsearch recovers?](https://discuss.elastic.co/t/does-logstash-queue-data-until-crashed-elasticsearch-recovers/227518)
**Author:** [@isharamadhawa](https://discuss.elastic.co/u/isharamadhawa)\ **Replies:** 0\ **Last updated:** [April 10, 2020, 2:24pm UTC](https://discuss.elastic.co/t/does-logstash-queue-data-until-crashed-elasticsearch-recovers/227518 "2020-04-10T14:24:45Z")
I'm performing duplicate data removing process using fingerprint filter in logstash. I have an elastic cloud setup. Logstash reads data from my 3 node elastic cloud cluster and writes to the same cluster without any dupl… --- ## [Logstash 7.6.1 Exception: Sequel::DatabaseConnectionError](https://discuss.elastic.co/t/logstash-7-6-1-exception-sequel-databaseconnectionerror/227516)
**Author:** [@NitinLogiciel](https://discuss.elastic.co/u/NitinLogiciel)\ **Replies:** 0\ **Last updated:** [April 10, 2020, 2:15pm UTC](https://discuss.elastic.co/t/logstash-7-6-1-exception-sequel-databaseconnectionerror/227516 "2020-04-10T14:15:23Z")
I am using docker-elk official image to setup. My logstah conf file is input { jdbc { jdbc\_driver\_library =\> "/opt/logstash/logstash-core/lib/jars/mysql-connector-java-5.1.48-bin.jar" jdbc\_driver\_class =\> "com.mysq… --- ## [Logstash date filter timezone not working](https://discuss.elastic.co/t/logstash-date-filter-timezone-not-working/227451)
**Author:** [@srinika](https://discuss.elastic.co/u/srinika)\ **Replies:** 1\ **Last updated:** [April 10, 2020, 1:48pm UTC](https://discuss.elastic.co/t/logstash-date-filter-timezone-not-working/227451 "2020-04-10T13:48:14Z")
Hi I'm working on date filter and have used timezone. I get the output but the timezone is not working. Can someone please help me? Here's the config file input { stdin { } } filter { mutate { strip =\> "message" }… --- ## [Logstash parsing from filebeat with ECS](https://discuss.elastic.co/t/logstash-parsing-from-filebeat-with-ecs/227494)
**Author:** [@Rodrigo\_Jimenez](https://discuss.elastic.co/u/Rodrigo_Jimenez)\ **Replies:** 0\ **Last updated:** [April 10, 2020, 11:53am UTC](https://discuss.elastic.co/t/logstash-parsing-from-filebeat-with-ecs/227494 "2020-04-10T11:53:47Z")
Hi!, I saw some questions around talking about "moving forward" towards Logstash ECS compliance. So, I have now a setup with filebeat shipping logs using the nginx and system modules to an Logstash instance. I found th… --- ## [LogStash and FileBeats Architecture For the Log Monitoring Use Case](https://discuss.elastic.co/t/logstash-and-filebeats-architecture-for-the-log-monitoring-use-case/227487)
**Author:** [@ambut](https://discuss.elastic.co/u/ambut)\ **Replies:** 0\ **Last updated:** [April 10, 2020, 11:23am UTC](https://discuss.elastic.co/t/logstash-and-filebeats-architecture-for-the-log-monitoring-use-case/227487 "2020-04-10T11:23:41Z")
If I have 5 machines, those five machins needs to prcessed by logstassh. And each machines feeds needs to be sent to a different Kafka Topic. What all things are required: - File Beats instaltion at 5 machines ? - Sin… --- ## [JDBC Input Plugin: "date" input plugin can't apply known timezone to incoming DATETIMEs](https://discuss.elastic.co/t/jdbc-input-plugin-date-input-plugin-cant-apply-known-timezone-to-incoming-datetimes/227409)
**Author:** [@Jonathan\_Rice](https://discuss.elastic.co/u/Jonathan_Rice)\ **Replies:** 2\ **Last updated:** [April 9, 2020, 10:36pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-date-input-plugin-cant-apply-known-timezone-to-incoming-datetimes/227409 "2020-04-09T22:36:07Z")
Using Logstash 7.6.2 with the JDBC Input Plugin, and a MySQL database, I'm trying to import a DATETIME field, but apply a known timezone to that field on input, using the "date" filter. But "date" is always throwing an e… --- ## [Conditional Pipeline Logic](https://discuss.elastic.co/t/conditional-pipeline-logic/227390)
**Author:** [@InfiniteLoops](https://discuss.elastic.co/u/InfiniteLoops)\ **Replies:** 1\ **Last updated:** [April 9, 2020, 10:32pm UTC](https://discuss.elastic.co/t/conditional-pipeline-logic/227390 "2020-04-09T22:32:43Z")
Logstash with an input A sending to outputs B and C. If B or C fails everything becomes blocked. If the loss of data is acceptable is the below setup possible. So if a failure occurs in the first pipeline the system swit… --- ## [Delete and insert to elastic from logstash](https://discuss.elastic.co/t/delete-and-insert-to-elastic-from-logstash/227399)
**Author:** [@krishna\_bhargav](https://discuss.elastic.co/u/krishna_bhargav)\ **Replies:** 0\ **Last updated:** [April 9, 2020, 7:51pm UTC](https://discuss.elastic.co/t/delete-and-insert-to-elastic-from-logstash/227399 "2020-04-09T19:51:31Z")
We have intraday updates to our records, so we need to delete and insert all child records. logstash output : This is not deleting and insert , it's insert or update. output { elasticsearch{ hosts =\> \[ "xxxxxx" \] in… --- ## [JDBC Input Plugin: Behavior if job runs longer than schedule period? Just waits until next schedule time?](https://discuss.elastic.co/t/jdbc-input-plugin-behavior-if-job-runs-longer-than-schedule-period-just-waits-until-next-schedule-time/227382)
**Author:** [@Jonathan\_Rice](https://discuss.elastic.co/u/Jonathan_Rice)\ **Replies:** 0\ **Last updated:** [April 9, 2020, 5:59pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-behavior-if-job-runs-longer-than-schedule-period-just-waits-until-next-schedule-time/227382 "2020-04-09T17:59:34Z")
Say I give the JDBC Input Plugin a schedule to run every 5 minutes, but the Logstash job takes 6 minutes to run. Does the next job then run immediately, or does it wait until the next 5-minute wall-clock time is reached … --- ## [Mulitple pipeline with common output file](https://discuss.elastic.co/t/mulitple-pipeline-with-common-output-file/226910)
**Author:** [@priya\_yuvaraj](https://discuss.elastic.co/u/priya_yuvaraj)\ **Replies:** 1\ **Last updated:** [April 9, 2020, 5:32pm UTC](https://discuss.elastic.co/t/mulitple-pipeline-with-common-output-file/226910 "2020-04-09T17:32:57Z")
Hi all, I need a help in the following logstash implementation. I am new to logstash. I am having multiple pipeline configured in pipeline.yml. All the pipelines have to send data to the same output file ( commonfile\_d… --- ## [Duplicate log events with different \_id generated by logstash](https://discuss.elastic.co/t/duplicate-log-events-with-different-id-generated-by-logstash/227135)
**Author:** [@nityaraj06](https://discuss.elastic.co/u/nityaraj06)\ **Replies:** 1\ **Last updated:** [April 9, 2020, 5:29pm UTC](https://discuss.elastic.co/t/duplicate-log-events-with-different-id-generated-by-logstash/227135 "2020-04-09T17:29:34Z")
Hi, I am able to see duplicate log events with different \_id generated by logstash in Kibana. Below is my logstash config. Could someone help me with this? input { file { path =\> "E:/console-2020\*.log" sinced… --- ## [Problem when i run logstash as a service](https://discuss.elastic.co/t/problem-when-i-run-logstash-as-a-service/227312)
**Author:** [@mimimike](https://discuss.elastic.co/u/mimimike)\ **Replies:** 2\ **Last updated:** [April 9, 2020, 5:13pm UTC](https://discuss.elastic.co/t/problem-when-i-run-logstash-as-a-service/227312 "2020-04-09T17:13:20Z")
Hi everyone, I don't know why cannot run logstash as a service. If i run logstash in command line works well but when i run as a service appears the next error in status service: Apr 09 14:03:02 kafka02 logstash\[60398\]… --- ## [Grok create an array not a string](https://discuss.elastic.co/t/grok-create-an-array-not-a-string/227352)
**Author:** [@nico88](https://discuss.elastic.co/u/nico88)\ **Replies:** 2\ **Last updated:** [April 9, 2020, 4:14pm UTC](https://discuss.elastic.co/t/grok-create-an-array-not-a-string/227352 "2020-04-09T16:14:08Z")
Hello, I try to grok a tacacs authentification log file from tac plus. Ex of log line : 2020-04-09 16:13:26 +0200 10.2.0.163 test tty2 8.8.8.8 shell login succeeded This is my filter : filter { if \[type\] == … --- ## [Out of memory error with logstash 7.6.2](https://discuss.elastic.co/t/out-of-memory-error-with-logstash-7-6-2/227286)
**Author:** [@Sevy](https://discuss.elastic.co/u/Sevy)\ **Replies:** 1\ **Last updated:** [April 9, 2020, 3:31pm UTC](https://discuss.elastic.co/t/out-of-memory-error-with-logstash-7-6-2/227286 "2020-04-09T15:31:54Z")
Hi everyone, I have a Logstash 7.6.2 docker that stops running because of memory leak. After each pipeline execution, it looks like Logstash doesn't release memory. What should I do to identify the source of the proble… --- ## [Prevent the loss of existing fields while using document\_id of elasticsearch output plugin](https://discuss.elastic.co/t/prevent-the-loss-of-existing-fields-while-using-document-id-of-elasticsearch-output-plugin/227327)
**Author:** [@oula\_oula](https://discuss.elastic.co/u/oula_oula)\ **Replies:** 2\ **Last updated:** [April 9, 2020, 3:17pm UTC](https://discuss.elastic.co/t/prevent-the-loss-of-existing-fields-while-using-document-id-of-elasticsearch-output-plugin/227327 "2020-04-09T15:17:28Z")
Hi, I need to inject multiple logs in a common document according to a specific field value. So, I specify document\_id's value in elasticsearch plugin output. But instead of adding fields after parsing logs it deletes… --- ## [Dissect json fields in a specific field](https://discuss.elastic.co/t/dissect-json-fields-in-a-specific-field/227301)
**Author:** [@oula\_oula](https://discuss.elastic.co/u/oula_oula)\ **Replies:** 1\ **Last updated:** [April 9, 2020, 11:28am UTC](https://discuss.elastic.co/t/dissect-json-fields-in-a-specific-field/227301 "2020-04-09T11:28:59Z")
Hi, I would like to know if it is possible to use the json parser to add thes fields result in a chosen field instead of just in the "\_source" field. for example: message = "{'step': 1, 'completed': true}" filter{ … --- ## [Same Grok Filter for two somehow similar logs](https://discuss.elastic.co/t/same-grok-filter-for-two-somehow-similar-logs/227242)
**Author:** [@Ismail\_Zam\_Zam](https://discuss.elastic.co/u/Ismail_Zam_Zam)\ **Replies:** 2\ **Last updated:** [April 9, 2020, 10:51am UTC](https://discuss.elastic.co/t/same-grok-filter-for-two-somehow-similar-logs/227242 "2020-04-09T10:51:15Z")
type=EXECVE msg=audit(1585994474.430:1397): argc=5 a0="script" a1="-q" a2="-c" a3=6563686F2044415441 a4="file\_to\_write" type=EXECVE msg=audit(1585994474.430:1397): argc=2 a0="apt-get" a1="install" I want to parse these… --- ## [Should we be allowed to use http filter to get log data?](https://discuss.elastic.co/t/should-we-be-allowed-to-use-http-filter-to-get-log-data/227298)
**Author:** [@SHINRA](https://discuss.elastic.co/u/SHINRA)\ **Replies:** 0\ **Last updated:** [April 9, 2020, 10:35am UTC](https://discuss.elastic.co/t/should-we-be-allowed-to-use-http-filter-to-get-log-data/227298 "2020-04-09T10:35:37Z")
Problem: We need to dynamically change http request URL to fetch data. I figured that one cannot simply change URL dynamically; instead I found a few ways to go around it: exec input filter with curl and some scriptin… --- ## [How to extract only required content from a logfile and send it to filter parsing in logstash config file](https://discuss.elastic.co/t/how-to-extract-only-required-content-from-a-logfile-and-send-it-to-filter-parsing-in-logstash-config-file/226644)
**Author:** [@poddraj](https://discuss.elastic.co/u/poddraj)\ **Replies:** 1\ **Last updated:** [April 9, 2020, 10:39am UTC](https://discuss.elastic.co/t/how-to-extract-only-required-content-from-a-logfile-and-send-it-to-filter-parsing-in-logstash-config-file/226644 "2020-04-09T10:39:10Z")
Hi All, I am pretty new to Logstash and I have an unstructured C++ log file from which I want to send only selected content of log to kafka topic from logstash. Below is the sample log file, can someone help with the co… --- ## [Jdbc\_input\_plugin : extra count query with paging disabled](https://discuss.elastic.co/t/jdbc-input-plugin-extra-count-query-with-paging-disabled/227212)
**Author:** [@deslauriersp](https://discuss.elastic.co/u/deslauriersp)\ **Replies:** 2\ **Last updated:** [April 9, 2020, 3:19am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-extra-count-query-with-paging-disabled/227212 "2020-04-09T03:19:58Z")
I am trying to setup a JDBC input plugin. I have turned off paging : jdbc\_paging\_enabled =\> false jdbc\_page\_size =\> 0 Yet, in the "debug" logs I can see SELECT count(\*) "COUNT" FROM ( \[my\_query\] ) "T1" FETCH NEXT… --- ## [Logstash Mapping](https://discuss.elastic.co/t/logstash-mapping/227198)
**Author:** [@Antonio\_Antovski](https://discuss.elastic.co/u/Antonio_Antovski)\ **Replies:** 3\ **Last updated:** [April 8, 2020, 11:26pm UTC](https://discuss.elastic.co/t/logstash-mapping/227198 "2020-04-08T23:26:36Z")
Hello, I'm new to logstash, and while experimenting with it I faced some problems for which I need some help to solve them. I'm reading the data I want to put into ES, from MySQL db. I have created a template (with temp… --- ## [Logstash to logstash communication - beat.hostname](https://discuss.elastic.co/t/logstash-to-logstash-communication-beat-hostname/227168)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 5\ **Last updated:** [April 8, 2020, 10:44pm UTC](https://discuss.elastic.co/t/logstash-to-logstash-communication-beat-hostname/227168 "2020-04-08T22:44:36Z")
Hi, I'm actually have a configuration with Filebeat sending logs to logstash, then logstash parsing information and sending it to differente pipelines, one of the to elasticsearch. In the parsing i use the field beat.hos… --- ## [Logstash grok pattern question when sql statement has commas in it](https://discuss.elastic.co/t/logstash-grok-pattern-question-when-sql-statement-has-commas-in-it/227022)
**Author:** [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\ **Replies:** 4\ **Last updated:** [April 8, 2020, 8:55pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-question-when-sql-statement-has-commas-in-it/227022 "2020-04-08T20:55:09Z")
I am trying to get a grok pattern to match the following and i have it besides the fact that i had to leave a field out to get it done. this is postgresql csv pattern and i needed to splice out the duration from the que… --- ## [Filling value into placeholder in a field](https://discuss.elastic.co/t/filling-value-into-placeholder-in-a-field/226126)
**Author:** [@Saurabh\_Singh1](https://discuss.elastic.co/u/Saurabh_Singh1)\ **Replies:** 4\ **Last updated:** [April 8, 2020, 7:51pm UTC](https://discuss.elastic.co/t/filling-value-into-placeholder-in-a-field/226126 "2020-04-08T19:51:00Z")
Hi , I have a field named 'message'. This field contains data like, message : 'My IP is {%source\_ip}.This {%source.ip} is external ip.' I want to replace all the instance of this placeholder {%source\_ip} , with a v… --- ## [Capture Message](https://discuss.elastic.co/t/capture-message/227164)
**Author:** [@manikandanb87](https://discuss.elastic.co/u/manikandanb87)\ **Replies:** 2\ **Last updated:** [April 8, 2020, 5:25pm UTC](https://discuss.elastic.co/t/capture-message/227164 "2020-04-08T17:25:02Z")
Hi Team, I am testing my log stash output and found that it is stashing as below. But, I would like to know how can I match and capture only the entries in the log - "Sign on javacode failed for user USERA" Kindly hel… --- ## [Nmap plugin/module? still working](https://discuss.elastic.co/t/nmap-plugin-module-still-working/226479)
**Author:** [@stcdarrell](https://discuss.elastic.co/u/stcdarrell)\ **Replies:** 10\ **Last updated:** [April 8, 2020, 3:22pm UTC](https://discuss.elastic.co/t/nmap-plugin-module-still-working/226479 "2020-04-08T15:22:21Z")
anyone had any luck getting the nmap scan xml codec/plugin to work with v7? i've got it reading the data in with the plugin, but the template wont import.. and the data is pretty rough around the edges. any suggestions… --- ## [Error Installing the Plugin](https://discuss.elastic.co/t/error-installing-the-plugin/226989)
**Author:** [@hlogan](https://discuss.elastic.co/u/hlogan)\ **Replies:** 4\ **Last updated:** [April 8, 2020, 2:58pm UTC](https://discuss.elastic.co/t/error-installing-the-plugin/226989 "2020-04-08T14:58:52Z")
Hi , I am getting error while installing the plugin #bin/logstash-plugin install logstash-input-salesforce Validating logstash-input-salesforce Installing logstash-input-salesforce Plugin version conflict, aborting E… --- ## [Stuck at Successfully started Logstasg API endpoint](https://discuss.elastic.co/t/stuck-at-successfully-started-logstasg-api-endpoint/227155)
**Author:** [@Robert\_Ga](https://discuss.elastic.co/u/Robert_Ga)\ **Replies:** 0\ **Last updated:** [April 8, 2020, 2:38pm UTC](https://discuss.elastic.co/t/stuck-at-successfully-started-logstasg-api-endpoint/227155 "2020-04-08T14:38:51Z")
Hello everyone, I just installed Logstash 7.6 on ubuntu 18. My end goal is to add some json files to an elasticsearch instance. Unfortunately, when I run "logstash -f cfile.conf", I get stuck at phrase : Stuck at Succ… --- ## [Extracting field from Events in LOGSTASH](https://discuss.elastic.co/t/extracting-field-from-events-in-logstash/227102)
**Author:** [@nb03briceno](https://discuss.elastic.co/u/nb03briceno)\ **Replies:** 1\ **Last updated:** [April 8, 2020, 2:17pm UTC](https://discuss.elastic.co/t/extracting-field-from-events-in-logstash/227102 "2020-04-08T14:17:46Z")
Hello everyone, I come to the forum to ask if somebody could give a suggestion. I'm trying to take a field from a log event but I'm not having the results that I want. The field is for the words in bold; The log looks … --- ## [Exception on logstash when closing filebeat](https://discuss.elastic.co/t/exception-on-logstash-when-closing-filebeat/227142)
**Author:** [@billymerendina](https://discuss.elastic.co/u/billymerendina)\ **Replies:** 0\ **Last updated:** [April 8, 2020, 1:56pm UTC](https://discuss.elastic.co/t/exception-on-logstash-when-closing-filebeat/227142 "2020-04-08T13:56:52Z")
Hi everybody, I'm working with Elasticsearh, kibana, filebeat and logstash 7.6.2 on windows pc. Everythings seems to be right. But I see somethng strange when i kill (ctrl + c) the filebeat process (I don 't use it as… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=375) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=377)