# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=306 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 307 --- ## [Jdbc input plugin](https://discuss.elastic.co/t/jdbc-input-plugin/247757)
**Author:** [@Milan\_Suthar](https://discuss.elastic.co/u/Milan_Suthar)\ **Replies:** 0\ **Last updated:** [September 7, 2020, 10:21am UTC](https://discuss.elastic.co/t/jdbc-input-plugin/247757 "2020-09-07T10:21:57Z")
hello, i want mongodb as input to ES index as output i have used logstash jdbc input plugin but its not working it shows "no-suitable-driver-found-for-jdbcmongo " i have tried many examples but didt work for me , i'v… --- ## [Logstash JDBC Input not working, no error or output](https://discuss.elastic.co/t/logstash-jdbc-input-not-working-no-error-or-output/247154)
**Author:** [@netscape](https://discuss.elastic.co/u/netscape)\ **Replies:** 5\ **Last updated:** [September 7, 2020, 10:19am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-not-working-no-error-or-output/247154 "2020-09-07T10:19:57Z")
Hi, I have some problem with JDBC input plugin. Database connection seem to work (no error), but I don´t get any output. In the Logstash logfile I only get the SQL Statement, but no error or output. The logstash log: … --- ## [Logstash not updating last run metadata file](https://discuss.elastic.co/t/logstash-not-updating-last-run-metadata-file/247730)
**Author:** [@RiadhBH](https://discuss.elastic.co/u/RiadhBH)\ **Replies:** 0\ **Last updated:** [September 7, 2020, 8:42am UTC](https://discuss.elastic.co/t/logstash-not-updating-last-run-metadata-file/247730 "2020-09-07T08:42:55Z")
In my Logstash I want to download from a huge database table. To avoid overload, I tried to get data of each day separately (since 20 years) and copied it to elasticsearch. I used :sql\_last\_value as follow, but it does n… --- ## [Tags:\_grokparsefailure](https://discuss.elastic.co/t/tags-grokparsefailure/247754)
**Author:** [@Bashir\_Mihyar](https://discuss.elastic.co/u/Bashir_Mihyar)\ **Replies:** 0\ **Last updated:** [September 7, 2020, 10:07am UTC](https://discuss.elastic.co/t/tags-grokparsefailure/247754 "2020-09-07T10:07:30Z")
Hello, I am new to this stack, and i am looking for an answer for the below: When i am in kibana discover tab, i see records with tags: \_grokparsefailure, while i expect to see host: logs.myserver.me Details: my ng… --- ## [How to recieve response header in logstash 7.4?](https://discuss.elastic.co/t/how-to-recieve-response-header-in-logstash-7-4/247741)
**Author:** [@talbehat](https://discuss.elastic.co/u/talbehat)\ **Replies:** 0\ **Last updated:** [September 7, 2020, 9:25am UTC](https://discuss.elastic.co/t/how-to-recieve-response-header-in-logstash-7-4/247741 "2020-09-07T09:25:52Z")
i am sending key and value in request header at making connection from elasticsearch, but i am not getting respoanse header which was added in elasticsearch. how to get response header please let me know. --- ## [Logstash jdbc-input taking long time to load data from Oracle DataBase](https://discuss.elastic.co/t/logstash-jdbc-input-taking-long-time-to-load-data-from-oracle-database/246646)
**Author:** [@amitgupta](https://discuss.elastic.co/u/amitgupta)\ **Replies:** 21\ **Last updated:** [September 7, 2020, 8:55am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-taking-long-time-to-load-data-from-oracle-database/246646 "2020-09-07T08:55:37Z")
I am using ELK stack. I have to load Oracle data to ElasticSearch. There are around 5 million data to load at initial. Problem: The logstash is very very slow to load data, it loads around 50k in 30 mins. this speed is … --- ## [Minimum Privilege Required for Logstash Account](https://discuss.elastic.co/t/minimum-privilege-required-for-logstash-account/247731)
**Author:** [@hpicass0](https://discuss.elastic.co/u/hpicass0)\ **Replies:** 0\ **Last updated:** [September 7, 2020, 8:42am UTC](https://discuss.elastic.co/t/minimum-privilege-required-for-logstash-account/247731 "2020-09-07T08:42:58Z")
Hi, Given than I need to set the API key or username/password pair in Logstash config (and they might get exposed). I need to set the privilege to the absolute minimum for the relevant account I've went over the roles … --- ## [Logstash - not able to insert date in this format 'YYYY-MM-DD hh:mm:ss'](https://discuss.elastic.co/t/logstash-not-able-to-insert-date-in-this-format-yyyy-mm-dd-hhss/247405)
**Author:** [@SwatiD](https://discuss.elastic.co/u/SwatiD)\ **Replies:** 8\ **Last updated:** [September 7, 2020, 6:37am UTC](https://discuss.elastic.co/t/logstash-not-able-to-insert-date-in-this-format-yyyy-mm-dd-hhss/247405 "2020-09-07T06:37:10Z")
My date format is YYYY-MM-DD hh:mm:ss but logstash is changing this to something like 2020-09-03T05:03:00.000Z . I have use this filter in logstash config file filter { date { match =\> \[ "endDate", "YYYY-MM-DD hh:mm:… --- ## [Logstash - not pushing all data into elasticsearch](https://discuss.elastic.co/t/logstash-not-pushing-all-data-into-elasticsearch/247523)
**Author:** [@SwatiD](https://discuss.elastic.co/u/SwatiD)\ **Replies:** 2\ **Last updated:** [September 7, 2020, 4:39am UTC](https://discuss.elastic.co/t/logstash-not-pushing-all-data-into-elasticsearch/247523 "2020-09-07T04:39:48Z")
I have 7,00,000 records with 300 columns in my database,but when I am pushing it into elasticsearch through logstash,It is pushing records samething like 2,50,000 only. I have use below in logstash config file useCurs… --- ## [Gsub remove words less than 4 characters in length](https://discuss.elastic.co/t/gsub-remove-words-less-than-4-characters-in-length/247649)
**Author:** [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Replies:** 6\ **Last updated:** [September 6, 2020, 11:50pm UTC](https://discuss.elastic.co/t/gsub-remove-words-less-than-4-characters-in-length/247649 "2020-09-06T23:50:52Z")
I am attempting to use a gsub mutate to remove words that are less than 4 characters in length. mutate { gsub =\> \[ "\[tag\]\[cloud\]\[chat\]", "(\\S+){1,3}", "" \] } But it is removing everything. Any suggestions? --- ## [Pipelines.yml distributor pattern more than 1 pipeline same events](https://discuss.elastic.co/t/pipelines-yml-distributor-pattern-more-than-1-pipeline-same-events/247684)
**Author:** [@nino](https://discuss.elastic.co/u/nino)\ **Replies:** 2\ **Last updated:** [September 6, 2020, 9:35pm UTC](https://discuss.elastic.co/t/pipelines-yml-distributor-pattern-more-than-1-pipeline-same-events/247684 "2020-09-06T21:35:35Z")
Hello all, this is my pipelines.yml file and is working really fine - pipeline.id: distributor config.string: | input { beats { port =\> 5044 } } output { if "pc1" in \[tags\] { pipeline { sen… --- ## [Logstash Issue in finding the filter](https://discuss.elastic.co/t/logstash-issue-in-finding-the-filter/247683)
**Author:** [@tamilarasanbravo](https://discuss.elastic.co/u/tamilarasanbravo)\ **Replies:** 3\ **Last updated:** [September 6, 2020, 6:40pm UTC](https://discuss.elastic.co/t/logstash-issue-in-finding-the-filter/247683 "2020-09-06T18:40:38Z")
Hi Team, I am trying to get the logstash configured but I am unable to start the logstash with the below error. The file exists but still the logstash fails to open/read it. \[2020-09-06T12:50:11,299\]\[INFO \]\[org.reflec… --- ## [Logstash 7.9.1-1 installation errors - service not found](https://discuss.elastic.co/t/logstash-7-9-1-1-installation-errors-service-not-found/247644)
**Author:** [@finbarr996](https://discuss.elastic.co/u/finbarr996)\ **Replies:** 7\ **Last updated:** [September 6, 2020, 12:40pm UTC](https://discuss.elastic.co/t/logstash-7-9-1-1-installation-errors-service-not-found/247644 "2020-09-06T12:40:14Z")
Hi there, Just tried re-installing logstash having removed it on an ubuntu server and had the following errors: Get:1 https://artifacts.elastic.co/packages/7.x/apt stable/main amd64 logstash all 1:7.9.1-1 \[159 MB\] Fetc… --- ## [Using amazon\_es output plugin via proxy](https://discuss.elastic.co/t/using-amazon-es-output-plugin-via-proxy/247667)
**Author:** [@Abolurah](https://discuss.elastic.co/u/Abolurah)\ **Replies:** 0\ **Last updated:** [September 6, 2020, 10:26am UTC](https://discuss.elastic.co/t/using-amazon-es-output-plugin-via-proxy/247667 "2020-09-06T10:26:21Z")
Hi , This is the only knowledge sharing with the community, after long hours spent on issue solving of sending data via amazon\_es output plugin, only thing you need to do is to add your proxy like an example below, … --- ## [How to add first row first column value as a field in to other rows' documents of a csv file in logstash filters?](https://discuss.elastic.co/t/how-to-add-first-row-first-column-value-as-a-field-in-to-other-rows-documents-of-a-csv-file-in-logstash-filters/243806)
**Author:** [@DSMilestone](https://discuss.elastic.co/u/DSMilestone)\ **Replies:** 40\ **Last updated:** [September 6, 2020, 4:25am UTC](https://discuss.elastic.co/t/how-to-add-first-row-first-column-value-as-a-field-in-to-other-rows-documents-of-a-csv-file-in-logstash-filters/243806 "2020-09-06T04:25:00Z")
I was trying out logstash pipeline config file, in order to add the first row first column value to other documents as a new field. But I got no where! My csv file is, Suite/Test/Step Name,Browser,Description,Tag,Start… --- ## [Jdbc, for getting mssql to Logstash not working :-(](https://discuss.elastic.co/t/jdbc-for-getting-mssql-to-logstash-not-working/247627)
**Author:** [@Farid\_N](https://discuss.elastic.co/u/Farid_N)\ **Replies:** 4\ **Last updated:** [September 5, 2020, 12:45pm UTC](https://discuss.elastic.co/t/jdbc-for-getting-mssql-to-logstash-not-working/247627 "2020-09-05T12:45:07Z")
Hello Unfortunately, I can not able to input mssql data base to logstash... This is my configuration file: input { jdbc { jdbc\_driver\_library =\> "/usr/share/logstash/jdbc\_driver/sqljdbc\_8.4/enu/mssql-jdbc-8.4.1.jr… --- ## [Events Emitted more then Events Received](https://discuss.elastic.co/t/events-emitted-more-then-events-received/247479)
**Author:** [@Burga](https://discuss.elastic.co/u/Burga)\ **Replies:** 1\ **Last updated:** [September 5, 2020, 9:02am UTC](https://discuss.elastic.co/t/events-emitted-more-then-events-received/247479 "2020-09-05T09:02:15Z")
HI , suddenly I started to see backlog behavior and noticed that Events Emitted is 76M but Events Received are 61M , the average rates are different also , ## Events Emitted Rate (/s) --\> 1000 ## Events Received Rat… --- ## [How do I decide or evaluate the system/memory requirements for Logstash](https://discuss.elastic.co/t/how-do-i-decide-or-evaluate-the-system-memory-requirements-for-logstash/247607)
**Author:** [@Ashish1101](https://discuss.elastic.co/u/Ashish1101)\ **Replies:** 2\ **Last updated:** [September 4, 2020, 11:34pm UTC](https://discuss.elastic.co/t/how-do-i-decide-or-evaluate-the-system-memory-requirements-for-logstash/247607 "2020-09-04T23:34:32Z")
If I want to use logstash to collect logs in from server how to decide how much memory and disk space will be required for it? If you have any helpful links that are also welcome. --- ## [Is it necessary to use filebeat or logstash forwarder in ELK stack?](https://discuss.elastic.co/t/is-it-necessary-to-use-filebeat-or-logstash-forwarder-in-elk-stack/247606)
**Author:** [@Ashish1101](https://discuss.elastic.co/u/Ashish1101)\ **Replies:** 4\ **Last updated:** [September 4, 2020, 11:22pm UTC](https://discuss.elastic.co/t/is-it-necessary-to-use-filebeat-or-logstash-forwarder-in-elk-stack/247606 "2020-09-04T23:22:28Z")
Is it necessary to use filebeat in ELK stack for collecting logs like installing filebeat on all the servers from where we need to collect logs and installing Logstash, ES and Kibana in a single separate server. Or can … --- ## [Json Logstash](https://discuss.elastic.co/t/json-logstash/247602)
**Author:** [@HenriqueNogueira](https://discuss.elastic.co/u/HenriqueNogueira)\ **Replies:** 1\ **Last updated:** [September 4, 2020, 10:23pm UTC](https://discuss.elastic.co/t/json-logstash/247602 "2020-09-04T22:23:56Z")
Hi guys, I'm trying to add a json file to my ELK and I can't get it in any way, they could help me, I'm posting my confs. Thank you! input { file { path =\> "/tmp/teste/teste.json" start\_position =\> "beginning" sinc… --- ## [Parsing Nested JSON inside JSON getting NilClass](https://discuss.elastic.co/t/parsing-nested-json-inside-json-getting-nilclass/247593)
**Author:** [@mzubairsaleem](https://discuss.elastic.co/u/mzubairsaleem)\ **Replies:** 3\ **Last updated:** [September 4, 2020, 10:20pm UTC](https://discuss.elastic.co/t/parsing-nested-json-inside-json-getting-nilclass/247593 "2020-09-04T22:20:09Z")
\[ERROR\] 2020-09-04 21:58:24.155 \[\[main\]\>worker0\] ruby - Ruby exception occurred: undefined method 'each' for nil:NilClass \[ERROR\] 2020-09-04 21:58:24.155 \[\[main\]\>worker0\] ruby - Ruby exception occurred: undefined method… --- ## [Logstash 7.8.1 Beats Input Plugin does not reload TLS certificate on change](https://discuss.elastic.co/t/logstash-7-8-1-beats-input-plugin-does-not-reload-tls-certificate-on-change/247578)
**Author:** [@Sunil\_Chadha](https://discuss.elastic.co/u/Sunil_Chadha)\ **Replies:** 0\ **Last updated:** [September 4, 2020, 5:15pm UTC](https://discuss.elastic.co/t/logstash-7-8-1-beats-input-plugin-does-not-reload-tls-certificate-on-change/247578 "2020-09-04T17:15:38Z")
Hi Elastic Experts, I recently upgraded from Logstash 6.8.7 to Logstash 7.8.1, and notice the TLS Certificate on change is not reloaded by Beats Input. I need Logstash Beats Input to reload the renewed certificate, whe… --- ## [Logstash intermittently fails to start using jdbc-static filter](https://discuss.elastic.co/t/logstash-intermittently-fails-to-start-using-jdbc-static-filter/247575)
**Author:** [@Paal\_Aaserud](https://discuss.elastic.co/u/Paal_Aaserud)\ **Replies:** 0\ **Last updated:** [September 4, 2020, 4:08pm UTC](https://discuss.elastic.co/t/logstash-intermittently-fails-to-start-using-jdbc-static-filter/247575 "2020-09-04T16:08:11Z")
Environment: Red Hat Enterprise Linux release 8.2 (Ootpa) Logstash 7.9.0 I have configured logstash 7.9 with a multiple pipeline using the jdbc\_static filter to fill two local db's for enrichment. Some times starting … --- ## [Logstash can not start](https://discuss.elastic.co/t/logstash-can-not-start/247509)
**Author:** [@psanggabuana](https://discuss.elastic.co/u/psanggabuana)\ **Replies:** 1\ **Last updated:** [September 4, 2020, 3:12pm UTC](https://discuss.elastic.co/t/logstash-can-not-start/247509 "2020-09-04T15:12:38Z")
I have problem with my logstash. When I start the logstash service, I have this log: 2020-09-04 16:21:19,830 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender… --- ## [Object field starting or ending with a \[.\] makes object resolution ambiguous](https://discuss.elastic.co/t/object-field-starting-or-ending-with-a-makes-object-resolution-ambiguous/246733)
**Author:** [@mikewillis](https://discuss.elastic.co/u/mikewillis)\ **Replies:** 2\ **Last updated:** [September 4, 2020, 2:26pm UTC](https://discuss.elastic.co/t/object-field-starting-or-ending-with-a-makes-object-resolution-ambiguous/246733 "2020-09-04T14:26:46Z")
I'm trying to deal with an index that the Upgrade Assistant has said needs to be reindexed before an upgrade to Elasticsearch 7. The index was created with 5.6.14 and the cluster currently runs 6.8.9. Due to an issue wit… --- ## [Splitting multiple strings into subfields](https://discuss.elastic.co/t/splitting-multiple-strings-into-subfields/247549)
**Author:** [@ds2000](https://discuss.elastic.co/u/ds2000)\ **Replies:** 0\ **Last updated:** [September 4, 2020, 1:46pm UTC](https://discuss.elastic.co/t/splitting-multiple-strings-into-subfields/247549 "2020-09-04T13:46:23Z")
Hi all, I'm not sure if this is possible or not. I'm collecting a lot of data and collecting it into a single line in a logfile. For reference, this is the pre-log format moduleName1 ----- InstalledVersion ----- Ne… --- ## [Dynamic Timezone in Logstash csv](https://discuss.elastic.co/t/dynamic-timezone-in-logstash-csv/247502)
**Author:** [@Rakku](https://discuss.elastic.co/u/Rakku)\ **Replies:** 1\ **Last updated:** [September 4, 2020, 1:20pm UTC](https://discuss.elastic.co/t/dynamic-timezone-in-logstash-csv/247502 "2020-09-04T13:20:46Z")
Hi there, Im completely new to pretty much anything here and got introduced at my workplace to ELK with the words: "We use ELK here, we have this and this problem, fix it". So here I am, with little to none knowledge on… --- ## [Is it possible to pass Google Analytics data to Logstash?](https://discuss.elastic.co/t/is-it-possible-to-pass-google-analytics-data-to-logstash/246386)
**Author:** [@ckwm](https://discuss.elastic.co/u/ckwm)\ **Replies:** 1\ **Last updated:** [September 4, 2020, 1:13pm UTC](https://discuss.elastic.co/t/is-it-possible-to-pass-google-analytics-data-to-logstash/246386 "2020-09-04T13:13:22Z")
Hi, I'm new for ELK and I'm trying to pass the Google Analytics data to Logstash in order to visualize in Kibana. Is it possible to do it? Or is there a way to connect Google Analytics with Kibana directly? Hope someon… --- ## [Logstash not logging & failing after start of elasticsearch](https://discuss.elastic.co/t/logstash-not-logging-failing-after-start-of-elasticsearch/247501)
**Author:** [@apurvakhatri](https://discuss.elastic.co/u/apurvakhatri)\ **Replies:** 0\ **Last updated:** [September 4, 2020, 8:23am UTC](https://discuss.elastic.co/t/logstash-not-logging-failing-after-start-of-elasticsearch/247501 "2020-09-04T08:23:54Z")
Here is the process: sudo service logstash start sudo service logstash status logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset … --- ## [Duplicate records in Kibana](https://discuss.elastic.co/t/duplicate-records-in-kibana/245896)
**Author:** [@apurvakhatri](https://discuss.elastic.co/u/apurvakhatri)\ **Replies:** 3\ **Last updated:** [September 4, 2020, 8:08am UTC](https://discuss.elastic.co/t/duplicate-records-in-kibana/245896 "2020-09-04T08:08:26Z")
I noticed that there are multiple records for the same log. They have the same clientip, time & host. Issue: Duplicate records. (There are multiple copies of same record) Requirement: There should only be one record … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=305) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=307)