# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=336 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 337 --- ## [Create Empty array when logstash outputs](https://discuss.elastic.co/t/create-empty-array-when-logstash-outputs/239005)
**Author:** [@recordable542](https://discuss.elastic.co/u/recordable542)\ **Replies:** 0\ **Last updated:** [June 28, 2020, 12:10pm UTC](https://discuss.elastic.co/t/create-empty-array-when-logstash-outputs/239005 "2020-06-28T12:10:49Z")
input { jdbc { jdbc\_driver\_library =\> "..." jdbc\_driver\_class =\> "com.mysql.jdbc.Driver" jdbc\_paging\_enabled =\> true tracking\_column =\> "unix\_ts\_in\_secs" use\_column\_value =\> tr… --- ## [Logstash pipline optimization](https://discuss.elastic.co/t/logstash-pipline-optimization/236280)
**Author:** [@abhisheksimion](https://discuss.elastic.co/u/abhisheksimion)\ **Replies:** 3\ **Last updated:** [June 28, 2020, 6:20am UTC](https://discuss.elastic.co/t/logstash-pipline-optimization/236280 "2020-06-28T06:20:37Z")
I have multiple configurations defined in the pipelines.yml. All the configurations are responsible for pushing data from a jDBC source to Elasticsearch. I want to know what is the optimal settings required to do so. As … --- ## [Logstash install error in ubuntu](https://discuss.elastic.co/t/logstash-install-error-in-ubuntu/238987)
**Author:** [@Srini2](https://discuss.elastic.co/u/Srini2)\ **Replies:** 1\ **Last updated:** [June 28, 2020, 5:43am UTC](https://discuss.elastic.co/t/logstash-install-error-in-ubuntu/238987 "2020-06-28T05:43:55Z")
Hi, I'm new to ELK. I have successfully installed elasticsearch and kibana in Ubuntu 19.10 instance with OpenJDK 11 installed. But getting below error while installing Logstash. Any help on this please. $ sudo apt-get … --- ## [Unable to connect mysql using logstash](https://discuss.elastic.co/t/unable-to-connect-mysql-using-logstash/238969)
**Author:** [@sandy1](https://discuss.elastic.co/u/sandy1)\ **Replies:** 1\ **Last updated:** [June 28, 2020, 3:19am UTC](https://discuss.elastic.co/t/unable-to-connect-mysql-using-logstash/238969 "2020-06-28T03:19:49Z")
logstash.config: \> input { \> jdbc { \> jdbc\_connection\_string =\> "jdbc:mysql://localhost:3306/db\_name?ssl=false" \> jdbc\_user =\> "admin" \> jdbc\_password =\> "password" \> jdbc\_driver\_library =\> "D:/el… --- ## [Logstash delta pull from postgres](https://discuss.elastic.co/t/logstash-delta-pull-from-postgres/238501)
**Author:** [@tenet\_testuser1](https://discuss.elastic.co/u/tenet_testuser1)\ **Replies:** 5\ **Last updated:** [June 27, 2020, 4:15am UTC](https://discuss.elastic.co/t/logstash-delta-pull-from-postgres/238501 "2020-06-27T04:15:16Z")
I have a simple logstash pipeline job that is a SQL job that pulls data from a postgres db into ES. The initial pull was 1.5 million rows, which ran last night and took about 5 minutes. but the delta daily pull is about… --- ## [Logstash.instrument.periodicpoller.os\] PeriodicPoller: Timeout exception](https://discuss.elastic.co/t/logstash-instrument-periodicpoller-os-periodicpoller-timeout-exception/238923)
**Author:** [@Yeading](https://discuss.elastic.co/u/Yeading)\ **Replies:** 1\ **Last updated:** [June 27, 2020, 2:19am UTC](https://discuss.elastic.co/t/logstash-instrument-periodicpoller-os-periodicpoller-timeout-exception/238923 "2020-06-27T02:19:14Z")
\[2020-06-27T07:45:55,338\]\[DEBUG\]\[logstash.instrument.periodicpoller.os\] PeriodicPoller: Timeout exception {:poller=\>#\ **Author:** [@LukeJB](https://discuss.elastic.co/u/LukeJB)\ **Replies:** 0\ **Last updated:** [June 26, 2020, 10:08pm UTC](https://discuss.elastic.co/t/help-configuring-logstash-on-microsoft-azure/238919 "2020-06-26T22:08:25Z") Hi, I'm trying to get logstash configured correctly on Azure. I'm a bit less that a week old with this. Recently, when I try running logstash, I get the following warnings and errors: WARNING: Could not find logstash.ym… --- ## [Reingest Winlogbeat Events through Logstash](https://discuss.elastic.co/t/reingest-winlogbeat-events-through-logstash/238914)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 0\ **Last updated:** [June 26, 2020, 8:57pm UTC](https://discuss.elastic.co/t/reingest-winlogbeat-events-through-logstash/238914 "2020-06-26T20:57:50Z")
Not really sure where to put this question since it involves a lot of different parts of the stack.... I am currently ingesting windows event logs from a server --\> logstash --\> elasticsearch. I have modified my logsta… --- ## [Grok pattern not working](https://discuss.elastic.co/t/grok-pattern-not-working/238875)
**Author:** [@ndg](https://discuss.elastic.co/u/ndg)\ **Replies:** 5\ **Last updated:** [June 26, 2020, 7:21pm UTC](https://discuss.elastic.co/t/grok-pattern-not-working/238875 "2020-06-26T19:21:27Z")
Hi! I want to parse a Haproxy log sent by a rsyslog service. I have a grok pattern to parse the original rsyslog message (which send the message of the service to a syslog\_message field) and another that parse the syslog… --- ## [Problem in parsing json string](https://discuss.elastic.co/t/problem-in-parsing-json-string/238901)
**Author:** [@azteker](https://discuss.elastic.co/u/azteker)\ **Replies:** 1\ **Last updated:** [June 26, 2020, 6:45pm UTC](https://discuss.elastic.co/t/problem-in-parsing-json-string/238901 "2020-06-26T18:45:44Z")
This is how I parse json in filter json { source =\> "json\_content" target =\> "content" remove\_field=\>\["json\_content"\] } But I got the following error "type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse… --- ## [Logstash xpack monitoring with certificate instead of truststore](https://discuss.elastic.co/t/logstash-xpack-monitoring-with-certificate-instead-of-truststore/238885)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 0\ **Last updated:** [June 26, 2020, 4:19pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-with-certificate-instead-of-truststore/238885 "2020-06-26T16:19:43Z")
Hi, I have a cluster of elasticsearch and i have configured ssl/tls and set xpack.security.http.ssl.client\_authentication: required The i have logstash with xpack monitoring enabled, and i have to specify a certific… --- ## [Multiple logstash configs with different filters running at the same time](https://discuss.elastic.co/t/multiple-logstash-configs-with-different-filters-running-at-the-same-time/238749)
**Author:** [@Archie\_Crawford](https://discuss.elastic.co/u/Archie_Crawford)\ **Replies:** 6\ **Last updated:** [June 26, 2020, 3:14pm UTC](https://discuss.elastic.co/t/multiple-logstash-configs-with-different-filters-running-at-the-same-time/238749 "2020-06-26T15:14:21Z")
Hello, I have been having this issue when I try to send server logs to logstash.. Java stack trace server logs sent to Logstash with their own input.conf filter.conf and output.conf and Cisco ASA firewall logs going t… --- ## [Is there a way to use http input where you can keep the contents in sync like jdbc plugin?](https://discuss.elastic.co/t/is-there-a-way-to-use-http-input-where-you-can-keep-the-contents-in-sync-like-jdbc-plugin/238846)
**Author:** [@genehunter29009](https://discuss.elastic.co/u/genehunter29009)\ **Replies:** 1\ **Last updated:** [June 26, 2020, 11:37am UTC](https://discuss.elastic.co/t/is-there-a-way-to-use-http-input-where-you-can-keep-the-contents-in-sync-like-jdbc-plugin/238846 "2020-06-26T11:37:21Z")
Is there a way to use the http input in logstash where you are constantly polling to get the latest records in the same way you can do that with the jdbc plugin and tracking column? --- ## [GROK filter for specific keyword and values](https://discuss.elastic.co/t/grok-filter-for-specific-keyword-and-values/238808)
**Author:** [@Akshara](https://discuss.elastic.co/u/Akshara)\ **Replies:** 9\ **Last updated:** [June 26, 2020, 11:11am UTC](https://discuss.elastic.co/t/grok-filter-for-specific-keyword-and-values/238808 "2020-06-26T11:11:47Z")
LOG MSG : "2020-05-13 09:30:15.405 \[localhost\] \[Services\] \[Request Id : 06118032-ba2f-406e-b6b3-89f0511924ba\] \[User Id : HUNTER@UK.B-AND-Q.COM\] INFO com.logging.aspect.AspectController:120 - \[1242,124202,\] Exit LookupLa… --- ## [Running multiple logstash.conf file in same machine](https://discuss.elastic.co/t/running-multiple-logstash-conf-file-in-same-machine/238819)
**Author:** [@TUSHAR\_Sinha](https://discuss.elastic.co/u/TUSHAR_Sinha)\ **Replies:** 0\ **Last updated:** [June 26, 2020, 9:13am UTC](https://discuss.elastic.co/t/running-multiple-logstash-conf-file-in-same-machine/238819 "2020-06-26T09:13:35Z")
Hi, community, I have log files of different software running and the log pattern is different. I want to create two different logstash.conf file to filter them both and index into one elasticsearch server. How can I do… --- ## [Grok Filter - alternative patterns for same field?](https://discuss.elastic.co/t/grok-filter-alternative-patterns-for-same-field/238418)
**Author:** [@Ben\_Bryzak](https://discuss.elastic.co/u/Ben_Bryzak)\ **Replies:** 2\ **Last updated:** [June 26, 2020, 7:34am UTC](https://discuss.elastic.co/t/grok-filter-alternative-patterns-for-same-field/238418 "2020-06-26T07:34:13Z")
Hi - I'm trying to create a grok pattern to match a field that contains one or more words and assign a field name to it. When the field contains multiple words, it is wrapped in quotes: eg. FieldValue or "Field Value… --- ## [How to extract specific keyword from Logs in logstash](https://discuss.elastic.co/t/how-to-extract-specific-keyword-from-logs-in-logstash/238802)
**Author:** [@Magesh](https://discuss.elastic.co/u/Magesh)\ **Replies:** 0\ **Last updated:** [June 26, 2020, 7:14am UTC](https://discuss.elastic.co/t/how-to-extract-specific-keyword-from-logs-in-logstash/238802 "2020-06-26T07:14:52Z")
Hi I would like to extract specific keyword from Logs and store it in another variable using logstash pls find below the logs 6/26 00:33:04 -INFOS- {1:2} Decision PASSED\_WITH\_RESV applied on 1 message(s) api.service#… --- ## [Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71](https://discuss.elastic.co/t/handling-exception-org-logstash-beats-invalidframeprotocolexception-invalid-version-of-beats-protocol-71/238604)
**Author:** [@Magesh](https://discuss.elastic.co/u/Magesh)\ **Replies:** 2\ **Last updated:** [June 26, 2020, 7:08am UTC](https://discuss.elastic.co/t/handling-exception-org-logstash-beats-invalidframeprotocolexception-invalid-version-of-beats-protocol-71/238604 "2020-06-26T07:08:27Z")
Hi, I'm Using Filebeat 8.0/Logstash8.0/Elasticsearch8.0 version in windows 10 environment and i'm trying to send the logs from filebeat--\>logstash--\>elasticsearch. post start of filebeat / logstash/elasticsearch, i'm g… --- ## [Deleting documents from ElasticSearch when rows are deleted from CSV file](https://discuss.elastic.co/t/deleting-documents-from-elasticsearch-when-rows-are-deleted-from-csv-file/238794)
**Author:** [@damian.bulak](https://discuss.elastic.co/u/damian.bulak)\ **Replies:** 0\ **Last updated:** [June 26, 2020, 6:30am UTC](https://discuss.elastic.co/t/deleting-documents-from-elasticsearch-when-rows-are-deleted-from-csv-file/238794 "2020-06-26T06:30:29Z")
I have a CSV file which contains data like this: 111;Some description 112;Different description I run Logstash as a Linux service and it loaded the data to ElasticSearch correctly. When I add new lines to the CSV file… --- ## [Logstash http poller- custom header : error](https://discuss.elastic.co/t/logstash-http-poller-custom-header-error/237314)
**Author:** [@katara](https://discuss.elastic.co/u/katara)\ **Replies:** 6\ **Last updated:** [June 26, 2020, 5:52am UTC](https://discuss.elastic.co/t/logstash-http-poller-custom-header-error/237314 "2020-06-26T05:52:38Z")
I am trying to use an API url to get data from a source. My api has a custom Header for aw-tenant-code, which works well with a curl. Here is my sample data: input { http\_poller { urls =\> { snowsr =\> { … --- ## [Debugging Elasticsearch output in Logstash](https://discuss.elastic.co/t/debugging-elasticsearch-output-in-logstash/238163)
**Author:** [@Jezreel\_Ravina](https://discuss.elastic.co/u/Jezreel_Ravina)\ **Replies:** 8\ **Last updated:** [June 26, 2020, 5:31am UTC](https://discuss.elastic.co/t/debugging-elasticsearch-output-in-logstash/238163 "2020-06-26T05:31:06Z")
Hello. We have an ELK Stack v7.5.1 deployed in a k8s cluster. We have Filebeat gathering k8s/docker container logs. I've confirmed by using stdout that Filebeat is passing the needed logs and Logstash is receiving it. … --- ## [Logstash reading CLOB Data from oracle](https://discuss.elastic.co/t/logstash-reading-clob-data-from-oracle/238786)
**Author:** [@kamalgunda](https://discuss.elastic.co/u/kamalgunda)\ **Replies:** 0\ **Last updated:** [June 26, 2020, 5:30am UTC](https://discuss.elastic.co/t/logstash-reading-clob-data-from-oracle/238786 "2020-06-26T05:30:46Z")
Hi I want to read CLOB data type column with nested json in it from source oracle table and write to ES using the logstash. I have configured the configuration file with input 4 JDBC connections on same table based on p… --- ## [Hosting Logstash in Cloud](https://discuss.elastic.co/t/hosting-logstash-in-cloud/238777)
**Author:** [@aerstelle](https://discuss.elastic.co/u/aerstelle)\ **Replies:** 0\ **Last updated:** [June 26, 2020, 3:51am UTC](https://discuss.elastic.co/t/hosting-logstash-in-cloud/238777 "2020-06-26T03:51:39Z")
I'm a ElasticSearch cloud user, and want to push my Java Spring Boot logs into elasticsearch. The best solution that I have found so far is to use a logback-logstash-appender, and push the logs to logstash, which will p… --- ## [The Logstash configuration item is written repeatedly](https://discuss.elastic.co/t/the-logstash-configuration-item-is-written-repeatedly/238771)
**Author:** [@Richard-Tang](https://discuss.elastic.co/u/Richard-Tang)\ **Replies:** 0\ **Last updated:** [June 26, 2020, 3:15am UTC](https://discuss.elastic.co/t/the-logstash-configuration-item-is-written-repeatedly/238771 "2020-06-26T03:15:37Z")
Distinguish according to Tags But need to repeat it in if "hosts "，"user"，"password" I have a lot of "if" that I have to write multiple times output { if "server4-apache-access" in \[tags\] { elasticsearch … --- ## [Logstash is not sending the logfiles to Elasticsearch](https://discuss.elastic.co/t/logstash-is-not-sending-the-logfiles-to-elasticsearch/238617)
**Author:** [@Magesh](https://discuss.elastic.co/u/Magesh)\ **Replies:** 5\ **Last updated:** [June 26, 2020, 2:58am UTC](https://discuss.elastic.co/t/logstash-is-not-sending-the-logfiles-to-elasticsearch/238617 "2020-06-26T02:58:06Z")
Hi All, I'm new to ELK , I tried to send the log files from Logstash to Elasticsearch from my loacl windows10 machine, though the service started successfully i don't see the index /logs in Kibana. Logstash / Elasticse… --- ## [Data from mongodb using logstash via jdbc](https://discuss.elastic.co/t/data-from-mongodb-using-logstash-via-jdbc/235825)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 4\ **Last updated:** [June 25, 2020, 9:10pm UTC](https://discuss.elastic.co/t/data-from-mongodb-using-logstash-via-jdbc/235825 "2020-06-25T21:10:32Z")
I am trying to create mongodb-elk connection, read data from mongo and insert in to elk. decided to use jdbc because I can use date and only retrive new data every few minutes here is my jdbc connection setup. it makes … --- ## [Custom IIS logs into Logstash](https://discuss.elastic.co/t/custom-iis-logs-into-logstash/238726)
**Author:** [@andrew.campbell](https://discuss.elastic.co/u/andrew.campbell)\ **Replies:** 2\ **Last updated:** [June 25, 2020, 7:51pm UTC](https://discuss.elastic.co/t/custom-iis-logs-into-logstash/238726 "2020-06-25T19:51:16Z")
I've installed the stack on a Windows machine to be able to index and visualize IIS logs. Our logs have a custom field from the header and additional fields not in the standard W3C IIS log format. I'm trying to map the f… --- ## [How to ingest geo\_shape from MySQL database?](https://discuss.elastic.co/t/how-to-ingest-geo-shape-from-mysql-database/238173)
**Author:** [@edbautista](https://discuss.elastic.co/u/edbautista)\ **Replies:** 3\ **Last updated:** [June 25, 2020, 7:43pm UTC](https://discuss.elastic.co/t/how-to-ingest-geo-shape-from-mysql-database/238173 "2020-06-25T19:43:02Z")
We are currently working on ingesting multiple polygons from our MySQL database into Elasticsearch to be visualized in Kibana Maps. We have tried generating a geojson from the polygons we need but we encountered errors w… --- ## [How to create index by ingesting MongoDB data?](https://discuss.elastic.co/t/how-to-create-index-by-ingesting-mongodb-data/235389)
**Author:** [@FernandoMOTA](https://discuss.elastic.co/u/FernandoMOTA)\ **Replies:** 1\ **Last updated:** [June 25, 2020, 6:55pm UTC](https://discuss.elastic.co/t/how-to-create-index-by-ingesting-mongodb-data/235389 "2020-06-25T18:55:29Z")
All tutorials I used brought me the following error: \> D, \[2020-06-01T07: 33: 35.814000 # 276\] DEBUG -: MONGODB | \[202\] localhost: 27017 # 1 | example.find | STARTED | {"find" =\> "movie", "filter" =\> {"\_ id" =\> {"$ gt" … --- ## [Process single data to apply multiple split and store as array](https://discuss.elastic.co/t/process-single-data-to-apply-multiple-split-and-store-as-array/238189)
**Author:** [@karthick2020](https://discuss.elastic.co/u/karthick2020)\ **Replies:** 1\ **Last updated:** [June 25, 2020, 4:28pm UTC](https://discuss.elastic.co/t/process-single-data-to-apply-multiple-split-and-store-as-array/238189 "2020-06-25T16:28:24Z")
Hi Team, I am looking for clarifications on below sample log: 30338 /opt/IBM/WebSphere/AppServe 19.1 1.3~ 4609 /opt/BESClient/bin/BESClien 0.2 0.4~20235 /opt/IBM/WebSphere/AppServe 18.8 0.3~ 354 /opt/IBM/WebSphere/App… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=335) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=337)