# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=80 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 81 --- ## [GeoIP Manual Database Updating](https://discuss.elastic.co/t/geoip-manual-database-updating/330253)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 2\ **Last updated:** [April 20, 2023, 10:33am UTC](https://discuss.elastic.co/t/geoip-manual-database-updating/330253 "2023-04-20T10:33:48Z")
I am manually updating the GeoIP databases, but don't have the ability to host the updated DBs on a web server somewhere. Is it possible to utilize a Elasticsearch, Kibana, or Logstash to host the files? For instance, … --- ## [Logstash, parsing a localised date with HTTPDATE](https://discuss.elastic.co/t/logstash-parsing-a-localised-date-with-httpdate/330297)
**Author:** [@GreenEyed](https://discuss.elastic.co/u/GreenEyed)\ **Replies:** 3\ **Last updated:** [April 19, 2023, 6:15pm UTC](https://discuss.elastic.co/t/logstash-parsing-a-localised-date-with-httpdate/330297 "2023-04-19T18:15:06Z")
Hi there, We have a library that is sending access logs to logstash with a "similar" to Apache format. We have created the regexp in grok to parse it but I have detected that the library is using the default format, loc… --- ## [Filebaet and logstash encoding problem](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223)
**Author:** [@maks1001281](https://discuss.elastic.co/u/maks1001281)\ **Replies:** 15\ **Last updated:** [April 19, 2023, 3:00pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223 "2023-04-19T15:00:01Z")
Hello, I can't understand why Logstash doesn't analyze logs from filebeat correctly, I see strange errors like: JSON parsing error, source data now in message field {:message=\>"Unexpected character ('\*' (code 42)): expe… --- ## [Logstash-8.7 fails to load YAML larger than 3MB](https://discuss.elastic.co/t/logstash-8-7-fails-to-load-yaml-larger-than-3mb/330269)
**Author:** [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Replies:** 0\ **Last updated:** [April 19, 2023, 7:55am UTC](https://discuss.elastic.co/t/logstash-8-7-fails-to-load-yaml-larger-than-3mb/330269 "2023-04-19T07:55:24Z")
We are using Logstash translate plugin to add user information to IP addresses in logs/events in our organization. The user data is loaded via YAML. The file is large (5.5MB with around 15K entries). Till Logstash-8.6, … --- ## [Logstash JDBC Static Filter Can't Connect to SQLite DB](https://discuss.elastic.co/t/logstash-jdbc-static-filter-cant-connect-to-sqlite-db/330171)
**Author:** [@Dustin527](https://discuss.elastic.co/u/Dustin527)\ **Replies:** 6\ **Last updated:** [April 18, 2023, 4:52pm UTC](https://discuss.elastic.co/t/logstash-jdbc-static-filter-cant-connect-to-sqlite-db/330171 "2023-04-18T16:52:00Z")
Hi I am having trouble getting the JDBC static filter to work with an SQLite DB. I am using the xerial sqlite jdbc lib on Debian and the latest logstash package. I even have a small java program that can connect to and … --- ## [Logstash manages to send data to elasticsearch only in debugging mode](https://discuss.elastic.co/t/logstash-manages-to-send-data-to-elasticsearch-only-in-debugging-mode/330196)
**Author:** [@Skairik](https://discuss.elastic.co/u/Skairik)\ **Replies:** 3\ **Last updated:** [April 18, 2023, 11:53am UTC](https://discuss.elastic.co/t/logstash-manages-to-send-data-to-elasticsearch-only-in-debugging-mode/330196 "2023-04-18T11:53:06Z")
Hello everyone, I am currently trying a basic test setup with apache logs on logstash, but I have a problem, my data is received on kibana/elasticsearch only when I run the following command: /usr/share/logstash/bin/lo… --- ## [Tune logstash JDBC input for huge datasets](https://discuss.elastic.co/t/tune-logstash-jdbc-input-for-huge-datasets/330160)
**Author:** [@Marco\_Lagalla](https://discuss.elastic.co/u/Marco_Lagalla)\ **Replies:** 4\ **Last updated:** [April 18, 2023, 9:18am UTC](https://discuss.elastic.co/t/tune-logstash-jdbc-input-for-huge-datasets/330160 "2023-04-18T09:18:39Z")
Hi all! I am trying to migrate a very huge dataset from a source oracle database to Elasticsearch. The data that I am trying to migrate is contained in a single table, which is partitioned weekly, based on a timestamp … --- ## [Is it possible to limit http-poller to a single node in a Logstash cluster?](https://discuss.elastic.co/t/is-it-possible-to-limit-http-poller-to-a-single-node-in-a-logstash-cluster/330087)
**Author:** [@jba](https://discuss.elastic.co/u/jba)\ **Replies:** 5\ **Last updated:** [April 18, 2023, 8:06am UTC](https://discuss.elastic.co/t/is-it-possible-to-limit-http-poller-to-a-single-node-in-a-logstash-cluster/330087 "2023-04-18T08:06:35Z")
I have just discovered the http-poller plugin and it seems like it could replace some custom scripts that we have for getting monitoring data (e.g. index growth) into Elasticsearch. But it is not clear to me if I can res… --- ## [Use variable in logstash config](https://discuss.elastic.co/t/use-variable-in-logstash-config/330092)
**Author:** [@Indeed2000](https://discuss.elastic.co/u/Indeed2000)\ **Replies:** 2\ **Last updated:** [April 18, 2023, 6:20am UTC](https://discuss.elastic.co/t/use-variable-in-logstash-config/330092 "2023-04-18T06:20:52Z")
Hi I have logstash input like below, how can i set variable for "cpu" (it is name of table in database) and it has different values like "mem, disk,i/o,...". need to pass name of table as variable instead of define mul… --- ## [Logstash filter to extract key/values from curl result](https://discuss.elastic.co/t/logstash-filter-to-extract-key-values-from-curl-result/330083)
**Author:** [@Indeed2000](https://discuss.elastic.co/u/Indeed2000)\ **Replies:** 1\ **Last updated:** [April 18, 2023, 1:00am UTC](https://discuss.elastic.co/t/logstash-filter-to-extract-key-values-from-curl-result/330083 "2023-04-18T01:00:52Z")
Hi Here is the result of curl command that I need to extract key/values (columns,values) and here is the key/value that i need to send to elastic { "series": { "time": "2023-04-16T07:58:40Z", "cpu": "cpu-… --- ## [Kafka-Elasticsearch Logstash Configuration Error](https://discuss.elastic.co/t/kafka-elasticsearch-logstash-configuration-error/330130)
**Author:** [@Mustafa\_AYDOGDU](https://discuss.elastic.co/u/Mustafa_AYDOGDU)\ **Replies:** 1\ **Last updated:** [April 17, 2023, 2:52pm UTC](https://discuss.elastic.co/t/kafka-elasticsearch-logstash-configuration-error/330130 "2023-04-17T14:52:20Z")
I have a logstash pipeline which gets data from kafka and sends it to elasticsearch. However, in elasticsearch, data is not represented correctly. In this data I want it to be just field:value. But it is field:\[value,fi… --- ## [Help pattern for multiline logs](https://discuss.elastic.co/t/help-pattern-for-multiline-logs/330134)
**Author:** [@JackieLaFrite](https://discuss.elastic.co/u/JackieLaFrite)\ **Replies:** 4\ **Last updated:** [April 17, 2023, 2:41pm UTC](https://discuss.elastic.co/t/help-pattern-for-multiline-logs/330134 "2023-04-17T14:41:54Z")
What pattern should I use to retrieve correctly multi-lines logs ? Normally I use : file { path =\> "/var/log/appslogs/\*\*/\*.log" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" codec =\> multili… --- ## [Logstash docker cannot log into elasticsearch docker](https://discuss.elastic.co/t/logstash-docker-cannot-log-into-elasticsearch-docker/328336)
**Author:** [@kpankhurst](https://discuss.elastic.co/u/kpankhurst)\ **Replies:** 10\ **Last updated:** [April 17, 2023, 2:25pm UTC](https://discuss.elastic.co/t/logstash-docker-cannot-log-into-elasticsearch-docker/328336 "2023-04-17T14:25:01Z")
I have 2 dockers set up as follows: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.6.0 volumes: - ./config/elasticsearch/esdata:/usr/share/elasticsearch/data - ./config/elast… --- ## [Logstash throws java.lang.OutOfMemoryError: Java heap space no matter the heap size](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089)
**Author:** [@ste1](https://discuss.elastic.co/u/ste1)\ **Replies:** 6\ **Last updated:** [April 17, 2023, 10:57am UTC](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089 "2023-04-17T10:57:36Z")
Im attempting to parse a huge (few million lines) csv file with logstash and output it to elasticsearch. \[FATAL\] 2023-04-16 19:00:19.011 \[LogStash::Runner\] Logstash - java.lang.OutOfMemoryError: Java heap space … --- ## [Logstash add subfield to elasticsearch index](https://discuss.elastic.co/t/logstash-add-subfield-to-elasticsearch-index/329870)
**Author:** [@Utibeabasi\_Umanah](https://discuss.elastic.co/u/Utibeabasi_Umanah)\ **Replies:** 3\ **Last updated:** [April 13, 2023, 5:59pm UTC](https://discuss.elastic.co/t/logstash-add-subfield-to-elasticsearch-index/329870 "2023-04-13T17:59:00Z")
Hi, i want to add a sub field called prefix to a text field called title using a logstash filter plugin. how do i go about this? i need this because the sub fields are required in app search. here is my logstash config s… --- ## [Getting started with Logstash JDBC Integration on Windows](https://discuss.elastic.co/t/getting-started-with-logstash-jdbc-integration-on-windows/329784)
**Author:** [@Dale\_ander](https://discuss.elastic.co/u/Dale_ander)\ **Replies:** 3\ **Last updated:** [April 13, 2023, 4:20pm UTC](https://discuss.elastic.co/t/getting-started-with-logstash-jdbc-integration-on-windows/329784 "2023-04-13T16:20:15Z")
I'm just beginning my learning process on ELK but from what I've seen, I'd like to learn how to index data from an RDB table, I presume using the Logstash JDBC Integration plugin, so I can start trying to create differen… --- ## [Can I make two input and output in the logstash config file?](https://discuss.elastic.co/t/can-i-make-two-input-and-output-in-the-logstash-config-file/329933)
**Author:** [@lilyyy](https://discuss.elastic.co/u/lilyyy)\ **Replies:** 3\ **Last updated:** [April 13, 2023, 3:26pm UTC](https://discuss.elastic.co/t/can-i-make-two-input-and-output-in-the-logstash-config-file/329933 "2023-04-13T15:26:19Z")
Hello all. I want to get the two indexes from two input data in the one logstash config file. (One is from tshark file and the other one is filebeat so each data are different.) tshark data is changed to json file for … --- ## [Restarting logstash cloudwatch plugin](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681)
**Author:** [@mphilip9](https://discuss.elastic.co/u/mphilip9)\ **Replies:** 15\ **Last updated:** [April 13, 2023, 1:42pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681 "2023-04-13T13:42:41Z")
We have an ELK stack app that has been down for over a month due to a credentials issue in the logstash cloudwatch plugin. The plugin is digesting data again now, but what is strange is that it is digesting logs from the… --- ## [Error in Logstash - failed to parse date field with format strict\_date\_optional\_time||epoch\_millis date-time-parse-exception](https://discuss.elastic.co/t/error-in-logstash-failed-to-parse-date-field-with-format-strict-date-optional-time-epoch-millis-date-time-parse-exception/329797)
**Author:** [@sarath.sarepaka](https://discuss.elastic.co/u/sarath.sarepaka)\ **Replies:** 3\ **Last updated:** [April 13, 2023, 12:30pm UTC](https://discuss.elastic.co/t/error-in-logstash-failed-to-parse-date-field-with-format-strict-date-optional-time-epoch-millis-date-time-parse-exception/329797 "2023-04-13T12:30:34Z")
Hi, We are getting the below error in the logstash. We are using a field called "destination" for both time and string. We observed below issue when the destination field value is a string . ELasticsearch and Logstash … --- ## [Logstash is not reading data in Docker](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666)
**Author:** [@vvsenthil](https://discuss.elastic.co/u/vvsenthil)\ **Replies:** 7\ **Last updated:** [April 13, 2023, 8:08am UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666 "2023-04-13T08:08:14Z")
Hi, Someone would you be able to help me on setting up the logstah in docker. I am able to setup and push the message to Elasticsearch without docker. But if i move the logstah to docker i am not able to push the message… --- ## [Mongodb logstash data input](https://discuss.elastic.co/t/mongodb-logstash-data-input/329830)
**Author:** [@jskang](https://discuss.elastic.co/u/jskang)\ **Replies:** 2\ **Last updated:** [April 13, 2023, 5:34am UTC](https://discuss.elastic.co/t/mongodb-logstash-data-input/329830 "2023-04-13T05:34:09Z")
hello. After struggling for days, I finally connected mongodb and logstash. But another problem arose. I want to send only newly entered logs to the index using sql\_last\_value, but it doesn't work. Is this impossible… --- ## [Error during plugin bundling , while running gradlew gem command](https://discuss.elastic.co/t/error-during-plugin-bundling-while-running-gradlew-gem-command/329864)
**Author:** [@Theophila\_Vaiz\_R](https://discuss.elastic.co/u/Theophila_Vaiz_R)\ **Replies:** 0\ **Last updated:** [April 12, 2023, 6:08pm UTC](https://discuss.elastic.co/t/error-during-plugin-bundling-while-running-gradlew-gem-command/329864 "2023-04-12T18:08:47Z")
I created a logstash output plugin and tried building using ./gradlew gem command but locally its fine I changed it as a automated jenkins pipeline there I'm facing this error TypeError: Could not initialize copy of dig… --- ## [Log files getting accumulated in temporary\_directory path while reading logs from s3 buckets](https://discuss.elastic.co/t/log-files-getting-accumulated-in-temporary-directory-path-while-reading-logs-from-s3-buckets/329838)
**Author:** [@Anusha\_Kusanghi](https://discuss.elastic.co/u/Anusha_Kusanghi)\ **Replies:** 1\ **Last updated:** [April 12, 2023, 4:23pm UTC](https://discuss.elastic.co/t/log-files-getting-accumulated-in-temporary-directory-path-while-reading-logs-from-s3-buckets/329838 "2023-04-12T16:23:58Z")
Hi All, We have a logstash configuration to read logs from s3 bucket. Here is the configuration: input { s3 { access\_key\_id =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\*" secret\_access\_key =\> "hjiufaaaa" bucket =\> "test… --- ## [Parse XML sub tags as a separate log](https://discuss.elastic.co/t/parse-xml-sub-tags-as-a-separate-log/329832)
**Author:** [@Disha\_Bodade](https://discuss.elastic.co/u/Disha_Bodade)\ **Replies:** 1\ **Last updated:** [April 12, 2023, 3:43pm UTC](https://discuss.elastic.co/t/parse-xml-sub-tags-as-a-separate-log/329832 "2023-04-12T15:43:17Z")
Hi Team, I have a XML formatted as below \ \ \\101074476\\End of Sale 1403 and 1416\\clark13\\ \\ **Author:** [@serjio](https://discuss.elastic.co/u/serjio)\ **Replies:** 2\ **Last updated:** [April 12, 2023, 3:29pm UTC](https://discuss.elastic.co/t/apply-filters/329510 "2023-04-12T15:29:23Z") good afternoon. Recently I started to get acquainted with ELK and aot what is my problem: I use such a filter filter { if \[type\] == "syslog" { grok { match =\> { "message" =\> "\<%{POSINT:syslog\_pri}\>%{SYSLO… --- ## [Unable to create dead letter queue writer](https://discuss.elastic.co/t/unable-to-create-dead-letter-queue-writer/329688)
**Author:** [@tcapp24](https://discuss.elastic.co/u/tcapp24)\ **Replies:** 1\ **Last updated:** [April 12, 2023, 2:01pm UTC](https://discuss.elastic.co/t/unable-to-create-dead-letter-queue-writer/329688 "2023-04-12T14:01:10Z")
Logstash Version - 7.9.1 Currently we are unable to start Logstash properly without receiving error below: \[2023-04-10T20:18:52,978\]\[ERROR\]\[org.logstash.common.DeadLetterQueueFactory\] unable to create dead letter queue… --- ## [Aggregate secure/sshd syslog event based on selected events](https://discuss.elastic.co/t/aggregate-secure-sshd-syslog-event-based-on-selected-events/328249)
**Author:** [@jun.7.6](https://discuss.elastic.co/u/jun.7.6)\ **Replies:** 24\ **Last updated:** [April 12, 2023, 1:19pm UTC](https://discuss.elastic.co/t/aggregate-secure-sshd-syslog-event-based-on-selected-events/328249 "2023-04-12T13:19:55Z")
Hi I'm trying to filter out the login & logout events from linux ssh events send as syslog to Logstash and forward it to my firewall via syslog again. This setup is to allow my firewall to map the user-id to IP address i… --- ## [Multiple configuration or multiple codec](https://discuss.elastic.co/t/multiple-configuration-or-multiple-codec/329752)
**Author:** [@JackieLaFrite](https://discuss.elastic.co/u/JackieLaFrite)\ **Replies:** 2\ **Last updated:** [April 12, 2023, 9:44am UTC](https://discuss.elastic.co/t/multiple-configuration-or-multiple-codec/329752 "2023-04-12T09:44:11Z")
Here is my logstash.conf file input { file { path =\> "/var/log/appslogs/\*\*/\*.log" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" codec =\> plain { charset =\> "UTF-8" } type =\> "… --- ## [Help with this grok](https://discuss.elastic.co/t/help-with-this-grok/329817)
**Author:** [@roman-tasi](https://discuss.elastic.co/u/roman-tasi)\ **Replies:** 1\ **Last updated:** [April 12, 2023, 9:28am UTC](https://discuss.elastic.co/t/help-with-this-grok/329817 "2023-04-12T09:28:39Z")
Need a grok filter that parses out the account (the peacesat) from these two types of logs Case 1: Apr 11 14:26:55 mail saslauthd\[15405\]: auth\_zimbra: peacesat@uhtasi.org auth failed: authentication failed for \[peacesa… --- ## [Logstash plugin install : Error socket closed](https://discuss.elastic.co/t/logstash-plugin-install-error-socket-closed/328243)
**Author:** [@Julien069](https://discuss.elastic.co/u/Julien069)\ **Replies:** 22\ **Last updated:** [April 12, 2023, 7:50am UTC](https://discuss.elastic.co/t/logstash-plugin-install-error-socket-closed/328243 "2023-04-12T07:50:09Z")
Hi , I want to install a Stormshield plugin for Logstash I tried bin/logstash-plugin install --no-verify logstash-filter-SNS I have "ERROR : Something went wrong when installalling bin/logstash-filter-SNS , message s… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=79) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=81)