# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=82 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 83 --- ## [Is it possible to recover logs that failed to transfer to ElasticSearch?](https://discuss.elastic.co/t/is-it-possible-to-recover-logs-that-failed-to-transfer-to-elasticsearch/329401)
**Author:** [@r.fujii](https://discuss.elastic.co/u/r.fujii)\ **Replies:** 0\ **Last updated:** [April 5, 2023, 8:35am UTC](https://discuss.elastic.co/t/is-it-possible-to-recover-logs-that-failed-to-transfer-to-elasticsearch/329401 "2023-04-05T08:35:20Z")
The following configuration is used to obtain the server's audit logs and forward them to Elasticsearch. AuditBeat -\> Logstash -\> Elasticsearch However, on Elasticsearch, the number of shards exceeded max\_shards\_per\_no… --- ## [Why my geoip lookup is failing?](https://discuss.elastic.co/t/why-my-geoip-lookup-is-failing/329353)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 2\ **Last updated:** [April 5, 2023, 12:52am UTC](https://discuss.elastic.co/t/why-my-geoip-lookup-is-failing/329353 "2023-04-05T00:52:47Z")
Hi Team, This is my logstash config and surprisingly my geoip lookup is failed, I am not sure why. Can someone pls help? input { stdin {} } filter { json { source =\> "message" remove\_field =\> … --- ## [Logstash shutting down](https://discuss.elastic.co/t/logstash-shutting-down/329344)
**Author:** [@th\_shadoow](https://discuss.elastic.co/u/th_shadoow)\ **Replies:** 1\ **Last updated:** [April 4, 2023, 5:55pm UTC](https://discuss.elastic.co/t/logstash-shutting-down/329344 "2023-04-04T17:55:15Z")
after installing logstash in a windows machine and tried launching it with the command : bin\\logstash -f logstash.conf it start but immediatlly shows an error message and shut down the error meassge : org.jruby.excepti… --- ## [Logstash csv imports not all data into elastic search](https://discuss.elastic.co/t/logstash-csv-imports-not-all-data-into-elastic-search/329199)
**Author:** [@oleksiiorel](https://discuss.elastic.co/u/oleksiiorel)\ **Replies:** 1\ **Last updated:** [April 4, 2023, 3:50pm UTC](https://discuss.elastic.co/t/logstash-csv-imports-not-all-data-into-elastic-search/329199 "2023-04-04T15:50:36Z")
Hi guys! Do you have any idea why I can't import all the data from csv.file via logstash into Elasticsearch? it always imports the same amount equal to 6873 and should be more than 53k. at runtime, the console displays … --- ## [Logstash consumes persistent queue size when no events stored](https://discuss.elastic.co/t/logstash-consumes-persistent-queue-size-when-no-events-stored/329311)
**Author:** [@ferdose\_shaik](https://discuss.elastic.co/u/ferdose_shaik)\ **Replies:** 0\ **Last updated:** [April 4, 2023, 11:03am UTC](https://discuss.elastic.co/t/logstash-consumes-persistent-queue-size-when-no-events-stored/329311 "2023-04-04T11:03:25Z")
Hi, We are using persistent queue in Logstash to store the events when output is blocked. Please refer to the following configuration. - pipeline.id: syslog queue.type: persisted queue.max\_bytes: 128mb path.confi… --- ## [During scale in of logstash through HPA data remains in the persistence queue](https://discuss.elastic.co/t/during-scale-in-of-logstash-through-hpa-data-remains-in-the-persistence-queue/329307)
**Author:** [@prashant1](https://discuss.elastic.co/u/prashant1)\ **Replies:** 0\ **Last updated:** [April 4, 2023, 10:42am UTC](https://discuss.elastic.co/t/during-scale-in-of-logstash-through-hpa-data-remains-in-the-persistence-queue/329307 "2023-04-04T10:42:31Z")
Problem Statement :- In our environment we are sending application logs from Fluentd to logstash where persistent queues are enabled. HPA is enabled on the logstash pod so when the load increases so logstash pods scale … --- ## [Mongodb/logstash connect error](https://discuss.elastic.co/t/mongodb-logstash-connect-error/328799)
**Author:** [@jskang](https://discuss.elastic.co/u/jskang)\ **Replies:** 18\ **Last updated:** [April 4, 2023, 9:57am UTC](https://discuss.elastic.co/t/mongodb-logstash-connect-error/328799 "2023-04-04T09:57:48Z")
This is an error message when running logstash. error message: Using bundled JDK: /home/admin/lg-862/jdk Sending Logstash logs to /home/admin/lg-862/logs which is now configured via log4j2.properties \[2023-03-29T18:2… --- ## [Kafka 0 partition metadata cannot be read in logstash6.8, other partitions can. Sample configuration:](https://discuss.elastic.co/t/kafka-0-partition-metadata-cannot-be-read-in-logstash6-8-other-partitions-can-sample-configuration/329055)
**Author:** [@angus](https://discuss.elastic.co/u/angus)\ **Replies:** 1\ **Last updated:** [April 4, 2023, 1:45am UTC](https://discuss.elastic.co/t/kafka-0-partition-metadata-cannot-be-read-in-logstash6-8-other-partitions-can-sample-configuration/329055 "2023-04-04T01:45:44Z")
kafka 0 partition metadata cannot be read in logstash6.8, other partitions can. Sample configuration: input { kafka { client\_id =\> "ycUsrRdNews" consumer\_threads =\> 4 bootstrap\_servers =\> "${KAFKA\_BOOTSTRAP\_SERVE… --- ## [Streaming API to local folder using logstash](https://discuss.elastic.co/t/streaming-api-to-local-folder-using-logstash/329248)
**Author:** [@Reloef\_Khoza](https://discuss.elastic.co/u/Reloef_Khoza)\ **Replies:** 0\ **Last updated:** [April 3, 2023, 7:26pm UTC](https://discuss.elastic.co/t/streaming-api-to-local-folder-using-logstash/329248 "2023-04-03T19:26:54Z")
Any example of how to stream multiple API from a website into a local folder --- ## [If IP results in \_geoip\_lookup\_failure is it possible to fill geoip-related vields with a custom value?](https://discuss.elastic.co/t/if-ip-results-in-geoip-lookup-failure-is-it-possible-to-fill-geoip-related-vields-with-a-custom-value/329144)
**Author:** [@roman-tasi](https://discuss.elastic.co/u/roman-tasi)\ **Replies:** 3\ **Last updated:** [April 3, 2023, 4:57pm UTC](https://discuss.elastic.co/t/if-ip-results-in-geoip-lookup-failure-is-it-possible-to-fill-geoip-related-vields-with-a-custom-value/329144 "2023-04-03T16:57:54Z")
Basically if the IP cannot be found in the database, I want to fill the geoip.city\_name, geoip.region\_name, and geoip.country\_name with a custom value like "PRIVATE ADDRESS" or "IP NOT IN DATABASE" or something similar..… --- ## [Logstash does not creates nor updates index on elasticsearch](https://discuss.elastic.co/t/logstash-does-not-creates-nor-updates-index-on-elasticsearch/329069)
**Author:** [@Quentin\_Moisy](https://discuss.elastic.co/u/Quentin_Moisy)\ **Replies:** 5\ **Last updated:** [April 3, 2023, 3:41pm UTC](https://discuss.elastic.co/t/logstash-does-not-creates-nor-updates-index-on-elasticsearch/329069 "2023-04-03T15:41:40Z")
Hello, I new to the ELK flow and I have some issues with Logstash. Sometime my index will be populated sometime not. Furthermore it seems that logstash does not create index on elasticsearch. Can you help on that My .c… --- ## [All AWS WAF event goes to message field even after using correct mapping](https://discuss.elastic.co/t/all-aws-waf-event-goes-to-message-field-even-after-using-correct-mapping/329227)
**Author:** [@SSP1](https://discuss.elastic.co/u/SSP1)\ **Replies:** 0\ **Last updated:** [April 3, 2023, 2:55pm UTC](https://discuss.elastic.co/t/all-aws-waf-event-goes-to-message-field-even-after-using-correct-mapping/329227 "2023-04-03T14:55:22Z")
HI, I'm using Logstash to ingest AWS WAF Logs from S3 using S3 Input login with SQS and logs are going through to elasticsearch. I can see those in Kibana but all the waf event goes to message filed. I have tried to use … --- ## [Databricks lakehouse delta tables as data source](https://discuss.elastic.co/t/databricks-lakehouse-delta-tables-as-data-source/328591)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 2\ **Last updated:** [April 3, 2023, 2:44pm UTC](https://discuss.elastic.co/t/databricks-lakehouse-delta-tables-as-data-source/328591 "2023-04-03T14:44:47Z")
Is it possible to connect to the databricks lakehouse Delta tables to get the data to be indexed into elasticsearch? --- ## [Elastic Dissect](https://discuss.elastic.co/t/elastic-dissect/329117)
**Author:** [@oleksiiorel](https://discuss.elastic.co/u/oleksiiorel)\ **Replies:** 5\ **Last updated:** [April 3, 2023, 8:25am UTC](https://discuss.elastic.co/t/elastic-dissect/329117 "2023-04-03T08:25:13Z")
Hi everyone :slight\_smile: How to split a single cell with a different number of strings in CVS file into separate cells (fields). Assign the names of the fields from the string itself. If I manually write the names of… --- ## [Removing \\ from raw input log data](https://discuss.elastic.co/t/removing-from-raw-input-log-data/329062)
**Author:** [@Merdesz](https://discuss.elastic.co/u/Merdesz)\ **Replies:** 2\ **Last updated:** [April 3, 2023, 8:17am UTC](https://discuss.elastic.co/t/removing-from-raw-input-log-data/329062 "2023-04-03T08:17:59Z")
I have a device sending in logs which have "" before every string caracter and I would like to remove them or rewrite them to a simple ". So this " --\> " to this. Logs: srcintfrole="undefined" dstip=255.255.255.255 dst… --- ## [Logstash and mongodb connection error](https://discuss.elastic.co/t/logstash-and-mongodb-connection-error/329153)
**Author:** [@jskang](https://discuss.elastic.co/u/jskang)\ **Replies:** 0\ **Last updated:** [April 3, 2023, 5:50am UTC](https://discuss.elastic.co/t/logstash-and-mongodb-connection-error/329153 "2023-04-03T05:50:34Z")
input{ jdbc{ jdbc\_driver\_library =\> "/home/admin/lg-862/lgstash-core/lib/jars/mongojdbc4.8.jar" jdbc\_driver\_class =\> "Java::com.wisecoders.dbschema.mongodb.JdbcDriver" jdbc\_connection\_string =\> "mongodb://XXXXXX:XXXX… --- ## [Can't match string to format date](https://discuss.elastic.co/t/cant-match-string-to-format-date/329132)
**Author:** [@german](https://discuss.elastic.co/u/german)\ **Replies:** 4\ **Last updated:** [April 3, 2023, 3:30am UTC](https://discuss.elastic.co/t/cant-match-string-to-format-date/329132 "2023-04-03T03:30:23Z")
Hi everybody, I have a problem while I try to convert a string variable to timestamp. I'm using date module without successful result. Format date is dd/MM/yyyy hh:mm:ss.SSSSSS and originally the variable newDate is: … --- ## [Help with using Grok to parse these three log formats](https://discuss.elastic.co/t/help-with-using-grok-to-parse-these-three-log-formats/329107)
**Author:** [@roman-tasi](https://discuss.elastic.co/u/roman-tasi)\ **Replies:** 8\ **Last updated:** [April 2, 2023, 10:37pm UTC](https://discuss.elastic.co/t/help-with-using-grok-to-parse-these-three-log-formats/329107 "2023-04-02T22:37:55Z")
Hi I am experienced with Dissect but not Grok. I think I need to use Grok here because the log format varies between the logs, so dissect will only work on one format, not all three: Case 1: 2023-03-31 00:01:24,366 INF… --- ## [Elastic Filter](https://discuss.elastic.co/t/elastic-filter/329104)
**Author:** [@oleksiiorel](https://discuss.elastic.co/u/oleksiiorel)\ **Replies:** 8\ **Last updated:** [April 1, 2023, 4:57pm UTC](https://discuss.elastic.co/t/elastic-filter/329104 "2023-04-01T16:57:52Z")
I import a csv file via logstash "filter cvs" into Elasticsearch. One of the cells in a table (CVS file) contains several strings example: (categoty, subcategory, sub\_subcategory). I would like to split these strings int… --- ## [How can I join or paste array elements as of a one element?](https://discuss.elastic.co/t/how-can-i-join-or-paste-array-elements-as-of-a-one-element/329089)
**Author:** [@german](https://discuss.elastic.co/u/german)\ **Replies:** 2\ **Last updated:** [April 1, 2023, 3:08am UTC](https://discuss.elastic.co/t/how-can-i-join-or-paste-array-elements-as-of-a-one-element/329089 "2023-04-01T03:08:19Z")
Hi everybody, First of all, thanks for your time. I have a question regarding to Logstash. I would like to join some array elements as of a specific element. The log that I am processing, the first six fields have the … --- ## [Update Existing document through logstash](https://discuss.elastic.co/t/update-existing-document-through-logstash/329077)
**Author:** [@rubhamra](https://discuss.elastic.co/u/rubhamra)\ **Replies:** 4\ **Last updated:** [March 31, 2023, 8:40pm UTC](https://discuss.elastic.co/t/update-existing-document-through-logstash/329077 "2023-03-31T20:40:21Z")
logstash pipeline is not updating existing document for the same id, I have couples of fields which got updated frequestly for example Last Modified Date. I have below logstash config. output { elasticsearch { … --- ## [Kibana not give logs](https://discuss.elastic.co/t/kibana-not-give-logs/328695)
**Author:** [@Prabhath\_samarasingh](https://discuss.elastic.co/u/Prabhath_samarasingh)\ **Replies:** 7\ **Last updated:** [March 31, 2023, 8:13pm UTC](https://discuss.elastic.co/t/kibana-not-give-logs/328695 "2023-03-31T20:13:11Z")
Configured basic ELK set up.But my kibana interface had no logs. This is the guide I followed. What is the mistake I have done. Installing and Configuring Elasticsearch curl -fsSL https://artifacts.elastic.co/GPG-KEY… --- ## [Reading date format in logstash date filter](https://discuss.elastic.co/t/reading-date-format-in-logstash-date-filter/329070)
**Author:** [@UsmanNiazi](https://discuss.elastic.co/u/UsmanNiazi)\ **Replies:** 7\ **Last updated:** [March 31, 2023, 7:45pm UTC](https://discuss.elastic.co/t/reading-date-format-in-logstash-date-filter/329070 "2023-03-31T19:45:21Z")
Hi, I am unable to convert this string "03/31/2023 03:15 AM PDT" to date when using logstash date filter. Getting error dateparse failure. I am using below script date { match =\> \[ "start\_time", "mm/dd/yyyy HH:mm Z",… --- ## [CloudWatch input plugin not working with EC2 namespace](https://discuss.elastic.co/t/cloudwatch-input-plugin-not-working-with-ec2-namespace/329026)
**Author:** [@kanny](https://discuss.elastic.co/u/kanny)\ **Replies:** 0\ **Last updated:** [March 31, 2023, 8:19am UTC](https://discuss.elastic.co/t/cloudwatch-input-plugin-not-working-with-ec2-namespace/329026 "2023-03-31T08:19:16Z")
Hello, When I ran Logstash 8.6.2 version to collect EC2 CloudWatch metrics, the process returned "Exception: NameError". As far as I know by testing, this exception happends only for AWS/EC2 namespace, and setting for … --- ## [Logstash can't talk to Elasticsearch but I can access from the browser](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-but-i-can-access-from-the-browser/328970)
**Author:** [@crimson\_med](https://discuss.elastic.co/u/crimson_med)\ **Replies:** 0\ **Last updated:** [March 30, 2023, 6:18pm UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-but-i-can-access-from-the-browser/328970 "2023-03-30T18:18:52Z")
I have been trying to configure the ELK stack to use our wildcard certificate however this has been impossible until now. Logstash and kibana are unable to talk to elasticsearch however i can curl with no issues. Resul… --- ## [Why mutate will influence different pipelines?](https://discuss.elastic.co/t/why-mutate-will-influence-different-pipelines/328210)
**Author:** [@AlanChan](https://discuss.elastic.co/u/AlanChan)\ **Replies:** 9\ **Last updated:** [March 31, 2023, 6:44am UTC](https://discuss.elastic.co/t/why-mutate-will-influence-different-pipelines/328210 "2023-03-31T06:44:11Z")
Hi I'd like to process the same input data in different ways to get different results, so I'm trying to do with multiple pipelines. However, I notice that using mutate will influence other pipelines. Does anyone know ho… --- ## [Error parsing json but data still thrown to elastic](https://discuss.elastic.co/t/error-parsing-json-but-data-still-thrown-to-elastic/328922)
**Author:** [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Replies:** 5\ **Last updated:** [March 31, 2023, 4:03am UTC](https://discuss.elastic.co/t/error-parsing-json-but-data-still-thrown-to-elastic/328922 "2023-03-31T04:03:40Z")
Hi there, i want to ask about this error. anyone know what this error is trying to tell ? exception=\>java.lang.ClassCastException: class org.jruby.RubyHash cannot be cast to class org.jruby.RubyIO (org.jruby.RubyHash a… --- ## [Unable to start logstash on Amazon EC2 with Windows Server 2022](https://discuss.elastic.co/t/unable-to-start-logstash-on-amazon-ec2-with-windows-server-2022/328986)
**Author:** [@xavier76](https://discuss.elastic.co/u/xavier76)\ **Replies:** 1\ **Last updated:** [March 31, 2023, 5:01am UTC](https://discuss.elastic.co/t/unable-to-start-logstash-on-amazon-ec2-with-windows-server-2022/328986 "2023-03-31T05:01:58Z")
The erorr I'm getting is below. \[2023-03-30T23:40:36,465\]\[INFO \]\[logstash.runner \] Starting Logstash {"logstash.version"=\>"8.6.2", "jruby.version"=\>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit S… --- ## [How to get creation timestamp of input file](https://discuss.elastic.co/t/how-to-get-creation-timestamp-of-input-file/328533)
**Author:** [@Zak1](https://discuss.elastic.co/u/Zak1)\ **Replies:** 4\ **Last updated:** [March 31, 2023, 4:54am UTC](https://discuss.elastic.co/t/how-to-get-creation-timestamp-of-input-file/328533 "2023-03-31T04:54:48Z")
Am using logstash to parse an input logfile and subsequently sends data to elasticsearch. I would like to capture the creation date/timestamp of the input logfile as a field on the event. How best to go about this? Fyi, … --- ## [Is there anyway to store document to the index pattern {YYYY.MM.dd} but in CST time {YYYY.MM.dd} via logstash?](https://discuss.elastic.co/t/is-there-anyway-to-store-document-to-the-index-pattern-yyyy-mm-dd-but-in-cst-time-yyyy-mm-dd-via-logstash/328910)
**Author:** [@LongKang\_Fan](https://discuss.elastic.co/u/LongKang_Fan)\ **Replies:** 2\ **Last updated:** [March 31, 2023, 1:54am UTC](https://discuss.elastic.co/t/is-there-anyway-to-store-document-to-the-index-pattern-yyyy-mm-dd-but-in-cst-time-yyyy-mm-dd-via-logstash/328910 "2023-03-31T01:54:19Z")
Hi I have noticed Logstash stores documents to a {YYYY.MM.dd} index in UTC time. However, my local time is CST time. So, for example, my local time is now 2023/3/30 7:40 AM. The Logstash will store the data to index {20… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=81) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=83)