# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=22 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 23 --- ## [How to recycle the file after reading with logstash-input-file plugin](https://discuss.elastic.co/t/how-to-recycle-the-file-after-reading-with-logstash-input-file-plugin/365586)
**Author:** [@chun](https://discuss.elastic.co/u/chun)\ **Replies:** 3\ **Last updated:** [August 28, 2024, 4:23am UTC](https://discuss.elastic.co/t/how-to-recycle-the-file-after-reading-with-logstash-input-file-plugin/365586 "2024-08-28T04:23:15Z")
Hi, I have a logstash configuration as below. I'm expecting the json file is moved to archive path when reading is completed, but the json file is always there, and archive path is empty. Do I miss out anything in confi… --- ## [Using Logstash to read from a log file and then output each log lines to Kafka topic, cannot understand why it is not putting file content to kafka topic specified in conf file](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499)
**Author:** [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Replies:** 17\ **Last updated:** [August 26, 2024, 6:49pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499 "2024-08-26T18:49:57Z")
Details of the issue-: logstash version used-: 8.15.0 Kafka broker version - : 3.8.0- up at port 9092 Machine-: Windows Following is my logstash.conf-: input { file { path =\> "C:\\Users\\LENOVO\\Downloads\\ms1\\logs\\xe… --- ## [How can I use multiline for this log](https://discuss.elastic.co/t/how-can-i-use-multiline-for-this-log/365551)
**Author:** [@sumantapakira](https://discuss.elastic.co/u/sumantapakira)\ **Replies:** 3\ **Last updated:** [August 26, 2024, 3:49pm UTC](https://discuss.elastic.co/t/how-can-i-use-multiline-for-this-log/365551 "2024-08-26T15:49:20Z")
Log file - 25.08.2024 13:28:50.972 \*DEBUG\* \[127.0.0.1 \[1724326130527\] GET /content/smple/global/locations.html HTTP/1.1\] com.custom.global.core.models.impl.LocationCFListImpl damContentFragment name : em 25.08.2024 10:… --- ## [Save custom logs with logstash](https://discuss.elastic.co/t/save-custom-logs-with-logstash/365490)
**Author:** [@sandra\_a](https://discuss.elastic.co/u/sandra_a)\ **Replies:** 1\ **Last updated:** [August 24, 2024, 12:51pm UTC](https://discuss.elastic.co/t/save-custom-logs-with-logstash/365490 "2024-08-24T12:51:38Z")
I'm new to ELK. I have many containers from which I want to collect logs using Logstash. My custom log looks like this: {'agent\_id': 99, 'name': 'Cora', 'timestamp': '2024-08-21 06:26:09', 'operation\_name': 'Do\_smthin… --- ## [Data ingestion error](https://discuss.elastic.co/t/data-ingestion-error/365467)
**Author:** [@jaime\_solas](https://discuss.elastic.co/u/jaime_solas)\ **Replies:** 3\ **Last updated:** [August 23, 2024, 4:16pm UTC](https://discuss.elastic.co/t/data-ingestion-error/365467 "2024-08-23T16:16:12Z")
When I perform the ingestion everything seems to be fine, the information is displayed correctly in kibana but during the process the following errors appear \[ERROR\] 2024-08-23 15:08:25.370 \[\[main\]\>worker2\] ruby - Ruby … --- ## [Logstash timezone is not work](https://discuss.elastic.co/t/logstash-timezone-is-not-work/365466)
**Author:** [@Magali\_Romero](https://discuss.elastic.co/u/Magali_Romero)\ **Replies:** 2\ **Last updated:** [August 23, 2024, 3:26pm UTC](https://discuss.elastic.co/t/logstash-timezone-is-not-work/365466 "2024-08-23T15:26:09Z")
Hi, I have this configuration in logstash.conf, however the alerts continue to arrive in UTC filter { if "his-vm" in \[tags\] { # Agregar campos debug\_alert\_id y formatted\_timestamp mutate { add\_field =\> { "debug\_alert… --- ## [Logstash pulling in Date as a Timestamp](https://discuss.elastic.co/t/logstash-pulling-in-date-as-a-timestamp/365329)
**Author:** [@Vaesive](https://discuss.elastic.co/u/Vaesive)\ **Replies:** 8\ **Last updated:** [August 23, 2024, 3:01pm UTC](https://discuss.elastic.co/t/logstash-pulling-in-date-as-a-timestamp/365329 "2024-08-23T15:01:26Z")
I'm losing my mind here so I'm hoping someone can point out where I'm being dumb. My DB view has a date field and the dates are formatted "yyyy-MM-dd" (2023-08-15). However, when ingesting the data from the view with Lo… --- ## [Overwrite index when the new input file is discovered](https://discuss.elastic.co/t/overwrite-index-when-the-new-input-file-is-discovered/365438)
**Author:** [@chun](https://discuss.elastic.co/u/chun)\ **Replies:** 1\ **Last updated:** [August 23, 2024, 8:29am UTC](https://discuss.elastic.co/t/overwrite-index-when-the-new-input-file-is-discovered/365438 "2024-08-23T08:29:41Z")
I have a logstash configuration below reading a large jason file dumped by seperate python code. The ask is to refresh the index with the new data on daily basis. How to clear the old data from index before reading the n… --- ## [Authentication Failure 400 Logstash - Salesforce input plugin](https://discuss.elastic.co/t/authentication-failure-400-logstash-salesforce-input-plugin/365426)
**Author:** [@mosaadshaikh1998](https://discuss.elastic.co/u/mosaadshaikh1998)\ **Replies:** 0\ **Last updated:** [August 23, 2024, 6:34am UTC](https://discuss.elastic.co/t/authentication-failure-400-logstash-salesforce-input-plugin/365426 "2024-08-23T06:34:20Z")
Hi @Samuele\_Lolli, @Badger, @leandrojmp, @carly.richmond I am trying to use the salesforce input plugin in one of the pipeline on my logstash server. However I am getting an error saying 'Authentication Failure 400'. L… --- ## [Logstash has problem charset encoding non utf-8](https://discuss.elastic.co/t/logstash-has-problem-charset-encoding-non-utf-8/365418)
**Author:** [@sangwan1219](https://discuss.elastic.co/u/sangwan1219)\ **Replies:** 0\ **Last updated:** [August 22, 2024, 11:56pm UTC](https://discuss.elastic.co/t/logstash-has-problem-charset-encoding-non-utf-8/365418 "2024-08-22T23:56:59Z")
I use kafka - ELK stack for gathering logs. all the servers in our company use EUC-KR encoding. when the logstash input sets euc-kr and output stdout plain charset =\> "EUC-KR" (the logstash provides euc-kr in plain pl… --- ## [Not a valid Logstash keystore java exception problem](https://discuss.elastic.co/t/not-a-valid-logstash-keystore-java-exception-problem/365362)
**Author:** [@Abdarrahmane](https://discuss.elastic.co/u/Abdarrahmane)\ **Replies:** 3\ **Last updated:** [August 22, 2024, 4:34pm UTC](https://discuss.elastic.co/t/not-a-valid-logstash-keystore-java-exception-problem/365362 "2024-08-22T16:34:53Z")
hello i encrypted the username and password of my elasticsearch and call them as variables in the pipeline.conf but I wasn't able to get it to work properly here is the guide I followed: Secrets keystore for secure s… --- ## [Logstash security best practices: keystore, monitoring, ssl\_verification\_mode](https://discuss.elastic.co/t/logstash-security-best-practices-keystore-monitoring-ssl-verification-mode/365166)
**Author:** [@Abdarrahmane](https://discuss.elastic.co/u/Abdarrahmane)\ **Replies:** 2\ **Last updated:** [August 22, 2024, 3:58pm UTC](https://discuss.elastic.co/t/logstash-security-best-practices-keystore-monitoring-ssl-verification-mode/365166 "2024-08-22T15:58:30Z")
hello ! can you guys please guide me to the best security practices to secure the communication between Logstash and elasticsearch (logstash configuration (logstash.yml) and its pipelines in the conf.d) here is my curr… --- ## [Logstash input plugin with domain-joined service account](https://discuss.elastic.co/t/logstash-input-plugin-with-domain-joined-service-account/365338)
**Author:** [@Rakesh\_Mukherjee](https://discuss.elastic.co/u/Rakesh_Mukherjee)\ **Replies:** 2\ **Last updated:** [August 22, 2024, 3:44pm UTC](https://discuss.elastic.co/t/logstash-input-plugin-with-domain-joined-service-account/365338 "2024-08-22T15:44:07Z")
I'm using Logstash to pull logs from MySQL Database server. In the Logstash configuration file, I'm using a domain-joined service account for the DB server, but I'm encountering an authentication error in the log message… --- ## [Data is not coming through in Elastic](https://discuss.elastic.co/t/data-is-not-coming-through-in-elastic/365296)
**Author:** [@alexvp](https://discuss.elastic.co/u/alexvp)\ **Replies:** 10\ **Last updated:** [August 22, 2024, 9:22am UTC](https://discuss.elastic.co/t/data-is-not-coming-through-in-elastic/365296 "2024-08-22T09:22:51Z")
Hello, I am running the Elastic stack in docker desktop. This if the flow my data goes through : Filebeats -\> Logstash -\> Elastic However, when logstash is sending data to elastic I dont see any indexes being created… --- ## [Issue with regular expression](https://discuss.elastic.co/t/issue-with-regular-expression/365313)
**Author:** [@UMUT\_CAN\_ARGUN](https://discuss.elastic.co/u/UMUT_CAN_ARGUN)\ **Replies:** 2\ **Last updated:** [August 22, 2024, 5:37am UTC](https://discuss.elastic.co/t/issue-with-regular-expression/365313 "2024-08-22T05:37:30Z")
My regex for the sample log line looks line this: As you see, the endpoint section doesn’t match for “usr/admin/developer” although what I write in my regex. What is my mistake? --- ## [Convert string to array for new and old data in an index](https://discuss.elastic.co/t/convert-string-to-array-for-new-and-old-data-in-an-index/365242)
**Author:** [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\ **Replies:** 1\ **Last updated:** [August 21, 2024, 4:05pm UTC](https://discuss.elastic.co/t/convert-string-to-array-for-new-and-old-data-in-an-index/365242 "2024-08-21T16:05:17Z")
Our current Audit log entries contain a field called Ipaddress, which is a string IP address. Some services send this value as a single IP address while some services are sending it as an array with multiple comma sepa… --- ## [Logstash Kafka output fails on TLS post-handshake messags](https://discuss.elastic.co/t/logstash-kafka-output-fails-on-tls-post-handshake-messags/365297)
**Author:** [@natharran](https://discuss.elastic.co/u/natharran)\ **Replies:** 0\ **Last updated:** [August 21, 2024, 4:04pm UTC](https://discuss.elastic.co/t/logstash-kafka-output-fails-on-tls-post-handshake-messags/365297 "2024-08-21T16:04:30Z")
Hello, I have Logstash and Kafka deployed in OpenShift cluster via operators (ECK for Logstash and AMQ streams for Kafka) Logstash sends messages to Kafka without issue when unencrypted. To enable mTLS, I have custom k… --- ## [Fluentd is not able to send json logs to logstash](https://discuss.elastic.co/t/fluentd-is-not-able-to-send-json-logs-to-logstash/365264)
**Author:** [@upreddy](https://discuss.elastic.co/u/upreddy)\ **Replies:** 1\ **Last updated:** [August 21, 2024, 12:55pm UTC](https://discuss.elastic.co/t/fluentd-is-not-able-to-send-json-logs-to-logstash/365264 "2024-08-21T12:55:23Z")
Hi, I am trying to send json logs from application pods to logstash through fluentd agent. But i am getting one error and unable to send json logs to logstash. We mentioned the file extension in fluentd config is .json … --- ## [Logstash keystore key alias selection](https://discuss.elastic.co/t/logstash-keystore-key-alias-selection/365279)
**Author:** [@natharran](https://discuss.elastic.co/u/natharran)\ **Replies:** 0\ **Last updated:** [August 21, 2024, 12:29pm UTC](https://discuss.elastic.co/t/logstash-keystore-key-alias-selection/365279 "2024-08-21T12:29:36Z")
Hello, I'm trying to configure Kafka output module for Logstash. I have my PKCS12 keystore and truststore prepared with proper key and certs added, but I wonder - if there are more keys in the keystore, how does Logstas… --- ## [Continuous data read using Elasticsearch input plugin in logstash](https://discuss.elastic.co/t/continuous-data-read-using-elasticsearch-input-plugin-in-logstash/365198)
**Author:** [@venkatkumar229](https://discuss.elastic.co/u/venkatkumar229)\ **Replies:** 2\ **Last updated:** [August 21, 2024, 11:15am UTC](https://discuss.elastic.co/t/continuous-data-read-using-elasticsearch-input-plugin-in-logstash/365198 "2024-08-21T11:15:45Z")
Hi Team, I have a requirement where i need to read the data from an elasticsearch index(datastream) and perform "split" operation to split the array records into individual documents and store into a destination index(d… --- ## [Inserting text file containing key-value pairs into Elastic Search using Logstash](https://discuss.elastic.co/t/inserting-text-file-containing-key-value-pairs-into-elastic-search-using-logstash/365240)
**Author:** [@karankamat\_21](https://discuss.elastic.co/u/karankamat_21)\ **Replies:** 3\ **Last updated:** [August 21, 2024, 10:58am UTC](https://discuss.elastic.co/t/inserting-text-file-containing-key-value-pairs-into-elastic-search-using-logstash/365240 "2024-08-21T10:58:44Z")
Hello All, I'm relatively new to Elastic Stack. I wanted to understand how I can insert each record(line) as an individual document into Elastic Search from my text file using Logstash. Below is a sample content of my … --- ## [Logstash processing of files in subfolder's not working when having a large amount of subfolders](https://discuss.elastic.co/t/logstash-processing-of-files-in-subfolders-not-working-when-having-a-large-amount-of-subfolders/365266)
**Author:** [@Maretti](https://discuss.elastic.co/u/Maretti)\ **Replies:** 0\ **Last updated:** [August 21, 2024, 10:01am UTC](https://discuss.elastic.co/t/logstash-processing-of-files-in-subfolders-not-working-when-having-a-large-amount-of-subfolders/365266 "2024-08-21T10:01:22Z")
Hi Context: I am processing a dataset that contains a lot of subfolder's (2359 in total), when I run my logstash config on for example 5 subfolder's everything is getting processed correctly but on the full dataset this… --- ## [How to remove extra characters by using grok pattern in logstash](https://discuss.elastic.co/t/how-to-remove-extra-characters-by-using-grok-pattern-in-logstash/365187)
**Author:** [@upreddy](https://discuss.elastic.co/u/upreddy)\ **Replies:** 3\ **Last updated:** [August 20, 2024, 3:02pm UTC](https://discuss.elastic.co/t/how-to-remove-extra-characters-by-using-grok-pattern-in-logstash/365187 "2024-08-20T15:02:41Z")
Hi I trying to write grok pattern to remove some extra characters in between the message by using grok pattern. after removing these extra characters i want to apply json filter. I am sharing sample logs below. 2024-08… --- ## [Logs Received by Logstash Have Zero Length](https://discuss.elastic.co/t/logs-received-by-logstash-have-zero-length/364361)
**Author:** [@Manal\_A](https://discuss.elastic.co/u/Manal_A)\ **Replies:** 1\ **Last updated:** [August 19, 2024, 7:11pm UTC](https://discuss.elastic.co/t/logs-received-by-logstash-have-zero-length/364361 "2024-08-19T19:11:40Z")
Hello, I have an issue with the logs received by my Logstash. Their length is 0. I don't know why. Could it be related to Logstash? Thank you --- ## [Logstash Sending older logs](https://discuss.elastic.co/t/logstash-sending-older-logs/365019)
**Author:** [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Replies:** 5\ **Last updated:** [August 19, 2024, 2:44pm UTC](https://discuss.elastic.co/t/logstash-sending-older-logs/365019 "2024-08-19T14:44:27Z")
Logstash is sending older messages. Below is my pipeline. input { file { path =\> "C:/Program Files (x86)/dataops/ag/web/Logs/webSvr\*" type =\> "webSvr" start\_position =\> "beginning" } file { path =\> … --- ## [Not able to install Salesforce input plugin on Windows](https://discuss.elastic.co/t/not-able-to-install-salesforce-input-plugin-on-windows/365117)
**Author:** [@mosaadshaikh1998](https://discuss.elastic.co/u/mosaadshaikh1998)\ **Replies:** 0\ **Last updated:** [August 19, 2024, 8:13am UTC](https://discuss.elastic.co/t/not-able-to-install-salesforce-input-plugin-on-windows/365117 "2024-08-19T08:13:14Z")
Hi, I am trying to install the salesforce input plugin on the windows server 2019 Datacenter. Below is the command I am trying to run which i found in documentation: bin/logstash-plugin install logstash-input-salesfor… --- ## [Two logstash http\_poller (API query) configs are outputting documents to each other's indices](https://discuss.elastic.co/t/two-logstash-http-poller-api-query-configs-are-outputting-documents-to-each-others-indices/365066)
**Author:** [@taniumalloy](https://discuss.elastic.co/u/taniumalloy)\ **Replies:** 3\ **Last updated:** [August 17, 2024, 2:44pm UTC](https://discuss.elastic.co/t/two-logstash-http-poller-api-query-configs-are-outputting-documents-to-each-others-indices/365066 "2024-08-17T14:44:25Z")
I have a unique issue in Logstash where my data is not being routed to the correct index name appropriatly. I have two conf.d config files that each use http\_poller as input source and are gathering data from seperate AP… --- ## [Collecting logs from filebeats at k8s to logstash, can't filter fields](https://discuss.elastic.co/t/collecting-logs-from-filebeats-at-k8s-to-logstash-cant-filter-fields/365038)
**Author:** [@atchaikovski](https://discuss.elastic.co/u/atchaikovski)\ **Replies:** 5\ **Last updated:** [August 16, 2024, 4:44pm UTC](https://discuss.elastic.co/t/collecting-logs-from-filebeats-at-k8s-to-logstash-cant-filter-fields/365038 "2024-08-16T16:44:59Z")
hi there!! sorry i'm very new to this technology. ELK got installed at k8s cluster, working fine. from beats to logstash i get these things (see code below). which i'm happy about. but i need to get rid of several fields… --- ## [Can't modify the @timestamp field via logstash](https://discuss.elastic.co/t/cant-modify-the-timestamp-field-via-logstash/364009)
**Author:** [@Gur-Sin](https://discuss.elastic.co/u/Gur-Sin)\ **Replies:** 2\ **Last updated:** [August 16, 2024, 7:03am UTC](https://discuss.elastic.co/t/cant-modify-the-timestamp-field-via-logstash/364009 "2024-08-16T07:03:57Z")
Hi, I recently had to migrate data from one index to another and wanted to maintain the original timestamps as in the original index. I have used the date plugin to achieve the same and it seems to work when I pipe the … --- ## [Logstash: configure multiline configuration for specific type of logs only](https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700)
**Author:** [@pix9](https://discuss.elastic.co/u/pix9)\ **Replies:** 6\ **Last updated:** [August 9, 2024, 6:32pm UTC](https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700 "2024-08-09T18:32:44Z")
Hey Folks, We've an existing Logstash configuration where we use fields "logs\_type" to separate the different types of logs and parse them with their respective grok patterns into multiple respective indexes. Now we a… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=21) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=23)