# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=25 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 26 --- ## [Cannot send e-mail notifications SMTPAuthenticationError all the time](https://discuss.elastic.co/t/cannot-send-e-mail-notifications-smtpauthenticationerror-all-the-time/363431)
**Author:** [@Sainar\_Vekinel](https://discuss.elastic.co/u/Sainar_Vekinel)\ **Replies:** 0\ **Last updated:** [July 19, 2024, 12:10pm UTC](https://discuss.elastic.co/t/cannot-send-e-mail-notifications-smtpauthenticationerror-all-the-time/363431 "2024-07-19T12:10:03Z")
Hey guys i don't know if this problem already discussed? but i have the problem with email notification in logstash here is the error: \[2024-07-19T13:37:58,836\]\[ERROR\]\[logstash.outputs.email \]\[main\]\[aa5c77b7a64207e316… --- ## [How to Remove Prefixes and Ensure Parsing Integrity When Forwarding Logs to QRadar Using Logstash](https://discuss.elastic.co/t/how-to-remove-prefixes-and-ensure-parsing-integrity-when-forwarding-logs-to-qradar-using-logstash/363385)
**Author:** [@wangsubo](https://discuss.elastic.co/u/wangsubo)\ **Replies:** 2\ **Last updated:** [July 19, 2024, 6:34am UTC](https://discuss.elastic.co/t/how-to-remove-prefixes-and-ensure-parsing-integrity-when-forwarding-logs-to-qradar-using-logstash/363385 "2024-07-19T06:34:41Z")
I have two questions that need answers: When forwarding logs from Logstash to QRadar, the logs come with a Logstash prefix. How can we remove this prefix? Using Elastic-Agent to send logs to Elasticsearch allows au… --- ## [Logstash related nested json help](https://discuss.elastic.co/t/logstash-related-nested-json-help/363382)
**Author:** [@jgisler](https://discuss.elastic.co/u/jgisler)\ **Replies:** 6\ **Last updated:** [July 19, 2024, 5:27am UTC](https://discuss.elastic.co/t/logstash-related-nested-json-help/363382 "2024-07-19T05:27:06Z")
Hello Guru's, I have never ingested nested json before and I'm stuck. Log examples: {"system":"aa","logLevel":"\[INFO\]","log":{"0":"TT000000 joining provisioningEvent and will publish topics: provisioningEvent"}} {"sys… --- ## [Stream log4j2 logs to logstash hosted on an azure VM via tcp](https://discuss.elastic.co/t/stream-log4j2-logs-to-logstash-hosted-on-an-azure-vm-via-tcp/363377)
**Author:** [@asandhu](https://discuss.elastic.co/u/asandhu)\ **Replies:** 3\ **Last updated:** [July 18, 2024, 11:38pm UTC](https://discuss.elastic.co/t/stream-log4j2-logs-to-logstash-hosted-on-an-azure-vm-via-tcp/363377 "2024-07-18T23:38:56Z")
Hi All I'm a bit new to the ELK stack so greatly appreciate any help I have a mulesoft application on my local machine (looking to host the mule application on anypoint platform later) whose logs I am trying to send to… --- ## [Splitting Records With Start and End Times Into Separate Documents With A Timestamp Every X Minutes](https://discuss.elastic.co/t/splitting-records-with-start-and-end-times-into-separate-documents-with-a-timestamp-every-x-minutes/360997)
**Author:** [@nickbarry](https://discuss.elastic.co/u/nickbarry)\ **Replies:** 6\ **Last updated:** [July 18, 2024, 3:59pm UTC](https://discuss.elastic.co/t/splitting-records-with-start-and-end-times-into-separate-documents-with-a-timestamp-every-x-minutes/360997 "2024-07-18T15:59:26Z")
I'm currently attempting to split a single activity SQL record with a start and end time into multiple documents in an index. Each document will have a single timestamp (and duplicated data fields) and each timestamp is… --- ## [Delete Old Log elastic search](https://discuss.elastic.co/t/delete-old-log-elastic-search/363314)
**Author:** [@Agaaam](https://discuss.elastic.co/u/Agaaam)\ **Replies:** 4\ **Last updated:** [July 18, 2024, 12:56pm UTC](https://discuss.elastic.co/t/delete-old-log-elastic-search/363314 "2024-07-18T12:56:16Z")
Hallo, Currently im using Elastic Kibana Filebeat and logstash to colellect log accross server and docker , and suddenly the drive got too bigs and take so much disk space, what can i do to delete 3 old month logs ? t… --- ## [Update the json input file in Logstash](https://discuss.elastic.co/t/update-the-json-input-file-in-logstash/363331)
**Author:** [@mario\_kazela](https://discuss.elastic.co/u/mario_kazela)\ **Replies:** 0\ **Last updated:** [July 18, 2024, 8:18am UTC](https://discuss.elastic.co/t/update-the-json-input-file-in-logstash/363331 "2024-07-18T08:18:34Z")
Hi, I just want to asking, I has a json file which contains of several fields likes: { "ID": 12345 "title": "Example Post", "views\_count": 881, "author": "John Doe", "content": "This is an example post." } … --- ## [Kafka output in Logstash with authentification password and username](https://discuss.elastic.co/t/kafka-output-in-logstash-with-authentification-password-and-username/363285)
**Author:** [@Leo23](https://discuss.elastic.co/u/Leo23)\ **Replies:** 1\ **Last updated:** [July 17, 2024, 3:20pm UTC](https://discuss.elastic.co/t/kafka-output-in-logstash-with-authentification-password-and-username/363285 "2024-07-17T15:20:45Z")
I see that we can specify username and password in the output Kafka of Filebeat but these fields are not present in the Kafka output of Logstash. What can I do to specify username and password ? The exemple of input in … --- ## [Configuring "consumer\_threads" for a Kafka input with multiple Logstash instances](https://discuss.elastic.co/t/configuring-consumer-threads-for-a-kafka-input-with-multiple-logstash-instances/363279)
**Author:** [@mmfsilva](https://discuss.elastic.co/u/mmfsilva)\ **Replies:** 4\ **Last updated:** [July 17, 2024, 3:08pm UTC](https://discuss.elastic.co/t/configuring-consumer-threads-for-a-kafka-input-with-multiple-logstash-instances/363279 "2024-07-17T15:08:28Z")
I'm analysing the way a logging stack has been configured and am facing a particular issue. We have 2 Logstash instances. In those two instances, we have two pipelines which are consuming from Kafka. See the input secti… --- ## [Elasticsearch huge amount of duplicate with logstash](https://discuss.elastic.co/t/elasticsearch-huge-amount-of-duplicate-with-logstash/362085)
**Author:** [@AVMOps](https://discuss.elastic.co/u/AVMOps)\ **Replies:** 1\ **Last updated:** [July 15, 2024, 7:51am UTC](https://discuss.elastic.co/t/elasticsearch-huge-amount-of-duplicate-with-logstash/362085 "2024-07-15T07:51:18Z")
Hello, I'm facing a duplicate data issue with Elasticsearch (3 nodes v8.5.2 - green state), coupled with Logstash (1 node v 8.5.2). So basically we have multiple apps servers sending logs with NLog to Logstash on port … --- ## [Connect to AZURE SQL PAAS DB is getting failed](https://discuss.elastic.co/t/connect-to-azure-sql-paas-db-is-getting-failed/363033)
**Author:** [@krishna58](https://discuss.elastic.co/u/krishna58)\ **Replies:** 6\ **Last updated:** [July 17, 2024, 12:10pm UTC](https://discuss.elastic.co/t/connect-to-azure-sql-paas-db-is-getting-failed/363033 "2024-07-17T12:10:03Z")
Hi I am using the below script to connect to AZURE SQL PAAS DB from the logstash script. Service principal is created and added the RBAC role for both Service principal and SQL PAAS sever as a reader role. Please help on… --- ## [Logstash JDBC input plugin skips few scheduled runs](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-skips-few-scheduled-runs/363021)
**Author:** [@forabraham1](https://discuss.elastic.co/u/forabraham1)\ **Replies:** 6\ **Last updated:** [July 17, 2024, 7:24am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-skips-few-scheduled-runs/363021 "2024-07-17T07:24:37Z")
Hi All, We're using JDBC input plugin to load data from Oracle 19c DB. Totally 55 separate config files ( each per client ) . Each file having 4 different sections for JDBC input, scheduled to run every 10 secs. Some t… --- ## [Sending messages to Microsoft Teams](https://discuss.elastic.co/t/sending-messages-to-microsoft-teams/363214)
**Author:** [@Ryan5](https://discuss.elastic.co/u/Ryan5)\ **Replies:** 1\ **Last updated:** [July 16, 2024, 12:37pm UTC](https://discuss.elastic.co/t/sending-messages-to-microsoft-teams/363214 "2024-07-16T12:37:05Z")
With the new changes to webhooks Microsoft suddenly announced \[1\] what would be the best way to continue to send messages from Logstash into Teams? Before we had a small online payload but after their new changes the pa… --- ## [Logstash configurtaion for convert to Number](https://discuss.elastic.co/t/logstash-configurtaion-for-convert-to-number/363194)
**Author:** [@Shalinicts](https://discuss.elastic.co/u/Shalinicts)\ **Replies:** 5\ **Last updated:** [July 16, 2024, 10:30am UTC](https://discuss.elastic.co/t/logstash-configurtaion-for-convert-to-number/363194 "2024-07-16T10:30:28Z")
Hi , We are trying a configuration to convert the message field (which has number coming as Text) to integer as we have to create some line graphs based on the number . We tried below configuration but its converting on… --- ## [Logstash 7.7 can’t establish pipeline with ssl](https://discuss.elastic.co/t/logstash-7-7-can-t-establish-pipeline-with-ssl/363026)
**Author:** [@Ankit5](https://discuss.elastic.co/u/Ankit5)\ **Replies:** 2\ **Last updated:** [July 15, 2024, 5:27pm UTC](https://discuss.elastic.co/t/logstash-7-7-can-t-establish-pipeline-with-ssl/363026 "2024-07-15T17:27:06Z")
Hi Everyone, So I have been trying to push data from logstash to Elastic using ssl. I am facing the issue when I am doing this thing on Logstash which is running on 7.7 and When I am doing same configuration on Logstash… --- ## [Issue with BufferOverflowException from logstash agent](https://discuss.elastic.co/t/issue-with-bufferoverflowexception-from-logstash-agent/363156)
**Author:** [@Tachion1337](https://discuss.elastic.co/u/Tachion1337)\ **Replies:** 0\ **Last updated:** [July 15, 2024, 2:04pm UTC](https://discuss.elastic.co/t/issue-with-bufferoverflowexception-from-logstash-agent/363156 "2024-07-15T14:04:23Z")
Recently during logstash pod startup I've started getting the error trace below. Could you give me a hint where to look for a solution? ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::Pipe… --- ## [Logstash JDBC input cannot determine timezone from nil](https://discuss.elastic.co/t/logstash-jdbc-input-cannot-determine-timezone-from-nil/362996)
**Author:** [@chun](https://discuss.elastic.co/u/chun)\ **Replies:** 4\ **Last updated:** [July 15, 2024, 8:55am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-cannot-determine-timezone-from-nil/362996 "2024-07-15T08:55:07Z")
I'm getting an error while running logstash " (ArgumentError) Cannot determine timezone from nil\\n(secs:1720605029.799,utc~:"2024-07-10 09:50:29.7990000247955322",ltz~:nil)" I have tried the solution from other threads… --- ## [Convert embede xml in log to json in logstash](https://discuss.elastic.co/t/convert-embede-xml-in-log-to-json-in-logstash/363094)
**Author:** [@alex\_petrov](https://discuss.elastic.co/u/alex_petrov)\ **Replies:** 5\ **Last updated:** [July 14, 2024, 1:09pm UTC](https://discuss.elastic.co/t/convert-embede-xml-in-log-to-json-in-logstash/363094 "2024-07-14T13:09:25Z")
logs sent from filebeat to logstash and I write a pattern to parse them.developers added logs that have embeded xml inside logs and here is a sample : 2024-07-08 15:18:35,608 INFO |body=\ **Author:** [@Fdsm](https://discuss.elastic.co/u/Fdsm)\ **Replies:** 0\ **Last updated:** [July 12, 2024, 4:30am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-sometimes-does-not-work/362992 "2024-07-12T04:30:34Z") I have some integration logs coming from filebeat and I'm trying to use the aggregate filter to calculate the delta of the entire process. The code implemented seems to work properly, but looking at kibana some records a… --- ## [Editing dynamic field name in logstash](https://discuss.elastic.co/t/editing-dynamic-field-name-in-logstash/362843)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 1\ **Last updated:** [July 11, 2024, 9:22pm UTC](https://discuss.elastic.co/t/editing-dynamic-field-name-in-logstash/362843 "2024-07-11T21:22:07Z")
hi, I have a log in which each line has a field with different name like below. is it possible to change all of these field's names to just "day"? \<123\>day=.... ===\> day=.... \<678\>day=.... ===\> day=.... --- ## [Changing the key of kv outout](https://discuss.elastic.co/t/changing-the-key-of-kv-outout/362932)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 1\ **Last updated:** [July 11, 2024, 12:39pm UTC](https://discuss.elastic.co/t/changing-the-key-of-kv-outout/362932 "2024-07-11T12:39:52Z")
hi , I have below kv filter in my logstash conf file kv { field\_split =\> " " value\_split =\> "="" } and my data is as bellow: \<12\>d=2024-07 T=12:13 P=h c=12 \<13\>d=2024-07 T=12:15 P=l c=17 \<102\>d=2024… --- ## [Access to nested fields' values in logstash](https://discuss.elastic.co/t/access-to-nested-fields-values-in-logstash/362845)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 3\ **Last updated:** [July 11, 2024, 10:04am UTC](https://discuss.elastic.co/t/access-to-nested-fields-values-in-logstash/362845 "2024-07-11T10:04:31Z")
Hi, I have these fields and values in my log in differnet lines:: line 1: date ==\> "z\_ac" z\_ac ==\> "2024-07" line 2: date ==\> "z\_bc" z\_bc ==\> "2024-07" line 3: date ==\> "z\_dc" z\_dc ==\> "2024-07" ...............… --- ## [Isolation level](https://discuss.elastic.co/t/isolation-level/362818)
**Author:** [@Raul\_Uria](https://discuss.elastic.co/u/Raul_Uria)\ **Replies:** 2\ **Last updated:** [July 10, 2024, 8:40pm UTC](https://discuss.elastic.co/t/isolation-level/362818 "2024-07-10T20:40:54Z")
Hi, I´m trying to execute my queries against sql server using "read uncommitted" or "snapshot" isolation level cause I don´t want to lock others while long queries. I think there is no other way to do that but insert it… --- ## [Read huge elastic index with logstash](https://discuss.elastic.co/t/read-huge-elastic-index-with-logstash/360646)
**Author:** [@gueri](https://discuss.elastic.co/u/gueri)\ **Replies:** 1\ **Last updated:** [July 10, 2024, 4:04pm UTC](https://discuss.elastic.co/t/read-huge-elastic-index-with-logstash/360646 "2024-07-10T16:04:58Z")
Hi, I'm running Logstash 8.7 and reading from Elasticsearch 8.13. I want to retrieve all docs from an index with 18 millions logs and copy to Kafka. I thought Logstash was capable of doing that with the elasticsearch … --- ## [Logstash exec ruby script in docker](https://discuss.elastic.co/t/logstash-exec-ruby-script-in-docker/362895)
**Author:** [@Maeris](https://discuss.elastic.co/u/Maeris)\ **Replies:** 0\ **Last updated:** [July 10, 2024, 3:11pm UTC](https://discuss.elastic.co/t/logstash-exec-ruby-script-in-docker/362895 "2024-07-10T15:11:18Z")
Hi, I'm trying to execute a ruby script which prints json data to stdout. I have tried different versions, and right now testing with the most basic script: require 'json' json\_data = \[ { "uuid" =\> "noab14123dws11", … --- ## [Support for dynamic values in table\_prefix and date\_pattern in the BigQuery Logstash output plugin](https://discuss.elastic.co/t/support-for-dynamic-values-in-table-prefix-and-date-pattern-in-the-bigquery-logstash-output-plugin/362885)
**Author:** [@Deena\_Dayalan](https://discuss.elastic.co/u/Deena_Dayalan)\ **Replies:** 2\ **Last updated:** [July 10, 2024, 2:26pm UTC](https://discuss.elastic.co/t/support-for-dynamic-values-in-table-prefix-and-date-pattern-in-the-bigquery-logstash-output-plugin/362885 "2024-07-10T14:26:55Z")
Im using kafka and BigQuery as input and output plugins. Events coming from kafka are stored in BigQuery. But I want to attach the table\_prefix with some field in the events. For example, if the event contains "userId":… --- ## [Date Parse Logstash](https://discuss.elastic.co/t/date-parse-logstash/358533)
**Author:** [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)\ **Replies:** 2\ **Last updated:** [July 10, 2024, 12:25pm UTC](https://discuss.elastic.co/t/date-parse-logstash/358533 "2024-07-10T12:25:17Z")
Hi Team, I have a field request time whose value is as below: "response\_time" =\> "Wed, 17 Apr 2024 21:36:13 GMT". I need to fetch it in the yyyy-mm-dd format? How to do so? Kindly assist. //Ankita --- ## [Integration of Logstash with IBM QRadar](https://discuss.elastic.co/t/integration-of-logstash-with-ibm-qradar/362877)
**Author:** [@Rafey\_Zafar](https://discuss.elastic.co/u/Rafey_Zafar)\ **Replies:** 0\ **Last updated:** [July 10, 2024, 11:26am UTC](https://discuss.elastic.co/t/integration-of-logstash-with-ibm-qradar/362877 "2024-07-10T11:26:12Z")
I need to integrate Logstash with QRadar. While I am familiar with the configuration process on the QRadar end, I am new to setting up the integration on the Logstash side using the Syslog protocol. I am seeking detailed… --- ## [How can logs be forwarded from a Windows server (AD) to a centralized log server using Filebeat and Logstash?](https://discuss.elastic.co/t/how-can-logs-be-forwarded-from-a-windows-server-ad-to-a-centralized-log-server-using-filebeat-and-logstash/362830)
**Author:** [@Sehran\_Rasool](https://discuss.elastic.co/u/Sehran_Rasool)\ **Replies:** 0\ **Last updated:** [July 10, 2024, 12:39am UTC](https://discuss.elastic.co/t/how-can-logs-be-forwarded-from-a-windows-server-ad-to-a-centralized-log-server-using-filebeat-and-logstash/362830 "2024-07-10T00:39:37Z")
I'm seeking assistance with forwarding Windows Server Active Directory logs to my centralized Linux-based log server. I've set up Filebeat on the Windows Server and Logstash on the log server (linux). Despite configuring… --- ## [Logstash Mutate Arrays](https://discuss.elastic.co/t/logstash-mutate-arrays/362813)
**Author:** [@erikg](https://discuss.elastic.co/u/erikg)\ **Replies:** 1\ **Last updated:** [July 9, 2024, 5:42pm UTC](https://discuss.elastic.co/t/logstash-mutate-arrays/362813 "2024-07-09T17:42:07Z")
Hello, I have a simple question on arrays coming into Logstash. Lets say I have this example: Field Value data.root \[true, false\] This is expected behavior. What I want to do: Copy the first value of … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=24) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=26)