# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=91 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 92 --- ## [How to fetch out the array values](https://discuss.elastic.co/t/how-to-fetch-out-the-array-values/325001)
**Author:** [@prashant1](https://discuss.elastic.co/u/prashant1)\ **Replies:** 6\ **Last updated:** [February 8, 2023, 6:01pm UTC](https://discuss.elastic.co/t/how-to-fetch-out-the-array-values/325001 "2023-02-08T18:01:49Z")
Hi, I have below sample data { "logtype":"demo" ,"operation":"demoStatus", "Stats":\[ { "apiName":"a", "failedCount":0, "successCount":0 }, { "apiName":"b", "failedCount":0, "… --- ## [Is there is alternative for HTTP plugin in ECK version?](https://discuss.elastic.co/t/is-there-is-alternative-for-http-plugin-in-eck-version/325045)
**Author:** [@kinkkong](https://discuss.elastic.co/u/kinkkong)\ **Replies:** 0\ **Last updated:** [February 8, 2023, 4:28pm UTC](https://discuss.elastic.co/t/is-there-is-alternative-for-http-plugin-in-eck-version/325045 "2023-02-08T16:28:06Z")
We are migrating our platform to Kubernetes, one of the features that we currently use in ELK is the logstash HTTP plugin. Unfortunately, this will be no the case in Kubernetes. Is there an alternative plugin, or ingest … --- ## [Recovering data from a persistent queue page file](https://discuss.elastic.co/t/recovering-data-from-a-persistent-queue-page-file/325029)
**Author:** [@peter\_west](https://discuss.elastic.co/u/peter_west)\ **Replies:** 0\ **Last updated:** [February 8, 2023, 2:10pm UTC](https://discuss.elastic.co/t/recovering-data-from-a-persistent-queue-page-file/325029 "2023-02-08T14:10:59Z")
Is there any means by which you can reprocess data from a page file into an Elasticsearch index? My instincts tell me not because the likelihood is that the page file will probably have a partial record at the start so … --- ## [How to pick up certain Logstash events so I can ignore/work on them](https://discuss.elastic.co/t/how-to-pick-up-certain-logstash-events-so-i-can-ignore-work-on-them/324935)
**Author:** [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)\ **Replies:** 4\ **Last updated:** [February 8, 2023, 12:25pm UTC](https://discuss.elastic.co/t/how-to-pick-up-certain-logstash-events-so-i-can-ignore-work-on-them/324935 "2023-02-08T12:25:00Z")
So I've been working with a Logstash pipeline which deals with creating and updating a few documents. And when the schedule hits and the creation repeats the following line pop up: \[2023-02-07T17:36:07,915\]\[WARN \]\[logst… --- ## [Dynamic Generate CSV in Logstash csv output plugin](https://discuss.elastic.co/t/dynamic-generate-csv-in-logstash-csv-output-plugin/324301)
**Author:** [@Rakhshunda\_Noorein\_J](https://discuss.elastic.co/u/Rakhshunda_Noorein_J)\ **Replies:** 2\ **Last updated:** [February 8, 2023, 6:09am UTC](https://discuss.elastic.co/t/dynamic-generate-csv-in-logstash-csv-output-plugin/324301 "2023-02-08T06:09:19Z")
Hello, I want to dynamically generate my csv file with current date. My conf---- csv { fields =\> \["id" , "name"\] path =\> "test-%{+YYYY-MM-dd}.csv" } but it is not working.. It is generating file as : test-.… --- ## [Ssl\_certificate\_authorities seems to have no effect](https://discuss.elastic.co/t/ssl-certificate-authorities-seems-to-have-no-effect/324934)
**Author:** [@hexoffender](https://discuss.elastic.co/u/hexoffender)\ **Replies:** 1\ **Last updated:** [February 7, 2023, 10:44pm UTC](https://discuss.elastic.co/t/ssl-certificate-authorities-seems-to-have-no-effect/324934 "2023-02-07T22:44:36Z")
I'm getting an error in filebeat: ERROR \[publisher\_pipeline\_output\] pipeline/output.go:154 Failed to connect to backoff(async(tcp://localhost:5044)): x509: certificate signed by unknown authority Here is my logstash co… --- ## [Version Incompatibility Logstash and Opensearch service Elastic](https://discuss.elastic.co/t/version-incompatibility-logstash-and-opensearch-service-elastic/324928)
**Author:** [@cgcats](https://discuss.elastic.co/u/cgcats)\ **Replies:** 2\ **Last updated:** [February 7, 2023, 5:34pm UTC](https://discuss.elastic.co/t/version-incompatibility-logstash-and-opensearch-service-elastic/324928 "2023-02-07T17:34:32Z")
I am trying to bring up logstash in kubernetes using the v7.16.1 chart and connecting to v7.4 elasticsearch that is an aws opensearch service. According to the compatibility matrix, this should be fine. but I am seeing t… --- ## [About the integration between elasticsearch and logstash](https://discuss.elastic.co/t/about-the-integration-between-elasticsearch-and-logstash/324845)
**Author:** [@choilee](https://discuss.elastic.co/u/choilee)\ **Replies:** 5\ **Last updated:** [February 7, 2023, 1:07pm UTC](https://discuss.elastic.co/t/about-the-integration-between-elasticsearch-and-logstash/324845 "2023-02-07T13:07:23Z")
Hi, I am creating log analizing system using logstash and elasticsearch. But I couldn't send any data to elastic from logstash. (But Delete prune fillter, then could send data) I found under this error message, but i c… --- ## [Logstash CSV output plugin not flushing to disk](https://discuss.elastic.co/t/logstash-csv-output-plugin-not-flushing-to-disk/324413)
**Author:** [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)\ **Replies:** 4\ **Last updated:** [February 7, 2023, 11:31am UTC](https://discuss.elastic.co/t/logstash-csv-output-plugin-not-flushing-to-disk/324413 "2023-02-07T11:31:37Z")
Hey Everyone, I'm having some trouble with my Logstash config for exporting Elasticsearch data to CSV after upgrading my ELK stack from 7 to 8.6. When running my exporter.conf file it just never flushes to disk. It fil… --- ## [DeadLetterQueue Configration](https://discuss.elastic.co/t/deadletterqueue-configration/324869)
**Author:** [@Venkata\_Sai\_K](https://discuss.elastic.co/u/Venkata_Sai_K)\ **Replies:** 1\ **Last updated:** [February 7, 2023, 7:55am UTC](https://discuss.elastic.co/t/deadletterqueue-configration/324869 "2023-02-07T07:55:40Z")
I have done Deadletter queue configration in one file and given to that as volume for the docker service , everything we have dockerized it and here are my files persistent-queue and dlq config pipeline.batch.size: 125 … --- ## [How to parse the wrapper logs which contains timestamp value at each line](https://discuss.elastic.co/t/how-to-parse-the-wrapper-logs-which-contains-timestamp-value-at-each-line/324341)
**Author:** [@sai7276p](https://discuss.elastic.co/u/sai7276p)\ **Replies:** 2\ **Last updated:** [February 7, 2023, 6:58am UTC](https://discuss.elastic.co/t/how-to-parse-the-wrapper-logs-which-contains-timestamp-value-at-each-line/324341 "2023-02-07T06:58:05Z")
I have created grok pattern for single line entries, but I just want to remove the timestamp and extra fields before the java stack trace lines,,, to register a full stack trace to a single field 'MSG' "(%{LOGLEVEL:leve… --- ## [Logstash not inserted data into elasticsearch](https://discuss.elastic.co/t/logstash-not-inserted-data-into-elasticsearch/324737)
**Author:** [@mohanss08](https://discuss.elastic.co/u/mohanss08)\ **Replies:** 6\ **Last updated:** [February 7, 2023, 6:45am UTC](https://discuss.elastic.co/t/logstash-not-inserted-data-into-elasticsearch/324737 "2023-02-07T06:45:24Z")
Hello Team, I had Elasticsearch, Logstash and Kibana v7.16.2 with xpack security based login enabled, Yesterday i had upgraded my ELK versions to 8.6.1 using my docker-compose file. Current problem: My logstash fetched… --- ## [Error=\>logstash Pipeline worker error :"(EACCES) Permission denied](https://discuss.elastic.co/t/error-logstash-pipeline-worker-error-eacces-permission-denied/324788)
**Author:** [@vaibhav.ubale](https://discuss.elastic.co/u/vaibhav.ubale)\ **Replies:** 2\ **Last updated:** [February 7, 2023, 6:26am UTC](https://discuss.elastic.co/t/error-logstash-pipeline-worker-error-eacces-permission-denied/324788 "2023-02-07T06:26:23Z")
Hi Team/Everyone, I am facing a pipeline error and my pipeline is terminating randomly with error=\>"(EACCES) Permission denied - /var/myrepo/devops/mytask\_watcher.csv" My logstash output conf is as below output { csv… --- ## [Logstash microsoft-sentinel-logstash-output-plugin](https://discuss.elastic.co/t/logstash-microsoft-sentinel-logstash-output-plugin/324787)
**Author:** [@shadu88](https://discuss.elastic.co/u/shadu88)\ **Replies:** 2\ **Last updated:** [February 7, 2023, 5:27am UTC](https://discuss.elastic.co/t/logstash-microsoft-sentinel-logstash-output-plugin/324787 "2023-02-07T05:27:30Z")
Hello ELKs, Hope you doing well!! has anyone tried IF ELSE condition in "microsoft-sentinel-logstash-output-plugin" output logstash plugin? I'm trying to forward the logs based on log source type to respective DCR en… --- ## [Can use variables for output influxdb db and measurement field?](https://discuss.elastic.co/t/can-use-variables-for-output-influxdb-db-and-measurement-field/324508)
**Author:** [@AlanChan](https://discuss.elastic.co/u/AlanChan)\ **Replies:** 10\ **Last updated:** [February 7, 2023, 5:27am UTC](https://discuss.elastic.co/t/can-use-variables-for-output-influxdb-db-and-measurement-field/324508 "2023-02-07T05:27:16Z")
Hi I'm wondering if a configuration like this can work or not. filter { mutate { if \[topic\] == "xxxx" { add\_field =\> { "db" =\> "test2", "measurement" =\> "access\_logs" } } else { add\_field =\> { "db… --- ## [Work with xml in logstash](https://discuss.elastic.co/t/work-with-xml-in-logstash/324784)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 1\ **Last updated:** [February 6, 2023, 7:05pm UTC](https://discuss.elastic.co/t/work-with-xml-in-logstash/324784 "2023-02-06T19:05:39Z")
Hi all, I have a xml data as below which is harvesting by filebeat and sending to logstash. \ \ \5\ \ \ \123\ \ … --- ## [I am trying to add special character in logstash config](https://discuss.elastic.co/t/i-am-trying-to-add-special-character-in-logstash-config/324785)
**Author:** [@upreddy](https://discuss.elastic.co/u/upreddy)\ **Replies:** 1\ **Last updated:** [February 6, 2023, 6:02pm UTC](https://discuss.elastic.co/t/i-am-trying-to-add-special-character-in-logstash-config/324785 "2023-02-06T18:02:23Z")
Hi I am sharing log pattern in below. 2023-02-06T10:10:42.075890912Z stdout F 2023-02-06 10:10:42.075 \[DEBUG\] - {"logtype":"INFO","request":{"operation":"XXXXXX","trackingID":"abccc","usecase":"xyz"} I am trying to pu… --- ## [Grok fiels are removed by aggregate section](https://discuss.elastic.co/t/grok-fiels-are-removed-by-aggregate-section/324798)
**Author:** [@Miriam](https://discuss.elastic.co/u/Miriam)\ **Replies:** 1\ **Last updated:** [February 6, 2023, 5:54pm UTC](https://discuss.elastic.co/t/grok-fiels-are-removed-by-aggregate-section/324798 "2023-02-06T17:54:09Z")
I have follwoing file structure: id, iduser,datetimInit, dateTimeends 0001 0001 2023-02-03 04:45:16.78 2023-02-03 04:46:16.78 0002 0001 2023-02-03 08:45:16.78 2023-02-03 08:46:16.78 0003 0002 2023-02-04 04:45:16.78 2023… --- ## [Logstash slow processing of events](https://discuss.elastic.co/t/logstash-slow-processing-of-events/324392)
**Author:** [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Replies:** 11\ **Last updated:** [February 6, 2023, 5:02pm UTC](https://discuss.elastic.co/t/logstash-slow-processing-of-events/324392 "2023-02-06T17:02:58Z")
Hello, I'm trying to process events from logstash and I'm facing issue of slow processing of events.There are around 100k records.In logstash.yml I've enabled log.level debug. So far I can observe in 2 hours around 110… --- ## [Logstash and delete of gz files](https://discuss.elastic.co/t/logstash-and-delete-of-gz-files/324514)
**Author:** [@Rhh](https://discuss.elastic.co/u/Rhh)\ **Replies:** 2\ **Last updated:** [February 6, 2023, 10:07am UTC](https://discuss.elastic.co/t/logstash-and-delete-of-gz-files/324514 "2023-02-06T10:07:58Z")
Hi I have the following my conf file ... input { file { path =\> "C:/TDS.Extra/ConsoleApp13/ConsoleApp13/bin/Debug/test.gz" sincedb\_path =\> "nul" mode =\> "read" file\_completed\_action =\> "delete" codec =\> "json" } … --- ## [Required Privilege in Microsoft SQL Server to Connect With Elastic Integration and Logstash JDBC Plugin](https://discuss.elastic.co/t/required-privilege-in-microsoft-sql-server-to-connect-with-elastic-integration-and-logstash-jdbc-plugin/324510)
**Author:** [@OmFJ](https://discuss.elastic.co/u/OmFJ)\ **Replies:** 2\ **Last updated:** [February 6, 2023, 7:13am UTC](https://discuss.elastic.co/t/required-privilege-in-microsoft-sql-server-to-connect-with-elastic-integration-and-logstash-jdbc-plugin/324510 "2023-02-06T07:13:16Z")
Hi Everyone, i would like to ask related with Microsoft SQL Integration and JDBC Plugin in logstash. as we know we need username and password to access the database with JDBC or Microsoft SQL Integration. my question wou… --- ## [Filebeat stopped working after an hour after the install](https://discuss.elastic.co/t/filebeat-stopped-working-after-an-hour-after-the-install/324538)
**Author:** [@Joshua\_John\_Consulta](https://discuss.elastic.co/u/Joshua_John_Consulta)\ **Replies:** 1\ **Last updated:** [February 6, 2023, 3:08am UTC](https://discuss.elastic.co/t/filebeat-stopped-working-after-an-hour-after-the-install/324538 "2023-02-06T03:08:44Z")
Here's the error from the terminal: × filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch. Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; preset: enabled) Activ… --- ## [Invalid FieldReference](https://discuss.elastic.co/t/invalid-fieldreference/324365)
**Author:** [@ztony](https://discuss.elastic.co/u/ztony)\ **Replies:** 2\ **Last updated:** [February 5, 2023, 3:14pm UTC](https://discuss.elastic.co/t/invalid-fieldreference/324365 "2023-02-05T15:14:19Z")
Does anyone see this "Invalid FieldReference" error? we added some mutations to the pipeline, but new field with the same error came out. see the error log below: An unexpected error occurred! {:error=\>org.logstash.Fiel… --- ## [Http filter Vs elasticsearch ouput](https://discuss.elastic.co/t/http-filter-vs-elasticsearch-ouput/324718)
**Author:** [@mostafaelsayed](https://discuss.elastic.co/u/mostafaelsayed)\ **Replies:** 8\ **Last updated:** [February 5, 2023, 10:41am UTC](https://discuss.elastic.co/t/http-filter-vs-elasticsearch-ouput/324718 "2023-02-05T10:41:10Z")
Hello, I want to capture and process failures related to Elasticsearch being down and Elasticsearch output does not offer a way to handle this so, I want to do a POC and experiment with indexing events into Elasticsearc… --- ## [Logstash crash when starting after messing the queue files](https://discuss.elastic.co/t/logstash-crash-when-starting-after-messing-the-queue-files/324708)
**Author:** [@Baygon](https://discuss.elastic.co/u/Baygon)\ **Replies:** 2\ **Last updated:** [February 5, 2023, 3:34am UTC](https://discuss.elastic.co/t/logstash-crash-when-starting-after-messing-the-queue-files/324708 "2023-02-05T03:34:10Z")
I've just upgraded Logstash minor version from 7.13 to 7.17.9, but it doesn't restart, erroring about inability to create queues (there was a forced stop of Logstash, so I assume the files are corrupted). I tried to del… --- ## [SSL certificate embedding in winlogbeat or auditbeat - Need example](https://discuss.elastic.co/t/ssl-certificate-embedding-in-winlogbeat-or-auditbeat-need-example/324720)
**Author:** [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Replies:** 0\ **Last updated:** [February 4, 2023, 11:18pm UTC](https://discuss.elastic.co/t/ssl-certificate-embedding-in-winlogbeat-or-auditbeat-need-example/324720 "2023-02-04T23:18:33Z")
I'd like to know if someone can present a working example of their Beats config for Beats to Logstash with SSL, but using the embedding the certificate in the beats config as described in Configure SSL | Winlogbeat Refer… --- ## [JDBC Plugin - Missing Converter handling for full class name=com.ibm.db2.jcc.am.dc when parsing xml datatype in DB2](https://discuss.elastic.co/t/jdbc-plugin-missing-converter-handling-for-full-class-name-com-ibm-db2-jcc-am-dc-when-parsing-xml-datatype-in-db2/324631)
**Author:** [@khannaja](https://discuss.elastic.co/u/khannaja)\ **Replies:** 2\ **Last updated:** [February 4, 2023, 6:09am UTC](https://discuss.elastic.co/t/jdbc-plugin-missing-converter-handling-for-full-class-name-com-ibm-db2-jcc-am-dc-when-parsing-xml-datatype-in-db2/324631 "2023-02-04T06:09:12Z")
New to Logstash(ver 8.6.0), need to parse xml data type from db2 database. Keep getting "Missing Converter handling for full class" error. Read through most discussion post on this subject and tried a few things withou… --- ## [Write base64 decoded field from JSON message to a file](https://discuss.elastic.co/t/write-base64-decoded-field-from-json-message-to-a-file/324505)
**Author:** [@Arinjay\_Jain](https://discuss.elastic.co/u/Arinjay_Jain)\ **Replies:** 5\ **Last updated:** [February 3, 2023, 10:28pm UTC](https://discuss.elastic.co/t/write-base64-decoded-field-from-json-message-to-a-file/324505 "2023-02-03T22:28:19Z")
Hi Team, I have the following logstash pipeline configuration. input { tcp { port =\> 5102 codec =\> json } } filter { json { source =\> "message" remove\_field =\> \[ "message" \] } … --- ## [Grokparsefailure in processing a log file](https://discuss.elastic.co/t/grokparsefailure-in-processing-a-log-file/324682)
**Author:** [@Indrajit](https://discuss.elastic.co/u/Indrajit)\ **Replies:** 5\ **Last updated:** [February 3, 2023, 7:54pm UTC](https://discuss.elastic.co/t/grokparsefailure-in-processing-a-log-file/324682 "2023-02-03T19:54:15Z")
While processing a large log file with logstash, we are getting grokparsefailure & dateparsefailure. We would like to know which line in the log file is causing the failure so that we can look into more details with gro… --- ## [JSON format Decode error](https://discuss.elastic.co/t/json-format-decode-error/324652)
**Author:** [@djrshn2346](https://discuss.elastic.co/u/djrshn2346)\ **Replies:** 3\ **Last updated:** [February 3, 2023, 7:51pm UTC](https://discuss.elastic.co/t/json-format-decode-error/324652 "2023-02-03T19:51:49Z")
I am using : input { file { id =\> "my\_lt\_log" path =\> "/logs/logtransformer.log" type =\> "log" start\_position =\> "beginning" } } filter { if \[type\] == "log" { mutate { … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=90) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=92)