# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=20 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 21 --- ## [Logstash does not appear in Elastic Monitoring - fleet provisioned agent](https://discuss.elastic.co/t/logstash-does-not-appear-in-elastic-monitoring-fleet-provisioned-agent/366795)
**Author:** [@neoaddix](https://discuss.elastic.co/u/neoaddix)\ **Replies:** 6\ **Last updated:** [September 23, 2024, 5:16pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-in-elastic-monitoring-fleet-provisioned-agent/366795 "2024-09-23T17:16:43Z")
We have a fleet provisioned agent, installed it on our logstash server, however logstash does not appear in the monitoring dashboard. We do seem to have logging and metrics data in: logstash.stack\_monitoring.node\_stats… --- ## [Process two types of logs format](https://discuss.elastic.co/t/process-two-types-of-logs-format/366021)
**Author:** [@ELK\_enjoyer](https://discuss.elastic.co/u/ELK_enjoyer)\ **Replies:** 6\ **Last updated:** [September 23, 2024, 1:46pm UTC](https://discuss.elastic.co/t/process-two-types-of-logs-format/366021 "2024-09-23T13:46:13Z")
Hi everyone! I'm using the ELK stack with Kafka to collect and analyze logs from my K8s environment. We have different log formats, mainly plaintext and JSON. How can I process them in Logstash properly? Right now, p… --- ## [Logstash output Elastic App Search plugin](https://discuss.elastic.co/t/logstash-output-elastic-app-search-plugin/366884)
**Author:** [@Sanjay\_Samanaboina](https://discuss.elastic.co/u/Sanjay_Samanaboina)\ **Replies:** 0\ **Last updated:** [September 20, 2024, 2:43pm UTC](https://discuss.elastic.co/t/logstash-output-elastic-app-search-plugin/366884 "2024-09-20T14:43:00Z")
Hi, We have an application which reads emails and store it in Elastic Search using App search Engine where it is working previously with 7.17 Logstash version which is unable to connect to App search with newer versions… --- ## [Logstash--Need to create fields from Multi line log message which not able to get](https://discuss.elastic.co/t/logstash-need-to-create-fields-from-multi-line-log-message-which-not-able-to-get/366862)
**Author:** [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Replies:** 3\ **Last updated:** [September 23, 2024, 3:42am UTC](https://discuss.elastic.co/t/logstash-need-to-create-fields-from-multi-line-log-message-which-not-able-to-get/366862 "2024-09-23T03:42:50Z")
Hello, Below is my incoming single log message: \[2024-02-14T03:29:07.962+0000\]\[118\]\[safepoint \] Safepoint "Cleanup", Time since last: 1000174402 ns, Reaching safepoint: 9291 ns, Cleanup: 83971 ns, At safepoint: 14116… --- ## [Extract JSON fields from Message](https://discuss.elastic.co/t/extract-json-fields-from-message/366832)
**Author:** [@B\_H](https://discuss.elastic.co/u/B_H)\ **Replies:** 1\ **Last updated:** [September 20, 2024, 8:02pm UTC](https://discuss.elastic.co/t/extract-json-fields-from-message/366832 "2024-09-20T20:02:00Z")
I have all my desired fields coming into logstash under the message field, including the desired message. I want to pull them up one level. I am aware there are many similar topics and I have tried various techniques fro… --- ## [Provided sincedb file is ignored](https://discuss.elastic.co/t/provided-sincedb-file-is-ignored/366534)
**Author:** [@natbronislavska](https://discuss.elastic.co/u/natbronislavska)\ **Replies:** 2\ **Last updated:** [September 20, 2024, 5:43pm UTC](https://discuss.elastic.co/t/provided-sincedb-file-is-ignored/366534 "2024-09-20T17:43:23Z")
Hello, My logstash runs on EC2 and harvest data from s3. For some reasons, the logstash host machine is sometimes being terminated and replaced with new one (same AMI). In such cases lohstash's sincedb file is lost and … --- ## [Logstash to push Filebeat logs to Datadog](https://discuss.elastic.co/t/logstash-to-push-filebeat-logs-to-datadog/366849)
**Author:** [@Kalaivani\_C](https://discuss.elastic.co/u/Kalaivani_C)\ **Replies:** 0\ **Last updated:** [September 20, 2024, 6:19am UTC](https://discuss.elastic.co/t/logstash-to-push-filebeat-logs-to-datadog/366849 "2024-09-20T06:19:58Z")
I am currently working on configuring Logstash to push Filebeat logs to Datadog. I need some clarification regarding the necessity of deploying the Datadog Agent in our cluster for this purpose. Specifically, I would li… --- ## [7.17.7 logstash nokogiri-1.12.5 CVE-2022-23437](https://discuss.elastic.co/t/7-17-7-logstash-nokogiri-1-12-5-cve-2022-23437/366843)
**Author:** [@Jerry-zjl](https://discuss.elastic.co/u/Jerry-zjl)\ **Replies:** 0\ **Last updated:** [September 20, 2024, 2:47am UTC](https://discuss.elastic.co/t/7-17-7-logstash-nokogiri-1-12-5-cve-2022-23437/366843 "2024-09-20T02:47:56Z")
ask for help , do anyone know how to fix this question,I want to update nokogiri-1.12.5 to nokogiri-1.16.7,and i changed the version int the Gemfile,then i build a logstash docker image,however it doesn\`t work --- ## [Logstash-Not able to grok parse the incoming message field properly with condition](https://discuss.elastic.co/t/logstash-not-able-to-grok-parse-the-incoming-message-field-properly-with-condition/366792)
**Author:** [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Replies:** 3\ **Last updated:** [September 19, 2024, 2:14pm UTC](https://discuss.elastic.co/t/logstash-not-able-to-grok-parse-the-incoming-message-field-properly-with-condition/366792 "2024-09-19T14:14:21Z")
Hello, Below is my incoming single log message: \[2024-02-14T03:29:07.962+0000\]\[118\]\[safepoint \] Safepoint "Cleanup", Time since last: 1000174402 ns, Reaching safepoint: 9291 ns, Cleanup: 83971 ns, At safepoint: 14116… --- ## [How to shange JSON Output so its not wrapped in a field name](https://discuss.elastic.co/t/how-to-shange-json-output-so-its-not-wrapped-in-a-field-name/366738)
**Author:** [@Wilks](https://discuss.elastic.co/u/Wilks)\ **Replies:** 2\ **Last updated:** [September 18, 2024, 2:22pm UTC](https://discuss.elastic.co/t/how-to-shange-json-output-so-its-not-wrapped-in-a-field-name/366738 "2024-09-18T14:22:58Z")
How to I output my JSON so that it looks like this \[{"value":"main.abc.com","risk":"High","type":"fqdn","notes":"suspect"}\] Its currently outputting like this {"domain\_data":\[{"value":"main.abc.com","risk":"High","typ… --- ## [Salesforce Logstash pipeline shutting down after a single execution](https://discuss.elastic.co/t/salesforce-logstash-pipeline-shutting-down-after-a-single-execution/366590)
**Author:** [@mosaadshaikh1998](https://discuss.elastic.co/u/mosaadshaikh1998)\ **Replies:** 2\ **Last updated:** [September 18, 2024, 1:14pm UTC](https://discuss.elastic.co/t/salesforce-logstash-pipeline-shutting-down-after-a-single-execution/366590 "2024-09-18T13:14:47Z")
Hi Team, I'm using the Logstash Salesforce input plugin on a Windows server to fetch Salesforce application data. When I run it from the command line, the pipeline shuts down after fetching data once. Is there a way to… --- ## [How to Fix Unstable Logstash Event Rate at High EPS?](https://discuss.elastic.co/t/how-to-fix-unstable-logstash-event-rate-at-high-eps/366707)
**Author:** [@wangsubo](https://discuss.elastic.co/u/wangsubo)\ **Replies:** 0\ **Last updated:** [September 18, 2024, 5:08am UTC](https://discuss.elastic.co/t/how-to-fix-unstable-logstash-event-rate-at-high-eps/366707 "2024-09-18T05:08:44Z")
I am using logrun.pl to send 1000 log events per second from a CSV file to Elastic-agent, which then forwards them to Logstash When I set logrun.pl to send fewer than 100 events per second, the Event Received Rate in … --- ## [Fingerprint processor allowing duplicates](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696)
**Author:** [@mgordon](https://discuss.elastic.co/u/mgordon)\ **Replies:** 5\ **Last updated:** [September 18, 2024, 1:53am UTC](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696 "2024-09-18T01:53:05Z")
I'm ingesting log files that have known duplicates, so have implemented a Fingerprint processor in the ingest pipeline and setting that to \_id to remove the duplicates, which works perfectly. However, when the index roll… --- ## [Math filter plugin](https://discuss.elastic.co/t/math-filter-plugin/366676)
**Author:** [@Pavlo\_Pylypiv](https://discuss.elastic.co/u/Pavlo_Pylypiv)\ **Replies:** 1\ **Last updated:** [September 17, 2024, 1:38pm UTC](https://discuss.elastic.co/t/math-filter-plugin/366676 "2024-09-17T13:38:29Z")
Hello! I was searching for ability to perform arithmetic operations in logstash, found old topic about it - Arithmetic Operations in logstash - where where the math plugin was mentioned, but I haven't found any documenta… --- ## [Logstash installation paused in ubuntu server](https://discuss.elastic.co/t/logstash-installation-paused-in-ubuntu-server/366671)
**Author:** [@Raquel\_N\_Souza](https://discuss.elastic.co/u/Raquel_N_Souza)\ **Replies:** 2\ **Last updated:** [September 17, 2024, 12:37pm UTC](https://discuss.elastic.co/t/logstash-installation-paused-in-ubuntu-server/366671 "2024-09-17T12:37:10Z")
Hello, I'm installing logstash on the Ubuntu server, but when I get to this point, the terminal simply stops and I don't know what to do, could anyone help me? the command I'm using: sudo bin/logstash -f /etc/logstas… --- ## [Enable tls logstash](https://discuss.elastic.co/t/enable-tls-logstash/366665)
**Author:** [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Replies:** 0\ **Last updated:** [September 17, 2024, 10:27am UTC](https://discuss.elastic.co/t/enable-tls-logstash/366665 "2024-09-17T10:27:23Z")
if \[type\] == "traffic" { elasticsearch { ecs\_compatibility =\> disabled hosts =\> \["https://\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*:443"\] index =\> "${NODE\_ROLE}-traffic-%{+YYYY.MM.dd}" ssl =\> true… --- ## [Elasticsearch huge amount of duplicate with logstash](https://discuss.elastic.co/t/elasticsearch-huge-amount-of-duplicate-with-logstash/366628)
**Author:** [@AVMOps](https://discuss.elastic.co/u/AVMOps)\ **Replies:** 0\ **Last updated:** [September 16, 2024, 2:55pm UTC](https://discuss.elastic.co/t/elasticsearch-huge-amount-of-duplicate-with-logstash/366628 "2024-09-16T14:55:06Z")
Hello, I'm facing a duplicate data issue with Elasticsearch (3 nodes v8.5.2 - green state), coupled with Logstash (1 node v 8.5.2). So basically we have multiple apps servers sending logs with NLog to Logstash on port … --- ## [Unable to get the updated value of certain columns in logstash 8.6.1 version](https://discuss.elastic.co/t/unable-to-get-the-updated-value-of-certain-columns-in-logstash-8-6-1-version/366539)
**Author:** [@mj22manish](https://discuss.elastic.co/u/mj22manish)\ **Replies:** 1\ **Last updated:** [September 16, 2024, 1:41pm UTC](https://discuss.elastic.co/t/unable-to-get-the-updated-value-of-certain-columns-in-logstash-8-6-1-version/366539 "2024-09-16T13:41:16Z")
Why is the below query not getting the updated value of some columns even after there is an updated value for couple of columns ? Here is my logstash pipeline. input { jdbc { jdbc\_driver\_library =\> "mysql-connecto… --- ## [Logstash filter - match line pattern](https://discuss.elastic.co/t/logstash-filter-match-line-pattern/366598)
**Author:** [@Mark\_V](https://discuss.elastic.co/u/Mark_V)\ **Replies:** 0\ **Last updated:** [September 16, 2024, 3:56am UTC](https://discuss.elastic.co/t/logstash-filter-match-line-pattern/366598 "2024-09-16T03:56:21Z")
Hi, i have an input that is sending data in two different formats. format 1 16-Sep-2024 03:21:53.421 INFO \[pool-13-thread-18\] c.r.s.f.p.PropertyFacadeImpl \[PropertyFacadeImpl.java:209\] Getting PropertyList for client … --- ## [Logstash - Parsing Error](https://discuss.elastic.co/t/logstash-parsing-error/366564)
**Author:** [@slarosa89](https://discuss.elastic.co/u/slarosa89)\ **Replies:** 0\ **Last updated:** [September 14, 2024, 1:17am UTC](https://discuss.elastic.co/t/logstash-parsing-error/366564 "2024-09-14T01:17:20Z")
Hello, I am working on parsing manually imported office 365 audit logs for times when exporting a CSV manually (versus configuring the ELK integration) is necessarry. There is one column of the CSV called AuditData tha… --- ## [Logstash configuration parsing error](https://discuss.elastic.co/t/logstash-configuration-parsing-error/366543)
**Author:** [@temp\_data](https://discuss.elastic.co/u/temp_data)\ **Replies:** 4\ **Last updated:** [September 13, 2024, 5:15pm UTC](https://discuss.elastic.co/t/logstash-configuration-parsing-error/366543 "2024-09-13T17:15:52Z")
Hi, I'm new with Logstash and getting the error below, I think it's not parsing the json properly, does anyone see the problem in my Logstash config . test1.json \[ { "upsert": { "reference": "https://www.a… --- ## [Ruby event.cancel causing pipeline termination post processing 1000 events](https://discuss.elastic.co/t/ruby-event-cancel-causing-pipeline-termination-post-processing-1000-events/366540)
**Author:** [@vvavad](https://discuss.elastic.co/u/vvavad)\ **Replies:** 0\ **Last updated:** [September 13, 2024, 1:43pm UTC](https://discuss.elastic.co/t/ruby-event-cancel-causing-pipeline-termination-post-processing-1000-events/366540 "2024-09-13T13:43:27Z")
I am using logstash to read from an Elasticsearch index. I have to write custom correlation login in filter ruby code, which accumulates 100 records, does some processing and then goes for next records. For unknown rea… --- ## [\[HTTP Output Failure\] Encountered non-2xx HTTP code 413 {:response\_code=\>413, :url=\>"XXXXXX:XX", :event=\>#\}](https://discuss.elastic.co/t/http-output-failure-encountered-non-2xx-http-code-413-response-code-413-url-xxxxxx-xx-event-logstash-0x3b51e9f4/366531)
**Author:** [@Rakesh\_Panda](https://discuss.elastic.co/u/Rakesh_Panda)\ **Replies:** 0\ **Last updated:** [September 13, 2024, 11:52am UTC](https://discuss.elastic.co/t/http-output-failure-encountered-non-2xx-http-code-413-response-code-413-url-xxxxxx-xx-event-logstash-0x3b51e9f4/366531 "2024-09-13T11:52:18Z")
error logs in logstash indicating http Output failure , while using http plugin , any suggestion ? error appearing intermittently in pipeline using simple config output { http { http\_method =\> "post" url =\> "… --- ## [Logstash ECK Job](https://discuss.elastic.co/t/logstash-eck-job/366498)
**Author:** [@lkouts](https://discuss.elastic.co/u/lkouts)\ **Replies:** 0\ **Last updated:** [September 12, 2024, 8:54pm UTC](https://discuss.elastic.co/t/logstash-eck-job/366498 "2024-09-12T20:54:14Z")
My ECK implementation uses logstash with a JDBC driver with over 100 Inputs pushing to a 1:1 index. For it's initial crawl of each index we're looking at performing the initial crawl via a logstash job. When the job comp… --- ## [Logstash - High Network Bandwidth output using S3 Output](https://discuss.elastic.co/t/logstash-high-network-bandwidth-output-using-s3-output/366422)
**Author:** [@iseeldur](https://discuss.elastic.co/u/iseeldur)\ **Replies:** 4\ **Last updated:** [September 12, 2024, 6:29am UTC](https://discuss.elastic.co/t/logstash-high-network-bandwidth-output-using-s3-output/366422 "2024-09-12T06:29:39Z")
Hi, Thanks in advance. We have two aws logstash instances sending beats traffic to two s3 aws buckets using the s3 output plugin, however are seeing high outbound network traffic. Id expect to see 1:2 inbound to outbo… --- ## [About ETL processing failure (RSYSLOG)](https://discuss.elastic.co/t/about-etl-processing-failure-rsyslog/366438)
**Author:** [@nw-engineer](https://discuss.elastic.co/u/nw-engineer)\ **Replies:** 0\ **Last updated:** [September 12, 2024, 12:40am UTC](https://discuss.elastic.co/t/about-etl-processing-failure-rsyslog/366438 "2024-09-12T00:40:33Z")
I am using Logstash (version 8.4.3) to process and index log data received via RSYSLOG. RSYSLOG is configured to perform log rotation with the following parameters: hourly missingok compress rotate 1 nocopy nocrea… --- ## [Logstash csv parser with single and double quotes](https://discuss.elastic.co/t/logstash-csv-parser-with-single-and-double-quotes/366437)
**Author:** [@shmesh](https://discuss.elastic.co/u/shmesh)\ **Replies:** 1\ **Last updated:** [September 11, 2024, 11:36pm UTC](https://discuss.elastic.co/t/logstash-csv-parser-with-single-and-double-quotes/366437 "2024-09-11T23:36:47Z")
I'm having a hard time processing logs with the logstash csv filter, my logs are in a csv format with tab delimited, it has fields which may contain single quotes and also it contains json objects, http://www.example.co… --- ## [Logstash grok pattern not \_grokparsefailure](https://discuss.elastic.co/t/logstash-grok-pattern-not-grokparsefailure/366355)
**Author:** [@wmei](https://discuss.elastic.co/u/wmei)\ **Replies:** 3\ **Last updated:** [September 11, 2024, 1:18pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-not-grokparsefailure/366355 "2024-09-11T13:18:49Z")
Hi. I have a log that I am trying to parse with grok. I tested it on grokconstructor.appspot.com and it groks like i want but when I use the same snippet in my logstash grok it fails. log: I 2024-09-10T14:38:33,45… --- ## [Handle concurrency while updating different records of the same id from json file](https://discuss.elastic.co/t/handle-concurrency-while-updating-different-records-of-the-same-id-from-json-file/366318)
**Author:** [@vvavad](https://discuss.elastic.co/u/vvavad)\ **Replies:** 5\ **Last updated:** [September 11, 2024, 12:50pm UTC](https://discuss.elastic.co/t/handle-concurrency-while-updating-different-records-of-the-same-id-from-json-file/366318 "2024-09-11T12:50:55Z")
I am loading employee data from json files to elasticSearch using logstash The json file can have multiple records for an employee with different data e.g. addresses, languages etc. Each of these records can have same … --- ## [Retaining Logstash Keystore Password Across Upgrades Without Repeated Commands](https://discuss.elastic.co/t/retaining-logstash-keystore-password-across-upgrades-without-repeated-commands/366392)
**Author:** [@Michael\_Mathan\_S](https://discuss.elastic.co/u/Michael_Mathan_S)\ **Replies:** 0\ **Last updated:** [September 11, 2024, 8:34am UTC](https://discuss.elastic.co/t/retaining-logstash-keystore-password-across-upgrades-without-repeated-commands/366392 "2024-09-11T08:34:51Z")
Hi, my requirement is as follows: I installed Logstash on my server, and I successfully ran the commands logstash-keystore create and logstash-keystore add ES\_PWD to set the keystore password. After this, I backed up th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=19) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=21)