# Logstash

**URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=61

[Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md)

**Page:** 62

---

## [Elastic Serverless Forwarder for AWS SSL Authentication?](https://discuss.elastic.co/t/elastic-serverless-forwarder-for-aws-ssl-authentication/340576)

<div class="topic-metadata">

**Author:** [@stabbotco1](https://discuss.elastic.co/u/stabbotco1)\
**Replies:** 2\
**Last updated:** [August 30, 2023, 11:09pm UTC](https://discuss.elastic.co/t/elastic-serverless-forwarder-for-aws-ssl-authentication/340576 "2023-08-30T23:09:46Z")

</div>

Hi All, we are looking to use the Elastic Serverless Forwarder for AWS, sending to logstash. We've got the initial setup and working, and are wondering if there is support for ssl authentication when sending data to lo…

---

## [Help! Logstash send to Elasticsearch use XML file or JSON file](https://discuss.elastic.co/t/help-logstash-send-to-elasticsearch-use-xml-file-or-json-file/341841)

<div class="topic-metadata">

**Author:** [@hoaduy994](https://discuss.elastic.co/u/hoaduy994)\
**Replies:** 8\
**Last updated:** [August 30, 2023, 12:49pm UTC](https://discuss.elastic.co/t/help-logstash-send-to-elasticsearch-use-xml-file-or-json-file/341841 "2023-08-30T12:49:13Z")

</div>

hi everyone, can someone help me with this example? I have an XML file, I originally wanted to change it to JSON and use logstash to send it to elasticsearch, many times it didn't work so I decided to keep the XML and us…

---

## [Annotations not applied at pod level on logstash](https://discuss.elastic.co/t/annotations-not-applied-at-pod-level-on-logstash/341971)

<div class="topic-metadata">

**Author:** [@cdino](https://discuss.elastic.co/u/cdino)\
**Replies:** 0\
**Last updated:** [August 30, 2023, 10:18am UTC](https://discuss.elastic.co/t/annotations-not-applied-at-pod-level-on-logstash/341971 "2023-08-30T10:18:04Z")

</div>

I added some annotations as PodTemplate but those are not applied at pod level: --- apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: eck-logstash-beat spec: count: 3 version: 8.9.1 \[..…

---

## [Logstash Output to Kibana with SSL?](https://discuss.elastic.co/t/logstash-output-to-kibana-with-ssl/341905)

<div class="topic-metadata">

**Author:** [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\
**Replies:** 12\
**Last updated:** [August 29, 2023, 9:45pm UTC](https://discuss.elastic.co/t/logstash-output-to-kibana-with-ssl/341905 "2023-08-29T21:45:43Z")

</div>

I am new to Logstash, having previous experience with Filebeat and Winlogbeat. In the Filebeat/Winlogbeat configuration we have separate output sections for Elasticsearch and Kibana. We configure the Kibana output sect…

---

## [How to do float comparison](https://discuss.elastic.co/t/how-to-do-float-comparison/341881)

<div class="topic-metadata">

**Author:** [@lostsoul352](https://discuss.elastic.co/u/lostsoul352)\
**Replies:** 3\
**Last updated:** [August 29, 2023, 1:16pm UTC](https://discuss.elastic.co/t/how-to-do-float-comparison/341881 "2023-08-29T13:16:29Z")

</div>

I'm trying to drop events if the value of a float field is less than -90000 Here is a code snippet: if \[type\] == "node\_perf" { mutate { convert =\> { "nodeperf\_value" =\> "float"} …

---

## [Aggregate filter plugin - final event contains empty message](https://discuss.elastic.co/t/aggregate-filter-plugin-final-event-contains-empty-message/339702)

<div class="topic-metadata">

**Author:** [@Anca\_Linca](https://discuss.elastic.co/u/Anca_Linca)\
**Replies:** 8\
**Last updated:** [August 29, 2023, 1:02pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-final-event-contains-empty-message/339702 "2023-08-29T13:02:03Z")

</div>

Hello, Logstash version: 7.17 Aggregate filter plugin: v2.10.0 Contents for /var/log/logstash/input.log: {"timestamp": "2023-07-31T15:10:45.141Z", "parentOnly": 1, "logger\_name": "activity\_stream", "job": 102693, "ty…

---

## [Can Logstash support jvm security policy to restrict ruby exec policy](https://discuss.elastic.co/t/can-logstash-support-jvm-security-policy-to-restrict-ruby-exec-policy/341871)

<div class="topic-metadata">

**Author:** [@weizijun](https://discuss.elastic.co/u/weizijun)\
**Replies:** 0\
**Last updated:** [August 29, 2023, 9:35am UTC](https://discuss.elastic.co/t/can-logstash-support-jvm-security-policy-to-restrict-ruby-exec-policy/341871 "2023-08-29T09:35:44Z")

</div>

Since elasticsearch can configure security policies, can logstash do the same?

---

## [Logstash index is not having the autosuggestions](https://discuss.elastic.co/t/logstash-index-is-not-having-the-autosuggestions/341866)

<div class="topic-metadata">

**Author:** [@uma\_parvathy](https://discuss.elastic.co/u/uma_parvathy)\
**Replies:** 0\
**Last updated:** [August 29, 2023, 8:26am UTC](https://discuss.elastic.co/t/logstash-index-is-not-having-the-autosuggestions/341866 "2023-08-29T08:26:45Z")

</div>

i've been trying to populate the data from postgresql db to elasticsearch7.17 using logstash. The data is imported but it's missing the autosuggestions and fuzzy logic suggestions which is needed to query from django ap…

---

## [Aggregate filter の timeout\_timestamp\_field設定時の動作について （続き）](https://discuss.elastic.co/t/aggregate-filter-timeout-timestamp-field/341858)

<div class="topic-metadata">

**Author:** [@e-se](https://discuss.elastic.co/u/e-se)\
**Replies:** 0\
**Last updated:** [August 29, 2023, 7:18am UTC](https://discuss.elastic.co/t/aggregate-filter-timeout-timestamp-field/341858 "2023-08-29T07:18:38Z")

</div>

Continuing the discussion from Aggregate filter の timeout\_timestamp\_field設定時の動作について: 2 行目は 2 番目の集計フィルターを通過しますが、タイムアウト オプションが設定されていないため、タイムアウト処理は行われません。 とありますが、システム時間でタイムアウトを計測する場合、フィルターを通過するかどうかに関係なく、timeoutオプションに設定し…

---

## [Metricbeat - Limit of total fields \[1000\] has been exceeded-urgent](https://discuss.elastic.co/t/metricbeat-limit-of-total-fields-1000-has-been-exceeded-urgent/341824)

<div class="topic-metadata">

**Author:** [@EL\_MALKI\_MOHAMED](https://discuss.elastic.co/u/EL_MALKI_MOHAMED)\
**Replies:** 5\
**Last updated:** [August 29, 2023, 12:50am UTC](https://discuss.elastic.co/t/metricbeat-limit-of-total-fields-1000-has-been-exceeded-urgent/341824 "2023-08-29T00:50:03Z")

</div>

Hello, Can u help me I have problem. I am having errors trying to ingest system metrics (system module) .The logs are ingested via a common beats pipeline running having the following configuration: logstashPipeline: …

---

## [Logstash Logs output for jdbc](https://discuss.elastic.co/t/logstash-logs-output-for-jdbc/341826)

<div class="topic-metadata">

**Author:** [@nbrenke](https://discuss.elastic.co/u/nbrenke)\
**Replies:** 4\
**Last updated:** [August 28, 2023, 8:12pm UTC](https://discuss.elastic.co/t/logstash-logs-output-for-jdbc/341826 "2023-08-28T20:12:08Z")

</div>

I have a silly question:: I have several jdbc pipelines setup that pull data directly from a sql database. Short of the following that shows up in my logs \[2022-10-28T18:40:00,344\]\[INFO \]\[logstash.inputs.jdbc \] (0…

---

## [Disable logstash license check?](https://discuss.elastic.co/t/disable-logstash-license-check/341788)

<div class="topic-metadata">

**Author:** [@jacobdanielrose](https://discuss.elastic.co/u/jacobdanielrose)\
**Replies:** 1\
**Last updated:** [August 28, 2023, 4:40pm UTC](https://discuss.elastic.co/t/disable-logstash-license-check/341788 "2023-08-28T16:40:42Z")

</div>

Hi! I am trying to connect a logstash instance to an elasticseach which is part of a deployment of IBM Cloudpak for AIOps. It uses an elasticsearch instance to store related incident data from ticket systems. The versio…

---

## [Logstash connecting to more than 1 Database](https://discuss.elastic.co/t/logstash-connecting-to-more-than-1-database/341791)

<div class="topic-metadata">

**Author:** [@Ong](https://discuss.elastic.co/u/Ong)\
**Replies:** 1\
**Last updated:** [August 28, 2023, 1:10pm UTC](https://discuss.elastic.co/t/logstash-connecting-to-more-than-1-database/341791 "2023-08-28T13:10:00Z")

</div>

I have 2 separate MSSQL databases and would like to extract data from them, combine it and send it to ES for indexing. Can Logstash connect to more than 1 MSSQL database, retrieve certain data from them then combine the…

---

## [Atlassian access logs Index not getting created or data not sent / visible in Opensearch](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742)

<div class="topic-metadata">

**Author:** [@danmed](https://discuss.elastic.co/u/danmed)\
**Replies:** 29\
**Last updated:** [August 27, 2023, 8:04pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742 "2023-08-27T20:04:18Z")

</div>

I'm trying to use Logstash to send Atlassian access logs to opensearch. I'm absolutely new to the topic but can successfully send other logs and view them. It's the jira access logs that I cannot make work. Having tri…

---

## [Logstash ingesting Netflow traffic, the probability of parsing errors increases with larger data volumes](https://discuss.elastic.co/t/logstash-ingesting-netflow-traffic-the-probability-of-parsing-errors-increases-with-larger-data-volumes/340539)

<div class="topic-metadata">

**Author:** [@gaorui](https://discuss.elastic.co/u/gaorui)\
**Replies:** 1\
**Last updated:** [August 27, 2023, 12:02pm UTC](https://discuss.elastic.co/t/logstash-ingesting-netflow-traffic-the-probability-of-parsing-errors-increases-with-larger-data-volumes/340539 "2023-08-27T12:02:45Z")

</div>

I am using Logstash to ingest Netflow traffic and after parsing, I store the data in Kafka. As the Netflow traffic I am ingesting increases, the probability of incorrect structured data being parsed also increases. Howev…

---

## [Presumably udp input threads are soaking up cpu](https://discuss.elastic.co/t/presumably-udp-input-threads-are-soaking-up-cpu/341590)

<div class="topic-metadata">

**Author:** [@udp\_issues\_are\_one](https://discuss.elastic.co/u/udp_issues_are_one)\
**Replies:** 5\
**Last updated:** [August 27, 2023, 11:47am UTC](https://discuss.elastic.co/t/presumably-udp-input-threads-are-soaking-up-cpu/341590 "2023-08-27T11:47:32Z")

</div>

We have logstash running on ubuntu, logstash version 8.4.1 from the ubuntu repositories. We have a number of pipelines running, most of them work fine but the CPU utilization on the box is a bit high. From the linux co…

---

## [DNS Queries grok pattern working on devtools but not in kibana dashboards](https://discuss.elastic.co/t/dns-queries-grok-pattern-working-on-devtools-but-not-in-kibana-dashboards/341748)

<div class="topic-metadata">

**Author:** [@Poubelle\_Dirty](https://discuss.elastic.co/u/Poubelle_Dirty)\
**Replies:** 4\
**Last updated:** [August 27, 2023, 7:11am UTC](https://discuss.elastic.co/t/dns-queries-grok-pattern-working-on-devtools-but-not-in-kibana-dashboards/341748 "2023-08-27T07:11:37Z")

</div>

Hello, I'm new to ELK stack and I encounter a problem. I'm using the DNS grok pattern from here to parse dns queries from my bind server : https://github.com/cjslack/grok-debugger/blob/master/public/patterns/bind It w…

---

## [The issue of fetching duplicate data in Logstash](https://discuss.elastic.co/t/the-issue-of-fetching-duplicate-data-in-logstash/341643)

<div class="topic-metadata">

**Author:** [@inkweon7269](https://discuss.elastic.co/u/inkweon7269)\
**Replies:** 2\
**Last updated:** [August 26, 2023, 1:40pm UTC](https://discuss.elastic.co/t/the-issue-of-fetching-duplicate-data-in-logstash/341643 "2023-08-26T13:40:13Z")

</div>

I have written the following code within the input section, but I'm experiencing a problem where data is being fetched redundantly. Which part should I modify? logstash.conf input { beats { port =\> 5044 …

---

## [Syslog input plugin from Logstash, how to configure in Elastic agent?](https://discuss.elastic.co/t/syslog-input-plugin-from-logstash-how-to-configure-in-elastic-agent/341300)

<div class="topic-metadata">

**Author:** [@Craig\_Rodrigues](https://discuss.elastic.co/u/Craig_Rodrigues)\
**Replies:** 10\
**Last updated:** [August 25, 2023, 1:31pm UTC](https://discuss.elastic.co/t/syslog-input-plugin-from-logstash-how-to-configure-in-elastic-agent/341300 "2023-08-25T13:31:29Z")

</div>

I have about 2000 Elastic agents (version 8.9.0) connected to a system with 3 Fleet servers (version 8.9.0). We have about 20 different agent policies, because the various Elastic agents are sending slightly different …

---

## [Logstash configuration file for self join field with error object mapping found a concrete value](https://discuss.elastic.co/t/logstash-configuration-file-for-self-join-field-with-error-object-mapping-found-a-concrete-value/341071)

<div class="topic-metadata">

**Author:** [@uma\_parvathy](https://discuss.elastic.co/u/uma_parvathy)\
**Replies:** 8\
**Last updated:** [August 25, 2023, 9:08am UTC](https://discuss.elastic.co/t/logstash-configuration-file-for-self-join-field-with-error-object-mapping-found-a-concrete-value/341071 "2023-08-25T09:08:26Z")

</div>

I've a logstash integration with postgresql table. the table has a self join from incident\_parent\_id to incident\_number. incident\_number ( primary key) incident\_parent\_id ( self join with incident number). But the r…

---

## [Logstash 8.6 add a field "log.file.path"](https://discuss.elastic.co/t/logstash-8-6-add-a-field-log-file-path/341597)

<div class="topic-metadata">

**Author:** [@RobertC1](https://discuss.elastic.co/u/RobertC1)\
**Replies:** 3\
**Last updated:** [August 25, 2023, 3:33am UTC](https://discuss.elastic.co/t/logstash-8-6-add-a-field-log-file-path/341597 "2023-08-25T03:33:38Z")

</div>

Hi there I use .conf file to ingest data in my index. With the version 8.6 is added the field "log.file.path." I tried with mutate { remove\_field =\> \[ "message", "@version","host","log.file.path" \] } without suceed. …

---

## [Accurate decryption logstash data in Javascript](https://discuss.elastic.co/t/accurate-decryption-logstash-data-in-javascript/341538)

<div class="topic-metadata">

**Author:** [@Nik\_Ameer](https://discuss.elastic.co/u/Nik_Ameer)\
**Replies:** 2\
**Last updated:** [August 25, 2023, 3:26am UTC](https://discuss.elastic.co/t/accurate-decryption-logstash-data-in-javascript/341538 "2023-08-25T03:26:16Z")

</div>

I used logstash to encrypt my data using the cipher filter. My logstash.conf file are like so cipher { algorithm =\> "aes-256-cbc" cipher\_padding =\> 1 mode =\> "encrypt" so…

---

## [How do drop logs from being forwarded? My drop rule doesn't seem to be working](https://discuss.elastic.co/t/how-do-drop-logs-from-being-forwarded-my-drop-rule-doesnt-seem-to-be-working/341520)

<div class="topic-metadata">

**Author:** [@feo13](https://discuss.elastic.co/u/feo13)\
**Replies:** 4\
**Last updated:** [August 24, 2023, 6:48pm UTC](https://discuss.elastic.co/t/how-do-drop-logs-from-being-forwarded-my-drop-rule-doesnt-seem-to-be-working/341520 "2023-08-24T18:48:44Z")

</div>

I'm ingesting AWS WAF logs and would like to drop the 'ALLOW' logs. I have the following filter in place but it doesn't seem to be working: filter { if "\\"action\\":\\"ALLOW\\"" in \[message\] { drop {} } if \[ty…

---

## [Help with file name to date logstash grok](https://discuss.elastic.co/t/help-with-file-name-to-date-logstash-grok/341592)

<div class="topic-metadata">

**Author:** [@ethranes](https://discuss.elastic.co/u/ethranes)\
**Replies:** 2\
**Last updated:** [August 24, 2023, 5:07pm UTC](https://discuss.elastic.co/t/help-with-file-name-to-date-logstash-grok/341592 "2023-08-24T17:07:15Z")

</div>

Hi, the date isn't included in my log files. But the filename itself has the date. So I'm trying to extract the year month and day from the filename and then put that into a field. But logstash can't parse my filename, I…

---

## [Two nested Grok patterns not working](https://discuss.elastic.co/t/two-nested-grok-patterns-not-working/341533)

<div class="topic-metadata">

**Author:** [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)\
**Replies:** 3\
**Last updated:** [August 24, 2023, 3:54pm UTC](https://discuss.elastic.co/t/two-nested-grok-patterns-not-working/341533 "2023-08-24T15:54:46Z")

</div>

Hey everyone, I'm new to using Logstash and Elasticsearch. I've been working on collecting logs through Filebeat and then using Logstash for parsing and data cleaning. I have two Grok patterns in place. The first one ex…

---

## [How many records does jdbc input plugin can read at once?](https://discuss.elastic.co/t/how-many-records-does-jdbc-input-plugin-can-read-at-once/341561)

<div class="topic-metadata">

**Author:** [@Sreenivas1](https://discuss.elastic.co/u/Sreenivas1)\
**Replies:** 1\
**Last updated:** [August 24, 2023, 2:43pm UTC](https://discuss.elastic.co/t/how-many-records-does-jdbc-input-plugin-can-read-at-once/341561 "2023-08-24T14:43:13Z")

</div>

Hi Team, We have to pull data of 70million records from oracle database in a week and load into elastic cluster of 3 nodes. So we have a scheduler in database side which will load 50k or 1million records in 6min to a te…

---

## [Reload Logstash config on shutdown](https://discuss.elastic.co/t/reload-logstash-config-on-shutdown/341545)

<div class="topic-metadata">

**Author:** [@Ljapunov](https://discuss.elastic.co/u/Ljapunov)\
**Replies:** 2\
**Last updated:** [August 24, 2023, 2:25pm UTC](https://discuss.elastic.co/t/reload-logstash-config-on-shutdown/341545 "2023-08-24T14:25:19Z")

</div>

Hi everyone, we have a multiple-node logstash cluster using a distributor pipeline that distributes events depending on their types, eg input { beats { # ... } } output { if \[type\] == "foo" { pi…

---

## [Why my Logstash work normal with an recursively error log](https://discuss.elastic.co/t/why-my-logstash-work-normal-with-an-recursively-error-log/341537)

<div class="topic-metadata">

**Author:** [@waitspring](https://discuss.elastic.co/u/waitspring)\
**Replies:** 1\
**Last updated:** [August 24, 2023, 2:07pm UTC](https://discuss.elastic.co/t/why-my-logstash-work-normal-with-an-recursively-error-log/341537 "2023-08-24T14:07:52Z")

</div>

I have make my logstash conf as: ... ... filter { grok { match =\> { "message" =\> \[ "\\\<时间: (?\<timestamp\>.\*)\\\> \\\<进程号:(?\<process\>%{NUMBER}+)\\\>(?\<body\>.\*$)", "\\\<时间:(?\<t…

---

## [Logstash uses 80GB of memory with pipelines and 10 configurations](https://discuss.elastic.co/t/logstash-uses-80gb-of-memory-with-pipelines-and-10-configurations/341474)

<div class="topic-metadata">

**Author:** [@nilsen](https://discuss.elastic.co/u/nilsen)\
**Replies:** 11\
**Last updated:** [August 24, 2023, 1:20pm UTC](https://discuss.elastic.co/t/logstash-uses-80gb-of-memory-with-pipelines-and-10-configurations/341474 "2023-08-24T13:20:06Z")

</div>

A while back we had issues when we ran multiple Logstash instances, each using around 1GB of memory. We got advised to use pipelines instead. Keep in mind we are still in the POC stages, so very new to the ELK stack. We …

---

## [I am using Multiline codec input plugin but the events which are not matching with my PATTERN it also processing those Events](https://discuss.elastic.co/t/i-am-using-multiline-codec-input-plugin-but-the-events-which-are-not-matching-with-my-pattern-it-also-processing-those-events/341425)

<div class="topic-metadata">

**Author:** [@Subrato1](https://discuss.elastic.co/u/Subrato1)\
**Replies:** 1\
**Last updated:** [August 24, 2023, 8:13am UTC](https://discuss.elastic.co/t/i-am-using-multiline-codec-input-plugin-but-the-events-which-are-not-matching-with-my-pattern-it-also-processing-those-events/341425 "2023-08-24T08:13:59Z")

</div>

Below is the codec which am using for multiline events. codec =\> multiline { pattern =\> "%{TIMESTAMP\_ISO8601:syslogtime}\\s%{WORD:str}\\s%{WORD:s}\\s%{YEAR:yeaa}-%{MONTHNUM:ooo}-%{MONTHDAY:ppp}\\s%{TIME:trrrs}" #patte…

[Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=60)

[Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=62)