# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=70 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 71 --- ## [Multiple pipelines utilizing s3 output cause each other to terminate](https://discuss.elastic.co/t/multiple-pipelines-utilizing-s3-output-cause-each-other-to-terminate/336211)
**Author:** [@hughjarse](https://discuss.elastic.co/u/hughjarse)\ **Replies:** 5\ **Last updated:** [June 22, 2023, 12:54am UTC](https://discuss.elastic.co/t/multiple-pipelines-utilizing-s3-output-cause-each-other-to-terminate/336211 "2023-06-22T00:54:51Z")
In Logstash, I am using the s3 output plugin in multiple pipelines. Each plugin instance stores temporary files to disk and has the restore option set to true. I have problems with the restarting of other pipelines causi… --- ## [Parse failure (object mapping for \[trace.detail\] tried to parse field \[null\] as object, but found a concrete value)](https://discuss.elastic.co/t/parse-failure-object-mapping-for-trace-detail-tried-to-parse-field-null-as-object-but-found-a-concrete-value/336551)
**Author:** [@a.emrekaraman](https://discuss.elastic.co/u/a.emrekaraman)\ **Replies:** 1\ **Last updated:** [June 21, 2023, 7:20pm UTC](https://discuss.elastic.co/t/parse-failure-object-mapping-for-trace-detail-tried-to-parse-field-null-as-object-but-found-a-concrete-value/336551 "2023-06-21T19:20:17Z")
Hi Team, I basically use json filter to parse log. But somewhere in json have 2 different type of log that's why I get this error (object mapping for \[trace.detail\] tried to parse field \[null\] as object, but found a conc… --- ## [Logstash docker-compose non root user](https://discuss.elastic.co/t/logstash-docker-compose-non-root-user/336598)
**Author:** [@maehue](https://discuss.elastic.co/u/maehue)\ **Replies:** 0\ **Last updated:** [June 21, 2023, 2:41pm UTC](https://discuss.elastic.co/t/logstash-docker-compose-non-root-user/336598 "2023-06-21T14:41:07Z")
To date we have been running logstash 7.16.2 as a non-root user in docker using a docker-compose configuration similar to below: version: 3.3 services: logstash: image: logstash:7.16.2 user: 10002:1001 … --- ## [CVE-2022-1471 is not listed in Security Issues site](https://discuss.elastic.co/t/cve-2022-1471-is-not-listed-in-security-issues-site/336553)
**Author:** [@Ayushi\_bhardwaj](https://discuss.elastic.co/u/Ayushi_bhardwaj)\ **Replies:** 1\ **Last updated:** [June 21, 2023, 8:38am UTC](https://discuss.elastic.co/t/cve-2022-1471-is-not-listed-in-security-issues-site/336553 "2023-06-21T08:38:33Z")
Is there any fix for that in any Logstash version? Is there any plan to update the damaged package of snakeyaml 1.31=\>2.0? Can I manually change the snakeyaml version? if so then how? --- ## [If condition loop on array](https://discuss.elastic.co/t/if-condition-loop-on-array/336518)
**Author:** [@plus](https://discuss.elastic.co/u/plus)\ **Replies:** 3\ **Last updated:** [June 21, 2023, 8:22am UTC](https://discuss.elastic.co/t/if-condition-loop-on-array/336518 "2023-06-21T08:22:29Z")
{ Hello, I was reading several posts how to loop through with array but I don't know how to iterate on each value and then rename. I tried with split but it creates a document for each value ( I want a doc with all val… --- ## [Append a string to a field after mutate convert filter](https://discuss.elastic.co/t/append-a-string-to-a-field-after-mutate-convert-filter/336327)
**Author:** [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Replies:** 3\ **Last updated:** [June 21, 2023, 7:14am UTC](https://discuss.elastic.co/t/append-a-string-to-a-field-after-mutate-convert-filter/336327 "2023-06-21T07:14:11Z")
Hi I have the following log pattern \[19/Jun/2023:11:27:35 +0530\] | 503 | 1188 ms | 299 B | 172.31.40.179 | - | - | - | "GET /3dcomment/monitoring/healthcheck HTTP/1.1" I have applied grok to fetch the bytes field i.e 2… --- ## [Aggregate filter の timeout\_timestamp\_field設定時の動作について](https://discuss.elastic.co/t/aggregate-filter-timeout-timestamp-field/336283)
**Author:** [@e-se](https://discuss.elastic.co/u/e-se)\ **Replies:** 3\ **Last updated:** [June 20, 2023, 9:48pm UTC](https://discuss.elastic.co/t/aggregate-filter-timeout-timestamp-field/336283 "2023-06-20T21:48:19Z")
Aggregate filter pluginのオプションtimeout\_timestamp\_fieldについて、 機能追加の経緯やドキュメントの記載から設定すると、タイムアウトの判定がシステム時間からログのタイムスタンプに変わると思っていたが、実際に動かしてみると、システム時間で判定されたような挙動をした。 （私と同じ疑問を持った方が過去にいたよう。https://discuss.elastic.co/t/aggregate-fi… --- ## [How can I index only new documents without updating the older ones?](https://discuss.elastic.co/t/how-can-i-index-only-new-documents-without-updating-the-older-ones/336501)
**Author:** [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)\ **Replies:** 1\ **Last updated:** [June 20, 2023, 4:23pm UTC](https://discuss.elastic.co/t/how-can-i-index-only-new-documents-without-updating-the-older-ones/336501 "2023-06-20T16:23:15Z")
I am aware of the create action but when I use it a horrendous WARN log is printed in the logstash screen. The solution of create would fit perfect if it wasn't for it. So I've been wondering if there is another way to a… --- ## [Logstash output s3 prefix with date](https://discuss.elastic.co/t/logstash-output-s3-prefix-with-date/336498)
**Author:** [@kunalmohan](https://discuss.elastic.co/u/kunalmohan)\ **Replies:** 1\ **Last updated:** [June 20, 2023, 3:10pm UTC](https://discuss.elastic.co/t/logstash-output-s3-prefix-with-date/336498 "2023-06-20T15:10:27Z")
S3 output plugin | Logstash Reference \[8.8\] | Elastic mentions to use prefix = "%{+YYYY}/%{+MM}/%{+dd}" for creating folders based on event date. This doesn't work for my. It simply creates two nested folders with name /.… --- ## [Dynamically set s3 bucket name in logstash output](https://discuss.elastic.co/t/dynamically-set-s3-bucket-name-in-logstash-output/336484)
**Author:** [@kunalmohan](https://discuss.elastic.co/u/kunalmohan)\ **Replies:** 1\ **Last updated:** [June 20, 2023, 1:09pm UTC](https://discuss.elastic.co/t/dynamically-set-s3-bucket-name-in-logstash-output/336484 "2023-06-20T13:09:34Z")
Is there a way I can set s3 bucket name using a @metadata field? --- ## [Logstash not listening for second input](https://discuss.elastic.co/t/logstash-not-listening-for-second-input/336480)
**Author:** [@Shabu](https://discuss.elastic.co/u/Shabu)\ **Replies:** 2\ **Last updated:** [June 20, 2023, 1:01pm UTC](https://discuss.elastic.co/t/logstash-not-listening-for-second-input/336480 "2023-06-20T13:01:21Z")
I have set up a working ELK stack with input from winlogbeat. Now I want to add a second input for ingesting syslog logs from a switch. I configured my logstash to do so, but it still only listens on port 5044 after rest… --- ## [Not working TCP input with TLS](https://discuss.elastic.co/t/not-working-tcp-input-with-tls/336473)
**Author:** [@wedkarz014](https://discuss.elastic.co/u/wedkarz014)\ **Replies:** 0\ **Last updated:** [June 20, 2023, 11:41am UTC](https://discuss.elastic.co/t/not-working-tcp-input-with-tls/336473 "2023-06-20T11:41:28Z")
Hi, I want to send syslog events but with tls, unfortunately i have a problem with that. Logstash receive first event and that's all, i don't have any error logs. Here is output config: output { tcp { host =\> … --- ## [Logs related to database (postgres) pods are not moving to elastic](https://discuss.elastic.co/t/logs-related-to-database-postgres-pods-are-not-moving-to-elastic/336459)
**Author:** [@unais](https://discuss.elastic.co/u/unais)\ **Replies:** 0\ **Last updated:** [June 20, 2023, 9:29am UTC](https://discuss.elastic.co/t/logs-related-to-database-postgres-pods-are-not-moving-to-elastic/336459 "2023-06-20T09:29:37Z")
Hi community, I'm not able to see the logs related postgres even though I have mentioned the name of the pod in filebeat. postgres logs: (on running kubectl logs command) 2023-06-19 07:37:39.591 UTC \[73\] ERROR: "xyz" … --- ## [LogStash and parsing OPNSenser logs](https://discuss.elastic.co/t/logstash-and-parsing-opnsenser-logs/334234)
**Author:** [@LoggingJennfier](https://discuss.elastic.co/u/LoggingJennfier)\ **Replies:** 2\ **Last updated:** [June 20, 2023, 9:16am UTC](https://discuss.elastic.co/t/logstash-and-parsing-opnsenser-logs/334234 "2023-06-20T09:16:07Z")
My logs are coming in as follows: \<134\>May 24 14:39:32 edge.internal filterlog\[2535\]: 78,,,ffe6d10d1f27a42fc0edc3abb3a6d333,ovpnc1,match,pass,out,4,0x0,,63,61951,0,DF,6,tcp,60,10.8.0.2,20.44.17.5,44575,443,0,S,149708160… --- ## [Filebeat error for port ERROR: Address already in use](https://discuss.elastic.co/t/filebeat-error-for-port-error-address-already-in-use/336422)
**Author:** [@yogesh.gangwar](https://discuss.elastic.co/u/yogesh.gangwar)\ **Replies:** 1\ **Last updated:** [June 20, 2023, 8:54am UTC](https://discuss.elastic.co/t/filebeat-error-for-port-error-address-already-in-use/336422 "2023-06-20T08:54:33Z")
While running the logstash I'm getting an issue of "A plugin had an unrecoverable error. Will restart this plugin." \ **Author:** [@Indeed2000](https://discuss.elastic.co/u/Indeed2000)\ **Replies:** 5\ **Last updated:** [June 20, 2023, 6:25am UTC](https://discuss.elastic.co/t/logstash-config-for-transactions/336294 "2023-06-20T06:25:36Z") Hi need to write logstash config that parse log file from this path: "/tmp/logs/\*" store in elastic. here is the log: 09:54:37:566 R\[SRV1\]L\[477\]T\[0300\]ID\[696119\] 09:54:37:566 S\[SRV2\]L\[477\]T\[0300\]ID\[696119\] 09:54:55:28… --- ## [Logstash update sql\_last\_value while using paging](https://discuss.elastic.co/t/logstash-update-sql-last-value-while-using-paging/336362)
**Author:** [@zalseryani](https://discuss.elastic.co/u/zalseryani)\ **Replies:** 3\ **Last updated:** [June 20, 2023, 6:00am UTC](https://discuss.elastic.co/t/logstash-update-sql-last-value-while-using-paging/336362 "2023-06-20T06:00:52Z")
sql\_last\_value update with Paging I am configuring logstash to use jdbc input knowing that I am using jdbc\_paging with the configuration. what I am facing now is that sql\_last\_value is being updated after all record… --- ## [Logstash filling up disk space beyond queue.max\_bytes](https://discuss.elastic.co/t/logstash-filling-up-disk-space-beyond-queue-max-bytes/336388)
**Author:** [@Sindhu\_Bandi](https://discuss.elastic.co/u/Sindhu_Bandi)\ **Replies:** 4\ **Last updated:** [June 20, 2023, 5:19am UTC](https://discuss.elastic.co/t/logstash-filling-up-disk-space-beyond-queue-max-bytes/336388 "2023-06-20T05:19:39Z")
Logstash persistent volume size is increasing beyond configured queue.max\_bytes Scenario: Logstash : 7.17.3 Env : On Kubernetes cluster Persistence: Enabled logstash.yml: ---- http.host: "0.0.0.0" path.config: /usr/… --- ## [How Statsd output plugin work](https://discuss.elastic.co/t/how-statsd-output-plugin-work/336298)
**Author:** [@Indeed2000](https://discuss.elastic.co/u/Indeed2000)\ **Replies:** 3\ **Last updated:** [June 19, 2023, 10:55pm UTC](https://discuss.elastic.co/t/how-statsd-output-plugin-work/336298 "2023-06-19T22:55:12Z")
Hi I have logfile that need to count number of this string on it "connection failed" now question is log file created last day and continuously new log add to it. which of these Statsd output configuration options "co… --- ## [\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of \[ \\\\t\\\\r\\\\n\], \\"#\\", \\"input\\", \\"filter\\", \\"output\\" at line](https://discuss.elastic.co/t/error-logstash-agent-failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror-message-expected-one-of-t-r-n-input-filter-output-at-line/336352)
**Author:** [@dropp.dev.hamidreza](https://discuss.elastic.co/u/dropp.dev.hamidreza)\ **Replies:** 6\ **Last updated:** [June 19, 2023, 7:36pm UTC](https://discuss.elastic.co/t/error-logstash-agent-failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror-message-expected-one-of-t-r-n-input-filter-output-at-line/336352 "2023-06-19T19:36:41Z")
Hi, I'm trying to set up ELK stack with docker and docker compose and while setting up pipeline in logstash is gave me error in logstash container logs: \[INFO \]\[logstash.runner \] JVM bootstrap flags: \[-Xms4g, -… --- ## [Cannot initialize custom codec plugin](https://discuss.elastic.co/t/cannot-initialize-custom-codec-plugin/336371)
**Author:** [@ofekinger](https://discuss.elastic.co/u/ofekinger)\ **Replies:** 0\ **Last updated:** [June 19, 2023, 1:28pm UTC](https://discuss.elastic.co/t/cannot-initialize-custom-codec-plugin/336371 "2023-06-19T13:28:07Z")
Hello. I'm working on a new codec plugin that parses protobuf data in a unique way (meaning I can't use the existing protobuf plugin). Here's the plugin code: package com.ofekinger.logstash.plugins.mycodec; import co… --- ## [LogStash Configurations for Log4Net, Log4J etc](https://discuss.elastic.co/t/logstash-configurations-for-log4net-log4j-etc/336258)
**Author:** [@Tomahawk](https://discuss.elastic.co/u/Tomahawk)\ **Replies:** 2\ **Last updated:** [June 19, 2023, 11:55am UTC](https://discuss.elastic.co/t/logstash-configurations-for-log4net-log4j-etc/336258 "2023-06-19T11:55:34Z")
Bit of a left field question….. In a highly regulated space and restricted industry, log files coming from multiple apps (100-200) with Log4Net and Log4J, Python Native logging libraries. No real customisation done by t… --- ## [Errors Updating logstash from 8.5.3 to 8.8.0](https://discuss.elastic.co/t/errors-updating-logstash-from-8-5-3-to-8-8-0/335531)
**Author:** [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)\ **Replies:** 6\ **Last updated:** [June 19, 2023, 10:03am UTC](https://discuss.elastic.co/t/errors-updating-logstash-from-8-5-3-to-8-8-0/335531 "2023-06-19T10:03:52Z")
I have update all my Elasticsearch cluster, and kibana to the version 8.8.0 from 8.5.3, when update Logstash it doesnt start runing and show the next error \[2023-06-08T13:06:33,163\]\[WARN \]\[logstash.outputs.elasticsearch… --- ## [Logstash batch import nested objects](https://discuss.elastic.co/t/logstash-batch-import-nested-objects/336342)
**Author:** [@Joker\_Lu](https://discuss.elastic.co/u/Joker_Lu)\ **Replies:** 0\ **Last updated:** [June 19, 2023, 9:23am UTC](https://discuss.elastic.co/t/logstash-batch-import-nested-objects/336342 "2023-06-19T09:23:55Z")
Hi everyone, I want to batch import nested objects to ES, but when i paging my nested objects, it will cover my previous data. Can anyone have a solution for this. --- ## [Updating index is not working for existing data inside json object](https://discuss.elastic.co/t/updating-index-is-not-working-for-existing-data-inside-json-object/336291)
**Author:** [@J\_S](https://discuss.elastic.co/u/J_S)\ **Replies:** 1\ **Last updated:** [June 18, 2023, 7:15am UTC](https://discuss.elastic.co/t/updating-index-is-not-working-for-existing-data-inside-json-object/336291 "2023-06-18T07:15:53Z")
I am repeatedly fetching rows from a database. I insert them into elasticsearch using the unique key as the document\_id. For any fields not on the current document I want to add any missing columns to the exiting documen… --- ## [Event.remove method not working inside aggregate section in code block](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201)
**Author:** [@J\_S](https://discuss.elastic.co/u/J_S)\ **Replies:** 20\ **Last updated:** [June 18, 2023, 3:37am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201 "2023-06-18T03:37:16Z")
Hi All, I am newbie to ELK stack, I am trying to remove the field called "attributes" while aggregate the data inside code block. But it is not removing the already existing "attributes" in the corresponding "id" but on… --- ## [Logstash to Elasticsearch there is 10-20min for delay, also not all logs are indexed](https://discuss.elastic.co/t/logstash-to-elasticsearch-there-is-10-20min-for-delay-also-not-all-logs-are-indexed/336241)
**Author:** [@mayank\_singh](https://discuss.elastic.co/u/mayank_singh)\ **Replies:** 3\ **Last updated:** [June 17, 2023, 3:20pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-there-is-10-20min-for-delay-also-not-all-logs-are-indexed/336241 "2023-06-17T15:20:49Z")
Hi, I am sending logs from Logstash to Elasticsearch. The Elasticsearch seems to be working fine but there is 10-20mins of delay in logs index also getting below error on logstash, any help would be greatly appreciated. … --- ## [Possible issue with tracking\_column\_type =\> "timestamp" in 8.1.1](https://discuss.elastic.co/t/possible-issue-with-tracking-column-type-timestamp-in-8-1-1/336255)
**Author:** [@SrxDevOps](https://discuss.elastic.co/u/SrxDevOps)\ **Replies:** 1\ **Last updated:** [June 16, 2023, 8:18pm UTC](https://discuss.elastic.co/t/possible-issue-with-tracking-column-type-timestamp-in-8-1-1/336255 "2023-06-16T20:18:17Z")
After updating from 7x to 8.1.1 any pipeline that uses tracking\_column\_type =\> "timestamp" fails with the error \[2023-06-16T14:37:50,485\]\[ERROR\]\[logstash.javapipeline \]\[Questions\] Pipeline error {:pipeline\_id=\>"Quest… --- ## [Convert datetime to another timezone in logstash](https://discuss.elastic.co/t/convert-datetime-to-another-timezone-in-logstash/336214)
**Author:** [@ashokkrishna99\_Vemur](https://discuss.elastic.co/u/ashokkrishna99_Vemur)\ **Replies:** 8\ **Last updated:** [June 16, 2023, 8:06pm UTC](https://discuss.elastic.co/t/convert-datetime-to-another-timezone-in-logstash/336214 "2023-06-16T20:06:33Z")
I am getting logs from a firewall which are in GMT timezone. For example firewall sending rt=Jun 16 2023 11:24:40 GMT I want to convert that time to MYT rt=June 16 2023 19:24:40 MYT. How can I do that. filter { grok… --- ## [Logstash log clarification](https://discuss.elastic.co/t/logstash-log-clarification/336240)
**Author:** [@Karthik9099](https://discuss.elastic.co/u/Karthik9099)\ **Replies:** 1\ **Last updated:** [June 16, 2023, 3:22pm UTC](https://discuss.elastic.co/t/logstash-log-clarification/336240 "2023-06-16T15:22:54Z")
Logstash logs show - "Added to Object" - \<\> What does it mean? Is it reading the file or skipping out or accumulating data? [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=69) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=71)