# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=32 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 33 --- ## [HTTP output - Mapping reference to object make it string](https://discuss.elastic.co/t/http-output-mapping-reference-to-object-make-it-string/358721)
**Author:** [@AfiLouis](https://discuss.elastic.co/u/AfiLouis)\ **Replies:** 3\ **Last updated:** [May 6, 2024, 8:31am UTC](https://discuss.elastic.co/t/http-output-mapping-reference-to-object-make-it-string/358721 "2024-05-06T08:31:33Z")
Hi, I have made the following object using grok, ruby code and field mutation : { // ... "E2E"=\> { "scenarioId" =\> "drg", "XFT": { "name": "xft\_name", "RQ": "\", … --- ## [Is number of CPU defines number of workers in logstash?](https://discuss.elastic.co/t/is-number-of-cpu-defines-number-of-workers-in-logstash/358756)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 3\ **Last updated:** [May 5, 2024, 3:46pm UTC](https://discuss.elastic.co/t/is-number-of-cpu-defines-number-of-workers-in-logstash/358756 "2024-05-05T15:46:56Z")
In our logstash server 6 CPUs we have. Is this implies pipeline.worker value can be set to 6 ? --- ## [Avoid duplicates code in Logstash](https://discuss.elastic.co/t/avoid-duplicates-code-in-logstash/358695)
**Author:** [@Leo23](https://discuss.elastic.co/u/Leo23)\ **Replies:** 4\ **Last updated:** [May 4, 2024, 4:41pm UTC](https://discuss.elastic.co/t/avoid-duplicates-code-in-logstash/358695 "2024-05-04T16:41:14Z")
I have a problem I want to measure the execution time between each step of a log. So I use Elapsed Filter but I can have hundred step in a logs how can I delete duplicate code like this : elapsed { unique\_i… --- ## [AWS S3 Output - Failed to open TCP Connection to](https://discuss.elastic.co/t/aws-s3-output-failed-to-open-tcp-connection-to/358656)
**Author:** [@Alex\_Jamieson](https://discuss.elastic.co/u/Alex_Jamieson)\ **Replies:** 17\ **Last updated:** [May 4, 2024, 3:21am UTC](https://discuss.elastic.co/t/aws-s3-output-failed-to-open-tcp-connection-to/358656 "2024-05-04T03:21:45Z")
Having issues getting the pipeline running \[main\] Pipeline error {:pipeline\_id=\>"main", :exception=\>#\, Failing to understand wh… --- ## [Logstash re-processes files, incorrectly updates sincedb](https://discuss.elastic.co/t/logstash-re-processes-files-incorrectly-updates-sincedb/358640)
**Author:** [@antay](https://discuss.elastic.co/u/antay)\ **Replies:** 7\ **Last updated:** [May 3, 2024, 2:32pm UTC](https://discuss.elastic.co/t/logstash-re-processes-files-incorrectly-updates-sincedb/358640 "2024-05-03T14:32:55Z")
I've been struggling with this intermittent issue for months, where sometimes a log file is read twice into elasticsearch, and sincedb shows incorrect filenames while this is happening. Apologies for a long post, trying… --- ## [Kafka consumer\_threads](https://discuss.elastic.co/t/kafka-consumer-threads/358696)
**Author:** [@mcosta](https://discuss.elastic.co/u/mcosta)\ **Replies:** 0\ **Last updated:** [May 3, 2024, 10:05am UTC](https://discuss.elastic.co/t/kafka-consumer-threads/358696 "2024-05-03T10:05:38Z")
Hi all, We have 1 Kafka topic with 3 partitions and 2 Logstash (each one running on it own linux vm), that we can call logstas-A and logstash-B, both having same pipeline configured. Pior to this configuration we had o… --- ## [After updating logstash, errors started - Failed to perform request](https://discuss.elastic.co/t/after-updating-logstash-errors-started-failed-to-perform-request/357705)
**Author:** [@San9](https://discuss.elastic.co/u/San9)\ **Replies:** 9\ **Last updated:** [May 3, 2024, 9:42am UTC](https://discuss.elastic.co/t/after-updating-logstash-errors-started-failed-to-perform-request/357705 "2024-05-03T09:42:51Z")
The problem is in the operation of logstash. In the old version 8.9 there were no problems with pipelines, but as soon as they updated to 8.10 on one node I started getting errors: \[logstash.outputs.elasticsearch\]\[aws-… --- ## [Prevent data loss if elasticsearch not available](https://discuss.elastic.co/t/prevent-data-loss-if-elasticsearch-not-available/358624)
**Author:** [@Frances\_Chu](https://discuss.elastic.co/u/Frances_Chu)\ **Replies:** 2\ **Last updated:** [May 3, 2024, 7:31am UTC](https://discuss.elastic.co/t/prevent-data-loss-if-elasticsearch-not-available/358624 "2024-05-03T07:31:20Z")
If data send from filebeat to logstash then Elasticsearch. How to prevent data loss if case 1. Logstash not available case 2 elasticsearch not available any special configuration can be performed in both filebeat and… --- ## [Is logstash output plugin for slack is available?](https://discuss.elastic.co/t/is-logstash-output-plugin-for-slack-is-available/358634)
**Author:** [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\ **Replies:** 5\ **Last updated:** [May 3, 2024, 4:37am UTC](https://discuss.elastic.co/t/is-logstash-output-plugin-for-slack-is-available/358634 "2024-05-03T04:37:04Z")
sir is logstash output plugin for slack is available? --- ## [Beautifying the awslamda nested logs Using Logstash in Kibana](https://discuss.elastic.co/t/beautifying-the-awslamda-nested-logs-using-logstash-in-kibana/358261)
**Author:** [@Suman\_ISMT](https://discuss.elastic.co/u/Suman_ISMT)\ **Replies:** 40\ **Last updated:** [May 2, 2024, 8:07pm UTC](https://discuss.elastic.co/t/beautifying-the-awslamda-nested-logs-using-logstash-in-kibana/358261 "2024-05-02T20:07:54Z")
Hello Community, I am using the following Logstash configuration to monitor the aws serverless logs using elasticsearch and visualize it in kibana. The configuration is working no issue on that. input { cloudwatch\_lo… --- ## [Data enrichment](https://discuss.elastic.co/t/data-enrichment/358666)
**Author:** [@Rosanna\_Staiano](https://discuss.elastic.co/u/Rosanna_Staiano)\ **Replies:** 0\ **Last updated:** [May 2, 2024, 4:24pm UTC](https://discuss.elastic.co/t/data-enrichment/358666 "2024-05-02T16:24:59Z")
Hi everyone, we need some advice. We have an index with Kafka data source. Our goal is to enrich this index with information present on elasticsearch, that is updated every day. We need to do enrichment in real time. T… --- ## [Installation error](https://discuss.elastic.co/t/installation-error/358588)
**Author:** [@Manav\_Desai](https://discuss.elastic.co/u/Manav_Desai)\ **Replies:** 3\ **Last updated:** [May 1, 2024, 8:20pm UTC](https://discuss.elastic.co/t/installation-error/358588 "2024-05-01T20:20:04Z")
I was trying to install logstash for development contribution. While I was trying to see if it is installed properly, I ran bin/logstash -e 'input { stdin { } } output { stdout {} }' I am now getting the following err… --- ## [Logstash Aggregate plugin is passing by some events](https://discuss.elastic.co/t/logstash-aggregate-plugin-is-passing-by-some-events/358571)
**Author:** [@Delvin](https://discuss.elastic.co/u/Delvin)\ **Replies:** 4\ **Last updated:** [May 1, 2024, 4:34pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-is-passing-by-some-events/358571 "2024-05-01T16:34:19Z")
Good day everyone! I need some help, because I can't understand, why aggregate function is not working properly. The thing is, that some events are not just aggregated, though IDs are the same (in my case ID is web.trac… --- ## [Problem encrypting logs to logstash with TCP input](https://discuss.elastic.co/t/problem-encrypting-logs-to-logstash-with-tcp-input/358546)
**Author:** [@ITGUY](https://discuss.elastic.co/u/ITGUY)\ **Replies:** 1\ **Last updated:** [May 1, 2024, 3:28pm UTC](https://discuss.elastic.co/t/problem-encrypting-logs-to-logstash-with-tcp-input/358546 "2024-05-01T15:28:12Z")
Hello, I'm using logstash to receive logs from some equipments, I'm using these 3 input : UDP, TCP and beats. TCP will replace UDP to encrypt logs trafic but I have a problem with the TCP input : tcp { type =\>… --- ## [Switch tail to read mode, but logs not streaming to Kibana](https://discuss.elastic.co/t/switch-tail-to-read-mode-but-logs-not-streaming-to-kibana/356624)
**Author:** [@nandydesikan](https://discuss.elastic.co/u/nandydesikan)\ **Replies:** 0\ **Last updated:** [April 2, 2024, 3:43pm UTC](https://discuss.elastic.co/t/switch-tail-to-read-mode-but-logs-not-streaming-to-kibana/356624 "2024-04-02T15:43:32Z")
Thanks for insights I gained from this topic - Grokparsefailure randomly - #5 by fry2k I switched from default tail mode to read mode. However, logs are not streaming to kibana after the switch (restarted the server whi… --- ## [Method redefined; discarding old to\_int](https://discuss.elastic.co/t/method-redefined-discarding-old-to-int/358463)
**Author:** [@m\_pahlevanzadeh](https://discuss.elastic.co/u/m_pahlevanzadeh)\ **Replies:** 1\ **Last updated:** [April 30, 2024, 8:59am UTC](https://discuss.elastic.co/t/method-redefined-discarding-old-to-int/358463 "2024-04-30T08:59:49Z")
Hello, When I don't use any filter, logstash work fine. and 5044 is up. But when I use any filter , I get the following error: /usr/share/logstash/bin/logstash -t Using bundled JDK: /usr/share/logstash/jdk /usr/share/l… --- ## [Logstash Elasticsearch input plugin configuration to pull data from cluster](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-configuration-to-pull-data-from-cluster/358449)
**Author:** [@Sravani\_Sasubilli](https://discuss.elastic.co/u/Sravani_Sasubilli)\ **Replies:** 0\ **Last updated:** [April 29, 2024, 9:07pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-configuration-to-pull-data-from-cluster/358449 "2024-04-29T21:07:44Z")
Hi, I am trying to pull data from on-premises cluster is with SSL is enabled and I am working with Logstash -7.16.3 version and Elastic cluster is 7.17.11 version. Below is the error I am getting and Logstash configur… --- ## [Logstash OOM kills - k8s - 8.13.0](https://discuss.elastic.co/t/logstash-oom-kills-k8s-8-13-0/358423)
**Author:** [@d4nnyx](https://discuss.elastic.co/u/d4nnyx)\ **Replies:** 0\ **Last updated:** [April 29, 2024, 11:39am UTC](https://discuss.elastic.co/t/logstash-oom-kills-k8s-8-13-0/358423 "2024-04-29T11:39:37Z")
Hi everyone, We are using the logging pipeline Filebeat -\> Logstash -\> Dataset/Scalyr. The issue I'm dealing with is that the Logstash memory keeps increasing until the OOM kills the pod (so we are talking about OS OOM… --- ## [@timestamp not match with log timestamp](https://discuss.elastic.co/t/timestamp-not-match-with-log-timestamp/358403)
**Author:** [@Ken\_Pang](https://discuss.elastic.co/u/Ken_Pang)\ **Replies:** 1\ **Last updated:** [April 29, 2024, 9:47am UTC](https://discuss.elastic.co/t/timestamp-not-match-with-log-timestamp/358403 "2024-04-29T09:47:58Z")
I'm new to ELK, I've search the related post here, but still unable to get my event timestamp work, anyone can help? my sample data 2024/04/29 15:46:11.833 Dev = TTA10B6001-01, Data: Current State = ReadTagStatus, New… --- ## [JDBC password needs to be retrieved from CyberArk and cached locally](https://discuss.elastic.co/t/jdbc-password-needs-to-be-retrieved-from-cyberark-and-cached-locally/358193)
**Author:** [@forabraham1](https://discuss.elastic.co/u/forabraham1)\ **Replies:** 3\ **Last updated:** [April 29, 2024, 7:09am UTC](https://discuss.elastic.co/t/jdbc-password-needs-to-be-retrieved-from-cyberark-and-cached-locally/358193 "2024-04-29T07:09:24Z")
In our setup, there are ~ 30 JDBC configs \*.conf (one per client) within that it will have one input section with 4 different jdbc section (one per functionality). Every functionality JDBC sections are scheduled to run e… --- ## [Is logstash supports multiple grok pattern?](https://discuss.elastic.co/t/is-logstash-supports-multiple-grok-pattern/358393)
**Author:** [@abhishekacharya828](https://discuss.elastic.co/u/abhishekacharya828)\ **Replies:** 3\ **Last updated:** [April 29, 2024, 6:46am UTC](https://discuss.elastic.co/t/is-logstash-supports-multiple-grok-pattern/358393 "2024-04-29T06:46:31Z")
filter { grok { match =\> { "message" =\> \[ "%{TIMESTAMP\_ISO8601:timestamp} \[%{LOGLEVEL:loglevel}\] %{GREEDYDATA:message}", "\\\[%{LOGLEVEL:log\_level}\\\] System Metrics: CPU %{NUMBER:cpu\_usage}%\\, Memory %{NUMBER:memory\_… --- ## [Logstash API,Logstash doesn't update the data when mongodb data gets updated or delete records](https://discuss.elastic.co/t/logstash-api-logstash-doesnt-update-the-data-when-mongodb-data-gets-updated-or-delete-records/358183)
**Author:** [@anguri\_sudhakar](https://discuss.elastic.co/u/anguri_sudhakar)\ **Replies:** 4\ **Last updated:** [April 26, 2024, 12:54pm UTC](https://discuss.elastic.co/t/logstash-api-logstash-doesnt-update-the-data-when-mongodb-data-gets-updated-or-delete-records/358183 "2024-04-26T12:54:05Z")
As records are updated, modified and deleted in our MongoDB transactional database, those documents Logstsah API will not fetching the records in ES logstash: 8.5.1 Elasticsearch:8.9.1 Is there an option? What wou… --- ## [In logstash input, access to amazon msk(kafka)](https://discuss.elastic.co/t/in-logstash-input-access-to-amazon-msk-kafka/358276)
**Author:** [@nairobi](https://discuss.elastic.co/u/nairobi)\ **Replies:** 0\ **Last updated:** [April 26, 2024, 8:47am UTC](https://discuss.elastic.co/t/in-logstash-input-access-to-amazon-msk-kafka/358276 "2024-04-26T08:47:05Z")
I try to get data from amazon managed kafka service named msk. That msk using iam. I set input like below and also put "aws-msk-iam-auth-2.1.0-all.jar" file to logstash module directory. logstash input : input { ka… --- ## [Pipeline to Pipeline forked configuration to enable use of the aggregation filter](https://discuss.elastic.co/t/pipeline-to-pipeline-forked-configuration-to-enable-use-of-the-aggregation-filter/358202)
**Author:** [@finejason](https://discuss.elastic.co/u/finejason)\ **Replies:** 4\ **Last updated:** [April 25, 2024, 6:05pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-forked-configuration-to-enable-use-of-the-aggregation-filter/358202 "2024-04-25T18:05:21Z")
I've set up a pipeline to pipeline configuration following advice given in this topic Pipeline.workers configuration and aggregation filter What I'm trying to solve is to be able to use the aggregate filter but not degr… --- ## [Certificate logstash](https://discuss.elastic.co/t/certificate-logstash/358225)
**Author:** [@samsu](https://discuss.elastic.co/u/samsu)\ **Replies:** 1\ **Last updated:** [April 25, 2024, 4:25pm UTC](https://discuss.elastic.co/t/certificate-logstash/358225 "2024-04-25T16:25:15Z")
xpack.management.elasticsearch.ssl.certificate\_authority: "/usr/share/logstash/conf.d/ssl/elasticsearch-ca.pem" xpack.management.elasticsearch.ssl.verification\_mode: "certificate" where can i find this certificate plea… --- ## [Extract filed from specific message](https://discuss.elastic.co/t/extract-filed-from-specific-message/358198)
**Author:** [@Omid\_Mosayebi](https://discuss.elastic.co/u/Omid_Mosayebi)\ **Replies:** 2\ **Last updated:** [April 25, 2024, 4:19pm UTC](https://discuss.elastic.co/t/extract-filed-from-specific-message/358198 "2024-04-25T16:19:21Z")
Hi Everyone. I have a problem. How can I extract statusCodeValue from 2024-04-24 11:02:20 - \[http-nio-8082-exec-6\] - \[2e7e7b76-fbb6-47b2-b07e-7471175aacea\] fiscal-information ResponseData \[{"headers":{},"body":{"timest… --- ## [Extract filed](https://discuss.elastic.co/t/extract-filed/358204)
**Author:** [@Omid\_Mosayebi](https://discuss.elastic.co/u/Omid_Mosayebi)\ **Replies:** 1\ **Last updated:** [April 25, 2024, 4:13pm UTC](https://discuss.elastic.co/t/extract-filed/358204 "2024-04-25T16:13:45Z")
Hi Everyone. I have a problem. How can I extract statusCodeValue from 2024-04-24 11:02:20 - \[http-nio-8082-exec-6\] - \[2e7e7b76-fbb6-47b2-b07e-7471175aacea\] fiscal-information ResponseData \[{"headers":{},"body":{"timest… --- ## [Filter mutate replace a field for specific value](https://discuss.elastic.co/t/filter-mutate-replace-a-field-for-specific-value/357730)
**Author:** [@Armalyca](https://discuss.elastic.co/u/Armalyca)\ **Replies:** 4\ **Last updated:** [April 25, 2024, 2:51pm UTC](https://discuss.elastic.co/t/filter-mutate-replace-a-field-for-specific-value/357730 "2024-04-25T14:51:00Z")
Hello, I use logstash 7.17. I want to create (or replace if existing) a field when I have a specific value. Here is a snippet of my code : filter { grok { match =\> { "\[topic\]" =\> "%{GREEDYDATA… --- ## [Attempted to resurrect connection to dead ES instance](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance/358120)
**Author:** [@samsu](https://discuss.elastic.co/u/samsu)\ **Replies:** 5\ **Last updated:** [April 24, 2024, 2:19pm UTC](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance/358120 "2024-04-24T14:19:24Z")
i have this error : \[2024-04-24T12:24:55,861\]\[INFO \]\[logstash.outputs.elasticsearch\]\[main\] Failed to perform request {:message=\>"Connect to localhost:9200 \[localhost/127.0.0.1\] failed: Connection refused (Connection ref… --- ## [Error message](https://discuss.elastic.co/t/error-message/358032)
**Author:** [@samsu](https://discuss.elastic.co/u/samsu)\ **Replies:** 9\ **Last updated:** [April 24, 2024, 7:27am UTC](https://discuss.elastic.co/t/error-message/358032 "2024-04-24T07:27:32Z")
I am encountering an issue with establishing a connection from Logstash to my local Elasticsearch instance. Each time I attempt to send data from Logstash to Elasticsearch, I receive the following error messages: \[logst… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=31) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=33)