# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=29 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 30 --- ## [Problems with Fortigate integration](https://discuss.elastic.co/t/problems-with-fortigate-integration/360784)
**Author:** [@Rackhage](https://discuss.elastic.co/u/Rackhage)\ **Replies:** 0\ **Last updated:** [June 4, 2024, 12:23pm UTC](https://discuss.elastic.co/t/problems-with-fortigate-integration/360784 "2024-06-04T12:23:45Z")
Hi, As the titel suggests, I have some problems with the integration of Fortigate. Right now I receive logs from a fortigate through logstash. I did find the integration from Fortinet for the fortigate, but it seems li… --- ## [How to integrate the datadog output in the logstash](https://discuss.elastic.co/t/how-to-integrate-the-datadog-output-in-the-logstash/360772)
**Author:** [@Pravinz123](https://discuss.elastic.co/u/Pravinz123)\ **Replies:** 3\ **Last updated:** [June 4, 2024, 1:12pm UTC](https://discuss.elastic.co/t/how-to-integrate-the-datadog-output-in-the-logstash/360772 "2024-06-04T13:12:42Z")
Greetings, I have deployed only Logstash using helm chart in the Anthos Bare metal cluster. I want to integrate the Logstash with Datadog platform, for which plugin is installed using offline plugin management. Since ou… --- ## [Logstash pipeline terminated with logstash salesforce input plugin without any error logs](https://discuss.elastic.co/t/logstash-pipeline-terminated-with-logstash-salesforce-input-plugin-without-any-error-logs/359551)
**Author:** [@Pavani\_Reddy](https://discuss.elastic.co/u/Pavani_Reddy)\ **Replies:** 5\ **Last updated:** [June 4, 2024, 12:05pm UTC](https://discuss.elastic.co/t/logstash-pipeline-terminated-with-logstash-salesforce-input-plugin-without-any-error-logs/359551 "2024-06-04T12:05:06Z")
Hello I am trying to integrate salesforce logs to ELK platform. I used logstash salesforce input plugin to load the logs and output to a file. The logstash pipeline is getting terminated within within seconds without a… --- ## [Reason for the warning](https://discuss.elastic.co/t/reason-for-the-warning/360759)
**Author:** [@thisisnew](https://discuss.elastic.co/u/thisisnew)\ **Replies:** 0\ **Last updated:** [June 4, 2024, 6:29am UTC](https://discuss.elastic.co/t/reason-for-the-warning/360759 "2024-06-04T06:29:50Z")
I have an logstash conf file like input { s3 { access\_key\_id =\> "redacted" secret\_access\_key =\> "redacted" bucket =\> "aws-cloudtrail-logs-905418385260-a48e6842" delete =\> false interval =\> 30 # sec… --- ## [Elasticsearch output multiple hosts but no fault tolerance](https://discuss.elastic.co/t/elasticsearch-output-multiple-hosts-but-no-fault-tolerance/360667)
**Author:** [@Mahdi\_Moazami](https://discuss.elastic.co/u/Mahdi_Moazami)\ **Replies:** 2\ **Last updated:** [June 3, 2024, 8:56am UTC](https://discuss.elastic.co/t/elasticsearch-output-multiple-hosts-but-no-fault-tolerance/360667 "2024-06-03T08:56:26Z")
Hi Elastic team, We've designed an Elasticsearch cluster with 3 nodes and there is 2 independent Logstash instances that ingest data to this cluster. sometimes one of the ES cluster's node downs due to high load and can… --- ## [How to disable logstash add default field automatically?](https://discuss.elastic.co/t/how-to-disable-logstash-add-default-field-automatically/360695)
**Author:** [@Victor\_Lee](https://discuss.elastic.co/u/Victor_Lee)\ **Replies:** 1\ **Last updated:** [June 3, 2024, 8:49am UTC](https://discuss.elastic.co/t/how-to-disable-logstash-add-default-field-automatically/360695 "2024-06-03T08:49:22Z")
I am using Logstash and Elasticsearch as my backup database. Recently, I discovered an error related to Logstash using the JSON codec to decode my input file JSON. If the JSON does not contain a 'host' field, Logstash au… --- ## [Implement custom code in Filter plugin for Logstash](https://discuss.elastic.co/t/implement-custom-code-in-filter-plugin-for-logstash/360615)
**Author:** [@Manoilayans](https://discuss.elastic.co/u/Manoilayans)\ **Replies:** 3\ **Last updated:** [June 2, 2024, 2:21pm UTC](https://discuss.elastic.co/t/implement-custom-code-in-filter-plugin-for-logstash/360615 "2024-06-02T14:21:19Z")
While ingesting the data we can using Filters like filter { mutate { copy =\> { "source\_field" =\> "dest\_field" } } } But we need to implement as listed below.. · Proper case · Proper punctuation · Avoid duplicate… --- ## [Grok help not getting data](https://discuss.elastic.co/t/grok-help-not-getting-data/360664)
**Author:** [@wmei](https://discuss.elastic.co/u/wmei)\ **Replies:** 3\ **Last updated:** [June 2, 2024, 1:00pm UTC](https://discuss.elastic.co/t/grok-help-not-getting-data/360664 "2024-06-02T13:00:32Z")
Hi! Been scratching my head on this all day. Could use some help. I am using elasticsearch 7.17.6. I have filebeat set to read log file, use a dissect processor, sends to logstash. My problem is I am trying to grok thi… --- ## [Unable to create an Index and send logs to Elasticsearch from Logstash getting 404 error](https://discuss.elastic.co/t/unable-to-create-an-index-and-send-logs-to-elasticsearch-from-logstash-getting-404-error/360657)
**Author:** [@Guptha\_Bayyana](https://discuss.elastic.co/u/Guptha_Bayyana)\ **Replies:** 1\ **Last updated:** [June 1, 2024, 10:40am UTC](https://discuss.elastic.co/t/unable-to-create-an-index-and-send-logs-to-elasticsearch-from-logstash-getting-404-error/360657 "2024-06-01T10:40:33Z")
We are not able to create an Index on Elasticsearch using logstash. when I try the access indices I see 404 on the Elasticsearch \<{ "error" : { "root\_cause" : \[ { "type" : "index\_not\_found\_exception"… --- ## [Logstash Date does not match system](https://discuss.elastic.co/t/logstash-date-does-not-match-system/360459)
**Author:** [@iceman0410](https://discuss.elastic.co/u/iceman0410)\ **Replies:** 3\ **Last updated:** [June 1, 2024, 2:35am UTC](https://discuss.elastic.co/t/logstash-date-does-not-match-system/360459 "2024-06-01T02:35:29Z")
Hi everyone, I have 2 logs below need to parse. The out put timestamp match time but does not match date. Can anyone help me review m logstash conf ? Log message 1: \[13:35:32.727:\]ipr\_tcp\_open\_connect\_i(): fail openin… --- ## [How to swap field values](https://discuss.elastic.co/t/how-to-swap-field-values/360619)
**Author:** [@Zlobny\_Raven](https://discuss.elastic.co/u/Zlobny_Raven)\ **Replies:** 4\ **Last updated:** [May 31, 2024, 9:48pm UTC](https://discuss.elastic.co/t/how-to-swap-field-values/360619 "2024-05-31T21:48:42Z")
Hi, I need to swap values between two fields. I have: "field1" : "value1" "field2" : "value2" I need: "field1" : "value2" "field2" : "value1" Can I do it with help of some filter-plugin? --- ## [Logstash output S3 performance tunning](https://discuss.elastic.co/t/logstash-output-s3-performance-tunning/360640)
**Author:** [@Quan\_Le\_H\_i](https://discuss.elastic.co/u/Quan_Le_H_i)\ **Replies:** 0\ **Last updated:** [May 31, 2024, 5:06pm UTC](https://discuss.elastic.co/t/logstash-output-s3-performance-tunning/360640 "2024-05-31T17:06:38Z")
I have logstash server read events from input kafka and output to S3. My logstash pipeline have set pipeline.worker to 1 to set event in same order when write to file. After change other config such as pipeline.batchsi… --- ## [Aggregate documents](https://discuss.elastic.co/t/aggregate-documents/360599)
**Author:** [@reed](https://discuss.elastic.co/u/reed)\ **Replies:** 2\ **Last updated:** [May 31, 2024, 11:08am UTC](https://discuss.elastic.co/t/aggregate-documents/360599 "2024-05-31T11:08:18Z")
Hi all, I've created a pipeline that aggregate all input documents and generate a count for each group. in this moment I have two problems : at the first execution the aggregation works fine and the count is correct a… --- ## [Date parsing issue for ISO8601 in storing first line](https://discuss.elastic.co/t/date-parsing-issue-for-iso8601-in-storing-first-line/360608)
**Author:** [@prashant1](https://discuss.elastic.co/u/prashant1)\ **Replies:** 2\ **Last updated:** [May 31, 2024, 10:12am UTC](https://discuss.elastic.co/t/date-parsing-issue-for-iso8601-in-storing-first-line/360608 "2024-05-31T10:12:38Z")
We are reading the data from csv file and storing it on elasticsearch. Below is the filter used for date date { match =\> \[ "extractTimestamp", "ISO8601"\] target =\> "extractTimestamp" tag\_on\_failure =\> \["extr… --- ## [How to make field names lower case when using dynamic mapping](https://discuss.elastic.co/t/how-to-make-field-names-lower-case-when-using-dynamic-mapping/360566)
**Author:** [@bvoros](https://discuss.elastic.co/u/bvoros)\ **Replies:** 2\ **Last updated:** [May 31, 2024, 8:00am UTC](https://discuss.elastic.co/t/how-to-make-field-names-lower-case-when-using-dynamic-mapping/360566 "2024-05-31T08:00:28Z")
Hello, Is it possible to configure logstash to make all field names lowercase when using dynamic mapping? Thanks in advance once again, --- ## [How to append source system data into INDEX name Dynamically during push data from Logstash to ElasticSearch](https://discuss.elastic.co/t/how-to-append-source-system-data-into-index-name-dynamically-during-push-data-from-logstash-to-elasticsearch/360544)
**Author:** [@Manoilayans](https://discuss.elastic.co/u/Manoilayans)\ **Replies:** 1\ **Last updated:** [May 30, 2024, 10:24am UTC](https://discuss.elastic.co/t/how-to-append-source-system-data-into-index-name-dynamically-during-push-data-from-logstash-to-elasticsearch/360544 "2024-05-30T10:24:02Z")
output { elasticsearch { hosts =\>"http://localhost:9200" index =\> "control\_tower\_poc\_-%{+YYYY.MM.dd}" user =\> "elastic" password =\> "\*\*\*\*\*\*" } stdout { codec =\> rubydebug } } How to add … --- ## [Tomcat identity error log is not getting parsed](https://discuss.elastic.co/t/tomcat-identity-error-log-is-not-getting-parsed/360298)
**Author:** [@sudhir\_singh](https://discuss.elastic.co/u/sudhir_singh)\ **Replies:** 2\ **Last updated:** [May 30, 2024, 5:01am UTC](https://discuss.elastic.co/t/tomcat-identity-error-log-is-not-getting-parsed/360298 "2024-05-30T05:01:45Z")
Hello Everyone below is my log and when I'm trying to parse with multiline in logstash either it is getting all the events in one message field or the every line one events getting ingested in separate message. 05-27-20… --- ## [Force to output before processing](https://discuss.elastic.co/t/force-to-output-before-processing/360491)
**Author:** [@Keldane](https://discuss.elastic.co/u/Keldane)\ **Replies:** 1\ **Last updated:** [May 29, 2024, 6:48pm UTC](https://discuss.elastic.co/t/force-to-output-before-processing/360491 "2024-05-29T18:48:58Z")
Hi! Since i face some problems with nested fields in kibana lens, i decided to re-enginieer the ruby code for my filter, here is a snippet of it ruby { code =\> ' require "nokogiri" xml\_doc = Nokogiri::XM… --- ## [I'm having trouble integrating Logstash with the official Helm chart](https://discuss.elastic.co/t/im-having-trouble-integrating-logstash-with-the-official-helm-chart/360495)
**Author:** [@MrXY](https://discuss.elastic.co/u/MrXY)\ **Replies:** 0\ **Last updated:** [May 29, 2024, 4:40pm UTC](https://discuss.elastic.co/t/im-having-trouble-integrating-logstash-with-the-official-helm-chart/360495 "2024-05-29T16:40:27Z")
I'm attempting to integrate Logstash between Beats and Elasticsearch in Kubernetes. I'm utilizing the official Helm Chart available at artifacthub. I followed the guide provided by Elastic at secure-logstash-connections … --- ## [Joining 2 datasets together with the filter plugin does not work](https://discuss.elastic.co/t/joining-2-datasets-together-with-the-filter-plugin-does-not-work/360492)
**Author:** [@Ma\_G](https://discuss.elastic.co/u/Ma_G)\ **Replies:** 0\ **Last updated:** [May 29, 2024, 4:13pm UTC](https://discuss.elastic.co/t/joining-2-datasets-together-with-the-filter-plugin-does-not-work/360492 "2024-05-29T16:13:59Z")
Dear Elastic community, I am trying to join data from 2 indices together and write it to a third one. However, despite a correct Logstash configuration (I think) I do not get the desired result. Maybe one of you can spo… --- ## [How to handle key value pairs when keys and values are in different fields](https://discuss.elastic.co/t/how-to-handle-key-value-pairs-when-keys-and-values-are-in-different-fields/360460)
**Author:** [@bvoros](https://discuss.elastic.co/u/bvoros)\ **Replies:** 4\ **Last updated:** [May 29, 2024, 3:19pm UTC](https://discuss.elastic.co/t/how-to-handle-key-value-pairs-when-keys-and-values-are-in-different-fields/360460 "2024-05-29T15:19:35Z")
Hello, In the events that are being processed there are key value pairs where the keys and the values are in two fields. What is the best way to handle these? Example data: data.httpRequest.headers.name \[host, connect… --- ## [Need help to indentify what needs to be done in order to seperate and injest in elasticsearch](https://discuss.elastic.co/t/need-help-to-indentify-what-needs-to-be-done-in-order-to-seperate-and-injest-in-elasticsearch/360475)
**Author:** [@kishorkumar](https://discuss.elastic.co/u/kishorkumar)\ **Replies:** 0\ **Last updated:** [May 29, 2024, 1:38pm UTC](https://discuss.elastic.co/t/need-help-to-indentify-what-needs-to-be-done-in-order-to-seperate-and-injest-in-elasticsearch/360475 "2024-05-29T13:38:45Z")
I am trying to generate the logic in order to track last value injested in the elasticsearch for http\_poller. till now for testing version what i have done is i am using elastic index to elastic index pipeline. first i… --- ## [How to capture all entries that matches the same search pattern](https://discuss.elastic.co/t/how-to-capture-all-entries-that-matches-the-same-search-pattern/360473)
**Author:** [@thosch](https://discuss.elastic.co/u/thosch)\ **Replies:** 1\ **Last updated:** [May 29, 2024, 1:22pm UTC](https://discuss.elastic.co/t/how-to-capture-all-entries-that-matches-the-same-search-pattern/360473 "2024-05-29T13:22:47Z")
Hello, i am new at using logstash and its filters. I got log files looking like this: \ Started 'DB to PostgreSQL transfer' workflow at 2024.05.19 09:31:07 database v87\_orig\_rec\_8904\_03 ###########################… --- ## [How to drop the following documents?](https://discuss.elastic.co/t/how-to-drop-the-following-documents/360449)
**Author:** [@bvoros](https://discuss.elastic.co/u/bvoros)\ **Replies:** 3\ **Last updated:** [May 29, 2024, 11:55am UTC](https://discuss.elastic.co/t/how-to-drop-the-following-documents/360449 "2024-05-29T11:55:10Z")
Hello all, I would like to match and drop the following documents but my filter fails to match these. I am trying to match against the "message" field, could or should I match against the "event" field? Can someone exp… --- ## [Date filter results in empty documents and no error in logs](https://discuss.elastic.co/t/date-filter-results-in-empty-documents-and-no-error-in-logs/360397)
**Author:** [@bvoros](https://discuss.elastic.co/u/bvoros)\ **Replies:** 3\ **Last updated:** [May 29, 2024, 8:22am UTC](https://discuss.elastic.co/t/date-filter-results-in-empty-documents-and-no-error-in-logs/360397 "2024-05-29T08:22:42Z")
Hello, I am trying to apply the date filter against data that has been created by the json filter, the relevant field gets placed into \[data\]\[timestamp\] in text format, when the following filter is applied, logstash kee… --- ## [Logstash input data](https://discuss.elastic.co/t/logstash-input-data/360229)
**Author:** [@huanghaiqing1](https://discuss.elastic.co/u/huanghaiqing1)\ **Replies:** 4\ **Last updated:** [May 28, 2024, 12:16am UTC](https://discuss.elastic.co/t/logstash-input-data/360229 "2024-05-28T00:16:52Z")
A very strange question, I don't start any filebeat service, but just configure beats as input in my logstash.conf, and elasticsearch as my output. But in kibana I still can see data index created and data fetched in re… --- ## [Transport.go:125: SSL client failed to connect with: dial tcp xx.xx.xx.xx:5044: getsock…n refused](https://discuss.elastic.co/t/transport-go-ssl-client-failed-to-connect-with-dial-tcp-xx-xx-xx-xx-getsock-n-refused/360182)
**Author:** [@chandra\_sekhar](https://discuss.elastic.co/u/chandra_sekhar)\ **Replies:** 3\ **Last updated:** [May 27, 2024, 12:45pm UTC](https://discuss.elastic.co/t/transport-go-ssl-client-failed-to-connect-with-dial-tcp-xx-xx-xx-xx-getsock-n-refused/360182 "2024-05-27T12:45:09Z")
I am running ELK on 6.8.15 version which is pretty old version working as of now good. Since 2 weeks I am getting the following issues with filebeat is not able to communicate with logstatsh and getting the following SSL… --- ## [Need to parse multi-lines log message](https://discuss.elastic.co/t/need-to-parse-multi-lines-log-message/360206)
**Author:** [@iceman0410](https://discuss.elastic.co/u/iceman0410)\ **Replies:** 4\ **Last updated:** [May 27, 2024, 10:41am UTC](https://discuss.elastic.co/t/need-to-parse-multi-lines-log-message/360206 "2024-05-27T10:41:30Z")
Hi everyone, I am new on Grok. I am in stuck with multiple lines of log message below. Can anyone help me look at my grok. Thanks The log message: \[timestamp: 1621431760\] abort handler of pid 1823 thread 1848977280 \*… --- ## [Issue with the data ingestion/index creation using logstash](https://discuss.elastic.co/t/issue-with-the-data-ingestion-index-creation-using-logstash/360270)
**Author:** [@JimboSimbo](https://discuss.elastic.co/u/JimboSimbo)\ **Replies:** 0\ **Last updated:** [May 27, 2024, 9:57am UTC](https://discuss.elastic.co/t/issue-with-the-data-ingestion-index-creation-using-logstash/360270 "2024-05-27T09:57:18Z")
Hello Team, I am new to using this platform, hence posting my query here. I have a logstash conf file which is taking input from a csv file and I am parsing the data using csv filter plugin. Now the issue is I am using… --- ## [Can I do HA if my source of logs is kinesis for logstash](https://discuss.elastic.co/t/can-i-do-ha-if-my-source-of-logs-is-kinesis-for-logstash/360244)
**Author:** [@Drua\_Malik](https://discuss.elastic.co/u/Drua_Malik)\ **Replies:** 0\ **Last updated:** [May 26, 2024, 3:13pm UTC](https://discuss.elastic.co/t/can-i-do-ha-if-my-source-of-logs-is-kinesis-for-logstash/360244 "2024-05-26T15:13:02Z")
Please help me out [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=28) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=30)