# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=141 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 142 --- ## [Problem with date filter](https://discuss.elastic.co/t/problem-with-date-filter/304261)
**Author:** [@Thuunder7](https://discuss.elastic.co/u/Thuunder7)\ **Replies:** 5\ **Last updated:** [May 13, 2022, 8:29am UTC](https://discuss.elastic.co/t/problem-with-date-filter/304261 "2022-05-13T08:29:08Z")
Hello, I have been trying to use the date filter plugin but without success. I am trying to parse a field and target it into @timestamp field. My message field contains a date string like the following: 2022-05-09 09:… --- ## [Logstash running on machine has high load average](https://discuss.elastic.co/t/logstash-running-on-machine-has-high-load-average/304321)
**Author:** [@Thuunder7](https://discuss.elastic.co/u/Thuunder7)\ **Replies:** 3\ **Last updated:** [May 13, 2022, 8:28am UTC](https://discuss.elastic.co/t/logstash-running-on-machine-has-high-load-average/304321 "2022-05-13T08:28:15Z")
Hello guys, I am currently running into some issues regarding the usage of logstash. I am using the following architecture: kafka -\> logstash -\> elasticsearch We have 3 kafka brokers running on separated machines, 2 l… --- ## [Logstash config Alertmanager](https://discuss.elastic.co/t/logstash-config-alertmanager/304483)
**Author:** [@linuxxin](https://discuss.elastic.co/u/linuxxin)\ **Replies:** 3\ **Last updated:** [May 13, 2022, 2:53am UTC](https://discuss.elastic.co/t/logstash-config-alertmanager/304483 "2022-05-13T02:53:19Z")
Log resolution using logstash configure keywords to send alarms to Alertmanager for email or Webhook notification --- ## [Logstash KV filter escape ":"](https://discuss.elastic.co/t/logstash-kv-filter-escape/304624)
**Author:** [@Jim\_Thunder](https://discuss.elastic.co/u/Jim_Thunder)\ **Replies:** 1\ **Last updated:** [May 12, 2022, 10:54pm UTC](https://discuss.elastic.co/t/logstash-kv-filter-escape/304624 "2022-05-12T22:54:26Z")
Hello, I have a message here (all message will be like this): {"Time": "2022-05-12T04:18:46.077", "HostName": "IBMNOAH", "Cat": "JOBLOG", "Severity": "Err", "SAF": 1, "SAFD": "RACF", "Name": "TEST", "JobName": "DBS1MST… --- ## [Field Parse Issue](https://discuss.elastic.co/t/field-parse-issue/304596)
**Author:** [@hasnainNM](https://discuss.elastic.co/u/hasnainNM)\ **Replies:** 4\ **Last updated:** [May 12, 2022, 6:43pm UTC](https://discuss.elastic.co/t/field-parse-issue/304596 "2022-05-12T18:43:20Z")
I have CSV field & logstash ins't parsing it entirely, i have the first few fields parsed correctly yet once it comes across console it seems to stop parsing. Any idea of fixes? Here is my code filter { csv { … --- ## [WARN \]\[logstash.inputs.rabbitmq \] Error while setting up connection for rabbitmq input!](https://discuss.elastic.co/t/warn-logstash-inputs-rabbitmq-error-while-setting-up-connection-for-rabbitmq-input/304611)
**Author:** [@lopa](https://discuss.elastic.co/u/lopa)\ **Replies:** 5\ **Last updated:** [May 12, 2022, 6:42pm UTC](https://discuss.elastic.co/t/warn-logstash-inputs-rabbitmq-error-while-setting-up-connection-for-rabbitmq-input/304611 "2022-05-12T18:42:19Z")
Hi, I'm using Logstash6.8.23 and Logstash consumes RabbitMQ message as expected. But Logstash continuously flooding with many warnings as "\[logstash.inputs.rabbitmq \] Error while setting up connection for rabbitmq inp… --- ## [Logstach is broken when using syslog pipeline](https://discuss.elastic.co/t/logstach-is-broken-when-using-syslog-pipeline/304584)
**Author:** [@sbocquet](https://discuss.elastic.co/u/sbocquet)\ **Replies:** 5\ **Last updated:** [May 12, 2022, 4:39pm UTC](https://discuss.elastic.co/t/logstach-is-broken-when-using-syslog-pipeline/304584 "2022-05-12T16:39:13Z")
Hi, I've just installed a brand new ElacticSearch v8.2 with Logstach. Both a talking well together. Kibana is installed too. Background server are Debian 11 up to date. I try to send all the logs I collect with a syslo… --- ## [Update existing document](https://discuss.elastic.co/t/update-existing-document/304581)
**Author:** [@ChinigamiHunter](https://discuss.elastic.co/u/ChinigamiHunter)\ **Replies:** 2\ **Last updated:** [May 12, 2022, 4:24pm UTC](https://discuss.elastic.co/t/update-existing-document/304581 "2022-05-12T16:24:01Z")
I am facing the following issue with Elasticsearch and logstash. every day logstash create index with current date like : mydata\_2022.05.12 i have a index patterns name: mydata\_\* i get documents from api and index the… --- ## [Remove left whitespaces logstash](https://discuss.elastic.co/t/remove-left-whitespaces-logstash/304514)
**Author:** [@dannie-ml](https://discuss.elastic.co/u/dannie-ml)\ **Replies:** 7\ **Last updated:** [May 12, 2022, 4:01pm UTC](https://discuss.elastic.co/t/remove-left-whitespaces-logstash/304514 "2022-05-12T16:01:15Z")
Hi i have the following fields: "delivery\_quantity" : " \\s\\s\\s\\s4", "numerator" : "\\s\\s\\s1", "denominator" : "\\s\\s\\s1" I've tried to use mutate split and gsub but didnt work. I've seen that split works when you ha… --- ## [How to build grok filter](https://discuss.elastic.co/t/how-to-build-grok-filter/304537)
**Author:** [@smam](https://discuss.elastic.co/u/smam)\ **Replies:** 10\ **Last updated:** [May 12, 2022, 10:31am UTC](https://discuss.elastic.co/t/how-to-build-grok-filter/304537 "2022-05-12T10:31:42Z")
Hello, I want to understand how to build grok filter since other posts did not make it clear enough for me. Here are example lines I want to send: 30,05/10/22,07:30:27,DNS Update Request,\,\,,,0,6,,,,,,,,,0 11… --- ## [How to increase the performance of the kafka input in indexer](https://discuss.elastic.co/t/how-to-increase-the-performance-of-the-kafka-input-in-indexer/304549)
**Author:** [@Marco\_Bonardo](https://discuss.elastic.co/u/Marco_Bonardo)\ **Replies:** 0\ **Last updated:** [May 12, 2022, 9:42am UTC](https://discuss.elastic.co/t/how-to-increase-the-performance-of-the-kafka-input-in-indexer/304549 "2022-05-12T09:42:39Z")
Hello everyone, I have a problem I've been working on for days and I can't find a compatible solution. I need to consume about 150k messages per second via a kafka input / topic / groupid. I have 8 logstash on separate m… --- ## [Grok issue in Logstash input - data from Kafka logs using Filebeat Kafka module](https://discuss.elastic.co/t/grok-issue-in-logstash-input-data-from-kafka-logs-using-filebeat-kafka-module/303801)
**Author:** [@Sharon\_Hacham](https://discuss.elastic.co/u/Sharon_Hacham)\ **Replies:** 4\ **Last updated:** [May 12, 2022, 4:57am UTC](https://discuss.elastic.co/t/grok-issue-in-logstash-input-data-from-kafka-logs-using-filebeat-kafka-module/303801 "2022-05-12T04:57:44Z")
Hi , we are ingesting Kafka logs from on-premise servers running Kafka Connect and Kafka Rest-Proxy , We have 3 Logstash Windows servers ... we're using the built-in Dashboard you supply with the Filebeat Kafka module… --- ## [Http input and output google bigquery](https://discuss.elastic.co/t/http-input-and-output-google-bigquery/304323)
**Author:** [@dharminfadia](https://discuss.elastic.co/u/dharminfadia)\ **Replies:** 4\ **Last updated:** [May 12, 2022, 4:27am UTC](https://discuss.elastic.co/t/http-input-and-output-google-bigquery/304323 "2022-05-12T04:27:22Z")
input { http { host =\> "0.0.0.0" # default: 0.0.0.0 port =\> 0000 # default: 8080 } } filter { urldecode { all\_fields =\> true } } filter { mutate { gsub =\> \[ "message", "\\\]", "}", "message", "\\\[", "{" \] } … --- ## [How to remove additional comma in part of message](https://discuss.elastic.co/t/how-to-remove-additional-comma-in-part-of-message/304472)
**Author:** [@royalE](https://discuss.elastic.co/u/royalE)\ **Replies:** 1\ **Last updated:** [May 11, 2022, 3:36pm UTC](https://discuss.elastic.co/t/how-to-remove-additional-comma-in-part-of-message/304472 "2022-05-11T15:36:38Z")
I have string between quotation marks that is separated by comma example :"hello ,world" logstash separate them in different column which is not the output that I want I would like to remove the quotation and remove t… --- ## [Logstash configuration to load next pages automatically](https://discuss.elastic.co/t/logstash-configuration-to-load-next-pages-automatically/304457)
**Author:** [@Iss](https://discuss.elastic.co/u/Iss)\ **Replies:** 2\ **Last updated:** [May 11, 2022, 1:05pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-load-next-pages-automatically/304457 "2022-05-11T13:05:47Z")
I am new with ELK. I apologize for mistake I have to configure logstash in order to import data from an external source using an oai-pmh API. I try it using the following configuration. input{ http\_poller{ urls =\>{… --- ## [Logstash Illegal Argument Exception while parsing date](https://discuss.elastic.co/t/logstash-illegal-argument-exception-while-parsing-date/303142)
**Author:** [@b.ohmer](https://discuss.elastic.co/u/b.ohmer)\ **Replies:** 5\ **Last updated:** [May 10, 2022, 4:24pm UTC](https://discuss.elastic.co/t/logstash-illegal-argument-exception-while-parsing-date/303142 "2022-05-10T16:24:04Z")
Hi there, I've got three questions around Logstash: We're using an older version of Logstash (6.5.4) and experiencing currently an issue while parsing a date. I've got absolutely no idea anymore what I can do to make … --- ## [Incorrectly collected data in Logstash to Elasticsearch](https://discuss.elastic.co/t/incorrectly-collected-data-in-logstash-to-elasticsearch/303910)
**Author:** [@eduhernandezm](https://discuss.elastic.co/u/eduhernandezm)\ **Replies:** 11\ **Last updated:** [May 10, 2022, 4:17pm UTC](https://discuss.elastic.co/t/incorrectly-collected-data-in-logstash-to-elasticsearch/303910 "2022-05-10T16:17:53Z")
Hello, I have set up two processes to collect data and present it to ELK. On the one hand I have a process that collects log data from an iis and on the other I have another process that what it does is import the data… --- ## [Unable to Start Logstash with Proxy](https://discuss.elastic.co/t/unable-to-start-logstash-with-proxy/304350)
**Author:** [@hell\_storm2004](https://discuss.elastic.co/u/hell_storm2004)\ **Replies:** 0\ **Last updated:** [May 10, 2022, 1:23pm UTC](https://discuss.elastic.co/t/unable-to-start-logstash-with-proxy/304350 "2022-05-10T13:23:44Z")
I am currently trying to learn about Logstash. So I just download the zip file and put in some basic configuration and ran logstash.bat file. But for some reason I get Proxy Authentication exception. This is console out… --- ## [Logstash extract part of field](https://discuss.elastic.co/t/logstash-extract-part-of-field/304312)
**Author:** [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Replies:** 2\ **Last updated:** [May 10, 2022, 8:14am UTC](https://discuss.elastic.co/t/logstash-extract-part-of-field/304312 "2022-05-10T08:14:09Z")
Dears, ELK in version 7.16.3 What is the best way to extract part of mapped xml field to new field? if "app1xml" in \[tags\] { xml { source =\> "message" store\_xml =\> false force\_array =\> false xpath … --- ## [Logstash email output occur below error: \](https://discuss.elastic.co/t/logstash-email-output-occur-below-error-errno-broken-pipe-no-message-available/304301)
**Author:** [@haohao6683](https://discuss.elastic.co/u/haohao6683)\ **Replies:** 2\ **Last updated:** [May 10, 2022, 7:31am UTC](https://discuss.elastic.co/t/logstash-email-output-occur-below-error-errno-broken-pipe-no-message-available/304301 "2022-05-10T07:31:20Z")
Hi all, I am currently using Logstash-email output with the following configuration: (version: 7.3.1) email { from =\> "sender@qq.com" subject =\> "Ignore (test run)" body =\> "log-message: I… --- ## [How to remove event with "'\'" value on an IP type field](https://discuss.elastic.co/t/how-to-remove-event-with-nil-value-on-an-ip-type-field/304273)
**Author:** [@Thuunder7](https://discuss.elastic.co/u/Thuunder7)\ **Replies:** 6\ **Last updated:** [May 9, 2022, 7:45pm UTC](https://discuss.elastic.co/t/how-to-remove-event-with-nil-value-on-an-ip-type-field/304273 "2022-05-09T19:45:23Z")
Hello, I am trying to drop events every time the field \[dns.resolved\_ip\] is ''. I have tried multiple approachs but without successs. "reason"=\>"failed to parse field \[dns.resolved\_ip\] of type \[ip\] in document with id… --- ## [JSON codec plugin - target option (http input)](https://discuss.elastic.co/t/json-codec-plugin-target-option-http-input/304217)
**Author:** [@sectex](https://discuss.elastic.co/u/sectex)\ **Replies:** 4\ **Last updated:** [May 9, 2022, 3:12pm UTC](https://discuss.elastic.co/t/json-codec-plugin-target-option-http-input/304217 "2022-05-09T15:12:02Z")
I am having some issues using the JSON codec plugin in my Logstash (v8.1.2) pipeline. pipeline.conf: input { http { codec =\> json port =\> 5046 } } output { stdout { codec =\> rubydebu… --- ## [Logstash ECS Compatiblity Issues](https://discuss.elastic.co/t/logstash-ecs-compatiblity-issues/304260)
**Author:** [@teamomni](https://discuss.elastic.co/u/teamomni)\ **Replies:** 5\ **Last updated:** [May 9, 2022, 2:54pm UTC](https://discuss.elastic.co/t/logstash-ecs-compatiblity-issues/304260 "2022-05-09T14:54:16Z")
We have moved our Logstash indices to data stream recently but are having issues with the geopoints. As a result, the geofencing for everything including the Elastic agents are broken. At first there was the error: Pi… --- ## [Unable to add tag with filename](https://discuss.elastic.co/t/unable-to-add-tag-with-filename/304113)
**Author:** [@Fico125](https://discuss.elastic.co/u/Fico125)\ **Replies:** 4\ **Last updated:** [May 9, 2022, 1:50pm UTC](https://discuss.elastic.co/t/unable-to-add-tag-with-filename/304113 "2022-05-09T13:50:37Z")
Hey everyone, I'm quite stuck trying to add a custom field to my parsed data. Basically, I have a path from where I'm reading the content of csv files, and sending it to elastic. I'd like that content of .csv files rece… --- ## [Remove\_Field json path on logstash](https://discuss.elastic.co/t/remove-field-json-path-on-logstash/304146)
**Author:** [@adxalex](https://discuss.elastic.co/u/adxalex)\ **Replies:** 5\ **Last updated:** [May 9, 2022, 1:44pm UTC](https://discuss.elastic.co/t/remove-field-json-path-on-logstash/304146 "2022-05-09T13:44:52Z")
Hi comunity, I'm trying to remove some fileds into a json. The json is: {"recode":"VZ##","response-code":"4000","response":{"result":\[{"DetailsPageURL":"idsource":"0""Attribute":\[{"DISPLAYNAME":"Tiempo de respuesta ser… --- ## [ECS compatibility target info message](https://discuss.elastic.co/t/ecs-compatibility-target-info-message/304244)
**Author:** [@thibveni](https://discuss.elastic.co/u/thibveni)\ **Replies:** 1\ **Last updated:** [May 9, 2022, 1:28pm UTC](https://discuss.elastic.co/t/ecs-compatibility-target-info-message/304244 "2022-05-09T13:28:46Z")
Hello, i'm using logstash in v8.1.3 and since the upgrade in v8 i have this info message: logstash\_1 | \[2022-05-09T09:57:59,861\]\[INFO \]\[logstash.codecs.jsonlines\] ECS compatibility is enabled but \`target\` option was no… --- ## [Timeout in Logstash Elasticsearch Filter and Elasticsearch Output](https://discuss.elastic.co/t/timeout-in-logstash-elasticsearch-filter-and-elasticsearch-output/304253)
**Author:** [@sagarpatel](https://discuss.elastic.co/u/sagarpatel)\ **Replies:** 0\ **Last updated:** [May 9, 2022, 12:46pm UTC](https://discuss.elastic.co/t/timeout-in-logstash-elasticsearch-filter-and-elasticsearch-output/304253 "2022-05-09T12:46:00Z")
Elasticsearch version: 7.16 (running as elastic cloud) Logstash version: 7.16 I am using http\_poller input plugin which is scheduled every 15 mins. Based on the http\_poller API response I need to execute Elasticsearch … --- ## [How to remove fields exist in \_source array field using logstash filter](https://discuss.elastic.co/t/how-to-remove-fields-exist-in-source-array-field-using-logstash-filter/303350)
**Author:** [@Magesh\_02](https://discuss.elastic.co/u/Magesh_02)\ **Replies:** 1\ **Last updated:** [May 9, 2022, 8:39am UTC](https://discuss.elastic.co/t/how-to-remove-fields-exist-in-source-array-field-using-logstash-filter/303350 "2022-05-09T08:39:14Z")
I'm trying to use a logstash filter to remove unwanted fields from the \_source field, but the data isn't being sent to ES. filter { mutate { remove\_field =\> \[ "path", "host","@version","@timestamp","type"\] … --- ## [Loading unstructured data](https://discuss.elastic.co/t/loading-unstructured-data/304212)
**Author:** [@Naga\_Prudhvi](https://discuss.elastic.co/u/Naga_Prudhvi)\ **Replies:** 2\ **Last updated:** [May 9, 2022, 8:25am UTC](https://discuss.elastic.co/t/loading-unstructured-data/304212 "2022-05-09T08:25:33Z")
I am getting error while loading below line in elastic using gork filter 192.168.72.177 - - \[22/Dec/2002:23:32:14 -0400\] "GET /news/sports.html HTTP/1.1" 200 3500 www.yahoo.com "http://www.some.com/" "Mozilla/4.0 (com… --- ## [Logstash S3 input](https://discuss.elastic.co/t/logstash-s3-input/304118)
**Author:** [@shivendra95](https://discuss.elastic.co/u/shivendra95)\ **Replies:** 3\ **Last updated:** [May 9, 2022, 7:34am UTC](https://discuss.elastic.co/t/logstash-s3-input/304118 "2022-05-09T07:34:07Z")
Hi there, I'm trying to use the logstash S3 input plugin to fetch logs from S3 bucket. This is my input s3 { id =\> "terminal\_app" access\_key\_id =\> "${access\_key\_id}" secret\_access\_key =\> "${secret\_access\_k… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=140) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=142)