# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=206 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 207 --- ## [How would be filter plugin for this desired output](https://discuss.elastic.co/t/how-would-be-filter-plugin-for-this-desired-output/280301)
**Author:** [@Srujan\_L](https://discuss.elastic.co/u/Srujan_L)\ **Replies:** 0\ **Last updated:** [August 3, 2021, 10:25am UTC](https://discuss.elastic.co/t/how-would-be-filter-plugin-for-this-desired-output/280301 "2021-08-03T10:25:34Z")
my input file looks like {"msg1":"\\"abcd\\" : \\"msg2\\""} {"msg1":"\\"Dhhdk\\" : \\"msg3\\""} {"msg1":"\\"Djskdh\\" : \\"msg2\\""} {"msg1":"\\"ksgdh\\" : \\"msg\\""} {"msg1":"\\"hxvdb\\" : \\"msg\\""} my desired output should be like {… --- ## [Couldn't find any input plugin named 'google\_pubsub'. Are you sure this is correct?](https://discuss.elastic.co/t/couldnt-find-any-input-plugin-named-google-pubsub-are-you-sure-this-is-correct/280218)
**Author:** [@devashishsingh](https://discuss.elastic.co/u/devashishsingh)\ **Replies:** 3\ **Last updated:** [August 3, 2021, 6:51am UTC](https://discuss.elastic.co/t/couldnt-find-any-input-plugin-named-google-pubsub-are-you-sure-this-is-correct/280218 "2021-08-03T06:51:50Z")
Hello, I noticed Kibana is no longer having Google cloud Stackdriver logs. So I started my troubleshooting with logstash and debug the .conf file, here's what I got: \[2021-08-02T20:08:44,540\]\[DEBUG\]\[logstash.plugins.re… --- ## [Logstash "Rejected mapping update", not sure how to rectify the situation](https://discuss.elastic.co/t/logstash-rejected-mapping-update-not-sure-how-to-rectify-the-situation/280269)
**Author:** [@CameronCenic](https://discuss.elastic.co/u/CameronCenic)\ **Replies:** 5\ **Last updated:** [August 3, 2021, 4:22am UTC](https://discuss.elastic.co/t/logstash-rejected-mapping-update-not-sure-how-to-rectify-the-situation/280269 "2021-08-03T04:22:36Z")
I am seeing an error: Rejecting mapping update to \[logstash-INDEXNAME-2021.08.03\] as the final mapping would have more than 1 type: \[\_doc, doc\] I suspect this is due to me switching my template from using doc to usin… --- ## [Maximum load of a single logstash instance](https://discuss.elastic.co/t/maximum-load-of-a-single-logstash-instance/280205)
**Author:** [@Rahul\_Dey](https://discuss.elastic.co/u/Rahul_Dey)\ **Replies:** 1\ **Last updated:** [August 3, 2021, 4:17am UTC](https://discuss.elastic.co/t/maximum-load-of-a-single-logstash-instance/280205 "2021-08-03T04:17:15Z")
We have build a logging system using Elastic Stack which is currently ingesting log data from different applications for monitoring. We have plans of adding more applications to our logging system in the future. Only one… --- ## [Logstash Config Error-JAVA](https://discuss.elastic.co/t/logstash-config-error-java/125546)
**Author:** [@Nagu\_R\_Pujari](https://discuss.elastic.co/u/Nagu_R_Pujari)\ **Replies:** 10\ **Last updated:** [August 3, 2021, 3:24am UTC](https://discuss.elastic.co/t/logstash-config-error-java/125546 "2021-08-03T03:24:15Z")
i have performing basic setup of logstash getting JAVA error. not able to complete the config test. Error: ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/etc/logstash/logstash.yml, … --- ## [Treating Logs as Milk, not Wine (the art of playing catchup)](https://discuss.elastic.co/t/treating-logs-as-milk-not-wine-the-art-of-playing-catchup/280257)
**Author:** [@cknz](https://discuss.elastic.co/u/cknz)\ **Replies:** 9\ **Last updated:** [August 3, 2021, 3:04am UTC](https://discuss.elastic.co/t/treating-logs-as-milk-not-wine-the-art-of-playing-catchup/280257 "2021-08-03T03:04:29Z")
I'm have a logging pipeline that looks something like the following: Logstash/Rsyslog/etc (submission) -\> Kafka -\> Logstash (enrichment) -\> Elasticsearch Metrics tells me that the by-far slowest part of my pipeline in … --- ## [Very very new to this](https://discuss.elastic.co/t/very-very-new-to-this/280248)
**Author:** [@vinceisvince](https://discuss.elastic.co/u/vinceisvince)\ **Replies:** 0\ **Last updated:** [August 2, 2021, 6:14pm UTC](https://discuss.elastic.co/t/very-very-new-to-this/280248 "2021-08-02T18:14:18Z")
I'll do a high level overview. We are using IBM's APIC v10. It uses Kubernetes which I'm all but a few days new to it. They seemt o have used logstash on top of that for their propriety "analytics server". I feel l… --- ## [Logstash strip string from field](https://discuss.elastic.co/t/logstash-strip-string-from-field/280233)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 8\ **Last updated:** [August 2, 2021, 5:24pm UTC](https://discuss.elastic.co/t/logstash-strip-string-from-field/280233 "2021-08-02T17:24:16Z")
Hi guys. In my logstash stream, I have a some filter to add a fields which is the merging of other fields. as follow: if \[destinationUserName\] and \[sourceUserName\] { mutate { add\_field =\> { "userID" =\> "%{ad.login… --- ## [Drop incoming messages](https://discuss.elastic.co/t/drop-incoming-messages/280236)
**Author:** [@rosboc](https://discuss.elastic.co/u/rosboc)\ **Replies:** 6\ **Last updated:** [August 2, 2021, 5:22pm UTC](https://discuss.elastic.co/t/drop-incoming-messages/280236 "2021-08-02T17:22:44Z")
Hi, i am using logstash 6.1.1 with following configuration and need to drop messages containing "type= traffic" string and i am not able to get it working. input { udp { port =\> 2514 type =\> "syslog… --- ## [Elasticsearch index not matching stdout test](https://discuss.elastic.co/t/elasticsearch-index-not-matching-stdout-test/280221)
**Author:** [@kgeil](https://discuss.elastic.co/u/kgeil)\ **Replies:** 0\ **Last updated:** [August 2, 2021, 12:56pm UTC](https://discuss.elastic.co/t/elasticsearch-index-not-matching-stdout-test/280221 "2021-08-02T12:56:20Z")
Good morning. I'm currently vexed by a situation with my elastic stack (7.13) and how things are parsing. I am parsing some exported IIS logs, and if I test my config with stdout, things look smashing. When I change m… --- ## [Logstash exit with code 0](https://discuss.elastic.co/t/logstash-exit-with-code-0/279891)
**Author:** [@abdo37](https://discuss.elastic.co/u/abdo37)\ **Replies:** 4\ **Last updated:** [August 2, 2021, 7:25am UTC](https://discuss.elastic.co/t/logstash-exit-with-code-0/279891 "2021-08-02T07:25:39Z")
heey i'm trynig to use docker-compose for logstash,elasticsearch,postgresql and when i start docker-compose all containers are working except logstash container it gives me logstash exit with code 0 there is my docker… --- ## [Filebeat integration as file](https://discuss.elastic.co/t/filebeat-integration-as-file/279990)
**Author:** [@doragon](https://discuss.elastic.co/u/doragon)\ **Replies:** 2\ **Last updated:** [August 2, 2021, 7:25am UTC](https://discuss.elastic.co/t/filebeat-integration-as-file/279990 "2021-08-02T07:25:30Z")
Good afternoon. I tried to setup a disconnected architecture between production and monitoring, as I have non-connectivity between my production servers and my ELK stack (local network). I planed to work as follow: f… --- ## [How to set the index name to the value of tags set by filebeat](https://discuss.elastic.co/t/how-to-set-the-index-name-to-the-value-of-tags-set-by-filebeat/279820)
**Author:** [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Replies:** 6\ **Last updated:** [August 2, 2021, 1:22am UTC](https://discuss.elastic.co/t/how-to-set-the-index-name-to-the-value-of-tags-set-by-filebeat/279820 "2021-08-02T01:22:08Z")
I am using filebeat to collect some logs. I have set the value of tags for each log, and I want to use that value as the name of the index. Is that possible? vi /etc/filebeat/filebeat.yml ... snip ... filebeat.inputs:… --- ## [Create fields when a word followed by ':'](https://discuss.elastic.co/t/create-fields-when-a-word-followed-by/280164)
**Author:** [@Arkapravo\_Das](https://discuss.elastic.co/u/Arkapravo_Das)\ **Replies:** 3\ **Last updated:** [August 1, 2021, 11:15pm UTC](https://discuss.elastic.co/t/create-fields-when-a-word-followed-by/280164 "2021-08-01T23:15:11Z")
Hi, What I am trying to do is , if the unformatted text is like updating demand record with key:202107231924440412212356|current demand quantity:1|quantity getting reduced:1|shipmentno:170801380 then I want to create… --- ## [Need help - Cisco syslog events matching in GROK debugger but not showing up in](https://discuss.elastic.co/t/need-help-cisco-syslog-events-matching-in-grok-debugger-but-not-showing-up-in/280116)
**Author:** [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Replies:** 7\ **Last updated:** [August 1, 2021, 7:59pm UTC](https://discuss.elastic.co/t/need-help-cisco-syslog-events-matching-in-grok-debugger-but-not-showing-up-in/280116 "2021-08-01T19:59:20Z")
the actual grok pattern(s) used to match Cisco syslog event messages is no longer working despite the match is successful in the grok debugger UI . other pattern is working fine for Juniper routers though... not sure wh… --- ## [After lot of effort my geo\_ip is still not working](https://discuss.elastic.co/t/after-lot-of-effort-my-geo-ip-is-still-not-working/280031)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 2\ **Last updated:** [August 1, 2021, 8:16am UTC](https://discuss.elastic.co/t/after-lot-of-effort-my-geo-ip-is-still-not-working/280031 "2021-08-01T08:16:57Z")
Hi Team, Below is my config and after lot of tries my geo\_ip is still not reflecting in my indices. Am I doing anything wrong here? input { stdin { type =\> "json" codec =\> "json… --- ## [Best strategy to deal with one event that contain multiple events, and graph it on kibana](https://discuss.elastic.co/t/best-strategy-to-deal-with-one-event-that-contain-multiple-events-and-graph-it-on-kibana/280137)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 0\ **Last updated:** [July 31, 2021, 7:42pm UTC](https://discuss.elastic.co/t/best-strategy-to-deal-with-one-event-that-contain-multiple-events-and-graph-it-on-kibana/280137 "2021-07-31T19:42:38Z")
Hi, after parsing a string field . it, I got several fields with different names, this names will change constantly and probably will appear new ones { "mfemve"=\>"0,39 ", "vmtoolsd"=\>"0,39 ", "zabbix\_agentd"=\>"0,32 ", … --- ## [MySQL 8: Could not create connection to database server](https://discuss.elastic.co/t/mysql-8-could-not-create-connection-to-database-server/280064)
**Author:** [@Justin\_Morgan](https://discuss.elastic.co/u/Justin_Morgan)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 11:57am UTC](https://discuss.elastic.co/t/mysql-8-could-not-create-connection-to-database-server/280064 "2021-07-30T11:57:29Z")
After upgrading to MySQL 8 I get the following error from my logstash pod: Loading class \`com.mysql.jdbc.Driver'. This is deprecated. The new driver class is \`com.mysql.cj.jdbc.Driver'. The driver is automatically regis… --- ## [Ruby hash to fields of an event](https://discuss.elastic.co/t/ruby-hash-to-fields-of-an-event/280102)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 10:03pm UTC](https://discuss.elastic.co/t/ruby-hash-to-fields-of-an-event/280102 "2021-07-30T22:03:53Z")
Hi I got the following ruby hash after parsing a long string with ruby code, { "mfemve"=\>"0,39 ", "vmtoolsd"=\>"0,39 ", "zabbix\_agentd"=\>"0,39 ", "powershell"=\>"0,39 ", "ir\_agent"=\>"0,00 ", "wmiprvse"=\>"0,00 ", "unsecap… --- ## [Output to json saving only specific fields](https://discuss.elastic.co/t/output-to-json-saving-only-specific-fields/280098)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 3\ **Last updated:** [July 30, 2021, 7:57pm UTC](https://discuss.elastic.co/t/output-to-json-saving-only-specific-fields/280098 "2021-07-30T19:57:42Z")
Hello everyone I am new logstash and I am sorry if the question is so basic. I have been saving my syslog into a csv file, as follow: csv { path =\> "C:\\path\\Desktop\\Adib\\adib-syslogs-%{+yyyy.MM.dd}.csv" csv\_o… --- ## [Logstash mutate field](https://discuss.elastic.co/t/logstash-mutate-field/280097)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 7:56pm UTC](https://discuss.elastic.co/t/logstash-mutate-field/280097 "2021-07-30T19:56:27Z")
Hi everyone, I hope somebody can help me with this issue I am facing with logstash. I am receiving some syslog, and trying to put in place some filtering and merging together 2 fields. the simple structure that I want… --- ## [How to prevent clobbering in translate filter](https://discuss.elastic.co/t/how-to-prevent-clobbering-in-translate-filter/280008)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 4\ **Last updated:** [July 30, 2021, 4:50pm UTC](https://discuss.elastic.co/t/how-to-prevent-clobbering-in-translate-filter/280008 "2021-07-30T16:50:51Z")
Hello , I'm using translate filter with a set of mappings in dictionary . { "@timestamp" =\> 2021-07-23T11:52:08.000Z, "Flag" =\> "16464", "Flag\_Definition" =\> "liveness test timed out… --- ## [JDBC last run is updated, but logstash always start from the beggining and duplicate documents](https://discuss.elastic.co/t/jdbc-last-run-is-updated-but-logstash-always-start-from-the-beggining-and-duplicate-documents/280091)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 0\ **Last updated:** [July 30, 2021, 3:11pm UTC](https://discuss.elastic.co/t/jdbc-last-run-is-updated-but-logstash-always-start-from-the-beggining-and-duplicate-documents/280091 "2021-07-30T15:11:53Z")
Hi, I have a pipeline with JDBC input with schedule of 5 min, I ran logstash as a service sudo systemctl start logstash and the JDBC last run is updated, but when logstash start the last run is not used an duplicate do… --- ## [Extract selected properties from a nested JSON with logstash?](https://discuss.elastic.co/t/extract-selected-properties-from-a-nested-json-with-logstash/280090)
**Author:** [@Samurai1](https://discuss.elastic.co/u/Samurai1)\ **Replies:** 0\ **Last updated:** [July 30, 2021, 2:53pm UTC](https://discuss.elastic.co/t/extract-selected-properties-from-a-nested-json-with-logstash/280090 "2021-07-30T14:53:53Z")
{ "billToLocation": { "prop1": "iwant", "prop2": { "prop2prop1": "iwant", "prop2prop2": { }, "prop2prop3:"iDontWant }, … --- ## [Want to add multiple log files as a input in filebeat.yml and send to logstash](https://discuss.elastic.co/t/want-to-add-multiple-log-files-as-a-input-in-filebeat-yml-and-send-to-logstash/280087)
**Author:** [@Aboli\_77](https://discuss.elastic.co/u/Aboli_77)\ **Replies:** 0\ **Last updated:** [July 30, 2021, 2:23pm UTC](https://discuss.elastic.co/t/want-to-add-multiple-log-files-as-a-input-in-filebeat-yml-and-send-to-logstash/280087 "2021-07-30T14:23:38Z")
Hi All, My filebeat.yml file input is: filebeat.inputs: type: log enabled: true paths: /home/XXXX/XYZ/abc.send.2\* /home/XXXX/XYZ/abc\_xyz.target\* /home/XXXX/XYZ/abc.sender.6\* /home/XXXX/XYZ/abc.log\* fields: log… --- ## [Grok fails to write events on a file](https://discuss.elastic.co/t/grok-fails-to-write-events-on-a-file/280081)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 2\ **Last updated:** [July 30, 2021, 2:04pm UTC](https://discuss.elastic.co/t/grok-fails-to-write-events-on-a-file/280081 "2021-07-30T14:04:46Z")
I have the following test configuration. Basically I want to write the messages in either the fail or the notfail files depending on whether the grok filter would succeed. I've created the files proactively under a folde… --- ## [Logstash filter take everything after](https://discuss.elastic.co/t/logstash-filter-take-everything-after/280061)
**Author:** [@Betorov](https://discuss.elastic.co/u/Betorov)\ **Replies:** 3\ **Last updated:** [July 30, 2021, 9:56am UTC](https://discuss.elastic.co/t/logstash-filter-take-everything-after/280061 "2021-07-30T09:56:21Z")
Hi avrey one, I wanted to filter this type of message: ip ip text I don't know the format of the text and what is inside of it.I only know that there is 2 ip and the text. How i can geto something that I don't know i… --- ## [Centralization Logstash](https://discuss.elastic.co/t/centralization-logstash/279938)
**Author:** [@Ilya\_21bA](https://discuss.elastic.co/u/Ilya_21bA)\ **Replies:** 3\ **Last updated:** [July 30, 2021, 1:10pm UTC](https://discuss.elastic.co/t/centralization-logstash/279938 "2021-07-30T13:10:47Z")
Hi, tell me, is it necessary to install the entire ELK stack on a new virtual machine, if I want to collect logs from several machines, tell me how it can be done, there are instructions. --- ## [Concerns with scaling Logstash when using the Aggregate filter](https://discuss.elastic.co/t/concerns-with-scaling-logstash-when-using-the-aggregate-filter/279855)
**Author:** [@nsouth](https://discuss.elastic.co/u/nsouth)\ **Replies:** 5\ **Last updated:** [July 30, 2021, 11:26am UTC](https://discuss.elastic.co/t/concerns-with-scaling-logstash-when-using-the-aggregate-filter/279855 "2021-07-30T11:26:55Z")
The aggregate filter is working great in my testing, but I'm concerned about scale because it is stateful and requires setting filter workers to 1 (-w 1 flag). We have about 150 servers, each of which will generate ~2GB … --- ## [Limit ElasticSearch repetitive logs after exception](https://discuss.elastic.co/t/limit-elasticsearch-repetitive-logs-after-exception/279695)
**Author:** [@Stefan\_Speranta](https://discuss.elastic.co/u/Stefan_Speranta)\ **Replies:** 0\ **Last updated:** [July 27, 2021, 7:28am UTC](https://discuss.elastic.co/t/limit-elasticsearch-repetitive-logs-after-exception/279695 "2021-07-27T07:28:50Z")
Hi, We have developed a project using Elastic, Logstash and Kibana as log management. After some site attacks one area of application got into endless loops of exceptions and filled Elastic with the same log massage ag… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=205) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=207)