# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=112 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 113 --- ## [Exclude multiple patterns in logstash s3 input](https://discuss.elastic.co/t/exclude-multiple-patterns-in-logstash-s3-input/315917)
**Author:** [@Elias\_Ojeda](https://discuss.elastic.co/u/Elias_Ojeda)\ **Replies:** 1\ **Last updated:** [October 5, 2022, 11:55pm UTC](https://discuss.elastic.co/t/exclude-multiple-patterns-in-logstash-s3-input/315917 "2022-10-05T23:55:59Z")
Logstash S3 input has an option to exclude a pattern. The example shows how to exclude a single pattern. For example: "exclude\_pattern" =\> "/2020/04/" Is it possible to exclude multiple patterns? For example, exclude… --- ## [Count of files in sincedb and file\_completed\_log\_path](https://discuss.elastic.co/t/count-of-files-in-sincedb-and-file-completed-log-path/315907)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 3\ **Last updated:** [October 5, 2022, 7:07pm UTC](https://discuss.elastic.co/t/count-of-files-in-sincedb-and-file-completed-log-path/315907 "2022-10-05T19:07:09Z")
Hi Can You explain why I have a so much difference in count of particular daily file in sincedb and file\_complete\_log\_path. It seems that sincedb does not reflect the actual number of reprocessed files, hence there are n… --- ## [Null terminated message string](https://discuss.elastic.co/t/null-terminated-message-string/315905)
**Author:** [@LisaJ](https://discuss.elastic.co/u/LisaJ)\ **Replies:** 2\ **Last updated:** [October 5, 2022, 6:59pm UTC](https://discuss.elastic.co/t/null-terminated-message-string/315905 "2022-10-05T18:59:57Z")
I have a syslog message that contains a null terminated string: "syslog\_message":"A10\\u0000" -- these messages represent is-alive checks from a load balancer to the logstash servers. I would prefer not to have thousands … --- ## [How to partial update es in the output plugin of logstash](https://discuss.elastic.co/t/how-to-partial-update-es-in-the-output-plugin-of-logstash/315864)
**Author:** [@Juno163](https://discuss.elastic.co/u/Juno163)\ **Replies:** 0\ **Last updated:** [October 5, 2022, 11:03am UTC](https://discuss.elastic.co/t/how-to-partial-update-es-in-the-output-plugin-of-logstash/315864 "2022-10-05T11:03:22Z")
By default, logstash output cannot partial update Elasticsearch but fully update. In my case, I can only search some fields which need to update, when it comes to es, the fields which didn't searched will update as blan… --- ## [Can logstash\_admin built-in role make indices?](https://discuss.elastic.co/t/can-logstash-admin-built-in-role-make-indices/315834)
**Author:** [@mikhatanu](https://discuss.elastic.co/u/mikhatanu)\ **Replies:** 0\ **Last updated:** [October 5, 2022, 4:36am UTC](https://discuss.elastic.co/t/can-logstash-admin-built-in-role-make-indices/315834 "2022-10-05T04:36:38Z")
Hello, i'm planning to add a new user in kibana for logstash operation. I noticed the role logstash\_admin which is alerady built-in. My question is, can the logstash\_admin role make new indices (as configured in logstash… --- ## [Extract originals timestamp from logs?](https://discuss.elastic.co/t/extract-originals-timestamp-from-logs/315798)
**Author:** [@Nikolas1306](https://discuss.elastic.co/u/Nikolas1306)\ **Replies:** 6\ **Last updated:** [October 5, 2022, 3:06am UTC](https://discuss.elastic.co/t/extract-originals-timestamp-from-logs/315798 "2022-10-05T03:06:35Z")
hello i've see with send log with logstash not respect the originals timestamp this is my generic msg builded with spring-boot 2022-09-25 21:19:57,175 INFO \[it.unidoc.cdr.core.bean.ForwardProvideAndRegisterManager\] (C… --- ## [Timestamp pattern not working for logstash filtering](https://discuss.elastic.co/t/timestamp-pattern-not-working-for-logstash-filtering/315827)
**Author:** [@Chma](https://discuss.elastic.co/u/Chma)\ **Replies:** 1\ **Last updated:** [October 5, 2022, 2:53am UTC](https://discuss.elastic.co/t/timestamp-pattern-not-working-for-logstash-filtering/315827 "2022-10-05T02:53:24Z")
I have 3 lines of logs with different structure and i am constructing a grok pattern to filter the logs. But the pattern I have isn't working. Even when I use just the timestamp pattern, it doesn't match. \[2022-10-04 21… --- ## [Duplicated date in my elastic](https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506)
**Author:** [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Replies:** 5\ **Last updated:** [October 4, 2022, 10:56pm UTC](https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506 "2022-10-04T22:56:08Z")
Hello Team, Greeting, I have a situation when the data is being pushing into my elastic twice, a duplicate data. now both of them have different id and other than that everything is same. How do I get over this. My f… --- ## [Mutate convert multiple fields at once](https://discuss.elastic.co/t/mutate-convert-multiple-fields-at-once/315730)
**Author:** [@mikhatanu](https://discuss.elastic.co/u/mikhatanu)\ **Replies:** 1\ **Last updated:** [October 4, 2022, 3:09pm UTC](https://discuss.elastic.co/t/mutate-convert-multiple-fields-at-once/315730 "2022-10-04T15:09:58Z")
Hello, can i mutate convert multiple fields to a data type at once, without putting it one by one? something like: filter{ mutate{ convert(\["someField1","someField2","someField3"\] =\> "integer") } } I have a lar… --- ## [Filter first match only using Grok or ruby code](https://discuss.elastic.co/t/filter-first-match-only-using-grok-or-ruby-code/315758)
**Author:** [@rootk1d](https://discuss.elastic.co/u/rootk1d)\ **Replies:** 1\ **Last updated:** [October 4, 2022, 11:52am UTC](https://discuss.elastic.co/t/filter-first-match-only-using-grok-or-ruby-code/315758 "2022-10-04T11:52:44Z")
I have a field that contains the following data "REQUEST-941-APPLICATION-ATTACK-XSS, REQUEST-941-APPLICATION-ATTACK-XSS, REQUEST-941-APPLICATION-ATTACK-XSS, REQUEST-942-APPLICATION-ATTACK-SQLI, REQUEST-949-BLOCKING-EVAL… --- ## [Logstash Cloudwatch Input Plugin](https://discuss.elastic.co/t/logstash-cloudwatch-input-plugin/314694)
**Author:** [@connectgeeks](https://discuss.elastic.co/u/connectgeeks)\ **Replies:** 2\ **Last updated:** [October 4, 2022, 11:09am UTC](https://discuss.elastic.co/t/logstash-cloudwatch-input-plugin/314694 "2022-10-04T11:09:23Z")
Hi I'm using input cloudwatch plugin for aws cloudwatch ingestion using logstash. I want to reingest the entire data now. How can I achieve that ? Also, where (path) logstash keeps the registry file to maintain the seq… --- ## [Helm chart - logstash elasticsearch connection configuration?](https://discuss.elastic.co/t/helm-chart-logstash-elasticsearch-connection-configuration/315756)
**Author:** [@tirelibirefe](https://discuss.elastic.co/u/tirelibirefe)\ **Replies:** 0\ **Last updated:** [October 4, 2022, 9:34am UTC](https://discuss.elastic.co/t/helm-chart-logstash-elasticsearch-connection-configuration/315756 "2022-10-04T09:34:01Z")
Hello, I installed Logstash by using Helm Chart here and need to configure Elasticsearch connection. I looked for documentation but couldn't have found... How can I configure elasticsearch connection? Thanks & Regar… --- ## [Configuration for shipping Docker logs to a locally installed Logstash](https://discuss.elastic.co/t/configuration-for-shipping-docker-logs-to-a-locally-installed-logstash/315725)
**Author:** [@Chma](https://discuss.elastic.co/u/Chma)\ **Replies:** 0\ **Last updated:** [October 3, 2022, 10:27pm UTC](https://discuss.elastic.co/t/configuration-for-shipping-docker-logs-to-a-locally-installed-logstash/315725 "2022-10-03T22:27:30Z")
I have my ELK deployed on an ec2 instance and a dockerized application running on a different instance. I am trying to use gelf to collect the different service logs and send to logstash. But my current configuration doe… --- ## [CSV Filter - Backslash double quote parse failure](https://discuss.elastic.co/t/csv-filter-backslash-double-quote-parse-failure/315703)
**Author:** [@Michele\_De\_Benedet](https://discuss.elastic.co/u/Michele_De_Benedet)\ **Replies:** 1\ **Last updated:** [October 3, 2022, 4:54pm UTC](https://discuss.elastic.co/t/csv-filter-backslash-double-quote-parse-failure/315703 "2022-10-03T16:54:07Z")
Hi, I am using version 7.12.0 I'm parsing a csv which looks like this: colA;colB;colC x;"hello \\"world\\" ";0 y;"hello world";1 I cannot parse the x row because I get the following: Error parsing csv {:field=\>"messag… --- ## [Split the Field values by comma](https://discuss.elastic.co/t/split-the-field-values-by-comma/315673)
**Author:** [@Maruthappan\_Muthu](https://discuss.elastic.co/u/Maruthappan_Muthu)\ **Replies:** 1\ **Last updated:** [October 3, 2022, 4:06pm UTC](https://discuss.elastic.co/t/split-the-field-values-by-comma/315673 "2022-10-03T16:06:23Z")
My field contains multiple values, which separated by comma. How can I split the each comma separated values and put on the same filed name. Ex. The consider the below field and values, SerialNumber =\> \[abc23, cde56, … --- ## [How to read logs contineously from azure evnet hub](https://discuss.elastic.co/t/how-to-read-logs-contineously-from-azure-evnet-hub/315660)
**Author:** [@Anusha\_Kusanghi](https://discuss.elastic.co/u/Anusha_Kusanghi)\ **Replies:** 0\ **Last updated:** [October 3, 2022, 7:27am UTC](https://discuss.elastic.co/t/how-to-read-logs-contineously-from-azure-evnet-hub/315660 "2022-10-03T07:27:40Z")
Hie Everyone, I have an integration where I have to read logs that are generated/store in azure event hub. Will the below configuration will read logs as an when new log is generated in the hub ? If now what I'm loo… --- ## [Filtered Log messages show up as empty fields in Kibana](https://discuss.elastic.co/t/filtered-log-messages-show-up-as-empty-fields-in-kibana/315399)
**Author:** [@Chma](https://discuss.elastic.co/u/Chma)\ **Replies:** 1\ **Last updated:** [October 1, 2022, 2:02am UTC](https://discuss.elastic.co/t/filtered-log-messages-show-up-as-empty-fields-in-kibana/315399 "2022-10-01T02:02:28Z")
I have filtered my log message using grok. But when I check Kibana, I find the new fields on the left side of the page, but they are empty. I am also getting the \_grokparsefailure tag. Here's an example of my log messag… --- ## [How create 2 different indexs with same file source?](https://discuss.elastic.co/t/how-create-2-different-indexs-with-same-file-source/315594)
**Author:** [@Nikolas1306](https://discuss.elastic.co/u/Nikolas1306)\ **Replies:** 3\ **Last updated:** [September 30, 2022, 5:07pm UTC](https://discuss.elastic.co/t/how-create-2-different-indexs-with-same-file-source/315594 "2022-09-30T17:07:56Z")
this is my conf file and it's ok i've created an index with only errors match but now it's possible create a second indexs with file warning or custom? code =\> "event.cancel if not event.get('message').include? 'WARN' … --- ## [Custom filter in ruby?](https://discuss.elastic.co/t/custom-filter-in-ruby/315567)
**Author:** [@Nikolas1306](https://discuss.elastic.co/u/Nikolas1306)\ **Replies:** 2\ **Last updated:** [September 30, 2022, 3:49pm UTC](https://discuss.elastic.co/t/custom-filter-in-ruby/315567 "2022-09-30T15:49:39Z")
hello i've used this config but have error when executed filter { ruby { code =\> ' if event.get("message").include? ' INFO ' event.cancel end ' … --- ## [High CPU usage logstash](https://discuss.elastic.co/t/high-cpu-usage-logstash/315468)
**Author:** [@Cheroufa](https://discuss.elastic.co/u/Cheroufa)\ **Replies:** 10\ **Last updated:** [September 30, 2022, 3:27pm UTC](https://discuss.elastic.co/t/high-cpu-usage-logstash/315468 "2022-09-30T15:27:06Z")
Hi all, Whene i start logstash CPU usage jump to 99%, 100% and the logstash status is running but realy not started and no logs written can someone please help me to resolve this issue ? thank you. --- ## [Requirement to build logstah core module to generate a custom logstah for a specific purpose](https://discuss.elastic.co/t/requirement-to-build-logstah-core-module-to-generate-a-custom-logstah-for-a-specific-purpose/315200)
**Author:** [@rravitech](https://discuss.elastic.co/u/rravitech)\ **Replies:** 3\ **Last updated:** [September 30, 2022, 9:01am UTC](https://discuss.elastic.co/t/requirement-to-build-logstah-core-module-to-generate-a-custom-logstah-for-a-specific-purpose/315200 "2022-09-30T09:01:01Z")
I have a requirement to edit the logstash core, and generate a kind of build out of it. Need to customize logstash code to resist over the below mechanism. Logstash has an at-least-once delivery model. Every event is s… --- ## [JSON Plugin - Ignore Field](https://discuss.elastic.co/t/json-plugin-ignore-field/315510)
**Author:** [@QuestBevan](https://discuss.elastic.co/u/QuestBevan)\ **Replies:** 1\ **Last updated:** [September 30, 2022, 3:37am UTC](https://discuss.elastic.co/t/json-plugin-ignore-field/315510 "2022-09-30T03:37:20Z")
Hi All, Need some advise. I am currently using filebeat, to send logs to Elasticsearch via Logstash. Filebeat -\> Logstash -\> Elasticsearch Log files are in JSON, and these JSON documents are being expanded as expecte… --- ## [\[logstash.filters.xml Error parsing xml with XmlSimple {:source=\>"message" , :exception=\>#\ **Author:** [@erwin339](https://discuss.elastic.co/u/erwin339)\ **Replies:** 0\ **Last updated:** [September 29, 2022, 2:48pm UTC](https://discuss.elastic.co/t/logstash-filters-xml-error-parsing-xml-with-xmlsimple-source-message-exception-rexml-no-close-tag-for-log4j-event-log4j-message-and-truncated/315472 "2022-09-29T14:48:19Z") This log is giving me problems: Note: my log is much longer but you exceed the character limit that allows me to upload here. Just delete things from the log that are repeated as messages. The message exceeds 1000 lines… --- ## [Filtered Log messages show up as empty fields in Kibana](https://discuss.elastic.co/t/filtered-log-messages-show-up-as-empty-fields-in-kibana/315400)
**Author:** [@Chma](https://discuss.elastic.co/u/Chma)\ **Replies:** 7\ **Last updated:** [September 29, 2022, 5:01pm UTC](https://discuss.elastic.co/t/filtered-log-messages-show-up-as-empty-fields-in-kibana/315400 "2022-09-29T17:01:49Z")
I have filtered my log message using grok. But when I check Kibana, I find the new fields on the left side of the page, but they are empty. I am also getting the \_grokparsefailure tag. Here's an example of my log messag… --- ## [Logstash logs](https://discuss.elastic.co/t/logstash-logs/314911)
**Author:** [@irivas95](https://discuss.elastic.co/u/irivas95)\ **Replies:** 2\ **Last updated:** [September 29, 2022, 4:39pm UTC](https://discuss.elastic.co/t/logstash-logs/314911 "2022-09-29T16:39:23Z")
Hi, I have installed logstash 7.16.3 with default settings in logstash.yml and log4j2.properties. My understanding is that with this configuration each log type, for example, logstash-plain.log should not be able to ex… --- ## [Logstash: Variable substitution FAILS for "api\_key" in elasticsearch output](https://discuss.elastic.co/t/logstash-variable-substitution-fails-for-api-key-in-elasticsearch-output/315339)
**Author:** [@George\_Kossionis](https://discuss.elastic.co/u/George_Kossionis)\ **Replies:** 1\ **Last updated:** [September 29, 2022, 10:29am UTC](https://discuss.elastic.co/t/logstash-variable-substitution-fails-for-api-key-in-elasticsearch-output/315339 "2022-09-29T10:29:14Z")
Hello, I have configured logstash to use logstash.keystore to store secure credentials. The ${ES\_PASSWD} substitutions are working perfectly on the following configuration: elasticsearch { ... user… --- ## [Index origination date based on logs age](https://discuss.elastic.co/t/index-origination-date-based-on-logs-age/315159)
**Author:** [@booboo](https://discuss.elastic.co/u/booboo)\ **Replies:** 2\ **Last updated:** [September 29, 2022, 7:59am UTC](https://discuss.elastic.co/t/index-origination-date-based-on-logs-age/315159 "2022-09-29T07:59:11Z")
Hello everyone, I am trying to set a dynamic lifecycle "origination\_date" in order in inject old logs into Elastic, does anybody know how to do this ? Here is my situation. I need monthly indexes so I can't use "parse\_… --- ## [Logstash 8.4.2 Extremely High CPU Loading](https://discuss.elastic.co/t/logstash-8-4-2-extremely-high-cpu-loading/315417)
**Author:** [@ivanchak](https://discuss.elastic.co/u/ivanchak)\ **Replies:** 3\ **Last updated:** [September 29, 2022, 4:35am UTC](https://discuss.elastic.co/t/logstash-8-4-2-extremely-high-cpu-loading/315417 "2022-09-29T04:35:38Z")
My Logstash loading has been about avg. ~10% loading for couple months. Without any change on grokking and configuration, even event/s is about the same rate, just upgrading my Logstash to v8.4.2 causing CPU loading jump… --- ## [Logstash Failed to install template Error 400](https://discuss.elastic.co/t/logstash-failed-to-install-template-error-400/315403)
**Author:** [@fformoso](https://discuss.elastic.co/u/fformoso)\ **Replies:** 0\ **Last updated:** [September 28, 2022, 8:34pm UTC](https://discuss.elastic.co/t/logstash-failed-to-install-template-error-400/315403 "2022-09-28T20:34:39Z")
Dear all, I need help to understand root cause, I can add the template via api, but not using logstash 8.3 I've tried to fix it adding diferent roles to user =\> "logstash\_writer" and didnt work Error: \[2022-09-28T17… --- ## [Logstash on docker not listening to port](https://discuss.elastic.co/t/logstash-on-docker-not-listening-to-port/315285)
**Author:** [@Juan\_Fernandez\_Bern1](https://discuss.elastic.co/u/Juan_Fernandez_Bern1)\ **Replies:** 0\ **Last updated:** [September 27, 2022, 3:33pm UTC](https://discuss.elastic.co/t/logstash-on-docker-not-listening-to-port/315285 "2022-09-27T15:33:49Z")
Hello, im running logstash with docker-compose and for some reason im unable to listen to port 9015 in one of the nodes. image: docker.elastic.co/logstash/logstash:7.12.0 container\_name: logstash-node03 port… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=111) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=113)