# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=249 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 250 --- ## [Logstash Performance Leak When Consuming Kafka Topic into Elasticsearch](https://discuss.elastic.co/t/logstash-performance-leak-when-consuming-kafka-topic-into-elasticsearch/266685)
**Author:** [@davidb](https://discuss.elastic.co/u/davidb)\ **Replies:** 0\ **Last updated:** [March 9, 2021, 11:39am UTC](https://discuss.elastic.co/t/logstash-performance-leak-when-consuming-kafka-topic-into-elasticsearch/266685 "2021-03-09T11:39:13Z")
Greetings, I am facing a problem regarding logstash perfomance degradation over time. Any advice or hint is welcome. Thanks very much in advance. Short summary Logstash suffers from a perfomance leak over time. The data… --- ## [Logstash - In memory maps/Lookup tables?](https://discuss.elastic.co/t/logstash-in-memory-maps-lookup-tables/266673)
**Author:** [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Replies:** 0\ **Last updated:** [March 9, 2021, 10:56am UTC](https://discuss.elastic.co/t/logstash-in-memory-maps-lookup-tables/266673 "2021-03-09T10:56:44Z")
Filebeat sends data to logstash. I’m reading files that are up to about 500mb, about 2 million lines. On each longline I calculate certain fields based on patterns in the log file name, example, split the path, get the… --- ## [Ruby script and boolean](https://discuss.elastic.co/t/ruby-script-and-boolean/266535)
**Author:** [@logger](https://discuss.elastic.co/u/logger)\ **Replies:** 2\ **Last updated:** [March 9, 2021, 8:25am UTC](https://discuss.elastic.co/t/ruby-script-and-boolean/266535 "2021-03-09T08:25:43Z")
Hi there, I have a question about the ruby filter. I am getting logs in Json they are working quite good, but ofcourse I have inconsistence in the mapping. Usually the logs arrive with the fields arg0, arg1, arg,2 T… --- ## [CSV::MalformedCSVError: Unclosed quoted field on line 1](https://discuss.elastic.co/t/csv-unclosed-quoted-field-on-line-1/266509)
**Author:** [@padamrai](https://discuss.elastic.co/u/padamrai)\ **Replies:** 2\ **Last updated:** [March 9, 2021, 6:10am UTC](https://discuss.elastic.co/t/csv-unclosed-quoted-field-on-line-1/266509 "2021-03-09T06:10:07Z")
After running code, getting this error. How to resolve this error. In front of date I am getting this symbol ∩╗┐ while running my configuration file. It appears once(first line). Error parsing csv {:field=\>"message", :… --- ## [I want to output log to elasticsearch index from only specific "host.ip"](https://discuss.elastic.co/t/i-want-to-output-log-to-elasticsearch-index-from-only-specific-host-ip/266494)
**Author:** [@BlackCat](https://discuss.elastic.co/u/BlackCat)\ **Replies:** 4\ **Last updated:** [March 9, 2021, 3:51am UTC](https://discuss.elastic.co/t/i-want-to-output-log-to-elasticsearch-index-from-only-specific-host-ip/266494 "2021-03-09T03:51:57Z")
"host.ip" is at "\_source" displayed on Kibana. Please teach me how to resolve. logstash config input { beats { port =\> 5044 } } filter { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:Time} %{HO… --- ## [Logstash elastic as input](https://discuss.elastic.co/t/logstash-elastic-as-input/266280)
**Author:** [@Kiyoka](https://discuss.elastic.co/u/Kiyoka)\ **Replies:** 3\ **Last updated:** [March 8, 2021, 11:47pm UTC](https://discuss.elastic.co/t/logstash-elastic-as-input/266280 "2021-03-08T23:47:27Z")
Hi. I am using existing index as input and enriching it with the data from SQL server. This enriching process needs to run weekly for now. Currently I am loading past 10 days, but that cause duplicate. Preformatted … --- ## [JDBC\_Static Filter](https://discuss.elastic.co/t/jdbc-static-filter/266162)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 11\ **Last updated:** [March 8, 2021, 9:32pm UTC](https://discuss.elastic.co/t/jdbc-static-filter/266162 "2021-03-08T21:32:17Z")
I am pulling data from a database that has fields populated with numeric values. I need to perform subsequent lookups to additional tables to convert the numeric value to a named value. Since I'll be doing this frequen… --- ## [Logstash doesnt stop when config.reload.automatic is enabled](https://discuss.elastic.co/t/logstash-doesnt-stop-when-config-reload-automatic-is-enabled/266596)
**Author:** [@hayo](https://discuss.elastic.co/u/hayo)\ **Replies:** 2\ **Last updated:** [March 8, 2021, 7:39pm UTC](https://discuss.elastic.co/t/logstash-doesnt-stop-when-config-reload-automatic-is-enabled/266596 "2021-03-08T19:39:44Z")
Hello, I am running a logstash pipeline with JDBC input triggered via crontab, to ensure also a process runs afterwards. Besides that i have also a logstash pipeline with JDBC input but with schedule as a service. I w… --- ## [Logstash access dynamic field with ruby filter](https://discuss.elastic.co/t/logstash-access-dynamic-field-with-ruby-filter/266489)
**Author:** [@Ingram\_Gultom](https://discuss.elastic.co/u/Ingram_Gultom)\ **Replies:** 8\ **Last updated:** [March 8, 2021, 5:45pm UTC](https://discuss.elastic.co/t/logstash-access-dynamic-field-with-ruby-filter/266489 "2021-03-08T17:45:19Z")
Hello I have nested field looks like this \[field1\]\[field2\]\[field3\] I want to remove field1 and field2 so the expected output will be \[field3\] I already manage to remove field1 with this ruby filter ruby { … --- ## [Logstash Grok Pattern for Timestamp with Timezone](https://discuss.elastic.co/t/logstash-grok-pattern-for-timestamp-with-timezone/266268)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 3\ **Last updated:** [March 8, 2021, 4:59pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-for-timestamp-with-timezone/266268 "2021-03-08T16:59:35Z")
Hi, I am trying to match following logs for icinga2 using grok filter: \[2021-03-04 17:03:27 +0100\] warning/GraphiteWriter: Ignoring invalid perfdata for checkable 'host!service' and command 'by\_ssh' with value: /foo/ba… --- ## [Ruby syntax error only in logstash](https://discuss.elastic.co/t/ruby-syntax-error-only-in-logstash/266488)
**Author:** [@tangkalo](https://discuss.elastic.co/u/tangkalo)\ **Replies:** 3\ **Last updated:** [March 8, 2021, 4:57pm UTC](https://discuss.elastic.co/t/ruby-syntax-error-only-in-logstash/266488 "2021-03-08T16:57:10Z")
Hi, I am trying to use the ruby plugin to parse a field of array type and then generate new fields out of some of elements. However, I encountered the following error. Similar ruby codes(i.e. the statement to set event)… --- ## [Filebeat fields missing](https://discuss.elastic.co/t/filebeat-fields-missing/266375)
**Author:** [@bzak](https://discuss.elastic.co/u/bzak)\ **Replies:** 1\ **Last updated:** [March 8, 2021, 12:32pm UTC](https://discuss.elastic.co/t/filebeat-fields-missing/266375 "2021-03-08T12:32:45Z")
When I installed filebeat in a linux server I enabled nginx module. In Discover, I can see all the nginx fields but they are ALL empty. My configuration is as follows: input { beats:... } filter { ... } output { … --- ## [Verify whether logstash is receiving data or not](https://discuss.elastic.co/t/verify-whether-logstash-is-receiving-data-or-not/266515)
**Author:** [@user2416](https://discuss.elastic.co/u/user2416)\ **Replies:** 0\ **Last updated:** [March 8, 2021, 7:04am UTC](https://discuss.elastic.co/t/verify-whether-logstash-is-receiving-data-or-not/266515 "2021-03-08T07:04:37Z")
Hi, I am using beats input plugin and sending data to logstash. Logstash hangs frequently and putting back pressure on filebeat. As part of debugging I am using below curl command to test whether filebeat is able to co… --- ## [Logstash Load balancing configuration](https://discuss.elastic.co/t/logstash-load-balancing-configuration/266504)
**Author:** [@gurumurthy](https://discuss.elastic.co/u/gurumurthy)\ **Replies:** 0\ **Last updated:** [March 8, 2021, 3:58am UTC](https://discuss.elastic.co/t/logstash-load-balancing-configuration/266504 "2021-03-08T03:58:16Z")
For beats data we're trying to set up logstash load balancing. I see that I can only have one logstash SSL certificate in beats conf. Two logstash instances are created with separate SSL certificates, If I use respective… --- ## [Creating indices of older days too](https://discuss.elastic.co/t/creating-indices-of-older-days-too/266492)
**Author:** [@souji](https://discuss.elastic.co/u/souji)\ **Replies:** 0\ **Last updated:** [March 7, 2021, 11:15pm UTC](https://discuss.elastic.co/t/creating-indices-of-older-days-too/266492 "2021-03-07T23:15:16Z")
Hi I have deployed filebeat and logstash helm chart in eks. I am sending logs to amazon es. I am facing a similar issue https://github.com/uken/fluent-plugin-elasticsearch/issues/482, but in filebeat do we have optio… --- ## [Help me writing grook filter pattern for my log to injest into elasticsearch](https://discuss.elastic.co/t/help-me-writing-grook-filter-pattern-for-my-log-to-injest-into-elasticsearch/266423)
**Author:** [@abhishek\_s1](https://discuss.elastic.co/u/abhishek_s1)\ **Replies:** 4\ **Last updated:** [March 7, 2021, 5:36pm UTC](https://discuss.elastic.co/t/help-me-writing-grook-filter-pattern-for-my-log-to-injest-into-elasticsearch/266423 "2021-03-07T17:36:27Z")
These are few sample log lines 74128 2021-01-25T23:28:42.753582Z - xyz svn/repos get-latest-rev 74128 2021-01-25T23:28:43.030543Z - xyz svn/repos reparent /sm/branches/6.0r 74128 2021-01-25T23:28:43.307469Z - xyz svn/… --- ## [Logstash not working!](https://discuss.elastic.co/t/logstash-not-working/264983)
**Author:** [@Mohyden](https://discuss.elastic.co/u/Mohyden)\ **Replies:** 9\ **Last updated:** [March 7, 2021, 5:17am UTC](https://discuss.elastic.co/t/logstash-not-working/264983 "2021-03-07T05:17:27Z")
Hello, I'm running Ubuntu 16.04. When I try to start Logstash with systemctl start logstash, and then systemctl status logstash, it shows these: logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash… --- ## [Logstash on Docker ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml](https://discuss.elastic.co/t/logstash-on-docker-error-failed-to-read-pipelines-yaml-file-location-usr-share-logstash-config-pipelines-yml/266449)
**Author:** [@20031485](https://discuss.elastic.co/u/20031485)\ **Replies:** 0\ **Last updated:** [March 6, 2021, 4:40pm UTC](https://discuss.elastic.co/t/logstash-on-docker-error-failed-to-read-pipelines-yaml-file-location-usr-share-logstash-config-pipelines-yml/266449 "2021-03-06T16:40:23Z")
Hello! I am facing difficulties trying to configure my ELK stack in which Logstash should be running multiple pipelines, and the problem seems to be in the Logstash docker-compose.yml configuration. In fact, I get the fo… --- ## [Ingest pipeline differences with Logstash and Beats](https://discuss.elastic.co/t/ingest-pipeline-differences-with-logstash-and-beats/266302)
**Author:** [@b133](https://discuss.elastic.co/u/b133)\ **Replies:** 2\ **Last updated:** [March 5, 2021, 5:11pm UTC](https://discuss.elastic.co/t/ingest-pipeline-differences-with-logstash-and-beats/266302 "2021-03-05T17:11:27Z")
I want to bifurcate my logs using Logstash. When I sent logs directly from Filebeat to Elasticsearch, the log fields were all properly parsed and converted to appropriate variable types. I have since changed the Filebea… --- ## [Logstash - how to parse rsyslog date field from logs message into @timestamp (UTC)](https://discuss.elastic.co/t/logstash-how-to-parse-rsyslog-date-field-from-logs-message-into-timestamp-utc/265897)
**Author:** [@Jenisha\_Ramanathan](https://discuss.elastic.co/u/Jenisha_Ramanathan)\ **Replies:** 1\ **Last updated:** [March 5, 2021, 4:27pm UTC](https://discuss.elastic.co/t/logstash-how-to-parse-rsyslog-date-field-from-logs-message-into-timestamp-utc/265897 "2021-03-05T16:27:40Z")
Hi Team, I am trying to monitor Linux logs with rsyslog and kibana. Able to receive the logs from rsyslog client to rsyslog server and load the data to elastic. Getting the reported time in kibana ( UTC format ) but we … --- ## [S3snssqs - multiline support?](https://discuss.elastic.co/t/s3snssqs-multiline-support/266370)
**Author:** [@justroll](https://discuss.elastic.co/u/justroll)\ **Replies:** 1\ **Last updated:** [March 5, 2021, 4:21pm UTC](https://discuss.elastic.co/t/s3snssqs-multiline-support/266370 "2021-03-05T16:21:33Z")
Hi we are trying to read our log usign s3snssqs ; this all works great but the only thing is when we want to detect multiline logs .. The module doesn't seem to support codec =\> multiline in order to use a pattern mis… --- ## [Performing a conditional mutation to circumvent parsing issue](https://discuss.elastic.co/t/performing-a-conditional-mutation-to-circumvent-parsing-issue/266349)
**Author:** [@nectide](https://discuss.elastic.co/u/nectide)\ **Replies:** 1\ **Last updated:** [March 5, 2021, 4:13pm UTC](https://discuss.elastic.co/t/performing-a-conditional-mutation-to-circumvent-parsing-issue/266349 "2021-03-05T16:13:24Z")
Hi, I'm a newcomer to Logstash, so bear with me. My logstash logs are reporting: \[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"… --- ## [Helm logstash last\_run\_metadata\_path](https://discuss.elastic.co/t/helm-logstash-last-run-metadata-path/266362)
**Author:** [@josemiguelq](https://discuss.elastic.co/u/josemiguelq)\ **Replies:** 0\ **Last updated:** [March 5, 2021, 12:01pm UTC](https://discuss.elastic.co/t/helm-logstash-last-run-metadata-path/266362 "2021-03-05T12:01:46Z")
How can I define volumes bind for files used by last\_run\_metadata? In logstash.conf file a defined last\_run\_metadata\_path: "{$HOME}/states/.last\_run\_config" and I want this file persistent for any deploy. --- ## [\[0\] "\_dateparsefailure"](https://discuss.elastic.co/t/0-dateparsefailure/266226)
**Author:** [@padamrai](https://discuss.elastic.co/u/padamrai)\ **Replies:** 4\ **Last updated:** [March 5, 2021, 11:38am UTC](https://discuss.elastic.co/t/0-dateparsefailure/266226 "2021-03-05T11:38:09Z")
I want to replace timestamp with my field timestamp. I have tried but getting error so kindly share solution for this problem. "message" =\> "8/12/2020 0:00,jack,jazz,your recent attemt failed.,"9000.00 have been Debited… --- ## [How to add field to every element from an array with logstash ruby](https://discuss.elastic.co/t/how-to-add-field-to-every-element-from-an-array-with-logstash-ruby/266257)
**Author:** [@syrine\_chelly](https://discuss.elastic.co/u/syrine_chelly)\ **Replies:** 4\ **Last updated:** [March 5, 2021, 9:33am UTC](https://discuss.elastic.co/t/how-to-add-field-to-every-element-from-an-array-with-logstash-ruby/266257 "2021-03-05T09:33:35Z")
Hi I have an array such like this: "step": \[ "\> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIP A PRIS 00H00", "\> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20… --- ## [Logstash CIDR Filter : create new field with matched network](https://discuss.elastic.co/t/logstash-cidr-filter-create-new-field-with-matched-network/266282)
**Author:** [@Nybble](https://discuss.elastic.co/u/Nybble)\ **Replies:** 2\ **Last updated:** [March 5, 2021, 6:22am UTC](https://discuss.elastic.co/t/logstash-cidr-filter-create-new-field-with-matched-network/266282 "2021-03-05T06:22:03Z")
Hello, I'm currently using the Logstash CIDR filter to tag private and public IP. Now, I want to use this plugin with a list of all the networks used within my company and add a field or a tag with network information … --- ## [Unable to ingest data using different JDBC Drivers in one Conf file](https://discuss.elastic.co/t/unable-to-ingest-data-using-different-jdbc-drivers-in-one-conf-file/266261)
**Author:** [@ronaldom](https://discuss.elastic.co/u/ronaldom)\ **Replies:** 0\ **Last updated:** [March 4, 2021, 4:21pm UTC](https://discuss.elastic.co/t/unable-to-ingest-data-using-different-jdbc-drivers-in-one-conf-file/266261 "2021-03-04T16:21:31Z")
Hello, I am currently doing a proof of concept where I try to pull data from 2 different databases (SQL and Snowflake). I don't see any error in the logs when running the conf file and logstash never seem to complete eve… --- ## [Condition format for pipelines in Kibana](https://discuss.elastic.co/t/condition-format-for-pipelines-in-kibana/265985)
**Author:** [@maria\_k](https://discuss.elastic.co/u/maria_k)\ **Replies:** 0\ **Last updated:** [March 2, 2021, 4:34pm UTC](https://discuss.elastic.co/t/condition-format-for-pipelines-in-kibana/265985 "2021-03-02T16:34:43Z")
Hi! I'm trying to create a pipeline in Kibana (Stack Management -\> Ingest Node pipelines). I'm creating the pipeline with JSON processor, and it works well until I'm adding a condition to it. I've got error when I trie… --- ## [Interpolation in string should be happening but isn't](https://discuss.elastic.co/t/interpolation-in-string-should-be-happening-but-isnt/266275)
**Author:** [@Supermathie](https://discuss.elastic.co/u/Supermathie)\ **Replies:** 1\ **Last updated:** [March 4, 2021, 6:59pm UTC](https://discuss.elastic.co/t/interpolation-in-string-should-be-happening-but-isnt/266275 "2021-03-04T18:59:22Z")
I have the following document that went through logstash (v7.10.2): and message was constructed with the snippet: mutate { add\_field =\> { "message" =\> "%{\[http\]\[request\]\[method\]} %{\[url\]\[domain\]}%{\[url\]\[ori… --- ## [Logstash CSV filter - \_dateparsefailure from string to date](https://discuss.elastic.co/t/logstash-csv-filter-dateparsefailure-from-string-to-date/266189)
**Author:** [@pmaganti](https://discuss.elastic.co/u/pmaganti)\ **Replies:** 2\ **Last updated:** [March 4, 2021, 6:06pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-dateparsefailure-from-string-to-date/266189 "2021-03-04T18:06:57Z")
Hello, I am trying to set a field from CSV as @timestamp and is failing with a tag \_dateparsefailure. Following is the config. input { beats { port =\> 5044 } } filter { csv { separator =\> "," autodet… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=248) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=250)