# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=378 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 379 --- ## [Logstash filter on number](https://discuss.elastic.co/t/logstash-filter-on-number/223474)
**Author:** [@kamandohl](https://discuss.elastic.co/u/kamandohl)\ **Replies:** 2\ **Last updated:** [April 7, 2020, 10:42am UTC](https://discuss.elastic.co/t/logstash-filter-on-number/223474 "2020-04-07T10:42:30Z")
Trying to filter on an number, in this case 4624, in Logstash. But unable to get it to work. I was able to key off of event.kind, but that is a word and all events have that, so not an option. I have looked for days a… --- ## [How to make this json array as csv data](https://discuss.elastic.co/t/how-to-make-this-json-array-as-csv-data/226775)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [April 7, 2020, 3:42am UTC](https://discuss.elastic.co/t/how-to-make-this-json-array-as-csv-data/226775 "2020-04-07T03:42:45Z")
Hi All I have json data like this \[{ "jsondata": \["123456", "description1", "notes1", "information1", 9\] }, { "jsondata": \["44555", "description2", "notes2", "information2", 8\] } \] I need the output in csv forma… --- ## [Extracting the JSON fields from the message](https://discuss.elastic.co/t/extracting-the-json-fields-from-the-message/226590)
**Author:** [@Raed](https://discuss.elastic.co/u/Raed)\ **Replies:** 9\ **Last updated:** [April 7, 2020, 1:37am UTC](https://discuss.elastic.co/t/extracting-the-json-fields-from-the-message/226590 "2020-04-07T01:37:22Z")
I'm getting the below JSON as a message: { "requestUrl": "http://localhost:8080/frameworks/298", "requestUri": "/frameworks/298", "requestMethod": "PUT", "requestHeaders": { "authorization": "\*\*\*", "ac… --- ## [Grok Help - specificity?](https://discuss.elastic.co/t/grok-help-specificity/226802)
**Author:** [@almathden](https://discuss.elastic.co/u/almathden)\ **Replies:** 0\ **Last updated:** [April 7, 2020, 1:00am UTC](https://discuss.elastic.co/t/grok-help-specificity/226802 "2020-04-07T01:00:28Z")
Hey Team, trying to match my watchguard logs. Depending on the log entry the format varies pretty wildly so I have a few different matches set up. Problem is matching a new entry I'm working on is...too generic? And so i… --- ## [How to output records that come up as an error or not an error but not ingested properly into a separate file](https://discuss.elastic.co/t/how-to-output-records-that-come-up-as-an-error-or-not-an-error-but-not-ingested-properly-into-a-separate-file/226766)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 0\ **Last updated:** [April 6, 2020, 7:56pm UTC](https://discuss.elastic.co/t/how-to-output-records-that-come-up-as-an-error-or-not-an-error-but-not-ingested-properly-into-a-separate-file/226766 "2020-04-06T19:56:19Z")
How can I output records that come up either as errors or simply aren't ingested or read properly for whatever reason (ie syntax or something of the sort) into a separate file? --- ## [Tell Logstash to ignore old files in s3 bucket](https://discuss.elastic.co/t/tell-logstash-to-ignore-old-files-in-s3-bucket/226759)
**Author:** [@clarkritchie](https://discuss.elastic.co/u/clarkritchie)\ **Replies:** 2\ **Last updated:** [April 6, 2020, 7:47pm UTC](https://discuss.elastic.co/t/tell-logstash-to-ignore-old-files-in-s3-bucket/226759 "2020-04-06T19:47:25Z")
I'm using the Logstash s3 plugin to ingest logs from a partner's S3 bucket. Logstash itself runs in a container, so its sincedb is not persistent. When the container re-starts, Logstash restarts and begins ingesting lo… --- ## [Logstash working on Linux but not on Windows with same config files](https://discuss.elastic.co/t/logstash-working-on-linux-but-not-on-windows-with-same-config-files/223587)
**Author:** [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Replies:** 2\ **Last updated:** [April 6, 2020, 7:23pm UTC](https://discuss.elastic.co/t/logstash-working-on-linux-but-not-on-windows-with-same-config-files/223587 "2020-04-06T19:23:49Z")
I have one logstash configuration file which I am using for logs coming from both Windows and linux environment. I have the same filebeat configuration for both environment. The log file being shipped via filebeats is a… --- ## [Pipeline not reading csv files](https://discuss.elastic.co/t/pipeline-not-reading-csv-files/226749)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 2\ **Last updated:** [April 6, 2020, 7:19pm UTC](https://discuss.elastic.co/t/pipeline-not-reading-csv-files/226749 "2020-04-06T19:19:35Z")
I have a following configuration. when I run this from command line it works and reads file but when I run it via pipleline it never reads this file. pipeline shows it is running Am I missing something here ? I am g… --- ## [Mapping Issue in Index setting up manual mapping](https://discuss.elastic.co/t/mapping-issue-in-index-setting-up-manual-mapping/226485)
**Author:** [@Sergioc](https://discuss.elastic.co/u/Sergioc)\ **Replies:** 2\ **Last updated:** [April 6, 2020, 7:18pm UTC](https://discuss.elastic.co/t/mapping-issue-in-index-setting-up-manual-mapping/226485 "2020-04-06T19:18:22Z")
Hello, hope everybody is safe... I have a logstash pipeline that sends data from a SQL database to elastic. I have never configured mappings , I was letting elastic to set up the mapping for me. but I have a field tha… --- ## [Aggregate. Merging two messages in different order](https://discuss.elastic.co/t/aggregate-merging-two-messages-in-different-order/226423)
**Author:** [@Ilya\_Kornienko](https://discuss.elastic.co/u/Ilya_Kornienko)\ **Replies:** 4\ **Last updated:** [April 6, 2020, 4:51pm UTC](https://discuss.elastic.co/t/aggregate-merging-two-messages-in-different-order/226423 "2020-04-06T16:51:29Z")
Hello everyone! Filebeat 7.6.1 -\> logstash 7.6.1 Filebeat reads logfiles, processes multiline and sends to logstesh. (all OK) But at the time of file rotation, the filebeat cuts the message into two parts, the first i… --- ## [Add field in logstash of Winlogbeats input](https://discuss.elastic.co/t/add-field-in-logstash-of-winlogbeats-input/226672)
**Author:** [@Travis](https://discuss.elastic.co/u/Travis)\ **Replies:** 4\ **Last updated:** [April 6, 2020, 2:51pm UTC](https://discuss.elastic.co/t/add-field-in-logstash-of-winlogbeats-input/226672 "2020-04-06T14:51:23Z")
Hello there ! I have the following Winlogbeat ouput in Logstash : { "ecs" =\> { "version" =\> "1.4.0" }, "log" =\> { "level" =\> "information" }, "@version" =\> "1", … --- ## [\[Feature request\] Tag\_on\_failure for Logstash xml filter plugin](https://discuss.elastic.co/t/feature-request-tag-on-failure-for-logstash-xml-filter-plugin/226564)
**Author:** [@Ged](https://discuss.elastic.co/u/Ged)\ **Replies:** 4\ **Last updated:** [April 6, 2020, 2:29pm UTC](https://discuss.elastic.co/t/feature-request-tag-on-failure-for-logstash-xml-filter-plugin/226564 "2020-04-06T14:29:33Z")
HI Team, Could you please consider adding setting "tag\_on\_failure" for xml filter plugin as it is done for grok filter ? It could add a tag (for example "\_xmlfailure") in case of XML parsing error or when plugin crashe… --- ## [Catch Memcached Filter exceptions](https://discuss.elastic.co/t/catch-memcached-filter-exceptions/226432)
**Author:** [@Djamox](https://discuss.elastic.co/u/Djamox)\ **Replies:** 3\ **Last updated:** [April 6, 2020, 2:18pm UTC](https://discuss.elastic.co/t/catch-memcached-filter-exceptions/226432 "2020-04-06T14:18:15Z")
Hi, I'm trying to connect to a memcached server via the filter Memcached. The problem is that the pipeline failed if the server is not available, and it is not acceptable. I couldn't find anything to handler failure so… --- ## [Can I Extend the Logstash API?](https://discuss.elastic.co/t/can-i-extend-the-logstash-api/226714)
**Author:** [@bibblyb](https://discuss.elastic.co/u/bibblyb)\ **Replies:** 1\ **Last updated:** [April 6, 2020, 1:25pm UTC](https://discuss.elastic.co/t/can-i-extend-the-logstash-api/226714 "2020-04-06T13:25:38Z")
I'm looking to extend the API logstash provides to add an endpoint for scraping more in-depth metrics for our prometheus monitoring. I've been examining the source code and can get something working with adding a manual… --- ## [Logstash batch size query](https://discuss.elastic.co/t/logstash-batch-size-query/226651)
**Author:** [@vivektsb](https://discuss.elastic.co/u/vivektsb)\ **Replies:** 3\ **Last updated:** [April 6, 2020, 11:33am UTC](https://discuss.elastic.co/t/logstash-batch-size-query/226651 "2020-04-06T11:33:45Z")
Hi Team, We have observed that we are missing logs.Currently we have 24 workers and batch size 125 and batch delay 50 ms.Heap size is 8 GB.Currently we are getting 300000 logs per second.Do we have to increase batch siz… --- ## [Logstash 6.6.0 with syslog output 3.0.5 ssl-tcp issue](https://discuss.elastic.co/t/logstash-6-6-0-with-syslog-output-3-0-5-ssl-tcp-issue/226684)
**Author:** [@J.Timmer](https://discuss.elastic.co/u/J.Timmer)\ **Replies:** 0\ **Last updated:** [April 6, 2020, 11:04am UTC](https://discuss.elastic.co/t/logstash-6-6-0-with-syslog-output-3-0-5-ssl-tcp-issue/226684 "2020-04-06T11:04:53Z")
Hello all, I am having trouble getting the syslog output working with the ssl-tcp setting. Once I configure the pipeline to use ssl-tcp I get the following errors: Error registering plugin {:pipeline\_id=\>"\", :p… --- ## [How to replace ' to " with gsub](https://discuss.elastic.co/t/how-to-replace-to-with-gsub/226612)
**Author:** [@lusynda](https://discuss.elastic.co/u/lusynda)\ **Replies:** 4\ **Last updated:** [April 6, 2020, 9:11am UTC](https://discuss.elastic.co/t/how-to-replace-to-with-gsub/226612 "2020-04-06T09:11:28Z")
Hi all How can i use gsub to replace ' to become ", i need it to parse json data. So some how json can only parse data field with "" and not '' so i need to replace ' to ". Can anyone help me Thanks in advance --- ## [Logstash aggregate filter use with pagination](https://discuss.elastic.co/t/logstash-aggregate-filter-use-with-pagination/226530)
**Author:** [@kamboh](https://discuss.elastic.co/u/kamboh)\ **Replies:** 4\ **Last updated:** [April 5, 2020, 9:51am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-use-with-pagination/226530 "2020-04-05T09:51:23Z")
I have a logstash pipeline that fetches data from mysql using jdbc input connecter aggregates data for users based on user id pushes aggregated data to elasticsearch cluster It fetches large amount of data (e.g 2 mill… --- ## [Grok + Dissect - how to work properly?](https://discuss.elastic.co/t/grok-dissect-how-to-work-properly/226121)
**Author:** [@dmitryB](https://discuss.elastic.co/u/dmitryB)\ **Replies:** 3\ **Last updated:** [April 6, 2020, 8:18am UTC](https://discuss.elastic.co/t/grok-dissect-how-to-work-properly/226121 "2020-04-06T08:18:18Z")
Hello, everyone. I have a problem with running grok and dissect filtering at once. My config: grok { match =\> { 'message' =\> '\\\[%{TIMESTAMP\_ISO8601:time\_stamp}\\\] \\\[%{DATA:correlation\_id}\\\] \\\[%{LOGLEVEL:log\_le… --- ## [Logstash - java.lang.OutOfMemoryError: Java heap space](https://discuss.elastic.co/t/logstash-java-lang-outofmemoryerror-java-heap-space/226632)
**Author:** [@Elango\_Sakthi](https://discuss.elastic.co/u/Elango_Sakthi)\ **Replies:** 0\ **Last updated:** [April 6, 2020, 7:03am UTC](https://discuss.elastic.co/t/logstash-java-lang-outofmemoryerror-java-heap-space/226632 "2020-04-06T07:03:57Z")
I am using Logstash latest version 7.6.1 and I am using 4GB Ram in my machine. When I trying to push data from Redshift to Elasticsearch using Logstash I am getting the Error: java.lang.OutOfMemoryError: Java heap spa… --- ## [Problem when importing a CSV file](https://discuss.elastic.co/t/problem-when-importing-a-csv-file/226627)
**Author:** [@GuyMuller](https://discuss.elastic.co/u/GuyMuller)\ **Replies:** 1\ **Last updated:** [April 6, 2020, 6:48am UTC](https://discuss.elastic.co/t/problem-when-importing-a-csv-file/226627 "2020-04-06T06:48:16Z")
Good morning, sir, When importing, the columns described in the Filter section were not taken into account. They are all in column 1!?! Blockquote filter { csv { separator =\> "," … --- ## [How to create filter to handle spaces & commas in street addresses & person names?](https://discuss.elastic.co/t/how-to-create-filter-to-handle-spaces-commas-in-street-addresses-person-names/226604)
**Author:** [@hc-jgregory](https://discuss.elastic.co/u/hc-jgregory)\ **Replies:** 1\ **Last updated:** [April 6, 2020, 6:33am UTC](https://discuss.elastic.co/t/how-to-create-filter-to-handle-spaces-commas-in-street-addresses-person-names/226604 "2020-04-06T06:33:46Z")
Sample text from log (comes in a segment without quotes enclosing field and values as expected if true JSON): {Field1=1111 NE 1ST AVE,Field2=ATLANTA, GA 30309-1111,Field3=JOHN ALLEN DOE} filter { gr… --- ## [Splitting syslog/beat entries by host?](https://discuss.elastic.co/t/splitting-syslog-beat-entries-by-host/226596)
**Author:** [@almathden](https://discuss.elastic.co/u/almathden)\ **Replies:** 2\ **Last updated:** [April 5, 2020, 11:15pm UTC](https://discuss.elastic.co/t/splitting-syslog-beat-entries-by-host/226596 "2020-04-05T23:15:11Z")
Hey Team! Due to the wildly different syslog formats (And my not understanding how to grok/mutate/etc properly yet) I'm trying to split my syslog inputs by IP address so there's a different index for each. It seems to m… --- ## [Creating Multiple Logstash indexes from Filebeat logs of different machines](https://discuss.elastic.co/t/creating-multiple-logstash-indexes-from-filebeat-logs-of-different-machines/226588)
**Author:** [@Claire08](https://discuss.elastic.co/u/Claire08)\ **Replies:** 2\ **Last updated:** [April 5, 2020, 10:48pm UTC](https://discuss.elastic.co/t/creating-multiple-logstash-indexes-from-filebeat-logs-of-different-machines/226588 "2020-04-05T22:48:06Z")
I am fairly new to ES and I have been testing filebeat/logstash configurations. I am trying to get Logstash to output to multiple indexes based on the logs collected from another remote machine. For example, I wanted to … --- ## [Logstash geo\_point support](https://discuss.elastic.co/t/logstash-geo-point-support/226595)
**Author:** [@jimmymacGA](https://discuss.elastic.co/u/jimmymacGA)\ **Replies:** 1\ **Last updated:** [April 5, 2020, 10:38pm UTC](https://discuss.elastic.co/t/logstash-geo-point-support/226595 "2020-04-05T22:38:40Z")
Why is there no support on this forum for logstash and geo\_point? Every thread goes ignored and support for geojson in logstash is non-existant ? --- ## [Parse SNMP table data](https://discuss.elastic.co/t/parse-snmp-table-data/226552)
**Author:** [@Sketchy](https://discuss.elastic.co/u/Sketchy)\ **Replies:** 4\ **Last updated:** [April 5, 2020, 10:36pm UTC](https://discuss.elastic.co/t/parse-snmp-table-data/226552 "2020-04-05T22:36:24Z")
I am experimenting moving some snmp monitoring from telegraf to logstash snmp imput but having some problems with the layout of the output and cant figure out how to efficiently parse this into a useful format. Currentl… --- ## [Problem converting multi-delimited kv string](https://discuss.elastic.co/t/problem-converting-multi-delimited-kv-string/226527)
**Author:** [@adude946](https://discuss.elastic.co/u/adude946)\ **Replies:** 6\ **Last updated:** [April 5, 2020, 7:19pm UTC](https://discuss.elastic.co/t/problem-converting-multi-delimited-kv-string/226527 "2020-04-05T19:19:36Z")
I'm lost with converting 1 mysql column to ES via logstash. I can convert a single kv to the expected format but not multiple. Data is in 1 mysql column called "profile". It is going to ES under same field name, mappin… --- ## [How do I test my condition statement that is not working?](https://discuss.elastic.co/t/how-do-i-test-my-condition-statement-that-is-not-working/226504)
**Author:** [@newmember](https://discuss.elastic.co/u/newmember)\ **Replies:** 5\ **Last updated:** [April 5, 2020, 1:25pm UTC](https://discuss.elastic.co/t/how-do-i-test-my-condition-statement-that-is-not-working/226504 "2020-04-05T13:25:34Z")
This is my preferred filter to capture "error" from the tag that is added in the 'input' as part of filebeat. This filter fails. filter { if "error" in \[tags\] { grok { match =\> \[ 'message', '^.\*=%{INT… --- ## [Retrying failed action with response code: 503 unavailable\_shards\_exception, reason =\> logstash primary shard is not active](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-503-unavailable-shards-exception-reason-logstash-primary-shard-is-not-active/226437)
**Author:** [@vignesh.s2](https://discuss.elastic.co/u/vignesh.s2)\ **Replies:** 3\ **Last updated:** [April 5, 2020, 1:19pm UTC](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-503-unavailable-shards-exception-reason-logstash-primary-shard-is-not-active/226437 "2020-04-05T13:19:51Z")
After upgrading to the latest version of the ELK 7.6.2 , I tried to restart the all the services. Only elasticsearch service is up and running. Logstash and kibana failed. Below mentioned is the error Apr 03 21:56:46 C… --- ## [CSV Date column replace to timestamp](https://discuss.elastic.co/t/csv-date-column-replace-to-timestamp/226560)
**Author:** [@lohit18](https://discuss.elastic.co/u/lohit18)\ **Replies:** 1\ **Last updated:** [April 5, 2020, 10:05am UTC](https://discuss.elastic.co/t/csv-date-column-replace-to-timestamp/226560 "2020-04-05T10:05:14Z")
Dear Team, Kindly help me out how to repalce with CSV date format(3/27/2020 18:01 PM) with logstash timestamp . I am passing .conf like this below input { file { path=\>"C:Downloads/low\_data.csv" start\_position=\>"… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=377) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=379)