# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=909 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 910 --- ## [Logstash not filter but pattern works](https://discuss.elastic.co/t/logstash-not-filter-but-pattern-works/100503)
**Author:** [@Akosih](https://discuss.elastic.co/u/Akosih)\ **Replies:** 0\ **Last updated:** [September 14, 2017, 9:48am UTC](https://discuss.elastic.co/t/logstash-not-filter-but-pattern-works/100503 "2017-09-14T09:48:48Z")
Hi, My first topic i hope i will not make a duplicate :frowning: I got a problem I custom my logstash filter to ingest catalina.out logs, the pattern included doesn't fit me so i build this: Exemple of log: 2017-09… --- ## [Logstash not working on selected servers in production](https://discuss.elastic.co/t/logstash-not-working-on-selected-servers-in-production/99795)
**Author:** [@xyz\_12](https://discuss.elastic.co/u/xyz_12)\ **Replies:** 5\ **Last updated:** [September 14, 2017, 9:38am UTC](https://discuss.elastic.co/t/logstash-not-working-on-selected-servers-in-production/99795 "2017-09-14T09:38:42Z")
since the issue is related with PROD in terms of logstash relted and the same issue was occured in lower level environment. . However we have gone through the logs resepect to the server and noticed the below exception… --- ## [How to iterate multiple S3 inputs?](https://discuss.elastic.co/t/how-to-iterate-multiple-s3-inputs/100424)
**Author:** [@android.kc](https://discuss.elastic.co/u/android.kc)\ **Replies:** 1\ **Last updated:** [September 14, 2017, 9:36am UTC](https://discuss.elastic.co/t/how-to-iterate-multiple-s3-inputs/100424 "2017-09-14T09:36:03Z")
I am using s3 input plugin to download/unzip .gz file and feed to Logstash. Here is the input part of my working config for Logstash input { s3 { interval =\> 86400 bucket =\> "us-west-1" aws\_credentials\_fil… --- ## [Logstash filter not wroking](https://discuss.elastic.co/t/logstash-filter-not-wroking/100470)
**Author:** [@pope843](https://discuss.elastic.co/u/pope843)\ **Replies:** 6\ **Last updated:** [September 14, 2017, 9:20am UTC](https://discuss.elastic.co/t/logstash-filter-not-wroking/100470 "2017-09-14T09:20:19Z")
Hello I'm new with logstash and i'm having problems in filter condition, i would like to ship onlt llogs/lines with ERROR, DEBUG, INFO and WARN message. Here's the working config only one string on if condition. filte… --- ## [Comparing and Combining fields from " Different LOG TYPES " and form a new field](https://discuss.elastic.co/t/comparing-and-combining-fields-from-different-log-types-and-form-a-new-field/100476)
**Author:** [@sushanth](https://discuss.elastic.co/u/sushanth)\ **Replies:** 0\ **Last updated:** [September 14, 2017, 7:44am UTC](https://discuss.elastic.co/t/comparing-and-combining-fields-from-different-log-types-and-form-a-new-field/100476 "2017-09-14T07:44:05Z")
I have two different files one with CSV and one with XML Both has one common field and data has to be stored as a single set from both files To achieve that, I have now parsed and stored CSV first then while parsing X… --- ## [Logstash + historical import + oddball time stamps](https://discuss.elastic.co/t/logstash-historical-import-oddball-time-stamps/100087)
**Author:** [@mcdowej](https://discuss.elastic.co/u/mcdowej)\ **Replies:** 2\ **Last updated:** [September 14, 2017, 5:39am UTC](https://discuss.elastic.co/t/logstash-historical-import-oddball-time-stamps/100087 "2017-09-14T05:39:18Z")
I am experiencing challenges around getting any of my logs files to use the timestamp in the log file rather than the time it was sucked in. I have been googling, reading and even explored the "Your topic similar to..."… --- ## [How to add tags and add filter?](https://discuss.elastic.co/t/how-to-add-tags-and-add-filter/100105)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 5\ **Last updated:** [September 14, 2017, 5:31am UTC](https://discuss.elastic.co/t/how-to-add-tags-and-add-filter/100105 "2017-09-14T05:31:13Z")
Hi Guys, I have below kind of information and looking assistance from community for creating logstash filter and add tag like "malware" So that I am planning to start netflow on my devices and index the data and filter… --- ## [Need some help with custom grok pattern](https://discuss.elastic.co/t/need-some-help-with-custom-grok-pattern/100282)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 4\ **Last updated:** [September 14, 2017, 12:20am UTC](https://discuss.elastic.co/t/need-some-help-with-custom-grok-pattern/100282 "2017-09-14T00:20:17Z")
Hello, I'm working on a filter to make sense out of equipment logs I have but I'm stuck because of my own incompetence. Are there any resources online that explain, on a idiot level, how to write patterns? I've looked a… --- ## [Json filter dropping messages](https://discuss.elastic.co/t/json-filter-dropping-messages/100144)
**Author:** [@ikirani](https://discuss.elastic.co/u/ikirani)\ **Replies:** 4\ **Last updated:** [September 13, 2017, 11:01pm UTC](https://discuss.elastic.co/t/json-filter-dropping-messages/100144 "2017-09-13T23:01:35Z")
I have a set of 100 json messages that i send through Logstash Json filter. Each and every time I see a variable number of messages persisted to output file, anywhere between 60 to 100. Things I've tried so far: Upgr… --- ## [Writing to 2 different locations in S3](https://discuss.elastic.co/t/writing-to-2-different-locations-in-s3/100267)
**Author:** [@mateo41](https://discuss.elastic.co/u/mateo41)\ **Replies:** 2\ **Last updated:** [September 13, 2017, 10:01pm UTC](https://discuss.elastic.co/t/writing-to-2-different-locations-in-s3/100267 "2017-09-13T22:01:22Z")
Hi folks, I'm using logstash to read from 2 rabbitMQ queues and write to two different locations in S3. Data from one queue will go to one location in S3, and data from the other queue will go to a different location i… --- ## [\[FATAL\]\[logstash.runner\]](https://discuss.elastic.co/t/fatal-logstash-runner/100416)
**Author:** [@19831d72452f0be1a755](https://discuss.elastic.co/u/19831d72452f0be1a755)\ **Replies:** 11\ **Last updated:** [September 13, 2017, 9:42pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416 "2017-09-13T21:42:13Z")
Hello. I'm using Logstash 5.5 Elastic 5.5 Kibana 5.5 X-Pack 5.5 logstash - ssl - elastic Here's the error: \[FATAL\]\[logstash.runner \] An unexpected error occurred! {:error=\>#\ **Author:** [@JP2017](https://discuss.elastic.co/u/JP2017)\ **Replies:** 0\ **Last updated:** [September 13, 2017, 9:05pm UTC](https://discuss.elastic.co/t/send-logs-analysis-to-ibm-bluemix-how-and-which-elastic-component/100419 "2017-09-13T21:05:04Z") I am new with Elastic tools. I want to send logs analysis from Window server to IBM BLuemix. I used Logstash and Filebeat without success. I don't know which tool and how to? Please assist. Thank you, JP --- ## [Field Values from Pattern Directory](https://discuss.elastic.co/t/field-values-from-pattern-directory/100377)
**Author:** [@robscott27](https://discuss.elastic.co/u/robscott27)\ **Replies:** 4\ **Last updated:** [September 13, 2017, 5:28pm UTC](https://discuss.elastic.co/t/field-values-from-pattern-directory/100377 "2017-09-13T17:28:28Z")
Hello! I'm looking to pull field values from a pattern defined in /etc/logstash/conf.d/patterns Background: I'm pulling pfsense firewall logs into Kibana via Logstash. And for the most part, it works. But it's missing … --- ## [How to find lag between filebeat and logstash based on timestamp](https://discuss.elastic.co/t/how-to-find-lag-between-filebeat-and-logstash-based-on-timestamp/100391)
**Author:** [@Gangadhar\_Mahadevan](https://discuss.elastic.co/u/Gangadhar_Mahadevan)\ **Replies:** 0\ **Last updated:** [September 13, 2017, 4:58pm UTC](https://discuss.elastic.co/t/how-to-find-lag-between-filebeat-and-logstash-based-on-timestamp/100391 "2017-09-13T16:58:12Z")
Hi, I am trying to find lag between filebeat sending a message and time at which logstash reads and process it from topic and not sure how to achieve it. This is snippet of my filebeat config filebeat\_prospectors: pat… --- ## [Logstash KV plugin, convert string values to timestamp](https://discuss.elastic.co/t/logstash-kv-plugin-convert-string-values-to-timestamp/99791)
**Author:** [@shashwat](https://discuss.elastic.co/u/shashwat)\ **Replies:** 11\ **Last updated:** [September 13, 2017, 4:41pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-convert-string-values-to-timestamp/99791 "2017-09-13T16:41:54Z")
Hi, I am using Logstash 5.5 to parse my logs. I have following format of the logs: time taken for doing transfer once for all files in seconds=23 transfer start time= 201708030959 transfer end time = 201708030959. I a… --- ## [Json field parsing , multi value field](https://discuss.elastic.co/t/json-field-parsing-multi-value-field/97005)
**Author:** [@shaileshmali](https://discuss.elastic.co/u/shaileshmali)\ **Replies:** 10\ **Last updated:** [September 13, 2017, 4:11pm UTC](https://discuss.elastic.co/t/json-field-parsing-multi-value-field/97005 "2017-09-13T16:11:07Z")
I want to parse json message . I am using add\_field option as i do not need all fields from the message. landingPlatformCode field has mutiple values . given bellow exmample to pull first value. There might be 1 ,2 ,3 4… --- ## [How to concatenate fields inside an array using Logstash?](https://discuss.elastic.co/t/how-to-concatenate-fields-inside-an-array-using-logstash/100148)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 3\ **Last updated:** [September 13, 2017, 11:31am UTC](https://discuss.elastic.co/t/how-to-concatenate-fields-inside-an-array-using-logstash/100148 "2017-09-13T11:31:23Z")
Hi, I am trying to concatenate fields inside an array. For example, event\_data =\> { param1 =\> "request", param2 =\> "is signing off" } And fields inside event\_data may vary. For example it may be param1… --- ## [Jenkins Integration with Logstash](https://discuss.elastic.co/t/jenkins-integration-with-logstash/99716)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 3\ **Last updated:** [September 13, 2017, 10:13am UTC](https://discuss.elastic.co/t/jenkins-integration-with-logstash/99716 "2017-09-13T10:13:26Z")
Hi - I’d like to get the Jenkins pipeline Job build status along with the Timestamps and publish that data to the Elastic Search. Could you please let me know if you have suggestions. Thanks ! --- ## [Aggregation creating event on timeout, but not pushed to ES](https://discuss.elastic.co/t/aggregation-creating-event-on-timeout-but-not-pushed-to-es/100209)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 0\ **Last updated:** [September 12, 2017, 12:25pm UTC](https://discuss.elastic.co/t/aggregation-creating-event-on-timeout-but-not-pushed-to-es/100209 "2017-09-12T12:25:32Z")
Hi, I did: aggregate { task\_id =\> "%{calledNumber}" code =\> "map\['logNum'\] ||= 0; map\['logNum'\] += 1;" push\_map\_as\_event\_on\_timeout =\> true timeout\_task\_id\_field =\> "calledNumber" timeout =\> 60 … --- ## [Logstash with NSSM never ending when stop service](https://discuss.elastic.co/t/logstash-with-nssm-never-ending-when-stop-service/98882)
**Author:** [@APR](https://discuss.elastic.co/u/APR)\ **Replies:** 1\ **Last updated:** [August 30, 2017, 3:55pm UTC](https://discuss.elastic.co/t/logstash-with-nssm-never-ending-when-stop-service/98882 "2017-08-30T15:55:09Z")
Hello, I use NSSM for create a Windows Service to start/stop Logstash. I follow the instruction you can find here https://github.com/verbosemode/public-notes/blob/master/logstash-windows.md The Windows Service is creat… --- ## [Need Help to install Translate filter in Logstash](https://discuss.elastic.co/t/need-help-to-install-translate-filter-in-logstash/100276)
**Author:** [@Akrishna](https://discuss.elastic.co/u/Akrishna)\ **Replies:** 3\ **Last updated:** [September 13, 2017, 6:22am UTC](https://discuss.elastic.co/t/need-help-to-install-translate-filter-in-logstash/100276 "2017-09-13T06:22:25Z")
My server(centos) has logstash installed but I donot have internet connection there. 1)So I installed translate filter from below site in my local machine(Windows os). https://rubygems.org/gems/logstash-filter-translat… --- ## [Add attachment in elasticsearch using ignest attachment pipeline](https://discuss.elastic.co/t/add-attachment-in-elasticsearch-using-ignest-attachment-pipeline/100211)
**Author:** [@EvalueServe](https://discuss.elastic.co/u/EvalueServe)\ **Replies:** 0\ **Last updated:** [September 12, 2017, 12:33pm UTC](https://discuss.elastic.co/t/add-attachment-in-elasticsearch-using-ignest-attachment-pipeline/100211 "2017-09-12T12:33:46Z")
Hi all, Currently i am working on a project in which we are creating knowledge Repository. User can upload any type of attachment and can also provide some other info about that attachment. Other Info will save in SQL … --- ## [Csv filter can only be used for one file? (multiple files messes up fields/columns)](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 13\ **Last updated:** [September 13, 2017, 4:33am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995 "2017-09-13T04:33:48Z")
Hi, I'm having a big problem with the csv filter. I got four .csv files, each with a separate .conf. When I run those configs in stdout they are fine. When I run logstash and have only one of them active, data gets imp… --- ## [KV-Pairs and type conversion](https://discuss.elastic.co/t/kv-pairs-and-type-conversion/100046)
**Author:** [@desete](https://discuss.elastic.co/u/desete)\ **Replies:** 3\ **Last updated:** [September 13, 2017, 3:54am UTC](https://discuss.elastic.co/t/kv-pairs-and-type-conversion/100046 "2017-09-13T03:54:55Z")
Hello, I have an input like this \<14\>1 2017-09-11T14:00:27.582+02:00 SECRETHOSTNAME RT\_FLOW - RT\_FLOW\_SESSION\_CREATE \[junos@2636.1.1.1.2.40 source-address="1.1.1.1" source-port="53780" destination-address="2.2.2.2" des… --- ## [Some logs not being ingested from LS to ES](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147)
**Author:** [@theflakes](https://discuss.elastic.co/u/theflakes)\ **Replies:** 11\ **Last updated:** [September 12, 2017, 10:54pm UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147 "2017-09-12T22:54:24Z")
Some logs from the same LS instance are not being imported into ES. I see the following notification in ES when sending just the problem logs to ES; "the indices which match this index pattern don't contain any time fie… --- ## [Multiple .csv file data is being added together](https://discuss.elastic.co/t/multiple-csv-file-data-is-being-added-together/100145)
**Author:** [@cxbarbosa](https://discuss.elastic.co/u/cxbarbosa)\ **Replies:** 3\ **Last updated:** [September 12, 2017, 10:45pm UTC](https://discuss.elastic.co/t/multiple-csv-file-data-is-being-added-together/100145 "2017-09-12T22:45:52Z")
I have a wildcard to ingest multiple .csv files, however in the elastic search visualizations, its adding the same column items together instead of skipping repeat entries. I want to create a data dump where new logs up… --- ## [Prevent Logstash/JVM from shutting down when memory usage is high](https://discuss.elastic.co/t/prevent-logstash-jvm-from-shutting-down-when-memory-usage-is-high/100110)
**Author:** [@wakinov](https://discuss.elastic.co/u/wakinov)\ **Replies:** 3\ **Last updated:** [September 12, 2017, 10:42pm UTC](https://discuss.elastic.co/t/prevent-logstash-jvm-from-shutting-down-when-memory-usage-is-high/100110 "2017-09-12T22:42:11Z")
Besides using a buffer like Kafka or increasing the memory on the machines. Is there a way to prevent Logstash from using too much memory and shutting down. For example, have it make input plugins stop accepting new mess… --- ## [Help parsing \[06/Sep/2017:10:57:42 -0400\]](https://discuss.elastic.co/t/help-parsing-06-sep-2017-10-57-42-0400/99622)
**Author:** [@cchooks2](https://discuss.elastic.co/u/cchooks2)\ **Replies:** 23\ **Last updated:** [September 12, 2017, 7:10pm UTC](https://discuss.elastic.co/t/help-parsing-06-sep-2017-10-57-42-0400/99622 "2017-09-12T19:10:36Z")
Hello, I am using the following regex to parse out \[06/Sep/2017:10:57:42 -0400\] \[0-9\]{2}/\[A-z\]{3}/\[0-9\]{4}:\[0-9\]{2}:\[0-9\]{2}:\[0-9\]{2} -\[0-9\]{4} I used the HTTPDATE grok, but this is placing my timestamp in a CST timez… --- ## [Filebeat add system time in prospectors field](https://discuss.elastic.co/t/filebeat-add-system-time-in-prospectors-field/99913)
**Author:** [@Gangadhar\_Mahadevan](https://discuss.elastic.co/u/Gangadhar_Mahadevan)\ **Replies:** 4\ **Last updated:** [September 12, 2017, 6:53pm UTC](https://discuss.elastic.co/t/filebeat-add-system-time-in-prospectors-field/99913 "2017-09-12T18:53:11Z")
Filebeat version 5.2 Is there a way to add system current time (using date command) as a custom field in filebeat prosectors? I am able to add hardcoded values in custom fields but am looking for gathering system time a… --- ## [Logstash Ruby Parse CSV and Validate Its Fields](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643)
**Author:** [@Robert.S](https://discuss.elastic.co/u/Robert.S)\ **Replies:** 6\ **Last updated:** [September 12, 2017, 6:46pm UTC](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643 "2017-09-12T18:46:31Z")
Hello, I am trying to monitor a directory, parse csv files, and validate some of the fields’ values. I thought I could do the following but failed. Any suggestion? I am open for alternatives… input { file { path =\> … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=908) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=910)