# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=277 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 278 --- ## [Logstash does not forget the previous csv header coming from filebeat](https://discuss.elastic.co/t/logstash-does-not-forget-the-previous-csv-header-coming-from-filebeat/256910)
**Author:** [@jason.greffier](https://discuss.elastic.co/u/jason.greffier)\ **Replies:** 2\ **Last updated:** [November 27, 2020, 8:36pm UTC](https://discuss.elastic.co/t/logstash-does-not-forget-the-previous-csv-header-coming-from-filebeat/256910 "2020-11-27T20:36:40Z")
Hi, My issue: When I pass csv files from filebeat with different header, csv output does not match the right column with the right value. Does someone know to manage that, forget the previous csv header read by log … --- ## [Logstash throwing Failed to execute action java.lang.IllegalStateException: error when starting the agent](https://discuss.elastic.co/t/logstash-throwing-failed-to-execute-action-java-lang-illegalstateexception-error-when-starting-the-agent/256875)
**Author:** [@Rahul\_Ravichandran](https://discuss.elastic.co/u/Rahul_Ravichandran)\ **Replies:** 5\ **Last updated:** [November 27, 2020, 4:41pm UTC](https://discuss.elastic.co/t/logstash-throwing-failed-to-execute-action-java-lang-illegalstateexception-error-when-starting-the-agent/256875 "2020-11-27T16:41:30Z")
Hey guys Im trying to push logs from my server to Elasticsearch endpoint through logstash. My server is running a 2016 server OS and the im using a logstash version 7.9.0 This is my logstsh.conf file input { … --- ## [Date is not being parsed with AM / PM](https://discuss.elastic.co/t/date-is-not-being-parsed-with-am-pm/256852)
**Author:** [@shani](https://discuss.elastic.co/u/shani)\ **Replies:** 3\ **Last updated:** [November 27, 2020, 4:21pm UTC](https://discuss.elastic.co/t/date-is-not-being-parsed-with-am-pm/256852 "2020-11-27T16:21:45Z")
HI, this is my config file. I want to extract date from the log and want to create two field like time\_mentioned\_in\_log time\_when\_log\_was\_received\_at\_logstash Raw Log: 3/2/2020 10:14 AM TYPE=Information USER= COMP=AB… --- ## [Logstash warnings: illegal reflective access & pipelines.yml](https://discuss.elastic.co/t/logstash-warnings-illegal-reflective-access-pipelines-yml/256888)
**Author:** [@jaispirit](https://discuss.elastic.co/u/jaispirit)\ **Replies:** 0\ **Last updated:** [November 27, 2020, 2:15pm UTC](https://discuss.elastic.co/t/logstash-warnings-illegal-reflective-access-pipelines-yml/256888 "2020-11-27T14:15:07Z")
Using latest versions of: Ubuntu 20.04-Server mit 4 GB RAM und 2 CPUs OpenJDK 11 Elasticsearch Version: 7.10.0 Kibana Version: 7.10.0 Logstash Version: 7.10.0 Filebeat Version: 7.10.0 $ sudo /usr/share/logstash/bin/lo… --- ## [Logstash 7.9.2 @Docker doesn't start -](https://discuss.elastic.co/t/logstash-7-9-2-docker-doesnt-start/256806)
**Author:** [@Waxman](https://discuss.elastic.co/u/Waxman)\ **Replies:** 5\ **Last updated:** [November 27, 2020, 2:10pm UTC](https://discuss.elastic.co/t/logstash-7-9-2-docker-doesnt-start/256806 "2020-11-27T14:10:23Z")
I cannot start my logstash service on docker. Here is my config: input { file { path =\> "/usr/share/logstash/messages-host" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } beats { port… --- ## [Outdated logstash plugin](https://discuss.elastic.co/t/outdated-logstash-plugin/256881)
**Author:** [@cactus](https://discuss.elastic.co/u/cactus)\ **Replies:** 0\ **Last updated:** [November 27, 2020, 1:29pm UTC](https://discuss.elastic.co/t/outdated-logstash-plugin/256881 "2020-11-27T13:29:27Z")
Hello, guys! Does it safe to upgrade a particular logstash plugin to the latest version even when logstash itself too far behind the current master? Problem: memory leak on grok pattern Solution: update grok pat… --- ## [Help deciphering "message" field](https://discuss.elastic.co/t/help-deciphering-message-field/256880)
**Author:** [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)\ **Replies:** 0\ **Last updated:** [November 27, 2020, 1:17pm UTC](https://discuss.elastic.co/t/help-deciphering-message-field/256880 "2020-11-27T13:17:19Z")
Hello, What input plugin could I use to decipher the following on logstash? I want to be able to do some grok on the message field and it would be much easier and shorter if I deciphered it first... \\u00002\\u00000\\u000… --- ## [\[SOLVED\] Mutate string to integer when field type is already integer](https://discuss.elastic.co/t/solved-mutate-string-to-integer-when-field-type-is-already-integer/256825)
**Author:** [@nnet](https://discuss.elastic.co/u/nnet)\ **Replies:** 5\ **Last updated:** [November 27, 2020, 4:29am UTC](https://discuss.elastic.co/t/solved-mutate-string-to-integer-when-field-type-is-already-integer/256825 "2020-11-27T04:29:27Z")
Hi, using ELK 5.6.16 on Ubuntu 16.04. I'm trying to figure out a way to ingest json where a field learn is of type integer. Once in a while an event will come in and that field's value is not an integer but instead a st… --- ## [CONVERT GSM COORDINATE TO GD COORDINATE](https://discuss.elastic.co/t/convert-gsm-coordinate-to-gd-coordinate/256664)
**Author:** [@Daniel\_Lopez](https://discuss.elastic.co/u/Daniel_Lopez)\ **Replies:** 3\ **Last updated:** [November 26, 2020, 7:57pm UTC](https://discuss.elastic.co/t/convert-gsm-coordinate-to-gd-coordinate/256664 "2020-11-26T19:57:25Z")
Hi to all I'm just trying to convert GSM to GD For example GMS N 39° 56' 41.673'' O 3° 43' 47.872'' to GET GD 39.94490923478156 and -3.729964618217764 I have this config csv { columns =\> \["MCC","MNC","LAC","CI"… --- ## [Logstash conditional elasticsearch output plugin depends on success of S3 output plugin](https://discuss.elastic.co/t/logstash-conditional-elasticsearch-output-plugin-depends-on-success-of-s3-output-plugin/256800)
**Author:** [@Abolurah](https://discuss.elastic.co/u/Abolurah)\ **Replies:** 0\ **Last updated:** [November 26, 2020, 4:49pm UTC](https://discuss.elastic.co/t/logstash-conditional-elasticsearch-output-plugin-depends-on-success-of-s3-output-plugin/256800 "2020-11-26T16:49:19Z")
Hi All, In my Logstash configuration, I have 2 output plugins, one is S3 and another is Elasticsearch. My goal is to send logs to S3 and update the same logs with "uploadStatus = finished", everything works fine until … --- ## [Filter for content of specific field of second index](https://discuss.elastic.co/t/filter-for-content-of-specific-field-of-second-index/256796)
**Author:** [@elk51211](https://discuss.elastic.co/u/elk51211)\ **Replies:** 0\ **Last updated:** [November 26, 2020, 4:35pm UTC](https://discuss.elastic.co/t/filter-for-content-of-specific-field-of-second-index/256796 "2020-11-26T16:35:19Z")
Hello, I'm trying to setup a filter that creates unique fingerprints if username does not already exists in index identities. Otherwise look up the value of the created pseudoId and put this in the event index. How do a… --- ## [Running another logstash instance incase one logstash fails](https://discuss.elastic.co/t/running-another-logstash-instance-incase-one-logstash-fails/256656)
**Author:** [@mahi2](https://discuss.elastic.co/u/mahi2)\ **Replies:** 3\ **Last updated:** [November 26, 2020, 2:54pm UTC](https://discuss.elastic.co/t/running-another-logstash-instance-incase-one-logstash-fails/256656 "2020-11-26T14:54:35Z")
Hi, I have a scenario where one logstash is already running, and for some reason the machine fails. I want the process to keep going and not wait for the machine to be restarted.It must automatically start taking data f… --- ## [Persistent queue, drop?](https://discuss.elastic.co/t/persistent-queue-drop/256785)
**Author:** [@diecode](https://discuss.elastic.co/u/diecode)\ **Replies:** 0\ **Last updated:** [November 26, 2020, 2:42pm UTC](https://discuss.elastic.co/t/persistent-queue-drop/256785 "2020-11-26T14:42:59Z")
Hey, I am using persistent queue in my logstash with aggregate filter. I would like drop few incoming syslog message in filter, but when I use drop, then it will not removed from queue, thus my queue reach the maximum l… --- ## [S3 buckets data format unknown](https://discuss.elastic.co/t/s3-buckets-data-format-unknown/256773)
**Author:** [@shani](https://discuss.elastic.co/u/shani)\ **Replies:** 0\ **Last updated:** [November 26, 2020, 1:05pm UTC](https://discuss.elastic.co/t/s3-buckets-data-format-unknown/256773 "2020-11-26T13:05:13Z")
HI, I've 50K+ documents of unknown log format that can't be segregated until someone manually open and read the document content. Is there any way to take all documents data in input and make it parse them? What should … --- ## [Taking care of syslog PRI header in filter](https://discuss.elastic.co/t/taking-care-of-syslog-pri-header-in-filter/256732)
**Author:** [@Elitlogik](https://discuss.elastic.co/u/Elitlogik)\ **Replies:** 0\ **Last updated:** [November 26, 2020, 7:53am UTC](https://discuss.elastic.co/t/taking-care-of-syslog-pri-header-in-filter/256732 "2020-11-26T07:53:36Z")
I use kv filter on a syslog data on the form key1=value1 key2=value2 etc. However, every syslog row has a leading PRI header of key1=value1. Is there a way to: Parse the pri header and all key-value pairs. Remove… --- ## [What config will need to send mail using log4j.property settings in logstash?](https://discuss.elastic.co/t/what-config-will-need-to-send-mail-using-log4j-property-settings-in-logstash/256481)
**Author:** [@talbehat](https://discuss.elastic.co/u/talbehat)\ **Replies:** 4\ **Last updated:** [November 26, 2020, 5:51am UTC](https://discuss.elastic.co/t/what-config-will-need-to-send-mail-using-log4j-property-settings-in-logstash/256481 "2020-11-26T05:51:28Z")
currently logstash create plain.log, how can send mail using log4j2.property file. any idea? --- ## [Date type fields not being created though date format conversion works](https://discuss.elastic.co/t/date-type-fields-not-being-created-though-date-format-conversion-works/256709)
**Author:** [@Chris\_Stone](https://discuss.elastic.co/u/Chris_Stone)\ **Replies:** 2\ **Last updated:** [November 25, 2020, 8:21pm UTC](https://discuss.elastic.co/t/date-type-fields-not-being-created-though-date-format-conversion-works/256709 "2020-11-25T20:21:50Z")
Hi, I am collecting http access logs and wanting to place some date headers into date type fields. I've tried a couple of things, shown below, but while the date format is converted properly, the resulting field is still… --- ## [RequestElaspedTime](https://discuss.elastic.co/t/requestelaspedtime/256369)
**Author:** [@dfoot](https://discuss.elastic.co/u/dfoot)\ **Replies:** 8\ **Last updated:** [November 25, 2020, 8:01pm UTC](https://discuss.elastic.co/t/requestelaspedtime/256369 "2020-11-25T20:01:25Z")
Recently upgraded log message include field requestElapsedTime to replaced elapsed\_tme but from index pattern for requestElaspedTIme the type is 'string' only not 'number' for metric result. I've added below on logstash… --- ## [Logstash 7.10 fails with date filter UNIX\_MS - failed to parse date field strict\_date\_optional\_time](https://discuss.elastic.co/t/logstash-7-10-fails-with-date-filter-unix-ms-failed-to-parse-date-field-strict-date-optional-time/256583)
**Author:** [@dfrewin](https://discuss.elastic.co/u/dfrewin)\ **Replies:** 2\ **Last updated:** [November 25, 2020, 6:55pm UTC](https://discuss.elastic.co/t/logstash-7-10-fails-with-date-filter-unix-ms-failed-to-parse-date-field-strict-date-optional-time/256583 "2020-11-25T18:55:59Z")
Initially logstash was running when we were on 6.8 but as soon as we upgraded to 7.9.3 it started failing. Using salesforce logstash plugin error msg retrieves the time value successfully, but looks to fail when sending… --- ## [JSON Filter Making Events Disappear](https://discuss.elastic.co/t/json-filter-making-events-disappear/256199)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 7\ **Last updated:** [November 25, 2020, 5:08pm UTC](https://discuss.elastic.co/t/json-filter-making-events-disappear/256199 "2020-11-25T17:08:21Z")
I've been using the same filter configuration across multiple upgrades of Logstash. After upgrading to 7.10.0, the JSON filter now makes events...disappear. Events are sent to Logstash via filebeat (running v 7.10.0). … --- ## [Grok increase CPU usage to 700+%](https://discuss.elastic.co/t/grok-increase-cpu-usage-to-700/256633)
**Author:** [@vaclav1](https://discuss.elastic.co/u/vaclav1)\ **Replies:** 2\ **Last updated:** [November 25, 2020, 2:42pm UTC](https://discuss.elastic.co/t/grok-increase-cpu-usage-to-700/256633 "2020-11-25T14:42:28Z")
Hello team, we have implemented a following GROK multi pattern match but when we deploy the pipeline for this component it raises CPU from 100 to 700%. Is there a way how to improve its performance? One way would be t… --- ## [CPU usage for logstash hits over 300%](https://discuss.elastic.co/t/cpu-usage-for-logstash-hits-over-300/255174)
**Author:** [@powerinside](https://discuss.elastic.co/u/powerinside)\ **Replies:** 5\ **Last updated:** [November 25, 2020, 2:28pm UTC](https://discuss.elastic.co/t/cpu-usage-for-logstash-hits-over-300/255174 "2020-11-25T14:28:00Z")
Hello everybody, I am trying to parse logs using multiple grok filters. I have noticed that if i add following pattern, CPU usage for logstash hits over 300%. I have a couple of grok patterns for nginx logs as well wit… --- ## [User\_agent filter regexes.yaml location](https://discuss.elastic.co/t/user-agent-filter-regexes-yaml-location/256625)
**Author:** [@probson](https://discuss.elastic.co/u/probson)\ **Replies:** 2\ **Last updated:** [November 25, 2020, 2:11pm UTC](https://discuss.elastic.co/t/user-agent-filter-regexes-yaml-location/256625 "2020-11-25T14:11:21Z")
Hi, I am looking to use the user\_agent filter in logstash. In the guide (https://www.elastic.co/guide/en/logstash/current/plugins-filters-useragent.html#plugins-filters-useragent-regexes) it states If not specified, t… --- ## [Logstash remove after split](https://discuss.elastic.co/t/logstash-remove-after-split/256608)
**Author:** [@tamilarasanbravo](https://discuss.elastic.co/u/tamilarasanbravo)\ **Replies:** 1\ **Last updated:** [November 25, 2020, 1:45pm UTC](https://discuss.elastic.co/t/logstash-remove-after-split/256608 "2020-11-25T13:45:40Z")
Hi Team, I have configured a logstash http which receives JSON data. As part of the logstash filter I split a json field/value of array into 5 different key values. i.e., original Field commits : { author : TJ email… --- ## [Logstash with multiple conf files](https://discuss.elastic.co/t/logstash-with-multiple-conf-files/255664)
**Author:** [@Kfiro](https://discuss.elastic.co/u/Kfiro)\ **Replies:** 6\ **Last updated:** [November 25, 2020, 1:37pm UTC](https://discuss.elastic.co/t/logstash-with-multiple-conf-files/255664 "2020-11-25T13:37:41Z")
Hi ELK, I'm trying to configure my Logstash to run with 2 "conf" files, First: winlogbeat.conf input { beats { port =\> 5044 } } output { elasticsearch { hosts=\>\["https://elk01:9200"\] index=\>"logstas… --- ## [Logstash logging](https://discuss.elastic.co/t/logstash-logging/256655)
**Author:** [@hzarrabi](https://discuss.elastic.co/u/hzarrabi)\ **Replies:** 0\ **Last updated:** [November 25, 2020, 12:10pm UTC](https://discuss.elastic.co/t/logstash-logging/256655 "2020-11-25T12:10:26Z")
Hi Guys, I would like to configure logstash log4j2.properties in order to fix the log size to 10M and to delete those older then 7 days. Could you please tell me which parameters to set log4j2.properties. Thanks, Bes… --- ## [Aggregate multiple events based on file](https://discuss.elastic.co/t/aggregate-multiple-events-based-on-file/256650)
**Author:** [@Muhammad\_Faisal](https://discuss.elastic.co/u/Muhammad_Faisal)\ **Replies:** 0\ **Last updated:** [November 25, 2020, 11:50am UTC](https://discuss.elastic.co/t/aggregate-multiple-events-based-on-file/256650 "2020-11-25T11:50:21Z")
hi , i have below directory structure and each subnode contains multiple log files which are generated daily and closed and then next day or some time after couple of days another file is generated and closed....i am ca… --- ## [Map serivce url to service/operation](https://discuss.elastic.co/t/map-serivce-url-to-service-operation/256642)
**Author:** [@rmrfchik](https://discuss.elastic.co/u/rmrfchik)\ **Replies:** 0\ **Last updated:** [November 25, 2020, 11:05am UTC](https://discuss.elastic.co/t/map-serivce-url-to-service-operation/256642 "2020-11-25T11:05:34Z")
I have a number of services across servers, each service serves urls like /service1/operation1 /service1/operation2?data /service2/path/operation3/data1 /service2/path/operation3/data2 I need to measure metrics for gi… --- ## [Net::ReadTimeout, how to avoid going to artifacts.elastic.co/downloads for customizedlogstash java-plugin](https://discuss.elastic.co/t/net-readtimeout-how-to-avoid-going-to-artifacts-elastic-co-downloads-for-customizedlogstash-java-plugin/256626)
**Author:** [@hellopcz](https://discuss.elastic.co/u/hellopcz)\ **Replies:** 1\ **Last updated:** [November 25, 2020, 9:40am UTC](https://discuss.elastic.co/t/net-readtimeout-how-to-avoid-going-to-artifacts-elastic-co-downloads-for-customizedlogstash-java-plugin/256626 "2020-11-25T09:40:13Z")
I developed some logstash java plugin and it run well under good network condition. But under some bad network condition, I have to use proxy for internet surfing, the plugin shows the Net::ReadTimeout. --- ## [BadRequirementError, Illformed requirement, fail to install offline logstash java plugin for the version reason or others?](https://discuss.elastic.co/t/badrequirementerror-illformed-requirement-fail-to-install-offline-logstash-java-plugin-for-the-version-reason-or-others/256619)
**Author:** [@hellopcz](https://discuss.elastic.co/u/hellopcz)\ **Replies:** 1\ **Last updated:** [November 25, 2020, 9:01am UTC](https://discuss.elastic.co/t/badrequirementerror-illformed-requirement-fail-to-install-offline-logstash-java-plugin-for-the-version-reason-or-others/256619 "2020-11-25T09:01:13Z")
I've packed logstash offline java plugin but the plugin can not be well installed. it may be caused by wrong version. The offline plugin was built by prepare-offline-pack command but plugin name has to be set to logstas… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=276) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=278)