# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=394 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 395 --- ## [Logstash if message is empty, drop the entire row](https://discuss.elastic.co/t/logstash-if-message-is-empty-drop-the-entire-row/221986)
**Author:** [@katara](https://discuss.elastic.co/u/katara)\ **Replies:** 9\ **Last updated:** [March 6, 2020, 1:38pm UTC](https://discuss.elastic.co/t/logstash-if-message-is-empty-drop-the-entire-row/221986 "2020-03-06T13:38:40Z")
Hello, I have a scenario where my Log messages are empty in a few cases: So what I want to do is, If message is empty, then drop the whole row. I tried filter { if \[Message\] == "" { drop { } … --- ## [Jdbc\_static local\_lookup not working with Azure data](https://discuss.elastic.co/t/jdbc-static-local-lookup-not-working-with-azure-data/222303)
**Author:** [@spate](https://discuss.elastic.co/u/spate)\ **Replies:** 1\ **Last updated:** [March 6, 2020, 1:17pm UTC](https://discuss.elastic.co/t/jdbc-static-local-lookup-not-working-with-azure-data/222303 "2020-03-06T13:17:30Z")
Hi, I'm working with Azure data being piped into ELK, and I want to add the subscription name to the event. Currently it only has the subscription ID. We have an external postgreql DB with subscription name and ID (as … --- ## [Graphite input and output logstash](https://discuss.elastic.co/t/graphite-input-and-output-logstash/222365)
**Author:** [@jalajgujral](https://discuss.elastic.co/u/jalajgujral)\ **Replies:** 3\ **Last updated:** [March 6, 2020, 11:07am UTC](https://discuss.elastic.co/t/graphite-input-and-output-logstash/222365 "2020-03-06T11:07:04Z")
Hi Guys , I am listing to one port where i am getting graphite and i want to send metrics to another graphite instance i am using graphite input and output plugins This my config input { graphite { port =\> 8686 } o… --- ## [Use of filebeat templates - single template vs multiple templates](https://discuss.elastic.co/t/use-of-filebeat-templates-single-template-vs-multiple-templates/222437)
**Author:** [@mvienne](https://discuss.elastic.co/u/mvienne)\ **Replies:** 1\ **Last updated:** [March 6, 2020, 10:27am UTC](https://discuss.elastic.co/t/use-of-filebeat-templates-single-template-vs-multiple-templates/222437 "2020-03-06T10:27:35Z")
I'm confused about the use of filebeat templates with the file fields.yml. If I want to store two types of logs into elasticsearch, let's say apache logs and postgres logs, is it better to have a single template in the … --- ## [Grok parse error in Kibana](https://discuss.elastic.co/t/grok-parse-error-in-kibana/222391)
**Author:** [@kaj](https://discuss.elastic.co/u/kaj)\ **Replies:** 1\ **Last updated:** [March 6, 2020, 7:40am UTC](https://discuss.elastic.co/t/grok-parse-error-in-kibana/222391 "2020-03-06T07:40:46Z")
I've been constructing a grok filter for a log that has lines like the one below: 2018-09-19 03:48:46.900-05:00 \[HOST:hostname.domain.com\]\[SERVER:VariableString\]\[PID:35570\]\[THR:3726157568\]\[Kernel XML API\]\[Trace\] XML Com… --- ## [Parsing CSV file with multiple key values in one line](https://discuss.elastic.co/t/parsing-csv-file-with-multiple-key-values-in-one-line/221959)
**Author:** [@david4](https://discuss.elastic.co/u/david4)\ **Replies:** 3\ **Last updated:** [March 5, 2020, 11:07pm UTC](https://discuss.elastic.co/t/parsing-csv-file-with-multiple-key-values-in-one-line/221959 "2020-03-05T23:07:38Z")
Hello, I'm new to the ELK stack and have been struggling to solve one problem. I'm trying to write a Logstash configuration to parse a log file of csv format as follow: "time\_stamp", 2020-02-28 16:35:16.048, "name", "D… --- ## [Remove old data from database](https://discuss.elastic.co/t/remove-old-data-from-database/222374)
**Author:** [@coi6644](https://discuss.elastic.co/u/coi6644)\ **Replies:** 7\ **Last updated:** [March 5, 2020, 9:57pm UTC](https://discuss.elastic.co/t/remove-old-data-from-database/222374 "2020-03-05T21:57:35Z")
Hi, How can I remove data older from 12 months from database? I have a ELK+Elastiflow to collect logs from some equipments, but I need to do a cleanup of records older than 12 monts. --- ## [Selectively dedot some fields](https://discuss.elastic.co/t/selectively-dedot-some-fields/218489)
**Author:** [@Todd\_Lyons](https://discuss.elastic.co/u/Todd_Lyons)\ **Replies:** 3\ **Last updated:** [March 5, 2020, 5:55pm UTC](https://discuss.elastic.co/t/selectively-dedot-some-fields/218489 "2020-03-05T17:55:30Z")
TL;DR: How to detect subfields with dots, converting those dots to underscores, but ONLY in fields under that subfield. Detail: We use filebeats, logstash, and feed messages to an Elastic Stack, of which all pieces are… --- ## [How to get first match regex grok](https://discuss.elastic.co/t/how-to-get-first-match-regex-grok/222339)
**Author:** [@nino](https://discuss.elastic.co/u/nino)\ **Replies:** 0\ **Last updated:** [March 5, 2020, 4:27pm UTC](https://discuss.elastic.co/t/how-to-get-first-match-regex-grok/222339 "2020-03-05T16:27:48Z")
Dear all, I want to retrieve only "14" from this message 3/03/0 EE 14 GFR 20 AAA XXXXX 50 3365.00 this is my grok code grok{ match =\> { field1 =\> \[ "(?\\\d{0}\\s\\d{1,3}\\s{1})" … --- ## [RabbitMQ input using a non standard port](https://discuss.elastic.co/t/rabbitmq-input-using-a-non-standard-port/222069)
**Author:** [@damianconnolly](https://discuss.elastic.co/u/damianconnolly)\ **Replies:** 2\ **Last updated:** [March 5, 2020, 4:01pm UTC](https://discuss.elastic.co/t/rabbitmq-input-using-a-non-standard-port/222069 "2020-03-05T16:01:20Z")
Hi, I'm working on a project to containerise all of our applications to remove the reliance on physical devices (as I'm sure half the world is doing right now!). We've moved most of our application estate to Docker/K8s … --- ## [Data missing with logstash elasticsearch-input-plugin](https://discuss.elastic.co/t/data-missing-with-logstash-elasticsearch-input-plugin/222327)
**Author:** [@higee](https://discuss.elastic.co/u/higee)\ **Replies:** 0\ **Last updated:** [March 5, 2020, 3:09pm UTC](https://discuss.elastic.co/t/data-missing-with-logstash-elasticsearch-input-plugin/222327 "2020-03-05T15:09:31Z")
Hi, I was using Logstash (7.4.2) to transfer documents between Elasticsearch indices (7.4.2). First have a look at the Logstash.conf input { elasticsearch { hosts =\> "" index =\> "haproxy-\*" scroll =\> "10m… --- ## [Logstash log is static and not moving with input data](https://discuss.elastic.co/t/logstash-log-is-static-and-not-moving-with-input-data/222319)
**Author:** [@lnitin](https://discuss.elastic.co/u/lnitin)\ **Replies:** 0\ **Last updated:** [March 5, 2020, 2:42pm UTC](https://discuss.elastic.co/t/logstash-log-is-static-and-not-moving-with-input-data/222319 "2020-03-05T14:42:53Z")
I’m installing Logstash 7.5.1 on cent os 7 machine, Logstash logs are not showing any data being digested when passed Apache logs. If shows successful and api is up on 9600 and port 5400. But however Logstash shows few… --- ## [Grok to parse Json in a Json](https://discuss.elastic.co/t/grok-to-parse-json-in-a-json/222258)
**Author:** [@aashish](https://discuss.elastic.co/u/aashish)\ **Replies:** 1\ **Last updated:** [March 5, 2020, 2:34pm UTC](https://discuss.elastic.co/t/grok-to-parse-json-in-a-json/222258 "2020-03-05T14:34:42Z")
Hi Team, I'm facing issue to parse json in a json. Below is my sample log. Using Json formate in logstash it is working fine. But now I want to parse fields that are in "txn\_pl\_resp" (i.e. code & message) Using ELK ve… --- ## [Using external document\_id](https://discuss.elastic.co/t/using-external-document-id/221200)
**Author:** [@jwinth](https://discuss.elastic.co/u/jwinth)\ **Replies:** 8\ **Last updated:** [March 5, 2020, 2:16pm UTC](https://discuss.elastic.co/t/using-external-document-id/221200 "2020-03-05T14:16:17Z")
Hi, Whenever i try to assign the document\_id with some attribute from the documents, it creates just one new document on ElasticSearch with the document\_id as the name of the attribute i'm trying to use. What i want is … --- ## [Create fields by splitting a log](https://discuss.elastic.co/t/create-fields-by-splitting-a-log/221891)
**Author:** [@prasadlele](https://discuss.elastic.co/u/prasadlele)\ **Replies:** 7\ **Last updated:** [March 5, 2020, 2:11pm UTC](https://discuss.elastic.co/t/create-fields-by-splitting-a-log/221891 "2020-03-05T14:11:29Z")
Hi , I have configure logstash to read my log file which gives me below input. Now I want to split the message and create a field for each comma separated values against each. Tried to use mutate in my logstash.conf f… --- ## [If condition not working](https://discuss.elastic.co/t/if-condition-not-working/222122)
**Author:** [@wifi](https://discuss.elastic.co/u/wifi)\ **Replies:** 8\ **Last updated:** [March 5, 2020, 1:36pm UTC](https://discuss.elastic.co/t/if-condition-not-working/222122 "2020-03-05T13:36:36Z")
I use Logstash + Filebeat + Elasticsearch. In filebeat the elasticsearch plugin is enabled so that the logs of elasticsearch get collected. Now i want to define a simple condition in a logstash pipeline. I want to do s… --- ## [Understanding Logstash configuration naming](https://discuss.elastic.co/t/understanding-logstash-configuration-naming/222295)
**Author:** [@tbhaxor](https://discuss.elastic.co/u/tbhaxor)\ **Replies:** 0\ **Last updated:** [March 5, 2020, 1:34pm UTC](https://discuss.elastic.co/t/understanding-logstash-configuration-naming/222295 "2020-03-05T13:34:13Z")
In one of the online documentation, I found this name10-suricata.conf. Also, the prefix number changes for some other configurations. So apart from prioritizing configs, what else this numbering means? --- ## [Logstash not parsing data properly](https://discuss.elastic.co/t/logstash-not-parsing-data-properly/221912)
**Author:** [@Vishnu\_mk](https://discuss.elastic.co/u/Vishnu_mk)\ **Replies:** 3\ **Last updated:** [March 5, 2020, 12:20pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-data-properly/221912 "2020-03-05T12:20:39Z")
My csv file contains ^M character ( ^ M ) characters are introduced when you use lines of text from a windows computer to Linux or Unix machine How should i get rid of ^M character in logstash --- ## [Change the regexes useragent filter plugin uses by default](https://discuss.elastic.co/t/change-the-regexes-useragent-filter-plugin-uses-by-default/222282)
**Author:** [@mikewillis](https://discuss.elastic.co/u/mikewillis)\ **Replies:** 0\ **Last updated:** [March 5, 2020, 12:11pm UTC](https://discuss.elastic.co/t/change-the-regexes-useragent-filter-plugin-uses-by-default/222282 "2020-03-05T12:11:42Z")
This question is about version of Logstash 5.6 onwards as myself and colleagues are still working on moving away from 5 (yes we know it's EOL). At time of writing the Logstash useragent filter plugin identifies Chromium… --- ## [The index pattern access\* does not contain any of the following compatible field types: geo\_point](https://discuss.elastic.co/t/the-index-pattern-access-does-not-contain-any-of-the-following-compatible-field-types-geo-point/221989)
**Author:** [@vishal001](https://discuss.elastic.co/u/vishal001)\ **Replies:** 2\ **Last updated:** [March 5, 2020, 11:52am UTC](https://discuss.elastic.co/t/the-index-pattern-access-does-not-contain-any-of-the-following-compatible-field-types-geo-point/221989 "2020-03-05T11:52:53Z")
HI, I am facing the below error. any one have idea how to resolve the same. I almost try everything which is available on google but not sure where I am wrong. The index pattern access\* does not contain any of the foll… --- ## [Multiline pattern for date is not working](https://discuss.elastic.co/t/multiline-pattern-for-date-is-not-working/222272)
**Author:** [@Sudheer\_T](https://discuss.elastic.co/u/Sudheer_T)\ **Replies:** 0\ **Last updated:** [March 5, 2020, 11:24am UTC](https://discuss.elastic.co/t/multiline-pattern-for-date-is-not-working/222272 "2020-03-05T11:24:08Z")
input { file { path =\> "C:/ELK/logs/\*.log" codec =\> multiline { pattern =\> "^%{TIMESTAMP\_ISO8601}" negate =\> true what =\> "previous" } } } --- ## [Getting Memory Leak Errors in Logstatsh 7.4.0](https://discuss.elastic.co/t/getting-memory-leak-errors-in-logstatsh-7-4-0/222255)
**Author:** [@msunilreddy](https://discuss.elastic.co/u/msunilreddy)\ **Replies:** 0\ **Last updated:** [March 5, 2020, 9:58am UTC](https://discuss.elastic.co/t/getting-memory-leak-errors-in-logstatsh-7-4-0/222255 "2020-03-05T09:58:27Z")
Hi Team, I am using logstash 7.4.0 version and getting memory leak errors while staring of logstash but logs are successfully sending to ES and able to view them in Kibana. I am getting memory leak errors only on Prod … --- ## [Filter conditions, config](https://discuss.elastic.co/t/filter-conditions-config/222221)
**Author:** [@111291](https://discuss.elastic.co/u/111291)\ **Replies:** 1\ **Last updated:** [March 5, 2020, 9:30am UTC](https://discuss.elastic.co/t/filter-conditions-config/222221 "2020-03-05T09:30:01Z")
We have three servers. Each sends logs of one format. I am trying to make conditions for filtering logs from these servers using tags. But that does not work. input { beats { port =\> 5044 type =\> beats } u… --- ## [Grok filter not being applied but logstash logs being collected](https://discuss.elastic.co/t/grok-filter-not-being-applied-but-logstash-logs-being-collected/221112)
**Author:** [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Replies:** 9\ **Last updated:** [March 5, 2020, 9:24am UTC](https://discuss.elastic.co/t/grok-filter-not-being-applied-but-logstash-logs-being-collected/221112 "2020-03-05T09:24:10Z")
I am passing logs from server to Elasticsearch and they are indexed properly but the grok filter I have applied in not extracting the information. The grok pattern works perfectly when i use it in grok debugger against t… --- ## [Attempted to resurrect connection to dead ES instance with automatic config reload 6.7](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance-with-automatic-config-reload-6-7/222234)
**Author:** [@AmirZ](https://discuss.elastic.co/u/AmirZ)\ **Replies:** 0\ **Last updated:** [March 5, 2020, 8:33am UTC](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance-with-automatic-config-reload-6-7/222234 "2020-03-05T08:33:34Z")
Hi, i have the following config: output { elasticsearch { hosts =\> \["localhost:9200"\] index =\> "%{\[@metadata\]\[index\]}-logs-%{+YYYY.MM.dd}" } elasticsearch { hosts =\> \["remote:9200"\] index =\> "%{\[@metadata\]\[index… --- ## [Remove a specific field from the log and overwrite the message field](https://discuss.elastic.co/t/remove-a-specific-field-from-the-log-and-overwrite-the-message-field/221820)
**Author:** [@nmoham](https://discuss.elastic.co/u/nmoham)\ **Replies:** 7\ **Last updated:** [March 5, 2020, 8:07am UTC](https://discuss.elastic.co/t/remove-a-specific-field-from-the-log-and-overwrite-the-message-field/221820 "2020-03-05T08:07:14Z")
trying to parse the following log using the KV filter ; but there are some fields "message", "type" and "index" in the log message, which is overwriting the original "message", "type", "index" fields dropping the entire … --- ## [What is the best way to forward sysmon logs to ELK Stack](https://discuss.elastic.co/t/what-is-the-best-way-to-forward-sysmon-logs-to-elk-stack/222214)
**Author:** [@My\_Miki](https://discuss.elastic.co/u/My_Miki)\ **Replies:** 0\ **Last updated:** [March 5, 2020, 7:20am UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-forward-sysmon-logs-to-elk-stack/222214 "2020-03-05T07:20:05Z")
Hi, Current practice is to forward all sysmon logs to syslog. How should i forward these logs from syslog server to ELK stack ? We dont want to install winlogbeats on every system. --- ## [Separate indexing from multiple sources in one logstash file](https://discuss.elastic.co/t/separate-indexing-from-multiple-sources-in-one-logstash-file/222206)
**Author:** [@Kanika\_Agarwal](https://discuss.elastic.co/u/Kanika_Agarwal)\ **Replies:** 0\ **Last updated:** [March 5, 2020, 6:32am UTC](https://discuss.elastic.co/t/separate-indexing-from-multiple-sources-in-one-logstash-file/222206 "2020-03-05T06:32:58Z")
Hi, I am trying to pull data from Git using flask and then uploading it on Elastic. We are using 3 APIs and the flask is giving us the required results. But when we try to index it using http\_poller plugin in one .conf f… --- ## [Logstash TCP unable to work](https://discuss.elastic.co/t/logstash-tcp-unable-to-work/222092)
**Author:** [@kopilim96](https://discuss.elastic.co/u/kopilim96)\ **Replies:** 4\ **Last updated:** [March 5, 2020, 6:17am UTC](https://discuss.elastic.co/t/logstash-tcp-unable-to-work/222092 "2020-03-05T06:17:15Z")
I would like to work with 2 logstash and I have made the setting in the file pipeline.yml My pipeline.yml: - pipeline.id: input\_pipeline\_1 path.config: "config/Logstash1.conf" pipeline.workers: 3 - pipeline.id: fo… --- ## [How to load json file into ES using logstash](https://discuss.elastic.co/t/how-to-load-json-file-into-es-using-logstash/218620)
**Author:** [@Deepak\_Dandotiya](https://discuss.elastic.co/u/Deepak_Dandotiya)\ **Replies:** 12\ **Last updated:** [March 5, 2020, 6:06am UTC](https://discuss.elastic.co/t/how-to-load-json-file-into-es-using-logstash/218620 "2020-03-05T06:06:38Z")
Hi Guys, can anyone provide me sample logstash conf file to parse and document the json formatted data into elasticsearch using logstash. There are multiple fields which needs to parsed. Any quick help is appreciated so… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=393) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=395)