# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=472 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 473 --- ## [Exclude null itens from aggregation](https://discuss.elastic.co/t/exclude-null-itens-from-aggregation/200916)
**Author:** [@fernando.vicente](https://discuss.elastic.co/u/fernando.vicente)\ **Replies:** 2\ **Last updated:** [September 24, 2019, 6:10pm UTC](https://discuss.elastic.co/t/exclude-null-itens-from-aggregation/200916 "2019-09-24T18:10:40Z")
Greetings, I'm trying to do an aggregation like that; map\['sub\_menu'\] ||= \[\] checkedFieldSubMenu = { 'sub\_menu\_id' =\> event.get('sub\_menu\_id'), 'cms\_menu\_name' =\>… --- ## [Trouble with ilm policy when using logstash](https://discuss.elastic.co/t/trouble-with-ilm-policy-when-using-logstash/200894)
**Author:** [@rjette](https://discuss.elastic.co/u/rjette)\ **Replies:** 0\ **Last updated:** [September 24, 2019, 2:44pm UTC](https://discuss.elastic.co/t/trouble-with-ilm-policy-when-using-logstash/200894 "2019-09-24T14:44:54Z")
Good morning, I've been using several beats with elasticsearch and kibana for a while. Everything works well. When I attempt to use logstash things no longer work as expected. As a test I setup winlog beat on a machi… --- ## [Logstash: How to replace field values in JSON based on ID](https://discuss.elastic.co/t/logstash-how-to-replace-field-values-in-json-based-on-id/200850)
**Author:** [@Igor\_Belyakov](https://discuss.elastic.co/u/Igor_Belyakov)\ **Replies:** 2\ **Last updated:** [September 24, 2019, 3:05pm UTC](https://discuss.elastic.co/t/logstash-how-to-replace-field-values-in-json-based-on-id/200850 "2019-09-24T15:05:32Z")
Hello How to replace all occurrences of ID in such JSON: "\_source": { "rules": \[ { "ruleID": "001" "action": "count" } { "ruleID": "003" "action": "count" } \] } Provided that all identifiers are known: 001… --- ## [SOAP call in HTTP\_POLLER with body](https://discuss.elastic.co/t/soap-call-in-http-poller-with-body/200885)
**Author:** [@fernrguez](https://discuss.elastic.co/u/fernrguez)\ **Replies:** 2\ **Last updated:** [September 24, 2019, 2:19pm UTC](https://discuss.elastic.co/t/soap-call-in-http-poller-with-body/200885 "2019-09-24T14:19:04Z")
Hello, I'm trying to retrieve logs from a SOAP WS, I tried several options but no luck so far. The main issue is related with the namespaces that requires quotations. I tested several options: Escaping quotes with \\… --- ## [Logstash pipeline with Elasticsearch on Openshift](https://discuss.elastic.co/t/logstash-pipeline-with-elasticsearch-on-openshift/200842)
**Author:** [@ShahafSW](https://discuss.elastic.co/u/ShahafSW)\ **Replies:** 1\ **Last updated:** [September 24, 2019, 2:07pm UTC](https://discuss.elastic.co/t/logstash-pipeline-with-elasticsearch-on-openshift/200842 "2019-09-24T14:07:19Z")
Hey everyone, Usually, I am using curl command to elasticsearch with the following flags: curl --key /PATH/admin-key --cert /PATH/admin-cert --cacert /PATH/admin-ca -XGET "https://ROUTE:PORT" This curl command w… --- ## [\_dataparsefailure in tags](https://discuss.elastic.co/t/dataparsefailure-in-tags/200824)
**Author:** [@zebu14](https://discuss.elastic.co/u/zebu14)\ **Replies:** 8\ **Last updated:** [September 24, 2019, 2:00pm UTC](https://discuss.elastic.co/t/dataparsefailure-in-tags/200824 "2019-09-24T14:00:10Z")
Hello, I think this issue is very common but I am still searching for a solution. I have this log line : Mar 9 02:20:42 dvgospxi1 (squid-1): 09/Mar/2017:02:20:42 +0100 - TCP\_DENIED 403 10.117.56.12:47151 3529 GET htt… --- ## [Import external ruby file for aggregate filter - performance tuning](https://discuss.elastic.co/t/import-external-ruby-file-for-aggregate-filter-performance-tuning/200804)
**Author:** [@isharamadhawa](https://discuss.elastic.co/u/isharamadhawa)\ **Replies:** 0\ **Last updated:** [September 24, 2019, 7:04am UTC](https://discuss.elastic.co/t/import-external-ruby-file-for-aggregate-filter-performance-tuning/200804 "2019-09-24T07:04:17Z")
In my logstash configuration file I have following code in serveral places with slight changes. aggregate { task\_id =\> "%{tid}" code =\> " require '/etc/log… --- ## [Drop message if tag is search and message has multiple strings](https://discuss.elastic.co/t/drop-message-if-tag-is-search-and-message-has-multiple-strings/200492)
**Author:** [@Abinnd\_G](https://discuss.elastic.co/u/Abinnd_G)\ **Replies:** 2\ **Last updated:** [September 24, 2019, 10:46am UTC](https://discuss.elastic.co/t/drop-message-if-tag-is-search-and-message-has-multiple-strings/200492 "2019-09-24T10:46:55Z")
I am looking for dropping a message from the tag search if message has the following strings. if "WARN" in \[message\] { drop{ } } if ".java:" in \[message\] { drop{ } } if "has open connections " in \[message\] How can I… --- ## [How to add MikroTik router in ELK v7](https://discuss.elastic.co/t/how-to-add-mikrotik-router-in-elk-v7/200507)
**Author:** [@lokmanopt](https://discuss.elastic.co/u/lokmanopt)\ **Replies:** 0\ **Last updated:** [September 21, 2019, 2:29am UTC](https://discuss.elastic.co/t/how-to-add-mikrotik-router-in-elk-v7/200507 "2019-09-21T02:29:17Z")
Hello Dear, Hope you are well, I want to add MikroTik router all event log in to the ELK v7, already i added some Linux machine, They are working fine. Please give me some advice how to add MikroTik router in ELK v7. … --- ## [Comparing data from two different source](https://discuss.elastic.co/t/comparing-data-from-two-different-source/200616)
**Author:** [@Donny\_Nightfrost](https://discuss.elastic.co/u/Donny_Nightfrost)\ **Replies:** 2\ **Last updated:** [September 24, 2019, 9:51am UTC](https://discuss.elastic.co/t/comparing-data-from-two-different-source/200616 "2019-09-24T09:51:46Z")
Greetings, everyone. I have a trouble on comparing two data sources. What i want to do is check message field on "/var/log/auth.log" and see if it contains a word / string from "test" database table. Is this the right wa… --- ## [Time aggregation error](https://discuss.elastic.co/t/time-aggregation-error/200829)
**Author:** [@NewToThis](https://discuss.elastic.co/u/NewToThis)\ **Replies:** 0\ **Last updated:** [September 24, 2019, 9:19am UTC](https://discuss.elastic.co/t/time-aggregation-error/200829 "2019-09-24T09:19:33Z")
Hey people, I running a time aggregation where I am adding a run time of a process to @timestamp. @timestamp format is YYYY-MM-dd'T'HH:MM:ss.SSS'Z' I add the run time as a float in seconds to @timestamp and get a 'sol… --- ## [Extracting fields from path and adding them to new elapsed events](https://discuss.elastic.co/t/extracting-fields-from-path-and-adding-them-to-new-elapsed-events/200820)
**Author:** [@Alisa\_Faingold](https://discuss.elastic.co/u/Alisa_Faingold)\ **Replies:** 0\ **Last updated:** [September 24, 2019, 8:32am UTC](https://discuss.elastic.co/t/extracting-fields-from-path-and-adding-them-to-new-elapsed-events/200820 "2019-09-24T08:32:56Z")
Hi all, I have files stored as follows: c: /..../ agency / device / log.txt I want to extract the fields of agency and the device and add them to each event in the log. Now I'm doing this using a grok filter, and it… --- ## [Fails to index due to conflicting doc type](https://discuss.elastic.co/t/fails-to-index-due-to-conflicting-doc-type/200748)
**Author:** [@stefws](https://discuss.elastic.co/u/stefws)\ **Replies:** 8\ **Last updated:** [September 24, 2019, 6:57am UTC](https://discuss.elastic.co/t/fails-to-index-due-to-conflicting-doc-type/200748 "2019-09-24T06:57:28Z")
Got the classic issue with a 6.8.3 cluster, I defined an index template and created it with ?include\_type\_name=false { "index\_patterns": \[ "esxi\_firewall-\*" \], "mappings": { "properties": { "@timestamp": {… --- ## [Logstash behaves like a portscanner](https://discuss.elastic.co/t/logstash-behaves-like-a-portscanner/200797)
**Author:** [@hoggle](https://discuss.elastic.co/u/hoggle)\ **Replies:** 0\ **Last updated:** [September 24, 2019, 6:39am UTC](https://discuss.elastic.co/t/logstash-behaves-like-a-portscanner/200797 "2019-09-24T06:39:33Z")
Hi there, our firewall monitoring has identified the logstash process behaving very weird: The java process connects to another machine on about 230 different TCP Ports. So basically it behaves like a portscanner. The… --- ## [Logstash fingerprint not able to remove duplicate](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663)
**Author:** [@sharry007](https://discuss.elastic.co/u/sharry007)\ **Replies:** 7\ **Last updated:** [September 24, 2019, 5:45am UTC](https://discuss.elastic.co/t/logstash-fingerprint-not-able-to-remove-duplicate/200663 "2019-09-24T05:45:24Z")
Hi, We have been using logstash fingerprint plugin since quite sometime. We now have a specific requirement of creating fingerprint hashkey using multiple fields of Elasticsearch indexes, but we are stuck in two cases a… --- ## [Not able to uninstall/remove LS plugin](https://discuss.elastic.co/t/not-able-to-uninstall-remove-ls-plugin/200592)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 4\ **Last updated:** [September 23, 2019, 8:28am UTC](https://discuss.elastic.co/t/not-able-to-uninstall-remove-ls-plugin/200592 "2019-09-23T08:28:04Z")
Not able to uninstall/remove LS plugin This is what i am using bin\>logstash-plugin.bat remove logstash-filter-csv I am using windows package This issue is mentioned here as well. But not sure how to fix this in windo… --- ## [Grok Filter Pattern Hostname OR IP](https://discuss.elastic.co/t/grok-filter-pattern-hostname-or-ip/200768)
**Author:** [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Replies:** 2\ **Last updated:** [September 24, 2019, 12:40am UTC](https://discuss.elastic.co/t/grok-filter-pattern-hostname-or-ip/200768 "2019-09-24T00:40:50Z")
Newb question. Parsing a Cisco ASA log. I've noticed some of my logs don't match the built in parsers. Here is one in particular, 305011. This is the default pattern. CISCOFW305011 %{CISCO\_ACTION:action} %{CISCO\_XLATE… --- ## [Logstash Kafka Input backpressure from Persistent Queue](https://discuss.elastic.co/t/logstash-kafka-input-backpressure-from-persistent-queue/200766)
**Author:** [@Suryatej\_Mukkamalla](https://discuss.elastic.co/u/Suryatej_Mukkamalla)\ **Replies:** 2\ **Last updated:** [September 23, 2019, 11:48pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-backpressure-from-persistent-queue/200766 "2019-09-23T23:48:47Z")
This is my current ELK setup: Logstash (7.2.0): Input: Kafka Queue: Persistent Queue (16 GB) Output: Elasticsearch Elasticsearch (7.2.0) I am currently load testing Elasticsearch (trying to insert a billion documen… --- ## [Make Json flattening more dynamic to process nested json arrays/objects](https://discuss.elastic.co/t/make-json-flattening-more-dynamic-to-process-nested-json-arrays-objects/200760)
**Author:** [@baselai](https://discuss.elastic.co/u/baselai)\ **Replies:** 2\ **Last updated:** [September 23, 2019, 11:09pm UTC](https://discuss.elastic.co/t/make-json-flattening-more-dynamic-to-process-nested-json-arrays-objects/200760 "2019-09-23T23:09:17Z")
Hi, I'm trying to flatten a complex json file, that contains a nested - multilevel - arrays/objects, something like this: { "body":{ "reg\_dicts":\[ { "sectionsA":\[ … --- ## [Update nested object in ElasticSearch](https://discuss.elastic.co/t/update-nested-object-in-elasticsearch/200751)
**Author:** [@MMH](https://discuss.elastic.co/u/MMH)\ **Replies:** 3\ **Last updated:** [September 23, 2019, 9:15pm UTC](https://discuss.elastic.co/t/update-nested-object-in-elasticsearch/200751 "2019-09-23T21:15:09Z")
In one of my pipelines I am using logstash with JSON input and output to Elasticsearch with action =\> "update". One of the document fields contains nested data type. If in input JSON I receive following: "nested\_field… --- ## [Log pattern for AWS EMR logs](https://discuss.elastic.co/t/log-pattern-for-aws-emr-logs/200756)
**Author:** [@sk545](https://discuss.elastic.co/u/sk545)\ **Replies:** 0\ **Last updated:** [September 23, 2019, 8:39pm UTC](https://discuss.elastic.co/t/log-pattern-for-aws-emr-logs/200756 "2019-09-23T20:39:14Z")
Can anyone please provide the example of log stash's AWS S3 configuration settings.? --- ## [Opaque Logstash error when setting up SSL](https://discuss.elastic.co/t/opaque-logstash-error-when-setting-up-ssl/200537)
**Author:** [@tao](https://discuss.elastic.co/u/tao)\ **Replies:** 2\ **Last updated:** [September 23, 2019, 7:52pm UTC](https://discuss.elastic.co/t/opaque-logstash-error-when-setting-up-ssl/200537 "2019-09-23T19:52:53Z")
I'm attempting to set up Logstash to communicate with ES over SSL. It's failing with a very opaque error -- no error message; only a stack trace: \[2019-09-21T18:51:12,494\]\[INFO \]\[logstash.runner \] Starting Logs… --- ## [Grok pattern does not work in logstash however it works in kibana](https://discuss.elastic.co/t/grok-pattern-does-not-work-in-logstash-however-it-works-in-kibana/200148)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 11\ **Last updated:** [September 23, 2019, 7:12pm UTC](https://discuss.elastic.co/t/grok-pattern-does-not-work-in-logstash-however-it-works-in-kibana/200148 "2019-09-23T19:12:56Z")
Hello, I have a long used grok pattern match that stopped working since upgraded to ELK 7.3.0. grok { match =\> { "message" =\> \["%{MONTH:month} %{MONTHDAY:day}, %{YEAR:year} %{TIME:time} %{WORD:day\_period} %{NOTSPACE… --- ## [Problem with http output](https://discuss.elastic.co/t/problem-with-http-output/200396)
**Author:** [@ovie](https://discuss.elastic.co/u/ovie)\ **Replies:** 1\ **Last updated:** [September 23, 2019, 5:40pm UTC](https://discuss.elastic.co/t/problem-with-http-output/200396 "2019-09-23T17:40:43Z")
Hello, I'm facing a problem with sending Input from SQL server to Power BI input { jdbc { jdbc\_connection\_string =\> "jdbc:sqlserver://db.local:1433;database=test" jdbc\_user =\> "test" jdbc\_password =\> "t… --- ## [How can I get a notification of when my elasticsearch output fails to index an event?](https://discuss.elastic.co/t/how-can-i-get-a-notification-of-when-my-elasticsearch-output-fails-to-index-an-event/200727)
**Author:** [@brandoncordell](https://discuss.elastic.co/u/brandoncordell)\ **Replies:** 1\ **Last updated:** [September 23, 2019, 4:34pm UTC](https://discuss.elastic.co/t/how-can-i-get-a-notification-of-when-my-elasticsearch-output-fails-to-index-an-event/200727 "2019-09-23T16:34:30Z")
We have some external data that we do not control that gets compiled and inserted into a SQL Server table. I wrote a logstash pipeline to query that table and process the data for insertion into our ElasticSearch index. … --- ## [S3 input plugin: since\_db doesn't work properly: high CPU usage of logstash when more files in folder](https://discuss.elastic.co/t/s3-input-plugin-since-db-doesnt-work-properly-high-cpu-usage-of-logstash-when-more-files-in-folder/200717)
**Author:** [@Pista](https://discuss.elastic.co/u/Pista)\ **Replies:** 0\ **Last updated:** [September 23, 2019, 3:19pm UTC](https://discuss.elastic.co/t/s3-input-plugin-since-db-doesnt-work-properly-high-cpu-usage-of-logstash-when-more-files-in-folder/200717 "2019-09-23T15:19:22Z")
Issue: CPU Usage too high when thousands of files in AWS S3 log folder (s3 input plugin in use) ELK/Logstash setup AWS S3 bucket is scanned by S3 input plugin and it sends data to Logstash Issue description: Seems t… --- ## [Errors in logstash 7.3.2](https://discuss.elastic.co/t/errors-in-logstash-7-3-2/200609)
**Author:** [@sk545](https://discuss.elastic.co/u/sk545)\ **Replies:** 2\ **Last updated:** [September 23, 2019, 4:07pm UTC](https://discuss.elastic.co/t/errors-in-logstash-7-3-2/200609 "2019-09-23T16:07:31Z")
\[2019-09-23T00:02:55,264\]\[ERROR\]\[logstash.licensechecker.licensereader\] Unable to retrieve license information from license server {:message=\>"No Available connections"} \[2019-09-23T00:02:57,038\]\[WARN \]\[logstash.outputs… --- ## [Issue reading text file with grok](https://discuss.elastic.co/t/issue-reading-text-file-with-grok/200707)
**Author:** [@Hyperion0](https://discuss.elastic.co/u/Hyperion0)\ **Replies:** 0\ **Last updated:** [September 23, 2019, 2:33pm UTC](https://discuss.elastic.co/t/issue-reading-text-file-with-grok/200707 "2019-09-23T14:33:16Z")
I'm very new to logstash and even newer to the idea of grok statements. I have a config file that I've written to use a grok statement to read in a text file. I'm using custom statements which I tested using the grok deb… --- ## [Unable to start pipeline after SSL implementation](https://discuss.elastic.co/t/unable-to-start-pipeline-after-ssl-implementation/200724)
**Author:** [@gdamico](https://discuss.elastic.co/u/gdamico)\ **Replies:** 0\ **Last updated:** [September 23, 2019, 3:47pm UTC](https://discuss.elastic.co/t/unable-to-start-pipeline-after-ssl-implementation/200724 "2019-09-23T15:47:09Z")
Hi, I'm trying to enable SSL on my demo logstash installation, but after applying the documented settings the demo pipeline is unable to start, because of an certificate format error (that´s what log says). I already r… --- ## [Using \[log\]\[file\]\[path\] in mail output for logstash](https://discuss.elastic.co/t/using-log-file-path-in-mail-output-for-logstash/200692)
**Author:** [@AndriesN](https://discuss.elastic.co/u/AndriesN)\ **Replies:** 0\ **Last updated:** [September 23, 2019, 12:37pm UTC](https://discuss.elastic.co/t/using-log-file-path-in-mail-output-for-logstash/200692 "2019-09-23T12:37:16Z")
Hi guys, I'm using Filebeat, Logstash and Elasticsearch and I'm sending mail when a certain log file has passed a condition. Now, I would like to add the path of the log file to my mail so that the recipient knows whe… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=471) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=473)