# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=154 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 155 --- ## [Reference a value from new added field within grok match setting](https://discuss.elastic.co/t/reference-a-value-from-new-added-field-within-grok-match-setting/299619)
**Author:** [@mstojanovic](https://discuss.elastic.co/u/mstojanovic)\ **Replies:** 0\ **Last updated:** [March 14, 2022, 12:33pm UTC](https://discuss.elastic.co/t/reference-a-value-from-new-added-field-within-grok-match-setting/299619 "2022-03-14T12:33:59Z")
Hi, I'm trying to reference a value from a new field that was added with mutate plugin. I try to reference it within a grok match settings. Part of the filter pipeline looks like: filter { mutate { add\_… --- ## [HTTP Output Could not fetch url (Illegal character)](https://discuss.elastic.co/t/http-output-could-not-fetch-url-illegal-character/299439)
**Author:** [@josemariafrupm](https://discuss.elastic.co/u/josemariafrupm)\ **Replies:** 2\ **Last updated:** [March 14, 2022, 8:23am UTC](https://discuss.elastic.co/t/http-output-could-not-fetch-url-illegal-character/299439 "2022-03-14T08:23:58Z")
Hi! I'm an university student working on his final degree project, and I can't find any info about the problem I'm having. I need to send logs through Logstash with http but it says it can't fetch the url. I think the pr… --- ## [Is it possible to update Elasticsearch using logstash conf file with field other than document id](https://discuss.elastic.co/t/is-it-possible-to-update-elasticsearch-using-logstash-conf-file-with-field-other-than-document-id/299203)
**Author:** [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Replies:** 1\ **Last updated:** [March 14, 2022, 3:52am UTC](https://discuss.elastic.co/t/is-it-possible-to-update-elasticsearch-using-logstash-conf-file-with-field-other-than-document-id/299203 "2022-03-14T03:52:21Z")
Hello if\[sid\]!="null" and \[sid\]!="" { elasticsearch { hosts =\> \["localhost:9200"\] document\_id =\> "%{sid}" index =\> "dashboard\_write" … --- ## [Logstash output using environment variable](https://discuss.elastic.co/t/logstash-output-using-environment-variable/299569)
**Author:** [@Amit\_Halle](https://discuss.elastic.co/u/Amit_Halle)\ **Replies:** 1\ **Last updated:** [March 13, 2022, 4:23pm UTC](https://discuss.elastic.co/t/logstash-output-using-environment-variable/299569 "2022-03-13T16:23:02Z")
hi I'm trying to send my logs to an additional output when a certain field's value is a member in a given list. The way I go about this is wrapping the additional output part in the pipeline with an "if" condition and … --- ## [Records are getting dropped while creating map using logstash](https://discuss.elastic.co/t/records-are-getting-dropped-while-creating-map-using-logstash/299509)
**Author:** [@Deepak\_Dandotiya](https://discuss.elastic.co/u/Deepak_Dandotiya)\ **Replies:** 1\ **Last updated:** [March 12, 2022, 3:26pm UTC](https://discuss.elastic.co/t/records-are-getting-dropped-while-creating-map-using-logstash/299509 "2022-03-12T15:26:42Z")
Hi Experts, I need your help here. Context is, I am trying to read the data from DB after multiple joins and then forming a Map in Elasticsearch using ID column as agreegate and trying to handle the duplicate event from… --- ## [Logstash filter loglevel info warn and error](https://discuss.elastic.co/t/logstash-filter-loglevel-info-warn-and-error/299306)
**Author:** [@waelboss](https://discuss.elastic.co/u/waelboss)\ **Replies:** 7\ **Last updated:** [March 12, 2022, 9:23am UTC](https://discuss.elastic.co/t/logstash-filter-loglevel-info-warn-and-error/299306 "2022-03-12T09:23:05Z")
i have this JSON data: "result": \[ { "payload": { "context": "default", "level": "DEBUG", "logger": "org.forgerock.opendj.ldap.CachedConnectionPool", "me… --- ## [Need help with a NOT in if statement, is it "not" or "!"?](https://discuss.elastic.co/t/need-help-with-a-not-in-if-statement-is-it-not-or/299486)
**Author:** [@teebu](https://discuss.elastic.co/u/teebu)\ **Replies:** 1\ **Last updated:** [March 11, 2022, 6:58pm UTC](https://discuss.elastic.co/t/need-help-with-a-not-in-if-statement-is-it-not-or/299486 "2022-03-11T18:58:41Z")
\# filter out local ips if !(\[http\]\[request\]\[headers\]\[CF-Connecting-IP\] =~ "^10.0.\*" or \[http\]\[request\]\[headers\]\[CF-Connecting-IP\] =~ "^127.0.\*" or \[http\]\[request\]\[headers\]\[CF-Connecting-IP\] == "0.0.0.0") … --- ## [Installed logstash rpm, service keeps stopping with no clear reason](https://discuss.elastic.co/t/installed-logstash-rpm-service-keeps-stopping-with-no-clear-reason/299446)
**Author:** [@jelmew](https://discuss.elastic.co/u/jelmew)\ **Replies:** 2\ **Last updated:** [March 11, 2022, 4:56pm UTC](https://discuss.elastic.co/t/installed-logstash-rpm-service-keeps-stopping-with-no-clear-reason/299446 "2022-03-11T16:56:28Z")
Hi all, We are trying to setup logstash to connect to elastic cloud. We downloaded the rpm and installed it. But even without any configuration pointing to our elastic cloud instance, the service just stops with the mes… --- ## [Grok pattern for stashing Ansible logs](https://discuss.elastic.co/t/grok-pattern-for-stashing-ansible-logs/299227)
**Author:** [@santy1](https://discuss.elastic.co/u/santy1)\ **Replies:** 6\ **Last updated:** [March 11, 2022, 4:06pm UTC](https://discuss.elastic.co/t/grok-pattern-for-stashing-ansible-logs/299227 "2022-03-11T16:06:57Z")
Hello, I am new to Elasticsearch. I am on Elasticsearch 7.17.1. Below are the logs that I am trying to parse. 2022-03-08 14:52:50,672 p=1654827 u=ansible n=ansible | TASK \[lvm : Formatting xfs filesystem\] \*\* 2022-03-… --- ## [Logstash crashes when kafka producer sends less fields than excepted](https://discuss.elastic.co/t/logstash-crashes-when-kafka-producer-sends-less-fields-than-excepted/299475)
**Author:** [@guriv](https://discuss.elastic.co/u/guriv)\ **Replies:** 0\ **Last updated:** [March 11, 2022, 3:48pm UTC](https://discuss.elastic.co/t/logstash-crashes-when-kafka-producer-sends-less-fields-than-excepted/299475 "2022-03-11T15:48:15Z")
Hello we are using logstash-kafka-integration plugin to read messages from a kafka topic using avro encoding If my logstash consumer refers to a schema registry file that defines 2 optional fields but receive a message… --- ## [Create new file every 60seconds in Logstash CSV output](https://discuss.elastic.co/t/create-new-file-every-60seconds-in-logstash-csv-output/299380)
**Author:** [@gnam](https://discuss.elastic.co/u/gnam)\ **Replies:** 2\ **Last updated:** [March 11, 2022, 2:01pm UTC](https://discuss.elastic.co/t/create-new-file-every-60seconds-in-logstash-csv-output/299380 "2022-03-11T14:01:31Z")
I have following filter configuration in my logstatsh pipeline. What it does is, at the start of the event first filter creates a CSV file with header and sets the file name to metadata. Second filter writes the output … --- ## [Hadoop env Hive Connectivity through logstash](https://discuss.elastic.co/t/hadoop-env-hive-connectivity-through-logstash/299452)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 0\ **Last updated:** [March 11, 2022, 1:02pm UTC](https://discuss.elastic.co/t/hadoop-env-hive-connectivity-through-logstash/299452 "2022-03-11T13:02:37Z")
Is it possible to connect to Hadoop hive environment using hive jdbc driver, through logstash jdbc connector? --- ## [Connecting to the Teradata Data Source through logstash](https://discuss.elastic.co/t/connecting-to-the-teradata-data-source-through-logstash/299420)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 0\ **Last updated:** [March 11, 2022, 9:22am UTC](https://discuss.elastic.co/t/connecting-to-the-teradata-data-source-through-logstash/299420 "2022-03-11T09:22:07Z")
Is it possible to use the logstash JDBC connector to connect to the Teradata data platform? --- ## [Can't run Logstash as a Service](https://discuss.elastic.co/t/cant-run-logstash-as-a-service/299202)
**Author:** [@Simone1](https://discuss.elastic.co/u/Simone1)\ **Replies:** 26\ **Last updated:** [March 11, 2022, 8:06am UTC](https://discuss.elastic.co/t/cant-run-logstash-as-a-service/299202 "2022-03-11T08:06:57Z")
If i run logstash manually it works fine and sends all data in the right way. But when i try to run it as a service it shows this error: Cmd that i use: sudo systemctl start logstash.service OS Ubuntu 20.04.1 Log… --- ## [Want to disable Logstash http://localhost:9600/ URL JSON output](https://discuss.elastic.co/t/want-to-disable-logstash-http-localhost-9600-url-json-output/299352)
**Author:** [@shahidraza](https://discuss.elastic.co/u/shahidraza)\ **Replies:** 6\ **Last updated:** [March 11, 2022, 5:51am UTC](https://discuss.elastic.co/t/want-to-disable-logstash-http-localhost-9600-url-json-output/299352 "2022-03-11T05:51:51Z")
Hi Team, We don't want to see JSON output as it contains Hostname and other sensitive information and it's accessible without user id and password too. Please share details to disable this so that security vulnerabilit… --- ## [Logstash Helm install with jdbc connection to MySQL/Mariadb](https://discuss.elastic.co/t/logstash-helm-install-with-jdbc-connection-to-mysql-mariadb/299396)
**Author:** [@Matthew\_Ong](https://discuss.elastic.co/u/Matthew_Ong)\ **Replies:** 0\ **Last updated:** [March 11, 2022, 4:53am UTC](https://discuss.elastic.co/t/logstash-helm-install-with-jdbc-connection-to-mysql-mariadb/299396 "2022-03-11T04:53:51Z")
Hi, I am new to logstash(2 weeks). Please help me with these questions. Thanks very much in advance. How to perform JDBC connection to mysql with helm chart installation and how to upload the jdbc driver after the inst… --- ## [Renaming multiple fields that are nested](https://discuss.elastic.co/t/renaming-multiple-fields-that-are-nested/299376)
**Author:** [@itschobot](https://discuss.elastic.co/u/itschobot)\ **Replies:** 3\ **Last updated:** [March 11, 2022, 12:05am UTC](https://discuss.elastic.co/t/renaming-multiple-fields-that-are-nested/299376 "2022-03-11T00:05:25Z")
I need help renaming multiple fields that are nested, here is my problem, I can only rename "did" within example.devices but when I attempt to rename others I would assume it's the same pattern but the results are what i… --- ## [Unable to get terminal for logstash 8.0.1](https://discuss.elastic.co/t/unable-to-get-terminal-for-logstash-8-0-1/299370)
**Author:** [@uma\_rengasamy](https://discuss.elastic.co/u/uma_rengasamy)\ **Replies:** 0\ **Last updated:** [March 10, 2022, 9:38pm UTC](https://discuss.elastic.co/t/unable-to-get-terminal-for-logstash-8-0-1/299370 "2022-03-10T21:38:21Z")
Getting below error OCI runtime exec failed: exec failed: container\_linux.go:380: starting container process caused: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown --- ## [Prune filter does not work](https://discuss.elastic.co/t/prune-filter-does-not-work/299274)
**Author:** [@Juan\_David\_Jaramillo](https://discuss.elastic.co/u/Juan_David_Jaramillo)\ **Replies:** 6\ **Last updated:** [March 10, 2022, 8:15pm UTC](https://discuss.elastic.co/t/prune-filter-does-not-work/299274 "2022-03-10T20:15:00Z")
good morning, hope you are well I have a problem with the "prune" filter in logstash and I have a configuration file in logstash in which the objective is to take the data from filebeat-netflow and go through the pipe i… --- ## [Encountered a retryable error. Will Retry with exponential backoff {:code=\>500,](https://discuss.elastic.co/t/encountered-a-retryable-error-will-retry-with-exponential-backoff-code-500/299138)
**Author:** [@venky123](https://discuss.elastic.co/u/venky123)\ **Replies:** 4\ **Last updated:** [March 10, 2022, 7:26pm UTC](https://discuss.elastic.co/t/encountered-a-retryable-error-will-retry-with-exponential-backoff-code-500/299138 "2022-03-10T19:26:57Z")
Continuing the discussion from Encountered a retryable error. Will Retry with exponential backoff code=\>413: \[2022-03-07T02:18:15,991\]\[ERROR\]\[logstash.outputs.Elasticsearch\] Encountered a retryable error. Will Retry wit… --- ## [Query Separate Pipeline for Logstash SNMP Input Hosts](https://discuss.elastic.co/t/query-separate-pipeline-for-logstash-snmp-input-hosts/299355)
**Author:** [@bennrtc](https://discuss.elastic.co/u/bennrtc)\ **Replies:** 1\ **Last updated:** [March 10, 2022, 4:29pm UTC](https://discuss.elastic.co/t/query-separate-pipeline-for-logstash-snmp-input-hosts/299355 "2022-03-10T16:29:27Z")
Hello! I'm using the Logstash SNMP Input to query a large number of hosts (\> 5,000). The list of host ip addresses changes frequently and requires us to regularly add and remove hosts to ensure our pipeline output is u… --- ## [Logstash Failed to Parse Date (6 milliseconds)](https://discuss.elastic.co/t/logstash-failed-to-parse-date-6-milliseconds/299271)
**Author:** [@userR](https://discuss.elastic.co/u/userR)\ **Replies:** 4\ **Last updated:** [March 10, 2022, 4:23pm UTC](https://discuss.elastic.co/t/logstash-failed-to-parse-date-6-milliseconds/299271 "2022-03-10T16:23:13Z")
"reason"=\>"failed to parse date field \[Sun Feb 20 03:36:11.782065 2022\] with format \[strict\_date\_optional\_time||epoch\_millis\]", "caused\_by"=\>{"type"=\>"date\_time\_parse\_exception", "reason"=\>"Failed to parse with all enclo… --- ## [Facing issue in logstash after applying basic authentication on ES](https://discuss.elastic.co/t/facing-issue-in-logstash-after-applying-basic-authentication-on-es/299308)
**Author:** [@Sandeep\_Thakur](https://discuss.elastic.co/u/Sandeep_Thakur)\ **Replies:** 1\ **Last updated:** [March 10, 2022, 3:51pm UTC](https://discuss.elastic.co/t/facing-issue-in-logstash-after-applying-basic-authentication-on-es/299308 "2022-03-10T15:51:35Z")
Hi Team, We are trying to applying basic authentication on ELk stack, when we enable security true in Elasticsearch.yml then logstash stopping pushing log on ES and give following error \[2022-03-10T08:39:43,365\]\[WARN \]… --- ## [ILM error killing all indexes](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993)
**Author:** [@cstone1492](https://discuss.elastic.co/u/cstone1492)\ **Replies:** 9\ **Last updated:** [March 10, 2022, 3:06pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993 "2022-03-10T15:06:59Z")
The es instance I'm working with crashed, and attempting to restart the kernel leads to the run command immediately being killed (runES.sh: line 1: 92654 Killed ./bin/Elasticsearch). The log shows this … --- ## [Ruby vs translate plugin performance](https://discuss.elastic.co/t/ruby-vs-translate-plugin-performance/299305)
**Author:** [@skhadraoui](https://discuss.elastic.co/u/skhadraoui)\ **Replies:** 0\ **Last updated:** [March 10, 2022, 8:21am UTC](https://discuss.elastic.co/t/ruby-vs-translate-plugin-performance/299305 "2022-03-10T08:21:22Z")
Hi folks, does anyone have experience regarding the performance of translate/plugins vs. Ruby scripts (https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html ) and can share some insights on that. I … --- ## [Curl ssl error from Logstash Docker](https://discuss.elastic.co/t/curl-ssl-error-from-logstash-docker/299301)
**Author:** [@paspao](https://discuss.elastic.co/u/paspao)\ **Replies:** 0\ **Last updated:** [March 10, 2022, 7:59am UTC](https://discuss.elastic.co/t/curl-ssl-error-from-logstash-docker/299301 "2022-03-10T07:59:26Z")
Hello, I am doing test with official ELK Docker images. I have a cluster setup with Docker compose working I start a Logstash Docker then enter in it's shell to test if it can connect to cluster so I run a : curl -k … --- ## [Bitnami Logstash Image 7.16.3 Reported by Azure Defender Still Affected by CVE-2021-44832](https://discuss.elastic.co/t/bitnami-logstash-image-7-16-3-reported-by-azure-defender-still-affected-by-cve-2021-44832/299295)
**Author:** [@chferng](https://discuss.elastic.co/u/chferng)\ **Replies:** 0\ **Last updated:** [March 10, 2022, 5:51am UTC](https://discuss.elastic.co/t/bitnami-logstash-image-7-16-3-reported-by-azure-defender-still-affected-by-cve-2021-44832/299295 "2022-03-10T05:51:16Z")
We have deployed the Bitnami Logstash image based on Logstash 7.16.3 in our environment on Azure AKS, as a remediation to the CVE-2021-44832 log4j vulnerability. After redeploying Logstash, we noticed the Azure Defender … --- ## [Where does logstash plugins get installed inside the logstash?](https://discuss.elastic.co/t/where-does-logstash-plugins-get-installed-inside-the-logstash/299290)
**Author:** [@Nikhitha](https://discuss.elastic.co/u/Nikhitha)\ **Replies:** 0\ **Last updated:** [March 10, 2022, 4:49am UTC](https://discuss.elastic.co/t/where-does-logstash-plugins-get-installed-inside-the-logstash/299290 "2022-03-10T04:49:35Z")
Currently since TLS 1.3 is not supporting for logstash input beats plugin, I am trying to modify the beats plugin source code. In this regard , to replace the original installed beats plugin tar file, I would like to rep… --- ## [Does Logstash8.0 support modern authentication to access Outlook with token?](https://discuss.elastic.co/t/does-logstash8-0-support-modern-authentication-to-access-outlook-with-token/299278)
**Author:** [@Albert\_Jia](https://discuss.elastic.co/u/Albert_Jia)\ **Replies:** 0\ **Last updated:** [March 10, 2022, 2:35am UTC](https://discuss.elastic.co/t/does-logstash8-0-support-modern-authentication-to-access-outlook-with-token/299278 "2022-03-10T02:35:44Z")
input { imap { host =\> "outlook.office365.com" user =\> "$USER" password =\> "$PASSWORD" port =\> $PORT folder =\> "Inbox" } } Currently I'm using username/password way to pul… --- ## [Some index rollover\_alias are not updating](https://discuss.elastic.co/t/some-index-rollover-alias-are-not-updating/299194)
**Author:** [@V\_Steeph](https://discuss.elastic.co/u/V_Steeph)\ **Replies:** 0\ **Last updated:** [March 9, 2022, 9:56am UTC](https://discuss.elastic.co/t/some-index-rollover-alias-are-not-updating/299194 "2022-03-09T09:56:39Z")
Hello, We are currently using ELK to centralize every log of our system. There are lots of elements and everything is retrieved through Logstash with one big config file. Each element has the same structure : else if… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=153) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=155)