# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=195 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 196 --- ## [Logstash refuses to bind to port 9600](https://discuss.elastic.co/t/logstash-refuses-to-bind-to-port-9600/284183)
**Author:** [@Dan\_Sputnikk](https://discuss.elastic.co/u/Dan_Sputnikk)\ **Replies:** 3\ **Last updated:** [September 14, 2021, 2:04pm UTC](https://discuss.elastic.co/t/logstash-refuses-to-bind-to-port-9600/284183 "2021-09-14T14:04:20Z")
Hi all, Using Logstash 6. I have: http.host: "0.0.0.0" http.port: 9600-9700 xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.hosts: \["http://10.1.96.89:9200", "http://10.1.96.90:9200", "http://10.1.96.9… --- ## [Need help with conditional only if field contains a period](https://discuss.elastic.co/t/need-help-with-conditional-only-if-field-contains-a-period/284126)
**Author:** [@pcharles1](https://discuss.elastic.co/u/pcharles1)\ **Replies:** 2\ **Last updated:** [September 14, 2021, 1:04pm UTC](https://discuss.elastic.co/t/need-help-with-conditional-only-if-field-contains-a-period/284126 "2021-09-14T13:04:44Z")
I'm trying to grok the host name field only if it contains a period using the conditional below: if (\[host\]\[name\] =~ ".") { grok host.name field } For some reason, when the host name field doesn't contain a period, I g… --- ## [Change event name based on type](https://discuss.elastic.co/t/change-event-name-based-on-type/284129)
**Author:** [@antonisnyc94](https://discuss.elastic.co/u/antonisnyc94)\ **Replies:** 1\ **Last updated:** [September 14, 2021, 12:48pm UTC](https://discuss.elastic.co/t/change-event-name-based-on-type/284129 "2021-09-14T12:48:30Z")
Hello, I am having a field which is an object but sometimes it's a string. I'm using the below ruby code but for some reason im still getting an error.. Does anyone know why? if event.get("\[data\]\[aws\]\[configuration\]\[s… --- ## [Logstash kafka input plugin mutate problem](https://discuss.elastic.co/t/logstash-kafka-input-plugin-mutate-problem/284156)
**Author:** [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Replies:** 1\ **Last updated:** [September 14, 2021, 10:15am UTC](https://discuss.elastic.co/t/logstash-kafka-input-plugin-mutate-problem/284156 "2021-09-14T10:15:16Z")
Dears, I'm trying to load data from kafka topic but have some problem with if statement in filter. This part of the setup doesn't work and I don't know why: replace =\> { "\[@metadata\]\[index\_prefix\]" =\> "kafka-%{+YYYY.M… --- ## [Logstash s3 output plugin not working properly](https://discuss.elastic.co/t/logstash-s3-output-plugin-not-working-properly/284159)
**Author:** [@vishakha.tyagi.blaze](https://discuss.elastic.co/u/vishakha.tyagi.blaze)\ **Replies:** 0\ **Last updated:** [September 14, 2021, 8:53am UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-not-working-properly/284159 "2021-09-14T08:53:24Z")
So we are trying to use Logstash to send out some logs to s3 and its not working for some reason. This is the code we are using. output { amazon\_es { hosts =\> \["xxxx"\] … --- ## [Problem installing ruby gem active\_support](https://discuss.elastic.co/t/problem-installing-ruby-gem-active-support/283466)
**Author:** [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\ **Replies:** 1\ **Last updated:** [September 14, 2021, 8:33am UTC](https://discuss.elastic.co/t/problem-installing-ruby-gem-active-support/283466 "2021-09-14T08:33:57Z")
Hi all, Some specs: Logstash 7.13.1 Ruby 2.5.0 activesupport 6.1.4.1 I'd like to use some active\_support functionality in my ruby filters that I list in my logstash config files. I'm having a problem however with my… --- ## [Logstash persistent queue settings](https://discuss.elastic.co/t/logstash-persistent-queue-settings/284144)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 6\ **Last updated:** [September 14, 2021, 6:35am UTC](https://discuss.elastic.co/t/logstash-persistent-queue-settings/284144 "2021-09-14T06:35:32Z")
Using logstash 7.13 We are running logstash on linux server which has 4 cpu(s) 15gb of ram of 100gb of ssd. We are using persistent queue settings in logstash but we actually can't sure is this right configuration or c… --- ## [Logstash api endpoints are not accessible outside the logstash container](https://discuss.elastic.co/t/logstash-api-endpoints-are-not-accessible-outside-the-logstash-container/284135)
**Author:** [@vijayebpzr](https://discuss.elastic.co/u/vijayebpzr)\ **Replies:** 3\ **Last updated:** [September 14, 2021, 4:49am UTC](https://discuss.elastic.co/t/logstash-api-endpoints-are-not-accessible-outside-the-logstash-container/284135 "2021-09-14T04:49:49Z")
We deployed the ELK stack in kubernetes cluster and everything is working fine. Now we are trying to monitor logstash from kibana stack monitoring using metricbeat. We could see that logstash section is not visible in st… --- ## [How to prevent automatic deletion of indices](https://discuss.elastic.co/t/how-to-prevent-automatic-deletion-of-indices/283965)
**Author:** [@asan](https://discuss.elastic.co/u/asan)\ **Replies:** 4\ **Last updated:** [September 14, 2021, 4:45am UTC](https://discuss.elastic.co/t/how-to-prevent-automatic-deletion-of-indices/283965 "2021-09-14T04:45:32Z")
hi After checking history of my logs in Discovery i found out that my logs just stored for 2 weeks and after that history of my logs is deleted. i didn't set up any policy in Index Lifecycle Policies. how can i fix thi… --- ## [Parsing SQL queries using ruby; any recommended gem package?](https://discuss.elastic.co/t/parsing-sql-queries-using-ruby-any-recommended-gem-package/284132)
**Author:** [@SHINRA](https://discuss.elastic.co/u/SHINRA)\ **Replies:** 0\ **Last updated:** [September 14, 2021, 2:44am UTC](https://discuss.elastic.co/t/parsing-sql-queries-using-ruby-any-recommended-gem-package/284132 "2021-09-14T02:44:08Z")
Hi, I'm new to Ruby language and I thought I could still do a simple task of normalizing + parsing SQL query using some third party library; but I had no luck finding any that works to my intention. Here's what I'm try… --- ## [How to match lines in unstructured log starting with specific string](https://discuss.elastic.co/t/how-to-match-lines-in-unstructured-log-starting-with-specific-string/284112)
**Author:** [@ansamHox](https://discuss.elastic.co/u/ansamHox)\ **Replies:** 3\ **Last updated:** [September 13, 2021, 7:18pm UTC](https://discuss.elastic.co/t/how-to-match-lines-in-unstructured-log-starting-with-specific-string/284112 "2021-09-13T19:18:35Z")
I have a very big unstructured file. Firstly, I want to parse all lines that starts with ABC or CDE and store them as one document in Elasticsearch. One file should be one document in index, so @message should look like … --- ## [Exec command Linux and extract specific field](https://discuss.elastic.co/t/exec-command-linux-and-extract-specific-field/283254)
**Author:** [@Mira\_9](https://discuss.elastic.co/u/Mira_9)\ **Replies:** 2\ **Last updated:** [September 6, 2021, 2:13pm UTC](https://discuss.elastic.co/t/exec-command-linux-and-extract-specific-field/283254 "2021-09-06T14:13:17Z")
Hi Community, i'm executing several linux commands in Logstash. and i want to extract specific fields to build some graphs in Kibana. for Example this is the result of an executed command \[2021-02-13 19:28:49.200\] chan… --- ## [Is it possible to add the "number of replicas" in logstash code , rather than in index template?](https://discuss.elastic.co/t/is-it-possible-to-add-the-number-of-replicas-in-logstash-code-rather-than-in-index-template/284064)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 1\ **Last updated:** [September 13, 2021, 1:00pm UTC](https://discuss.elastic.co/t/is-it-possible-to-add-the-number-of-replicas-in-logstash-code-rather-than-in-index-template/284064 "2021-09-13T13:00:13Z")
Hello, I wanted to know if its possible to mention "number of replicas" in the logstash code , rather than in index template , so when i rollover the index , this is inherited by the newer index. Mine is a single node … --- ## [Can't connect to http-input of Logstash on Elastic Cloud](https://discuss.elastic.co/t/cant-connect-to-http-input-of-logstash-on-elastic-cloud/283822)
**Author:** [@Samuel\_Wehrli](https://discuss.elastic.co/u/Samuel_Wehrli)\ **Replies:** 5\ **Last updated:** [September 13, 2021, 12:44am UTC](https://discuss.elastic.co/t/cant-connect-to-http-input-of-logstash-on-elastic-cloud/283822 "2021-09-13T00:44:25Z")
Hi, i just deployed an elastic cloud instance on google cloud. I want to use the http input plugin of Logstash to receive sensor data (from "the things network") via webhook requests. I configured the Logstash pipeline o… --- ## [Pulling data from two different DB with Logstash JDBC Input and merging input events data](https://discuss.elastic.co/t/pulling-data-from-two-different-db-with-logstash-jdbc-input-and-merging-input-events-data/283973)
**Author:** [@e-fo](https://discuss.elastic.co/u/e-fo)\ **Replies:** 1\ **Last updated:** [September 12, 2021, 3:46pm UTC](https://discuss.elastic.co/t/pulling-data-from-two-different-db-with-logstash-jdbc-input-and-merging-input-events-data/283973 "2021-09-12T15:46:14Z")
Hi and thanks for your great community. I have two database that I want configuring a pipeline with Logstash that pulling data from these two database and merging the input events based on one same field in these events… --- ## [Logstash error to finding tracking\_column in jdbc input for mongodb](https://discuss.elastic.co/t/logstash-error-to-finding-tracking-column-in-jdbc-input-for-mongodb/283329)
**Author:** [@e-fo](https://discuss.elastic.co/u/e-fo)\ **Replies:** 1\ **Last updated:** [September 12, 2021, 9:35am UTC](https://discuss.elastic.co/t/logstash-error-to-finding-tracking-column-in-jdbc-input-for-mongodb/283329 "2021-09-12T09:35:44Z")
Hello, I using logstash jdbc for pulling data from mongodb and pushing to elasticsearch. I using this pipeline configuration for logstash: input { jdbc { type =\> "bb\_purchaselog" jdbc\_driver\_library =\> "/usr/… --- ## [Update nested array in existing document](https://discuss.elastic.co/t/update-nested-array-in-existing-document/283957)
**Author:** [@zeoiioze](https://discuss.elastic.co/u/zeoiioze)\ **Replies:** 0\ **Last updated:** [September 11, 2021, 8:08pm UTC](https://discuss.elastic.co/t/update-nested-array-in-existing-document/283957 "2021-09-11T20:08:01Z")
Hello ! I have documents with fields like that : doc.site containing a string like "paris" doc.ips.values containing a list of strings \["10.0.0.1", "10.0.0.2"\], or which can sometimes be empty \[\] doc.ips.last\_che… --- ## [Match error grok filter](https://discuss.elastic.co/t/match-error-grok-filter/283851)
**Author:** [@mar-ro](https://discuss.elastic.co/u/mar-ro)\ **Replies:** 3\ **Last updated:** [September 11, 2021, 3:24pm UTC](https://discuss.elastic.co/t/match-error-grok-filter/283851 "2021-09-11T15:24:43Z")
Good morning! I'm trying to set up a grok filter to a pipeline which receives the following text: 09/10/2021, 8:30:00 AM \[Message\] therefore the date format is: Month/Day/Year, Hour:Minutes:Seconds The grok filter lo… --- ## [How do I fix "Invalid FieldReference" errors?](https://discuss.elastic.co/t/how-do-i-fix-invalid-fieldreference-errors/283912)
**Author:** [@mrled](https://discuss.elastic.co/u/mrled)\ **Replies:** 5\ **Last updated:** [September 10, 2021, 11:18pm UTC](https://discuss.elastic.co/t/how-do-i-fix-invalid-fieldreference-errors/283912 "2021-09-10T23:18:38Z")
Logstash is reporting errors like this every second or so: \[2021-09-10T19:09:56,647\]\[INFO \]\[org.logstash.beats.BeatsHandler\]\[main\]\[98ee7b7f6aa150efeb06cd1c765e51dc0b1e1cd81424a2538524f239b81fa590\] \[local: 10.0.3.241:504… --- ## [Is there a way to connect logstash to LinkedIn?](https://discuss.elastic.co/t/is-there-a-way-to-connect-logstash-to-linkedin/283915)
**Author:** [@jcarney](https://discuss.elastic.co/u/jcarney)\ **Replies:** 0\ **Last updated:** [September 10, 2021, 7:46pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-connect-logstash-to-linkedin/283915 "2021-09-10T19:46:14Z")
This is what I have found so far: --- ## [Logstash elasticsearch output plugin Document\_id and Upserts](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-document-id-and-upserts/283909)
**Author:** [@pkward](https://discuss.elastic.co/u/pkward)\ **Replies:** 1\ **Last updated:** [September 10, 2021, 7:27pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-document-id-and-upserts/283909 "2021-09-10T19:27:51Z")
Question, does a customized document\_id require an upsert to update the document or can this be accomplished with just the document\_id? I've read that upserts are used if you want create a new document if the document d… --- ## [Logstash filter plugin](https://discuss.elastic.co/t/logstash-filter-plugin/283847)
**Author:** [@dorinand](https://discuss.elastic.co/u/dorinand)\ **Replies:** 1\ **Last updated:** [September 10, 2021, 2:12pm UTC](https://discuss.elastic.co/t/logstash-filter-plugin/283847 "2021-09-10T14:12:20Z")
I am using Logstash version 7.14.0. I would like to parse message to the top level of message. So this is my configuration: json { skip\_on\_invalid\_json =\> true source =\> "message" t… --- ## [How to do source to "multi target" translate in logstash?](https://discuss.elastic.co/t/how-to-do-source-to-multi-target-translate-in-logstash/283623)
**Author:** [@kelk](https://discuss.elastic.co/u/kelk)\ **Replies:** 2\ **Last updated:** [September 10, 2021, 10:23am UTC](https://discuss.elastic.co/t/how-to-do-source-to-multi-target-translate-in-logstash/283623 "2021-09-10T10:23:10Z")
We have got a CSV lookup as follows EmpID,EmpName,EmpSalary,EmpDepartment 1001,Bob,10000,XX 1002,Joe,20000,YY 1003,Foo,30000,ZZ and in Logstash, we need to translate the EmpID to both EmpName & EmpSalary at the same ti… --- ## [Logstash + Telegraf](https://discuss.elastic.co/t/logstash-telegraf/283860)
**Author:** [@hassen\_k](https://discuss.elastic.co/u/hassen_k)\ **Replies:** 0\ **Last updated:** [September 10, 2021, 10:10am UTC](https://discuss.elastic.co/t/logstash-telegraf/283860 "2021-09-10T10:10:23Z")
Hi, I use Logstash to read a Kafka topic and to send the events to Influxdb. The events in the Kafka topic are sent by Telegraf. Here are a example of events received from Kafka : { "message" =\> "disk,all\_serve… --- ## [Creating an array of values via ruby on logstash config](https://discuss.elastic.co/t/creating-an-array-of-values-via-ruby-on-logstash-config/283649)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 4\ **Last updated:** [September 10, 2021, 9:51am UTC](https://discuss.elastic.co/t/creating-an-array-of-values-via-ruby-on-logstash-config/283649 "2021-09-10T09:51:04Z")
Hello, Im trying to get the cpu core values in a single array on my logstash config . I'm not familiar with ruby code, however based on my very limited understanding i was able to do this . It has the cpu core values , … --- ## [Logstash 7.11 ruby filter error](https://discuss.elastic.co/t/logstash-7-11-ruby-filter-error/283844)
**Author:** [@shharukh](https://discuss.elastic.co/u/shharukh)\ **Replies:** 0\ **Last updated:** [September 10, 2021, 7:09am UTC](https://discuss.elastic.co/t/logstash-7-11-ruby-filter-error/283844 "2021-09-10T07:09:51Z")
Hello World, I wanted to add 5 hours in my datetime through logstash. have found a way in logstash 6.4. ruby { code =\> 'event.set("@timestamp", LogStash::Timestamp.new(Time.at(event.get("@timestamp").to\_f+86400)))' … --- ## [Unable to install composable template from the logstash.outputs.elasticsearch plugin](https://discuss.elastic.co/t/unable-to-install-composable-template-from-the-logstash-outputs-elasticsearch-plugin/283835)
**Author:** [@Usman18](https://discuss.elastic.co/u/Usman18)\ **Replies:** 0\ **Last updated:** [September 10, 2021, 1:39am UTC](https://discuss.elastic.co/t/unable-to-install-composable-template-from-the-logstash-outputs-elasticsearch-plugin/283835 "2021-09-10T01:39:45Z")
When I try to load a composable template from the logstash into the elasticsearch using the logstash output elasticsearch plugin, I get the error Failed to install template. {:message=\>"Got response code '400' contactin… --- ## [Working with key:value arrays](https://discuss.elastic.co/t/working-with-key-value-arrays/283772)
**Author:** [@awer1967](https://discuss.elastic.co/u/awer1967)\ **Replies:** 4\ **Last updated:** [September 10, 2021, 1:06am UTC](https://discuss.elastic.co/t/working-with-key-value-arrays/283772 "2021-09-10T01:06:04Z")
Good day ! Accordingly to my logstash configuration I get a message with an array consists of key:value pairs. I have to mutate it on a very special way adding two fields and put there the key and the value . For examp… --- ## [Logstash - Difference between date into new field](https://discuss.elastic.co/t/logstash-difference-between-date-into-new-field/283818)
**Author:** [@stemons](https://discuss.elastic.co/u/stemons)\ **Replies:** 6\ **Last updated:** [September 9, 2021, 10:02pm UTC](https://discuss.elastic.co/t/logstash-difference-between-date-into-new-field/283818 "2021-09-09T22:02:17Z")
Hello team, I'm trying to add a new field to metricbeat data collection through logstash. The idea is to create a field from the difference between two dates (@timestamp and system.process.cpu.start\_date). I've made so… --- ## [Questions regarding auto-reload and encryption](https://discuss.elastic.co/t/questions-regarding-auto-reload-and-encryption/283758)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 6\ **Last updated:** [September 9, 2021, 7:40pm UTC](https://discuss.elastic.co/t/questions-regarding-auto-reload-and-encryption/283758 "2021-09-09T19:40:23Z")
Apologies beforehand for the dilettante question. I have an up and running Logstash configuration. Now and then I edit my grok parser and then I always have to type systemctl restart logstash so that the latest config i… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=194) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=196)