# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=447 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 448 --- ## [ENvironmental variable hostname in logstash.yml and pipelines.yml or logstash.conf](https://discuss.elastic.co/t/environmental-variable-hostname-in-logstash-yml-and-pipelines-yml-or-logstash-conf/207508)
**Author:** [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Replies:** 1\ **Last updated:** [November 13, 2019, 11:50am UTC](https://discuss.elastic.co/t/environmental-variable-hostname-in-logstash-yml-and-pipelines-yml-or-logstash-conf/207508 "2019-11-13T11:50:11Z")
I have pipeline id in pipelines.yml=\> pipe-common I want to give id as pipe-common\_ So the hostname should be dynamic from the environmentla variables of OS level. I can use ${hostname} variable in shell script. same … --- ## [Logstash (7.2.0) actively running but not listening on port 5043](https://discuss.elastic.co/t/logstash-7-2-0-actively-running-but-not-listening-on-port-5043/206772)
**Author:** [@koushikpal](https://discuss.elastic.co/u/koushikpal)\ **Replies:** 0\ **Last updated:** [November 6, 2019, 10:14am UTC](https://discuss.elastic.co/t/logstash-7-2-0-actively-running-but-not-listening-on-port-5043/206772 "2019-11-06T10:14:45Z")
I have configured Logstash to take input by listening to port 5043 from filebeat by creating a pipeline in /etc/logstash/conf.d for filebeat. Now when I start logstash and check the status, it shows active but when I che… --- ## [Logstash to parse custom exception log](https://discuss.elastic.co/t/logstash-to-parse-custom-exception-log/207685)
**Author:** [@bamboomy](https://discuss.elastic.co/u/bamboomy)\ **Replies:** 0\ **Last updated:** [November 13, 2019, 10:59am UTC](https://discuss.elastic.co/t/logstash-to-parse-custom-exception-log/207685 "2019-11-13T10:59:23Z")
Dears, I know there are a lot of tutorials and examples with almost an answer but I don't seem to get it right... I want to parse exception logs and they are in the following format: ERROR (some data that I like to p… --- ## [Filter pfsense Logs in Logstash](https://discuss.elastic.co/t/filter-pfsense-logs-in-logstash/207658)
**Author:** [@Snadro](https://discuss.elastic.co/u/Snadro)\ **Replies:** 0\ **Last updated:** [November 13, 2019, 8:54am UTC](https://discuss.elastic.co/t/filter-pfsense-logs-in-logstash/207658 "2019-11-13T08:54:20Z")
Hi erveryone. I looking for to use ELK-Stack as intrusion decetion system (IDS). i'm new in ELK and it is my first setup of en IDS. Our-Setup: Firewall: pfsense ELK-Stack Version: 6.8.4 Java-Version: 1.8.0\_222-8u222… --- ## [Error while sending logs to logstash with filebeats](https://discuss.elastic.co/t/error-while-sending-logs-to-logstash-with-filebeats/207652)
**Author:** [@David\_Samm](https://discuss.elastic.co/u/David_Samm)\ **Replies:** 0\ **Last updated:** [November 13, 2019, 8:16am UTC](https://discuss.elastic.co/t/error-while-sending-logs-to-logstash-with-filebeats/207652 "2019-11-13T08:16:54Z")
Getting following error while I try to fetch logs using filebeats to logstash \[logstash.outputs.elasticsearch\] Installing elasticsearch template to \_template/logstash \[2019-11-13T13:36:49,899\]\[ERROR\]\[logstash.outputs… --- ## [Design issue or simple limitation?](https://discuss.elastic.co/t/design-issue-or-simple-limitation/206944)
**Author:** [@pierreact](https://discuss.elastic.co/u/pierreact)\ **Replies:** 3\ **Last updated:** [November 13, 2019, 7:25am UTC](https://discuss.elastic.co/t/design-issue-or-simple-limitation/206944 "2019-11-13T07:25:24Z")
Hello, In lumberjack output, I see hosts is an array. 2 questions: Is it failover only or also load balancing? The documentation is not very talkative. ssl\_certificate... Are we unable to have one cert per host also? … --- ## [Run Multiple conf files in Windows Logstash service](https://discuss.elastic.co/t/run-multiple-conf-files-in-windows-logstash-service/207046)
**Author:** [@katara](https://discuss.elastic.co/u/katara)\ **Replies:** 2\ **Last updated:** [November 13, 2019, 7:21am UTC](https://discuss.elastic.co/t/run-multiple-conf-files-in-windows-logstash-service/207046 "2019-11-13T07:21:13Z")
Hi , I have installed Logstash 7.1.0 as a service in Windows through NSSM. I haven't provided any Arguments in NSSM. And In the pipelines.yml, I set different pipelines just by enabling - pipeline.id: dataupdate … --- ## [Remove a character with gsub](https://discuss.elastic.co/t/remove-a-character-with-gsub/207570)
**Author:** [@Kernel\_Panic](https://discuss.elastic.co/u/Kernel_Panic)\ **Replies:** 7\ **Last updated:** [November 12, 2019, 10:59pm UTC](https://discuss.elastic.co/t/remove-a-character-with-gsub/207570 "2019-11-12T22:59:11Z")
Hello there Im generating new field based on this entry a serial number and a status 34401910175945 ,ON The with this regular expression (?\.\*\[0-9\])\\s(?\\[^,\].\\w+) I get { "serial\_number": … --- ## [How to set and insert class variables in a hash with Ruby/Logstash](https://discuss.elastic.co/t/how-to-set-and-insert-class-variables-in-a-hash-with-ruby-logstash/207597)
**Author:** [@mogwaipr](https://discuss.elastic.co/u/mogwaipr)\ **Replies:** 0\ **Last updated:** [November 12, 2019, 10:52pm UTC](https://discuss.elastic.co/t/how-to-set-and-insert-class-variables-in-a-hash-with-ruby-logstash/207597 "2019-11-12T22:52:58Z")
I need to persist variables through class variables in ruby from Logstash so I can add fields to subsequent events such as timestamp can this be done initiating hash maps in Ruby then inserting values into the hash to p… --- ## [Incorrect Dates/Times After Parsing](https://discuss.elastic.co/t/incorrect-dates-times-after-parsing/207576)
**Author:** [@Jim\_Thunder](https://discuss.elastic.co/u/Jim_Thunder)\ **Replies:** 0\ **Last updated:** [November 12, 2019, 7:16pm UTC](https://discuss.elastic.co/t/incorrect-dates-times-after-parsing/207576 "2019-11-12T19:16:26Z")
I'm sending SMF records from a mainframe to ELK. It's just a CSV file with a bunch of values. Anyway, when I look at the unparsed message in kibana all of the times/dates are accurate. However after the record is parsed… --- ## [How to know the logstash indexing rate?](https://discuss.elastic.co/t/how-to-know-the-logstash-indexing-rate/207477)
**Author:** [@arunpmohan](https://discuss.elastic.co/u/arunpmohan)\ **Replies:** 2\ **Last updated:** [November 12, 2019, 6:25pm UTC](https://discuss.elastic.co/t/how-to-know-the-logstash-indexing-rate/207477 "2019-11-12T18:25:42Z")
I have implemented a pipeline like the below: kafka---\>logstash---\>Elasticsearch How to find the indexing rate? Is there any mechanism via logstash to get the indexing rate? Also, how to find indexing rate from Elasti… --- ## [JSON Array Split](https://discuss.elastic.co/t/json-array-split/207563)
**Author:** [@asingh-80](https://discuss.elastic.co/u/asingh-80)\ **Replies:** 0\ **Last updated:** [November 12, 2019, 5:25pm UTC](https://discuss.elastic.co/t/json-array-split/207563 "2019-11-12T17:25:20Z")
Hi, I am new to Logstash....I have the following data "status": "Request was successful", "nextpagetoken": "eyJsb2dfaWQiOiAiZjgzYmRlZDE0YwMTRmRmMzY5OTQ5NyIsICJkYXRldGltZSI6ICIyMDE5LTExLTEyVDAwOjUyOjE4LjM4ODAwMFoifQ==",… --- ## [Dynamic Data Type](https://discuss.elastic.co/t/dynamic-data-type/206639)
**Author:** [@zmink-pxc](https://discuss.elastic.co/u/zmink-pxc)\ **Replies:** 2\ **Last updated:** [November 12, 2019, 4:13pm UTC](https://discuss.elastic.co/t/dynamic-data-type/206639 "2019-11-12T16:13:45Z")
Hi all, I have a data format as shown in the attached image - I'm able to import the data via the CSV input plugin which works great. However, I'm stuck on the next step which is to map the rows to numbers so th… --- ## [Beat version in index name](https://discuss.elastic.co/t/beat-version-in-index-name/207092)
**Author:** [@pjanzen](https://discuss.elastic.co/u/pjanzen)\ **Replies:** 3\ **Last updated:** [November 12, 2019, 3:11pm UTC](https://discuss.elastic.co/t/beat-version-in-index-name/207092 "2019-11-12T15:11:04Z")
Hi All, I am using ELK 6.8.0 and I want to add the beat version to the indexname I have configured in an Logstash output. I have configured it currently like this clog-%{\[beat\]\[version\]}-%{+YYYY.MM.dd} But that give… --- ## [Unable to push data to ES from csv](https://discuss.elastic.co/t/unable-to-push-data-to-es-from-csv/207507)
**Author:** [@GuruTeja](https://discuss.elastic.co/u/GuruTeja)\ **Replies:** 0\ **Last updated:** [November 12, 2019, 12:05pm UTC](https://discuss.elastic.co/t/unable-to-push-data-to-es-from-csv/207507 "2019-11-12T12:05:49Z")
input{ file{ path =\> "F:/GuruTeja/KIBANA/LOGSTACS/adderss.csv" start\_position =\> "beginning" sincedb\_path =\> "NULL" } } output{ elasticsearch{ hosts =\> "http://localhost:9200" index =\> mytest } stdout{ } } … --- ## [Http\_poller in logstash](https://discuss.elastic.co/t/http-poller-in-logstash/207504)
**Author:** [@subash](https://discuss.elastic.co/u/subash)\ **Replies:** 0\ **Last updated:** [November 12, 2019, 11:45am UTC](https://discuss.elastic.co/t/http-poller-in-logstash/207504 "2019-11-12T11:45:20Z")
Hi, I am using http\_poller input plugin to get the data from the URL. I am receiving the data like below every 5 minutes. I want to visualize the data in Kibana in a line graph with timestamp in the x axis (every 5 mins… --- ## [Could not index event to Elasticsearch reason failed to parse field \[rule\] of type \[text\] in document](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-reason-failed-to-parse-field-rule-of-type-text-in-document/207470)
**Author:** [@nghia\_huynh](https://discuss.elastic.co/u/nghia_huynh)\ **Replies:** 1\ **Last updated:** [November 12, 2019, 11:25am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-reason-failed-to-parse-field-rule-of-type-text-in-document/207470 "2019-11-12T11:25:38Z")
Hi everyone, Today, I check log from logstash-plain.log and I see many event was spam from one error like that: \[2019-11-12T16:03:53,696\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {… --- ## [Use map field as index name](https://discuss.elastic.co/t/use-map-field-as-index-name/207494)
**Author:** [@Anuwa\_A](https://discuss.elastic.co/u/Anuwa_A)\ **Replies:** 0\ **Last updated:** [November 12, 2019, 11:00am UTC](https://discuss.elastic.co/t/use-map-field-as-index-name/207494 "2019-11-12T11:00:48Z")
In my logstash config file map\['first\_name'\] ||= event.get('first\_name') . when i try to use that as my index index =\> "%{first\_name}" index doesn't create. --- ## [Logstash is slow for mulitiple pipelines](https://discuss.elastic.co/t/logstash-is-slow-for-mulitiple-pipelines/207491)
**Author:** [@nuwancs](https://discuss.elastic.co/u/nuwancs)\ **Replies:** 0\ **Last updated:** [November 12, 2019, 10:41am UTC](https://discuss.elastic.co/t/logstash-is-slow-for-mulitiple-pipelines/207491 "2019-11-12T10:41:37Z")
Hi There, I am using two pipelines to process my data coming from same server but different set of files. let's say A and B. A has a very high tps and it goes over 1000EPS(events per sec) B has 40EPS max. They push data… --- ## [Rename doc\_id in logstash output](https://discuss.elastic.co/t/rename-doc-id-in-logstash-output/207479)
**Author:** [@Anuwa\_A](https://discuss.elastic.co/u/Anuwa_A)\ **Replies:** 0\ **Last updated:** [November 12, 2019, 9:36am UTC](https://discuss.elastic.co/t/rename-doc-id-in-logstash-output/207479 "2019-11-12T09:36:35Z")
i want to sub-string id and ädd it in front of the id with \_ and store that as doc\_id in es out put. i tried below but doesn't work id = %{id}\[0..1\]+""+%{id} elasticsearch { document\_id =\> "%{id}\[0..1\]+""+%{id}" } --- ## [How to read avro serialized file from s3](https://discuss.elastic.co/t/how-to-read-avro-serialized-file-from-s3/207455)
**Author:** [@Adarsh\_Kumar1](https://discuss.elastic.co/u/Adarsh_Kumar1)\ **Replies:** 0\ **Last updated:** [November 12, 2019, 7:40am UTC](https://discuss.elastic.co/t/how-to-read-avro-serialized-file-from-s3/207455 "2019-11-12T07:40:31Z")
I have avro files in s3 bucket and I am using s3 input plugin to read the file and passed by schema to codec but its throwing error and unable parse the file. When I looked into the code for input plugin, it seems not .… --- ## [Jdbc input plugin sql\_last\_value not updating](https://discuss.elastic.co/t/jdbc-input-plugin-sql-last-value-not-updating/207409)
**Author:** [@ma3](https://discuss.elastic.co/u/ma3)\ **Replies:** 1\ **Last updated:** [November 12, 2019, 6:26am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-sql-last-value-not-updating/207409 "2019-11-12T06:26:26Z")
I have multiple jdbc input plugins in logstash writing to different indices in elastic search. I'm using sql\_last\_value to send incremental updates. However, the timestamp in last\_run\_metadata\_path is only getting updat… --- ## [Json Target doesn't work](https://discuss.elastic.co/t/json-target-doesnt-work/207445)
**Author:** [@kollp](https://discuss.elastic.co/u/kollp)\ **Replies:** 0\ **Last updated:** [November 12, 2019, 6:07am UTC](https://discuss.elastic.co/t/json-target-doesnt-work/207445 "2019-11-12T06:07:17Z")
Hello, I am using logstash 7.3.1. I use an http input plugin to get data into logstash. I get an JSON payload which looks like {"version":"1.1","host":"myHost","short\_message":"Test Dev","full\_message":"Testlog Messag… --- ## [Logstash date not matching](https://discuss.elastic.co/t/logstash-date-not-matching/207266)
**Author:** [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\ **Replies:** 4\ **Last updated:** [November 12, 2019, 3:49am UTC](https://discuss.elastic.co/t/logstash-date-not-matching/207266 "2019-11-12T03:49:26Z")
Hey all, I'm trying to work out why I get a grok parse failure on date here when using these two as input 2014-10-05 02:59:25 Works 2015-10-05 02:59:25 Does not work timestamp.conf input { stdin { } }… --- ## [Logstash error using \`split\` on JSON array, where array is converted from hash](https://discuss.elastic.co/t/logstash-error-using-split-on-json-array-where-array-is-converted-from-hash/207395)
**Author:** [@DougR](https://discuss.elastic.co/u/DougR)\ **Replies:** 5\ **Last updated:** [November 11, 2019, 7:26pm UTC](https://discuss.elastic.co/t/logstash-error-using-split-on-json-array-where-array-is-converted-from-hash/207395 "2019-11-11T19:26:23Z")
TL;DR Receiving errors when using logstash split filter after using ruby filter to convert JSON dict (hash) to array. The rest of the story... I have Logstash polling an application for mbean metrics. The data set is co… --- ## [Extract keys from field name (not field value)](https://discuss.elastic.co/t/extract-keys-from-field-name-not-field-value/207406)
**Author:** [@vshankara](https://discuss.elastic.co/u/vshankara)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 5:54pm UTC](https://discuss.elastic.co/t/extract-keys-from-field-name-not-field-value/207406 "2019-11-11T17:54:24Z")
I have a requirement to retrieve filesystem utilization metrics for Linux servers via SNMP. I am utilizing logstash to do a snmp walk on HOST-RESOURCES-MIB and ingesting in to elastic. Here is how it is reported \*\*HOST… --- ## [Logstash SNMP over DTLS](https://discuss.elastic.co/t/logstash-snmp-over-dtls/207396)
**Author:** [@ANDREA\_CASARIN](https://discuss.elastic.co/u/ANDREA_CASARIN)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 5:03pm UTC](https://discuss.elastic.co/t/logstash-snmp-over-dtls/207396 "2019-11-11T17:03:24Z")
Hello all! I'm looking for a solution for gathering metrics from an snmp agent which uses DTLS as the security model. Apparently snmp-input-plugin doesn't have that configuration, i looked online but I didn't find anyth… --- ## [Using multiple config files](https://discuss.elastic.co/t/using-multiple-config-files/207384)
**Author:** [@Anuwa\_A](https://discuss.elastic.co/u/Anuwa_A)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 4:16pm UTC](https://discuss.elastic.co/t/using-multiple-config-files/207384 "2019-11-11T16:16:55Z")
I'm planing to migrate my mysql data to Elastic Search. Based on my data model it's complex to finish using one config file based on the complexity of the query. So i'm planing to use multiple configuration files. is… --- ## [Unable to send logs from NON-AWS infrastructure logstash shipper to SQS](https://discuss.elastic.co/t/unable-to-send-logs-from-non-aws-infrastructure-logstash-shipper-to-sqs/207374)
**Author:** [@medikonduruv](https://discuss.elastic.co/u/medikonduruv)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 3:04pm UTC](https://discuss.elastic.co/t/unable-to-send-logs-from-non-aws-infrastructure-logstash-shipper-to-sqs/207374 "2019-11-11T15:04:07Z")
Hi, I'm trying to implement the logstash to send logs from a NON-AWS infrastructure to SQS through SQS output plugin in shipper conf by passing the "access\_key\_id" and "secret\_access\_key", From the logs i'm able to see … --- ## [Constant restarts on bad configuration](https://discuss.elastic.co/t/constant-restarts-on-bad-configuration/207372)
**Author:** [@ogtool](https://discuss.elastic.co/u/ogtool)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 2:44pm UTC](https://discuss.elastic.co/t/constant-restarts-on-bad-configuration/207372 "2019-11-11T14:44:24Z")
Hey all, I've noticed the (apparently planned) behavior of logstash where it will restart if the configuration file isn't valid, even if the configuration file isn't changed. While the ideal situation is don't get the c… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=446) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=448)