# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=576 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 577 --- ## [Multiple inputs and outputs in logstash conf file](https://discuss.elastic.co/t/multiple-inputs-and-outputs-in-logstash-conf-file/171025)
**Author:** [@sak6070](https://discuss.elastic.co/u/sak6070)\ **Replies:** 11\ **Last updated:** [March 29, 2019, 9:05am UTC](https://discuss.elastic.co/t/multiple-inputs-and-outputs-in-logstash-conf-file/171025 "2019-03-29T09:05:25Z")
I have included multiple inputs and outputs in my logstash conf file (without filter for now). I have also created different indexes for each input. I am not able to see all the logs on kibana , also indices are not vi… --- ## [Kafka plugin try to describe all the topics when used \`topics\_pattern\`](https://discuss.elastic.co/t/kafka-plugin-try-to-describe-all-the-topics-when-used-topics-pattern/174508)
**Author:** [@skbly7](https://discuss.elastic.co/u/skbly7)\ **Replies:** 0\ **Last updated:** [March 29, 2019, 9:49am UTC](https://discuss.elastic.co/t/kafka-plugin-try-to-describe-all-the-topics-when-used-topics-pattern/174508 "2019-03-29T09:49:52Z")
Hi, We are using Logstash Kafka input plugin which is connecting to Kafka with below configuration. kafka { codec =\> "json" bootstrap\_servers =\> "1.2.3.4:9094" topics\_pattern =\> "^something-.\*$" group… --- ## [How i can match and replace the log file data in logstash](https://discuss.elastic.co/t/how-i-can-match-and-replace-the-log-file-data-in-logstash/174059)
**Author:** [@Mac099](https://discuss.elastic.co/u/Mac099)\ **Replies:** 11\ **Last updated:** [March 29, 2019, 9:45am UTC](https://discuss.elastic.co/t/how-i-can-match-and-replace-the-log-file-data-in-logstash/174059 "2019-03-29T09:45:10Z")
input { beats { port =\> 5044 } } filter { grok { match=\>{'message'=\>'%{DATESTAMP:time} %{LOGLEVEL:level} '} mutate { add\_tag =\> "string in field" } } } output { elasticsearch { hosts =\> \["http://localhost:9… --- ## [Bad performance after automatic reload configuration](https://discuss.elastic.co/t/bad-performance-after-automatic-reload-configuration/171125)
**Author:** [@florentmair](https://discuss.elastic.co/u/florentmair)\ **Replies:** 1\ **Last updated:** [March 29, 2019, 8:30am UTC](https://discuss.elastic.co/t/bad-performance-after-automatic-reload-configuration/171125 "2019-03-29T08:30:51Z")
Hello when i update the configuration of logstash filters, logstash i setup to reload automatically but after that i have an excessive jvm heap, high event latency and slow rates for the events received/emmitted what… --- ## [How to mutate sensitive data in logstash](https://discuss.elastic.co/t/how-to-mutate-sensitive-data-in-logstash/173441)
**Author:** [@manickalai](https://discuss.elastic.co/u/manickalai)\ **Replies:** 8\ **Last updated:** [March 29, 2019, 7:41am UTC](https://discuss.elastic.co/t/how-to-mutate-sensitive-data-in-logstash/173441 "2019-03-29T07:41:55Z")
Input : "refNo":"0015531891539643", cardNo=6055490201604225 Requirment: The requirement is to show the start 6 digits and last 4 digits and mask the remaining numbers of cardNo in logstash. I applied gsub/mutate filter … --- ## [Logstash end up with cluster\_block\_exception and is frequently occuring](https://discuss.elastic.co/t/logstash-end-up-with-cluster-block-exception-and-is-frequently-occuring/174342)
**Author:** [@Kaunya](https://discuss.elastic.co/u/Kaunya)\ **Replies:** 5\ **Last updated:** [March 29, 2019, 5:41am UTC](https://discuss.elastic.co/t/logstash-end-up-with-cluster-block-exception-and-is-frequently-occuring/174342 "2019-03-29T05:41:39Z")
Hi, We setup trial version of ELK stack (6.6.2) ( with 2 nodes of logstash, 3 nodes of Elastic cluster, 1 node of Kibana) and configure to flow messages from our Firewalls =\> Logstash =\> Elastic Search =\> Kibana. logst… --- ## [Logstash JDBC Input last\_run\_metadata\_path and Heroku](https://discuss.elastic.co/t/logstash-jdbc-input-last-run-metadata-path-and-heroku/174452)
**Author:** [@elseahawk](https://discuss.elastic.co/u/elseahawk)\ **Replies:** 0\ **Last updated:** [March 29, 2019, 1:47am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-last-run-metadata-path-and-heroku/174452 "2019-03-29T01:47:27Z")
Hi, I am considering running our Logstash in a series of dedicated dynos on Heroku. But heroku's filesystem is ephemeral and is not persisted across dyno reboots. So I am wondering how people use logstash on heroku an… --- ## [Rsyslogd replaces with Logstash and/or Filebeat?](https://discuss.elastic.co/t/rsyslogd-replaces-with-logstash-and-or-filebeat/173660)
**Author:** [@jasony](https://discuss.elastic.co/u/jasony)\ **Replies:** 1\ **Last updated:** [March 28, 2019, 11:12pm UTC](https://discuss.elastic.co/t/rsyslogd-replaces-with-logstash-and-or-filebeat/173660 "2019-03-28T23:12:43Z")
hello, I am managing one cluster and current service-flow is as below. SYSLOG -\> Rsyslogd (store data into disk) -\> Logstash (read from disk and transfer to Kafka) -\> Kafka -\> Logstash -\> Elasticsearch Problem is curr… --- ## [Assistance with multiple syslogs and creating multiple indexes](https://discuss.elastic.co/t/assistance-with-multiple-syslogs-and-creating-multiple-indexes/174406)
**Author:** [@FrostyDLL](https://discuss.elastic.co/u/FrostyDLL)\ **Replies:** 1\ **Last updated:** [March 28, 2019, 6:58pm UTC](https://discuss.elastic.co/t/assistance-with-multiple-syslogs-and-creating-multiple-indexes/174406 "2019-03-28T18:58:33Z")
Hi Everyone, I have an issue that many have probably encountered before. I have a server "DATA-MGT" that collects data from other systems, and formats that information as a custom SYSLOG file. With FileBeats installed … --- ## [Unable to preserve value in global variable](https://discuss.elastic.co/t/unable-to-preserve-value-in-global-variable/174304)
**Author:** [@avj1986](https://discuss.elastic.co/u/avj1986)\ **Replies:** 5\ **Last updated:** [March 28, 2019, 6:53pm UTC](https://discuss.elastic.co/t/unable-to-preserve-value-in-global-variable/174304 "2019-03-28T18:53:12Z")
Hello, After searching and surfing for hours and hours, I am still unable to preserve the value of an event field in a global variable. This is what I am trying to achive. I have a log file something like: INFO: Ser… --- ## [Stop on a specific character](https://discuss.elastic.co/t/stop-on-a-specific-character/174369)
**Author:** [@markov](https://discuss.elastic.co/u/markov)\ **Replies:** 1\ **Last updated:** [March 28, 2019, 3:25pm UTC](https://discuss.elastic.co/t/stop-on-a-specific-character/174369 "2019-03-28T15:25:53Z")
hello guys, i have a java log line "BusinessServiceImpl.java:7651)" and i want to stop just at the character ")" how can i do that with grok. thanks in advance. --- ## [Configuring Logstash Pipeline in Elasticsearch Cloud (elastic.co)](https://discuss.elastic.co/t/configuring-logstash-pipeline-in-elasticsearch-cloud-elastic-co/174359)
**Author:** [@amit.petpace](https://discuss.elastic.co/u/amit.petpace)\ **Replies:** 0\ **Last updated:** [March 28, 2019, 2:36pm UTC](https://discuss.elastic.co/t/configuring-logstash-pipeline-in-elasticsearch-cloud-elastic-co/174359 "2019-03-28T14:36:33Z")
Hi, We're using the Elasticsearch Cloud service (elastic.co). I'm trying to configure logstash pipeline via the Kibana UI. The idea is to pull message from RabbitMQ to Elasticsearch, I would appreciate if someone can … --- ## [Geoip.location field not generated](https://discuss.elastic.co/t/geoip-location-field-not-generated/174312)
**Author:** [@hannibal](https://discuss.elastic.co/u/hannibal)\ **Replies:** 5\ **Last updated:** [March 28, 2019, 2:23pm UTC](https://discuss.elastic.co/t/geoip-location-field-not-generated/174312 "2019-03-28T14:23:24Z")
Hi everyone, I'm trying to play with geoip data and I have a trouble with the field geoip.location. According to the documentation A \[geoip\]\[location\] field is created if the GeoIP lookup returns a latitude and longi… --- ## [Logstash query no output data](https://discuss.elastic.co/t/logstash-query-no-output-data/174174)
**Author:** [@hustlercoder](https://discuss.elastic.co/u/hustlercoder)\ **Replies:** 1\ **Last updated:** [March 28, 2019, 1:14pm UTC](https://discuss.elastic.co/t/logstash-query-no-output-data/174174 "2019-03-28T13:14:24Z")
Hi there i had some trouble with my logstash configuration.Basically i create a conf file with a basic jdbc connection to a postgresql database. I also had other 3 files...4 in total...2 works 2 no...but i cant explain … --- ## [Logstash to Logstash via TCP is slow/delayed/waiting for something](https://discuss.elastic.co/t/logstash-to-logstash-via-tcp-is-slow-delayed-waiting-for-something/173299)
**Author:** [@cawoodm](https://discuss.elastic.co/u/cawoodm)\ **Replies:** 5\ **Last updated:** [March 28, 2019, 12:35pm UTC](https://discuss.elastic.co/t/logstash-to-logstash-via-tcp-is-slow-delayed-waiting-for-something/173299 "2019-03-28T12:35:30Z")
We process logfiles with filebeat into logstash (CONF1). If the log entry is marked as an error, we clone it and send it to another instance of logstash (CONF2) via the tcp output plugin. The problem is that while we se… --- ## [Agent Address field of SNMPV1 Trap is coming as combination of hexadecimal and ASCII value](https://discuss.elastic.co/t/agent-address-field-of-snmpv1-trap-is-coming-as-combination-of-hexadecimal-and-ascii-value/174322)
**Author:** [@Nik\_1106](https://discuss.elastic.co/u/Nik_1106)\ **Replies:** 0\ **Last updated:** [March 28, 2019, 11:49am UTC](https://discuss.elastic.co/t/agent-address-field-of-snmpv1-trap-is-coming-as-combination-of-hexadecimal-and-ascii-value/174322 "2019-03-28T11:49:25Z")
I am trying to extract fields of SNMPv1 trap from logstash output. Here is my pipeline: And This is what I see in output: Here, agent address is coming as combination of hexadecimal and ASCII value. "agent\_address… --- ## [Error to send data to Elasticsearch](https://discuss.elastic.co/t/error-to-send-data-to-elasticsearch/174313)
**Author:** [@pad](https://discuss.elastic.co/u/pad)\ **Replies:** 0\ **Last updated:** [March 28, 2019, 11:01am UTC](https://discuss.elastic.co/t/error-to-send-data-to-elasticsearch/174313 "2019-03-28T11:01:41Z")
Hi, I'm using a Logstash (try 5.6.10 and currently 5.6.16) to execute sql queries on PostGresql and send the result to an ElasticSearch (5.6.10) but I've got this error when Logstash try to send to ES: \[2019-03-28T10:5… --- ## [How to parse xml file using logstash](https://discuss.elastic.co/t/how-to-parse-xml-file-using-logstash/174237)
**Author:** [@sowndharya.s](https://discuss.elastic.co/u/sowndharya.s)\ **Replies:** 1\ **Last updated:** [March 28, 2019, 10:47am UTC](https://discuss.elastic.co/t/how-to-parse-xml-file-using-logstash/174237 "2019-03-28T10:47:55Z")
how to parse xml file in logstash using conf file. --- ## [Remove unused Kibana fields/filters](https://discuss.elastic.co/t/remove-unused-kibana-fields-filters/174135)
**Author:** [@markov](https://discuss.elastic.co/u/markov)\ **Replies:** 2\ **Last updated:** [March 28, 2019, 9:48am UTC](https://discuss.elastic.co/t/remove-unused-kibana-fields-filters/174135 "2019-03-28T09:48:23Z")
hi, can someone please tell me how i can keep just my grok filters and remove those 62 fields, i tried using mutate plugin but not working. mutate { remove\_field =\> \[ "beat.name", "beat.version", "data", "host.archite… --- ## [Field logTime is 2019-Mar-28 02:39:24.778, 2019-Mar-28 02:39:24.778, why?](https://discuss.elastic.co/t/field-logtime-is-2019-mar-28-0224-778-2019-mar-28-0224-778-why/174218)
**Author:** [@kevin\_bolton](https://discuss.elastic.co/u/kevin_bolton)\ **Replies:** 2\ **Last updated:** [March 28, 2019, 8:02am UTC](https://discuss.elastic.co/t/field-logtime-is-2019-mar-28-0224-778-2019-mar-28-0224-778-why/174218 "2019-03-28T08:02:55Z")
I have multiple Logstash config(forti.conf, cisco.conf) in logstash conf directory Both configs have logTime field add\_field =\> { "logTime" =\> "%{year}-%{month}-%{day} %{time}" } In Kibans \> query \> index of cisco log… --- ## [How to Make a dashboard for cloud flare rate-limit graph and total web traffic bandwidth](https://discuss.elastic.co/t/how-to-make-a-dashboard-for-cloud-flare-rate-limit-graph-and-total-web-traffic-bandwidth/174254)
**Author:** [@Fire\_fire](https://discuss.elastic.co/u/Fire_fire)\ **Replies:** 0\ **Last updated:** [March 28, 2019, 7:27am UTC](https://discuss.elastic.co/t/how-to-make-a-dashboard-for-cloud-flare-rate-limit-graph-and-total-web-traffic-bandwidth/174254 "2019-03-28T07:27:22Z")
I have a cloud-flare enterprise logs and how to make a graph for rate-limit and total web traffic bandwidth like as a cloud flare dashboard. --- ## [Multiple pipelines or tags or if or else if?](https://discuss.elastic.co/t/multiple-pipelines-or-tags-or-if-or-else-if/173974)
**Author:** [@Matthew\_Quickenden](https://discuss.elastic.co/u/Matthew_Quickenden)\ **Replies:** 2\ **Last updated:** [March 27, 2019, 6:16pm UTC](https://discuss.elastic.co/t/multiple-pipelines-or-tags-or-if-or-else-if/173974 "2019-03-27T18:16:21Z")
I'm new to logstash and I have been looking through the help guides and through the forums and I can't seem to find the answers I need. I am running ELK 6.6.0. I have 3 inputs I'd like to get into Elasticsearch. Sys… --- ## [Sample logstash conf file to get logs from remote machine to ELK in another machine](https://discuss.elastic.co/t/sample-logstash-conf-file-to-get-logs-from-remote-machine-to-elk-in-another-machine/174236)
**Author:** [@sowndharya.s](https://discuss.elastic.co/u/sowndharya.s)\ **Replies:** 0\ **Last updated:** [March 28, 2019, 5:24am UTC](https://discuss.elastic.co/t/sample-logstash-conf-file-to-get-logs-from-remote-machine-to-elk-in-another-machine/174236 "2019-03-28T05:24:27Z")
Hi All, Can anyone please share conf file to connect to remote server to get logs and publish reports in separate machine. --- ## [Logstash error- Received an event that has a different character encoding](https://discuss.elastic.co/t/logstash-error-received-an-event-that-has-a-different-character-encoding/174231)
**Author:** [@sharon92](https://discuss.elastic.co/u/sharon92)\ **Replies:** 0\ **Last updated:** [March 28, 2019, 4:08am UTC](https://discuss.elastic.co/t/logstash-error-received-an-event-that-has-a-different-character-encoding/174231 "2019-03-28T04:08:53Z")
I use logspout v3.2.6 by gliderlab to route docker logs to Logstash 6.6.2. But the characters were scrambled. I tried many codes and charset nothing seems working. Need help. I will share the configurations and logs for… --- ## [Logstash doesn't work properly](https://discuss.elastic.co/t/logstash-doesnt-work-properly/174109)
**Author:** [@MaxON](https://discuss.elastic.co/u/MaxON)\ **Replies:** 1\ **Last updated:** [March 27, 2019, 10:07pm UTC](https://discuss.elastic.co/t/logstash-doesnt-work-properly/174109 "2019-03-27T22:07:30Z")
I had to reinstall the logstash service (I use windows). After that it stopped parsing and I can't find new logs in kibana, though Logstash service is running. I know that the problem is not in logstash.json file, becaus… --- ## [Cant install netflow plugin for kibana](https://discuss.elastic.co/t/cant-install-netflow-plugin-for-kibana/174189)
**Author:** [@Miu](https://discuss.elastic.co/u/Miu)\ **Replies:** 1\ **Last updated:** [March 27, 2019, 8:06pm UTC](https://discuss.elastic.co/t/cant-install-netflow-plugin-for-kibana/174189 "2019-03-27T20:06:50Z")
Good day! Im try install netflow plugin from this guide https://www.elastic.co/guide/en/logstash/current/netflow-module.html My environment: centos 7.6 - 192.168.20.56 - elk2 centos 7.6 - 192.168.20.57 - elk1 centos… --- ## [Environment variable interpolation in Logstash conf files](https://discuss.elastic.co/t/environment-variable-interpolation-in-logstash-conf-files/172411)
**Author:** [@01100010011001010110](https://discuss.elastic.co/u/01100010011001010110)\ **Replies:** 1\ **Last updated:** [March 27, 2019, 7:06pm UTC](https://discuss.elastic.co/t/environment-variable-interpolation-in-logstash-conf-files/172411 "2019-03-27T19:06:22Z")
We run logstash on k8s with centralized pipeline management, and I'm attempting to pass credentials into the containers as env vars. This has worked for everything but the azure\_eventhubs input plugin. Debug logging seem… --- ## [How can Logstash runs be triggered from automated tests in a separate Docker container?](https://discuss.elastic.co/t/how-can-logstash-runs-be-triggered-from-automated-tests-in-a-separate-docker-container/173786)
**Author:** [@m.carey](https://discuss.elastic.co/u/m.carey)\ **Replies:** 1\ **Last updated:** [March 27, 2019, 4:52pm UTC](https://discuss.elastic.co/t/how-can-logstash-runs-be-triggered-from-automated-tests-in-a-separate-docker-container/173786 "2019-03-27T16:52:29Z")
I'm trying to create automated tests for my Logstash pipeline config. I want to load multiple data sets from MySQL and for each data set, test that the raw data was transformed and filtered properly when loaded into Elas… --- ## [Logstash different .conf inputs are mixing up entries in indexes](https://discuss.elastic.co/t/logstash-different-conf-inputs-are-mixing-up-entries-in-indexes/174144)
**Author:** [@karo](https://discuss.elastic.co/u/karo)\ **Replies:** 3\ **Last updated:** [March 27, 2019, 3:21pm UTC](https://discuss.elastic.co/t/logstash-different-conf-inputs-are-mixing-up-entries-in-indexes/174144 "2019-03-27T15:21:58Z")
Before I go on, I have tried to look through the interwebs for an answer, I scoured the forums posts so much I feel I know the elk staff member names by heart (not really). I do apologise if my insufficient research ski… --- ## [Multiple input and multiple output](https://discuss.elastic.co/t/multiple-input-and-multiple-output/174069)
**Author:** [@Ana\_Thema](https://discuss.elastic.co/u/Ana_Thema)\ **Replies:** 4\ **Last updated:** [March 27, 2019, 3:17pm UTC](https://discuss.elastic.co/t/multiple-input-and-multiple-output/174069 "2019-03-27T15:17:56Z")
Good morning. I set up a log server with ELK stack on one machine. I need to ship to it logs from 16 other machines, and i'd like to have 16 named indexes. So i tried this (4 machines for now) input { beats { typ… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=575) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=577)