# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=580 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 581 --- ## [Firewalls -\> nginx (udp) -\> 4 x logstash losing device IP](https://discuss.elastic.co/t/firewalls-nginx-udp-4-x-logstash-losing-device-ip/173327)
**Author:** [@leo\_ponton](https://discuss.elastic.co/u/leo_ponton)\ **Replies:** 0\ **Last updated:** [March 21, 2019, 2:14pm UTC](https://discuss.elastic.co/t/firewalls-nginx-udp-4-x-logstash-losing-device-ip/173327 "2019-03-21T14:14:05Z")
I'm using Nginx configured as a load balancer to distribute firewall syslogs amongst my cluster's 4 logstash instances. The problem I have is that logstash sees the IP of the load balancer rather than the original devic… --- ## [Import data from mysql dump to Elasticsearch](https://discuss.elastic.co/t/import-data-from-mysql-dump-to-elasticsearch/173383)
**Author:** [@virgo714us](https://discuss.elastic.co/u/virgo714us)\ **Replies:** 5\ **Last updated:** [March 22, 2019, 8:44am UTC](https://discuss.elastic.co/t/import-data-from-mysql-dump-to-elasticsearch/173383 "2019-03-22T08:44:53Z")
I want to import data from .sql file (backup created from mysql db with extension .sql) to elasticsearch. What is the way of doing that? --- ## [JDBC full event - event.original ECS](https://discuss.elastic.co/t/jdbc-full-event-event-original-ecs/173437)
**Author:** [@Anabella\_Cristaldi](https://discuss.elastic.co/u/Anabella_Cristaldi)\ **Replies:** 0\ **Last updated:** [March 22, 2019, 7:15am UTC](https://discuss.elastic.co/t/jdbc-full-event-event-original-ecs/173437 "2019-03-22T07:15:06Z")
Hi all, I'm indexing data from an oracle db. It is an audit table from an specific application and I'm normalizing the fields into the ECS schema. My question: Is it possible to obtain the raw event from the jdbc input… --- ## [Logstash cmd error](https://discuss.elastic.co/t/logstash-cmd-error/173431)
**Author:** [@RoWesley](https://discuss.elastic.co/u/RoWesley)\ **Replies:** 0\ **Last updated:** [March 22, 2019, 6:40am UTC](https://discuss.elastic.co/t/logstash-cmd-error/173431 "2019-03-22T06:40:40Z")
Trying to run the logstash command getting an below error, please suggest C:\\Users\\Priyesh.Chourasia\\Desktop\\ElasticSearch\\logstash-6.5.2\\bin\>logstash -e "input { stdin { } } output { stdout {} }" Sending Logstash log… --- ## [Creating drop filters based on a string search in a message field](https://discuss.elastic.co/t/creating-drop-filters-based-on-a-string-search-in-a-message-field/64050)
**Author:** [@dvosbury](https://discuss.elastic.co/u/dvosbury)\ **Replies:** 12\ **Last updated:** [March 22, 2019, 4:40am UTC](https://discuss.elastic.co/t/creating-drop-filters-based-on-a-string-search-in-a-message-field/64050 "2019-03-22T04:40:23Z")
I'm new to the Elastic stack and Logstash. I have a new Logstash instance that is accepting logs from beats sending to Elasticsearch. I'm trying to create a filter that will drop some logs that we aren't that interested … --- ## [Logstash Config Flow](https://discuss.elastic.co/t/logstash-config-flow/173423)
**Author:** [@christopher](https://discuss.elastic.co/u/christopher)\ **Replies:** 0\ **Last updated:** [March 22, 2019, 3:34am UTC](https://discuss.elastic.co/t/logstash-config-flow/173423 "2019-03-22T03:34:22Z")
Hi Guys, I'm just new on ELK and trying things. Would anyone be so kind to help me understand the logstash process flow? I mean what config files is it reading first then going to what config file? e.g. logstash.yml \> pi… --- ## [Python Webapp Error Log — JSON?](https://discuss.elastic.co/t/python-webapp-error-log-json/173419)
**Author:** [@smlbiobot](https://discuss.elastic.co/u/smlbiobot)\ **Replies:** 0\ **Last updated:** [March 22, 2019, 2:42am UTC](https://discuss.elastic.co/t/python-webapp-error-log-json/173419 "2019-03-22T02:42:43Z")
I have a Python app that currently output regular console logs. I use this to monitor any potential issues with the app. I’m using Sentry atm to track errors but am wondering if Logstash would be a good use case to do th… --- ## [Parsing nested JSON with dynamic key](https://discuss.elastic.co/t/parsing-nested-json-with-dynamic-key/173110)
**Author:** [@mdobrin](https://discuss.elastic.co/u/mdobrin)\ **Replies:** 2\ **Last updated:** [March 22, 2019, 2:11am UTC](https://discuss.elastic.co/t/parsing-nested-json-with-dynamic-key/173110 "2019-03-22T02:11:55Z")
Given the following JSON, I would like to be able to extract the genreId. However, I am not sure how to do so with nested dynamic keys (in the example below, "610015") { "b\_shopId": 502192, "body": { "delivery": { … --- ## [Logstash s3 input plugin meta error](https://discuss.elastic.co/t/logstash-s3-input-plugin-meta-error/173228)
**Author:** [@miashah](https://discuss.elastic.co/u/miashah)\ **Replies:** 1\ **Last updated:** [March 21, 2019, 10:23pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-meta-error/173228 "2019-03-21T22:23:34Z")
Hi Folks, I am working with the s3 input plugin to ingest logs that are pushed my S3 bucket. I have encoded information in the s3 key of the object. I referenced the following in order to add the s3 key to each event: … --- ## [Logstash CSV - Changing columns settings based on first character](https://discuss.elastic.co/t/logstash-csv-changing-columns-settings-based-on-first-character/173388)
**Author:** [@Ethical2ak](https://discuss.elastic.co/u/Ethical2ak)\ **Replies:** 7\ **Last updated:** [March 21, 2019, 9:55pm UTC](https://discuss.elastic.co/t/logstash-csv-changing-columns-settings-based-on-first-character/173388 "2019-03-21T21:55:53Z")
Hi, I've came accross this thread https://discuss.elastic.co/t/changing-columns-based-on-first-character/112560 , but im not totally sure if its was resolved. I have a CSV file with plenty of entry like this : 7,155… --- ## [Logstash-netflow-codec question](https://discuss.elastic.co/t/logstash-netflow-codec-question/173398)
**Author:** [@sgnsys3](https://discuss.elastic.co/u/sgnsys3)\ **Replies:** 0\ **Last updated:** [March 21, 2019, 8:32pm UTC](https://discuss.elastic.co/t/logstash-netflow-codec-question/173398 "2019-03-21T20:32:47Z")
I have set up logstash cluster and use nginx to load balance flow. Do you have any solution for sharing flow template between logstash node? --- ## [Logstash on network volume](https://discuss.elastic.co/t/logstash-on-network-volume/173372)
**Author:** [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Replies:** 1\ **Last updated:** [March 21, 2019, 7:12pm UTC](https://discuss.elastic.co/t/logstash-on-network-volume/173372 "2019-03-21T19:12:02Z")
Hello, I have a working .conf file (it is working on my local disk). But when I change the path (I use file plugin) to my network disk Z, logstash doesn't detect any logs without sending any errors (with --verbose enabl… --- ## [Logstash - version 6.6.0- Scheduled JDBC input stop running](https://discuss.elastic.co/t/logstash-version-6-6-0-scheduled-jdbc-input-stop-running/173359)
**Author:** [@Jose\_Augusto\_Jales](https://discuss.elastic.co/u/Jose_Augusto_Jales)\ **Replies:** 0\ **Last updated:** [March 21, 2019, 5:05pm UTC](https://discuss.elastic.co/t/logstash-version-6-6-0-scheduled-jdbc-input-stop-running/173359 "2019-03-21T17:05:52Z")
Hi, we have a 6.2.2 Logstash running 8 pipelines on a linux machine, one of them uses a scheduled JDBC input and everything runs fine. We updated logstash to version 6.6.0 and now we are having problems, one of the sche… --- ## [Forward syslogs to another siem](https://discuss.elastic.co/t/forward-syslogs-to-another-siem/173045)
**Author:** [@bambam](https://discuss.elastic.co/u/bambam)\ **Replies:** 6\ **Last updated:** [March 21, 2019, 4:29pm UTC](https://discuss.elastic.co/t/forward-syslogs-to-another-siem/173045 "2019-03-21T16:29:28Z")
I have several machines that use filebeats to send syslog messages to the SIEM. I'd like to send these logs to another external machine besides the local machine. How can I do this? --- ## [Logstash error : elasticsearch host unreachable](https://discuss.elastic.co/t/logstash-error-elasticsearch-host-unreachable/173080)
**Author:** [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)\ **Replies:** 8\ **Last updated:** [March 21, 2019, 2:38pm UTC](https://discuss.elastic.co/t/logstash-error-elasticsearch-host-unreachable/173080 "2019-03-21T14:38:04Z")
Hello Community, I have a logstash running in a virtual box Centos system and purpose is for collecting a log file and parsing and sending it to elasticsearch in my other machine in a virtual box. I get \<\> not reachabl… --- ## [Reload pipeline from docker image](https://discuss.elastic.co/t/reload-pipeline-from-docker-image/169599)
**Author:** [@Hyllegaard](https://discuss.elastic.co/u/Hyllegaard)\ **Replies:** 2\ **Last updated:** [March 21, 2019, 1:06pm UTC](https://discuss.elastic.co/t/reload-pipeline-from-docker-image/169599 "2019-03-21T13:06:18Z")
Hi. we are using the Docker version of logstash. Is it possible to make it reload the configuration from the docker mounted volume. i tried using -e "CONFIG\_RELOAD\_AUTOMATiC=true" on the docker run command, but that doe… --- ## [IS elapsed Filter needs to be in single thread](https://discuss.elastic.co/t/is-elapsed-filter-needs-to-be-in-single-thread/173144)
**Author:** [@kcmenon](https://discuss.elastic.co/u/kcmenon)\ **Replies:** 3\ **Last updated:** [March 21, 2019, 12:03pm UTC](https://discuss.elastic.co/t/is-elapsed-filter-needs-to-be-in-single-thread/173144 "2019-03-21T12:03:20Z")
Is the elapsed filter plugin needs to be in single pipeline? I want to move the application to production soon. So don't have much data for verification. Do we need to specify single pipeline for elapsed to work. i.e. D… --- ## [Jdbc\_streaming with nested elements](https://discuss.elastic.co/t/jdbc-streaming-with-nested-elements/172208)
**Author:** [@bigster](https://discuss.elastic.co/u/bigster)\ **Replies:** 9\ **Last updated:** [March 21, 2019, 11:47am UTC](https://discuss.elastic.co/t/jdbc-streaming-with-nested-elements/172208 "2019-03-21T11:47:39Z")
Hi, I have 3 tables that has the described connections: A-\>B-\>C Basically i have FK from A to B and B to C. I'm trying to put all the data into a unique document with a list of B inside A and a list of C inside B. T… --- ## [How to differentiate log files sended from filebeat?](https://discuss.elastic.co/t/how-to-differentiate-log-files-sended-from-filebeat/173139)
**Author:** [@Deny7](https://discuss.elastic.co/u/Deny7)\ **Replies:** 3\ **Last updated:** [March 21, 2019, 9:28am UTC](https://discuss.elastic.co/t/how-to-differentiate-log-files-sended-from-filebeat/173139 "2019-03-21T09:28:34Z")
Hi, I'm trying to configure one logstash pipeline for my enviroments(test,dev,prd,acp). From each server I want to send log three log files(alfresco.log,share.log,access.log) using Filebeat. So my understanding is that … --- ## [Getting error while running .conf file](https://discuss.elastic.co/t/getting-error-while-running-conf-file/173262)
**Author:** [@RoWesley](https://discuss.elastic.co/u/RoWesley)\ **Replies:** 2\ **Last updated:** [March 21, 2019, 8:20am UTC](https://discuss.elastic.co/t/getting-error-while-running-conf-file/173262 "2019-03-21T08:20:12Z")
Below is the error while i run .CONF file logstash -f logconfig.conf C:\\Users\\Priyesh.Chourasia\\Desktop\\ElkOld\\logstash-6.5.2\\bin\>logstash -f logconf ig.conf Sending Logstash logs to C:/Users/Priyesh.Chourasia/Desktop… --- ## [Logstash Ruby Error with Local built Gem](https://discuss.elastic.co/t/logstash-ruby-error-with-local-built-gem/173256)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 0\ **Last updated:** [March 21, 2019, 7:11am UTC](https://discuss.elastic.co/t/logstash-ruby-error-with-local-built-gem/173256 "2019-03-21T07:11:23Z")
Hi- I have added the locally built gem in the logstash gem file, and did the bundle install and updated the plugins and so I could see the gem has been added to the gemfile.lock I have a .dll files which has been added… --- ## [Logstash ES-\>ES more documents out than in?](https://discuss.elastic.co/t/logstash-es-es-more-documents-out-than-in/171565)
**Author:** [@jrandombob](https://discuss.elastic.co/u/jrandombob)\ **Replies:** 4\ **Last updated:** [March 21, 2019, 6:41am UTC](https://discuss.elastic.co/t/logstash-es-es-more-documents-out-than-in/171565 "2019-03-21T06:41:26Z")
Hi All, Looking for a little guidance here, I've got a static ES index with 2.7 billion documents in it, I'm using Logstash to re-index that to a new empty index with an added fingerprint field so I can deduplicate the … --- ## [Logstash Fails To Startup](https://discuss.elastic.co/t/logstash-fails-to-startup/173022)
**Author:** [@drod](https://discuss.elastic.co/u/drod)\ **Replies:** 2\ **Last updated:** [March 21, 2019, 1:34am UTC](https://discuss.elastic.co/t/logstash-fails-to-startup/173022 "2019-03-21T01:34:11Z")
When I try to start up my logstash agent with a basic config, I get the following. \[ERROR\] 2019-03-19 16:50:26.469 \[main\] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (Loa… --- ## [Logstash SSL File does not contain a valid private key with Beats](https://discuss.elastic.co/t/logstash-ssl-file-does-not-contain-a-valid-private-key-with-beats/173229)
**Author:** [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Replies:** 1\ **Last updated:** [March 20, 2019, 11:24pm UTC](https://discuss.elastic.co/t/logstash-ssl-file-does-not-contain-a-valid-private-key-with-beats/173229 "2019-03-20T23:24:42Z")
I am trying to setup filebeat to use SSL, but I keep getting errors on the logstash server. I keep receiving an error when I have my input on logstash set to Beats. java.lang.IllegalArgumentException: File does not con… --- ## [How to add data to Elasticsearch DB as per log line timestamp](https://discuss.elastic.co/t/how-to-add-data-to-elasticsearch-db-as-per-log-line-timestamp/171207)
**Author:** [@dips](https://discuss.elastic.co/u/dips)\ **Replies:** 6\ **Last updated:** [March 20, 2019, 10:36pm UTC](https://discuss.elastic.co/t/how-to-add-data-to-elasticsearch-db-as-per-log-line-timestamp/171207 "2019-03-20T22:36:12Z")
Hello, I am pushing some log files to Elasticsearch DB every day. For example: on 6th March morning, I will push some log files with log lines: 03-02 10:21:13.042 227 I Test: Invalid Solid Color Type 03-02 10:21:13.04… --- ## [New index not created](https://discuss.elastic.co/t/new-index-not-created/173209)
**Author:** [@teamg](https://discuss.elastic.co/u/teamg)\ **Replies:** 2\ **Last updated:** [March 20, 2019, 7:36pm UTC](https://discuss.elastic.co/t/new-index-not-created/173209 "2019-03-20T19:36:28Z")
I want two of my hosts to write to a different index. Based on the articles I've read on here this output statement in /etc/logstash/conf.d/input.conf should create a new index when it finds these two hosts....but it's … --- ## [Access Plugin variable inside logstash ruby filter](https://discuss.elastic.co/t/access-plugin-variable-inside-logstash-ruby-filter/173196)
**Author:** [@Makra](https://discuss.elastic.co/u/Makra)\ **Replies:** 1\ **Last updated:** [March 20, 2019, 6:39pm UTC](https://discuss.elastic.co/t/access-plugin-variable-inside-logstash-ruby-filter/173196 "2019-03-20T18:39:41Z")
Hi I want to access plugin variable inside ruby filter. For example, the logstash CSV plugin makes use of variable @columns. Is it possible to access the variable inside ruby filter ? I tried with using the variable in… --- ## [Using Centralized Pipeline Management and Multiple Pipelines](https://discuss.elastic.co/t/using-centralized-pipeline-management-and-multiple-pipelines/172998)
**Author:** [@jvsell](https://discuss.elastic.co/u/jvsell)\ **Replies:** 3\ **Last updated:** [March 20, 2019, 6:39pm UTC](https://discuss.elastic.co/t/using-centralized-pipeline-management-and-multiple-pipelines/172998 "2019-03-20T18:39:17Z")
Hi everyone, I am trying to use multiple pipelines using the centralized pipeline management from Kibana, but without successful, it possible to do what I'm trying to do? If yes, how? I don't find any topic or document… --- ## [Using cert base authentication with Elasticsearch filter plugin](https://discuss.elastic.co/t/using-cert-base-authentication-with-elasticsearch-filter-plugin/173206)
**Author:** [@Jason\_McFarland](https://discuss.elastic.co/u/Jason_McFarland)\ **Replies:** 0\ **Last updated:** [March 20, 2019, 6:35pm UTC](https://discuss.elastic.co/t/using-cert-base-authentication-with-elasticsearch-filter-plugin/173206 "2019-03-20T18:35:39Z")
We have successfully configured the Elasticsearch output plugin with cert based authentication for a while now. We now have a use case where we want to do log enrichment with the Elasticsearch filter plugin but I can't f… --- ## [Graylog not receiving messages sent with logstash gelf output, both running on Docker](https://discuss.elastic.co/t/graylog-not-receiving-messages-sent-with-logstash-gelf-output-both-running-on-docker/173199)
**Author:** [@strike](https://discuss.elastic.co/u/strike)\ **Replies:** 0\ **Last updated:** [March 20, 2019, 5:49pm UTC](https://discuss.elastic.co/t/graylog-not-receiving-messages-sent-with-logstash-gelf-output-both-running-on-docker/173199 "2019-03-20T17:49:14Z")
I have Logastash, Graylog, Elasticsearch and Mongodb each one running as docker service (I use docker-compose). I use logstash jdbc input plugin to retreive the logs from database table and gelf output to send logs to gr… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=579) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=581)