# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1008 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1009 --- ## [Logstash with x-pack authentication problems](https://discuss.elastic.co/t/logstash-with-x-pack-authentication-problems/77897)
**Author:** [@cisaksen](https://discuss.elastic.co/u/cisaksen)\ **Replies:** 9\ **Last updated:** [March 31, 2017, 1:44am UTC](https://discuss.elastic.co/t/logstash-with-x-pack-authentication-problems/77897 "2017-03-31T01:44:14Z")
I have x-pack installed and I keep getting errors in the logstash log WARN \]\[logstash.outputs.elasticsearch\] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>#\ **Author:** [@siq992](https://discuss.elastic.co/u/siq992)\ **Replies:** 3\ **Last updated:** [March 31, 2017, 12:37am UTC](https://discuss.elastic.co/t/cannot-start-logstash-using-bin-logstash-bat-f-logstash-lab-conf-r/80723 "2017-03-31T00:37:11Z") I am a newbie. I am trying to be as clear as I can. A team member recently attended elastic conference and brought back a zipped folder containing some training material and elastic stack 5.2.0. I am trying to follow th… --- ## [Potential data deletion issue when using the offline plugin pack command](https://discuss.elastic.co/t/potential-data-deletion-issue-when-using-the-offline-plugin-pack-command/80714)
**Author:** [@Vittorio](https://discuss.elastic.co/u/Vittorio)\ **Replies:** 2\ **Last updated:** [March 30, 2017, 8:11pm UTC](https://discuss.elastic.co/t/potential-data-deletion-issue-when-using-the-offline-plugin-pack-command/80714 "2017-03-30T20:11:08Z")
this for let everybody know how beautiful logstash is ! running this command bin/logstash-plugin prepare-offline-pack --output OUTPUT \[PLUGINS\] and specify as output =\> /home/vittorio , the result is just amazing.… --- ## [Field data from match results](https://discuss.elastic.co/t/field-data-from-match-results/80524)
**Author:** [@anshul\_pandey](https://discuss.elastic.co/u/anshul_pandey)\ **Replies:** 3\ **Last updated:** [March 30, 2017, 7:17pm UTC](https://discuss.elastic.co/t/field-data-from-match-results/80524 "2017-03-30T19:17:39Z")
I have following that matches incoming rabbit mq data, match=\> { "full\_message" =\> "(?\[^;\])\[;\](?\[^;\])\[;\]%{NUMBER:payloadSize};\[;\]%{GREEDYDATA:message\_text}" } My requirement is to extract more fields out of the message… --- ## [Installation problems with Logstash 5.3.0 rpm's](https://discuss.elastic.co/t/installation-problems-with-logstash-5-3-0-rpms/80717)
**Author:** [@nolan.peters](https://discuss.elastic.co/u/nolan.peters)\ **Replies:** 2\ **Last updated:** [March 30, 2017, 7:15pm UTC](https://discuss.elastic.co/t/installation-problems-with-logstash-5-3-0-rpms/80717 "2017-03-30T19:15:09Z")
Is anyone else having issues with installing the latest Kibana download of 5.3.0? I tried twice with two different downloads and both get an unpacking error. Is it just me or is the RPM broke? rpm -Uvh logstash-5.3.… --- ## [Logstash - Startup Script / Service Daemon](https://discuss.elastic.co/t/logstash-startup-script-service-daemon/80690)
**Author:** [@urbans](https://discuss.elastic.co/u/urbans)\ **Replies:** 4\ **Last updated:** [March 30, 2017, 6:37pm UTC](https://discuss.elastic.co/t/logstash-startup-script-service-daemon/80690 "2017-03-30T18:37:26Z")
Been trying to figure out how to get the right config to work on the /etc/init/logstash.conf file. That is the one that can be modified using the system-install. But it never does any of my settings, therefore the por… --- ## [Date issues with Logstash](https://discuss.elastic.co/t/date-issues-with-logstash/80715)
**Author:** [@Riaz\_Mohamed](https://discuss.elastic.co/u/Riaz_Mohamed)\ **Replies:** 1\ **Last updated:** [March 30, 2017, 5:12pm UTC](https://discuss.elastic.co/t/date-issues-with-logstash/80715 "2017-03-30T17:12:19Z")
All I have the following log \[Feb 15, 2017 09:35:43 AM GMT, ERROR\] UpdateForecastFieldsFromTemplatePostPX.java:doAction():359) My gork expression looks like %{TIMESTAMP\_ISO8601:logdate}%{LOGLEVEL:log-level}\]%{GREEDY… --- ## [Refactoring logstash config](https://discuss.elastic.co/t/refactoring-logstash-config/80502)
**Author:** [@rahul.dhammy](https://discuss.elastic.co/u/rahul.dhammy)\ **Replies:** 6\ **Last updated:** [March 30, 2017, 3:16pm UTC](https://discuss.elastic.co/t/refactoring-logstash-config/80502 "2017-03-30T15:16:37Z")
Being a OO developer and focusing on code maintainability and quality, I want to know is there way I can make my logstash config a bit nicer. For instance below is my config file and I hate to see those if statements. Al… --- ## [Move data from RRD(Round Robin Database) to Elasticsearch](https://discuss.elastic.co/t/move-data-from-rrd-round-robin-database-to-elasticsearch/63929)
**Author:** [@cb2015](https://discuss.elastic.co/u/cb2015)\ **Replies:** 3\ **Last updated:** [March 30, 2017, 2:00pm UTC](https://discuss.elastic.co/t/move-data-from-rrd-round-robin-database-to-elasticsearch/63929 "2017-03-30T14:00:53Z")
Hi Everyone, So I have been using Nagios/RRD for many years to collect/store data about my machines. I would like to be able to move some of this data over to elasticsearch so that I can do some analysis on it. Does… --- ## [SNMPTRAP OCTET String to String](https://discuss.elastic.co/t/snmptrap-octet-string-to-string/80689)
**Author:** [@xiaozhuqq48](https://discuss.elastic.co/u/xiaozhuqq48)\ **Replies:** 0\ **Last updated:** [March 30, 2017, 1:52pm UTC](https://discuss.elastic.co/t/snmptrap-octet-string-to-string/80689 "2017-03-30T13:52:02Z")
When I use SNMPTRAP to get SNMP messages. The message includes below "\\xE9\\x80\\x9A\\xE4\\xBF\\xA1\\xE8\\xB4\\xA8\\xE9\\x87\\x8F", which has been confirmed the type 'utf8' through the filter ruby command 'puts event.get('message')… --- ## [Elastic REST API calls?](https://discuss.elastic.co/t/elastic-rest-api-calls/80200)
**Author:** [@jcannell](https://discuss.elastic.co/u/jcannell)\ **Replies:** 2\ **Last updated:** [March 30, 2017, 1:32pm UTC](https://discuss.elastic.co/t/elastic-rest-api-calls/80200 "2017-03-30T13:32:13Z")
Hello! I'm looking for an Elastic solution that can: 1. Periodically send a web request to a REST API 2. Receive the JSON response and 3. Send it to logstash for processing Is there something offered in Elastic that … --- ## [Logstash if else](https://discuss.elastic.co/t/logstash-if-else/80672)
**Author:** [@Ravi\_Shanker\_Reddy](https://discuss.elastic.co/u/Ravi_Shanker_Reddy)\ **Replies:** 7\ **Last updated:** [March 30, 2017, 1:06pm UTC](https://discuss.elastic.co/t/logstash-if-else/80672 "2017-03-30T13:06:44Z")
I am pushing some example lines to check my if else. But Its not working as expected. Here is my configuaration input { stdin{} } filter { if "cdrs" in \[tags\] { json { source =\> "message" } csv { separat… --- ## [How to pass ruby variable to logstash field?](https://discuss.elastic.co/t/how-to-pass-ruby-variable-to-logstash-field/80664)
**Author:** [@laptop\_data](https://discuss.elastic.co/u/laptop_data)\ **Replies:** 4\ **Last updated:** [March 30, 2017, 12:57pm UTC](https://discuss.elastic.co/t/how-to-pass-ruby-variable-to-logstash-field/80664 "2017-03-30T12:57:16Z")
Hi, I'm using ruby filter in my logstash conf file where it will produce some output into a variable which is json data, so that i will filter that data using json filter. But the problem is i can't able to pass that var… --- ## [Logstash 5.2.2 offline install on Windows - Can't find file](https://discuss.elastic.co/t/logstash-5-2-2-offline-install-on-windows-cant-find-file/77542)
**Author:** [@franhoeyemac](https://discuss.elastic.co/u/franhoeyemac)\ **Replies:** 2\ **Last updated:** [March 30, 2017, 12:48pm UTC](https://discuss.elastic.co/t/logstash-5-2-2-offline-install-on-windows-cant-find-file/77542 "2017-03-30T12:48:57Z")
I'm trying to setup logstash on a server with no internet access to read windows event logs I've followed the instructions here - https://www.elastic.co/guide/en/logstash/current/offline-plugins.html Installed logst… --- ## [File output overwrite?](https://discuss.elastic.co/t/file-output-overwrite/80651)
**Author:** [@Mouss](https://discuss.elastic.co/u/Mouss)\ **Replies:** 1\ **Last updated:** [March 30, 2017, 10:13am UTC](https://discuss.elastic.co/t/file-output-overwrite/80651 "2017-03-30T10:13:57Z")
I would like to use the file output to overwrite an existing file. Right now the new content is appended at the end of the existing file which keeps this file growing --- ## [Datetime coming wrongly in logstash](https://discuss.elastic.co/t/datetime-coming-wrongly-in-logstash/80622)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 1\ **Last updated:** [March 30, 2017, 9:49am UTC](https://discuss.elastic.co/t/datetime-coming-wrongly-in-logstash/80622 "2017-03-30T09:49:24Z")
I am taking jdbc input and http output. in my database datefiled stored as but when i am posting this to http in logstash it is giving me as below i can say it is adding +05:30. i want as it is from database How c… --- ## [CHecking for existance of a field in Ruby filter?](https://discuss.elastic.co/t/checking-for-existance-of-a-field-in-ruby-filter/80527)
**Author:** [@Grigory\_Shamov](https://discuss.elastic.co/u/Grigory_Shamov)\ **Replies:** 1\ **Last updated:** [March 30, 2017, 9:35am UTC](https://discuss.elastic.co/t/checking-for-existance-of-a-field-in-ruby-filter/80527 "2017-03-30T09:35:40Z")
Hi, with the new Ruby event.Set/Get syntax, is there a way to check for the variable being defined? Trying to event.get for a field that does not exist (say, coming from an optional field in grok) makes a Ruby exceptio… --- ## [Iteration and loop in logstash](https://discuss.elastic.co/t/iteration-and-loop-in-logstash/80485)
**Author:** [@Mohamed](https://discuss.elastic.co/u/Mohamed)\ **Replies:** 4\ **Last updated:** [March 30, 2017, 8:05am UTC](https://discuss.elastic.co/t/iteration-and-loop-in-logstash/80485 "2017-03-30T08:05:02Z")
Hello, I have a list data in Logstash that has been parsed as that looks like "Attribute": \[ "\\N1\\Clement\\", "\\N2\\Bundle Name\\<… --- ## [Multiline doesn't read last line](https://discuss.elastic.co/t/multiline-doesnt-read-last-line/80621)
**Author:** [@bertvervaele](https://discuss.elastic.co/u/bertvervaele)\ **Replies:** 2\ **Last updated:** [March 30, 2017, 8:43am UTC](https://discuss.elastic.co/t/multiline-doesnt-read-last-line/80621 "2017-03-30T08:43:28Z")
I know this is a known problem but I can't find I there is already a solution to it. So I want to read the oracle log file. This looks like this: Sat Apr 09 10:13:17 2016 Archived Log entry 24018 added for thread 1 se… --- ## [How long will Logstash recollect data?](https://discuss.elastic.co/t/how-long-will-logstash-recollect-data/80587)
**Author:** [@Zhong\_Guan](https://discuss.elastic.co/u/Zhong_Guan)\ **Replies:** 4\ **Last updated:** [March 30, 2017, 8:31am UTC](https://discuss.elastic.co/t/how-long-will-logstash-recollect-data/80587 "2017-03-30T08:31:07Z")
Hi, I ran logstash to collect log file from azure storage. I noticed logstash will start to collect log every hour? Is it true? Could you please tell me how to shorten it? Regards, Zhong --- ## [Wait until there is a specific message](https://discuss.elastic.co/t/wait-until-there-is-a-specific-message/80600)
**Author:** [@yanivo](https://discuss.elastic.co/u/yanivo)\ **Replies:** 4\ **Last updated:** [March 30, 2017, 7:50am UTC](https://discuss.elastic.co/t/wait-until-there-is-a-specific-message/80600 "2017-03-30T07:50:54Z")
Hi, let's say I have the following lines in my logs: Message: msisdn: 111111111 Message: msisdn: 222222222 Answer: msisdn: 111111111 Answer: msisdn: 222222222 Now, whenever I get a Message, I'd like to wait for X se… --- ## [Invalid Config File Error for Logstash](https://discuss.elastic.co/t/invalid-config-file-error-for-logstash/80592)
**Author:** [@ganesh3](https://discuss.elastic.co/u/ganesh3)\ **Replies:** 3\ **Last updated:** [March 30, 2017, 7:07am UTC](https://discuss.elastic.co/t/invalid-config-file-error-for-logstash/80592 "2017-03-30T07:07:46Z")
Hi, I am trying to run salesforce integration with logstash. I am using the steps mentioned at https://logz.io/blog/analyze-salesforce-elk-stack/ to setup. I am on Ubuntu 16.04 and have only installed logstack. On… --- ## [Defining top-level key for grok matches?](https://discuss.elastic.co/t/defining-top-level-key-for-grok-matches/80549)
**Author:** [@mrbanzai](https://discuss.elastic.co/u/mrbanzai)\ **Replies:** 1\ **Last updated:** [March 30, 2017, 5:27am UTC](https://discuss.elastic.co/t/defining-top-level-key-for-grok-matches/80549 "2017-03-30T05:27:10Z")
I'm attempting to clean-up our Logstash configs to manage ingestion from numerous disparate sources (forwarded from other Logstash clients, filebeat, winlogbeat, etc.), and I encountered an issue with using some of the d… --- ## [Speed up when logstash is running](https://discuss.elastic.co/t/speed-up-when-logstash-is-running/80383)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 5\ **Last updated:** [March 30, 2017, 5:15am UTC](https://discuss.elastic.co/t/speed-up-when-logstash-is-running/80383 "2017-03-30T05:15:28Z")
Question1: My Current CPU utilization is only 10% in my 36-core aws server, how can i increase it to say 80%? I am using logstash and ES 5.1.1. ES is a standard server with 4 nodes. Input is S3 buckets, with thousands… --- ## [Wrong source file name](https://discuss.elastic.co/t/wrong-source-file-name/80344)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 1\ **Last updated:** [March 30, 2017, 1:11am UTC](https://discuss.elastic.co/t/wrong-source-file-name/80344 "2017-03-30T01:11:13Z")
The file name in the OOB source field is wrong. I see that in the output of the logstash. For example the message "\\n|I |17:03:22 |41 |server |Correlati......" is coming from source file ICSServerManager\_stdout.log0006… --- ## [Sincedb file movable?](https://discuss.elastic.co/t/sincedb-file-movable/80385)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 1\ **Last updated:** [March 30, 2017, 1:07am UTC](https://discuss.elastic.co/t/sincedb-file-movable/80385 "2017-03-30T01:07:26Z")
If i just upload my sincedb file from one aws instance server to another server to /$home, does logstash continue to recognize it and parse from the specific point in the file? --- ## [Can't charging data on elasticsearch](https://discuss.elastic.co/t/cant-charging-data-on-elasticsearch/80447)
**Author:** [@cibermatteo98](https://discuss.elastic.co/u/cibermatteo98)\ **Replies:** 1\ **Last updated:** [March 30, 2017, 1:06am UTC](https://discuss.elastic.co/t/cant-charging-data-on-elasticsearch/80447 "2017-03-30T01:06:18Z")
this is the error that logstash gave us 11:10:50.063 \[Ruby-0-Thread-5: C:/Users/matte/Desktop/dati/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch… --- ## [Handling running time duration format "ddd:hh:mm:ss"](https://discuss.elastic.co/t/handling-running-time-duration-format-ddd-hh-mm-ss/80395)
**Author:** [@bigtoerag](https://discuss.elastic.co/u/bigtoerag)\ **Replies:** 3\ **Last updated:** [March 30, 2017, 12:23am UTC](https://discuss.elastic.co/t/handling-running-time-duration-format-ddd-hh-mm-ss/80395 "2017-03-30T00:23:37Z")
Has anyone needed to handle a custom duration field from servers showing running time? I am just starting out and trying to get a handle on the best way to handle it without creating a grok mutate mess to convert to sec… --- ## [Log4j Logstash plugin not writing to stdout](https://discuss.elastic.co/t/log4j-logstash-plugin-not-writing-to-stdout/80560)
**Author:** [@ab-agni](https://discuss.elastic.co/u/ab-agni)\ **Replies:** 0\ **Last updated:** [March 29, 2017, 11:46pm UTC](https://discuss.elastic.co/t/log4j-logstash-plugin-not-writing-to-stdout/80560 "2017-03-29T23:46:58Z")
Hello, Here is my config: input { log4j { mode =\> "server" host =\> "0.0.0.0" port =\> 3456 type =\> "log4j" } } output { stdout { } } log4j.properties: log4j.rootLogger=debug,tcp l… --- ## [Speed up processing of logs](https://discuss.elastic.co/t/speed-up-processing-of-logs/80062)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 6\ **Last updated:** [March 29, 2017, 5:19am UTC](https://discuss.elastic.co/t/speed-up-processing-of-logs/80062 "2017-03-29T05:19:57Z")
Hi I am using logstash 5.1.1 to parse file size of 2-3G (7-8M records) and there are thousands of them from the S3 bucket. My current speed is barely one file/day (260 logs for sec), in m3-xlarge aws instance(4 cpu, 15G … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1007) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1009)