# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=885 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 886 --- ## [Azure WebApps Log ingestion to LogStash](https://discuss.elastic.co/t/azure-webapps-log-ingestion-to-logstash/105501)
**Author:** [@Sandeep\_Pokala](https://discuss.elastic.co/u/Sandeep_Pokala)\ **Replies:** 0\ **Last updated:** [October 27, 2017, 2:17am UTC](https://discuss.elastic.co/t/azure-webapps-log-ingestion-to-logstash/105501 "2017-10-27T02:17:47Z")
Hello, Is there anyway to ingest logs to logstash ( On-Premise data center) from Azure Webapps blob containers ( PAAS Environment ). I tried azure blob plugin from github but it didnt worked out. Appreciate if someone… --- ## [Failed to start logstash5.6.3 after I terminated it](https://discuss.elastic.co/t/failed-to-start-logstash5-6-3-after-i-terminated-it/105496)
**Author:** [@lwolf86](https://discuss.elastic.co/u/lwolf86)\ **Replies:** 0\ **Last updated:** [October 27, 2017, 1:25am UTC](https://discuss.elastic.co/t/failed-to-start-logstash5-6-3-after-i-terminated-it/105496 "2017-10-27T01:25:08Z")
I'm using logstash5.6.3 to dealing logs, after it show a retryable error which I post it here, then I stop it. But I can't start logstash again, show a error log like this: \[2017-10-27T08:59:00,544\]\[ERROR\]\[logstash.pip… --- ## [After upgrade to 5.6.3 Logstash processing much slower](https://discuss.elastic.co/t/after-upgrade-to-5-6-3-logstash-processing-much-slower/105492)
**Author:** [@JJerpseth](https://discuss.elastic.co/u/JJerpseth)\ **Replies:** 0\ **Last updated:** [October 27, 2017, 12:42am UTC](https://discuss.elastic.co/t/after-upgrade-to-5-6-3-logstash-processing-much-slower/105492 "2017-10-27T00:42:51Z")
After updating from V1.7 to V5.6.3, my ELK with Filebeat web log cluster is having performance problems processing the log files. I am hoping there are some configuration changes to Logstash or Filebeat that I can make … --- ## [Kv filter - Fortigate Logs](https://discuss.elastic.co/t/kv-filter-fortigate-logs/105281)
**Author:** [@dev\_disola](https://discuss.elastic.co/u/dev_disola)\ **Replies:** 4\ **Last updated:** [October 27, 2017, 12:03am UTC](https://discuss.elastic.co/t/kv-filter-fortigate-logs/105281 "2017-10-27T00:03:36Z")
Hello everyone, I met ELK a short time ago and started to analyze logs generated from a Fortigate switch. The message is coming in the following format: \<189\>date = 2017-10-25 time = 14:25:25 devname = ELSEN01 ... I'm… --- ## [Keep getting failed to load template error](https://discuss.elastic.co/t/keep-getting-failed-to-load-template-error/105481)
**Author:** [@magdieldominguez](https://discuss.elastic.co/u/magdieldominguez)\ **Replies:** 0\ **Last updated:** [October 26, 2017, 9:33pm UTC](https://discuss.elastic.co/t/keep-getting-failed-to-load-template-error/105481 "2017-10-26T21:33:29Z")
I am trying to send information from a logfile to elasticsearch using logstash. I cant seem to find the solution to this error. Any help would be greatly appreciated. The following is my config file: input { file { p… --- ## [Conditional formatting of JSON or serialized JSON](https://discuss.elastic.co/t/conditional-formatting-of-json-or-serialized-json/105131)
**Author:** [@danielmcquillen](https://discuss.elastic.co/u/danielmcquillen)\ **Replies:** 7\ **Last updated:** [October 26, 2017, 9:17pm UTC](https://discuss.elastic.co/t/conditional-formatting-of-json-or-serialized-json/105131 "2017-10-26T21:17:59Z")
Apologies if this is obvious. I'm pulling Open edX logs from AWS CloudWatch using the cloudwatch\_logs plugin (awesome!). I have an "event" key in my incoming JSON, which contains arbitrary data: sometimes the value is a… --- ## [Copy field and gsub it](https://discuss.elastic.co/t/copy-field-and-gsub-it/105141)
**Author:** [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\ **Replies:** 7\ **Last updated:** [October 26, 2017, 9:21am UTC](https://discuss.elastic.co/t/copy-field-and-gsub-it/105141 "2017-10-26T09:21:46Z")
I am trying to remove the translate filter to get this going, the original type is number so I'm not sure if i need the convert section, I have used gsub with strings without a problem. Can this be done? The new field i… --- ## [Filebeat kafka logstash configuration](https://discuss.elastic.co/t/filebeat-kafka-logstash-configuration/105459)
**Author:** [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\ **Replies:** 0\ **Last updated:** [October 26, 2017, 5:27pm UTC](https://discuss.elastic.co/t/filebeat-kafka-logstash-configuration/105459 "2017-10-26T17:27:33Z")
Hi, I am trying to configure ELK cluster with kafka. I am using filebeat to ship the logs to kafka and logstash to grep the logs from kafka, process the logs and send it to elasticsearch. I need to find out a way to per… --- ## [How does Logstash behave when Input file content is deleted?](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 9\ **Last updated:** [October 26, 2017, 4:19pm UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570 "2017-10-26T16:19:53Z")
Hi, I have a Logstash setup where it reads file(Linux server logs) as an input. But for every 5 days, content in that file will be deleted and new logs will be placed in the input file. Will Logstash be reading lines fr… --- ## [Logstash Kafka input plugin does not consume data](https://discuss.elastic.co/t/logstash-kafka-input-plugin-does-not-consume-data/105447)
**Author:** [@andybingo](https://discuss.elastic.co/u/andybingo)\ **Replies:** 0\ **Last updated:** [October 26, 2017, 3:35pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-plugin-does-not-consume-data/105447 "2017-10-26T15:35:24Z")
Hi all, Recently, I got into a problem that Logstash Kafka plugin does not consume data. Below is my config file: input { kafka { auto\_offset\_reset =\> "earliest" bootstrap\_servers =\> "kafka-proxy-1.phx02.dev.company… --- ## [Problem understanding the concept of filebeat -\> logstash](https://discuss.elastic.co/t/problem-understanding-the-concept-of-filebeat-logstash/105230)
**Author:** [@sd2j](https://discuss.elastic.co/u/sd2j)\ **Replies:** 5\ **Last updated:** [October 26, 2017, 3:26pm UTC](https://discuss.elastic.co/t/problem-understanding-the-concept-of-filebeat-logstash/105230 "2017-10-26T15:26:11Z")
Well I got everything running .. at least partially. I am able to get logs from filebeat -\> logstash -\> elastic -\> kibana I have to admit i haven't read all documentation - which is way to much for my short time. But I … --- ## [Help for grok RFC3339 pattern](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274)
**Author:** [@stefanocog](https://discuss.elastic.co/u/stefanocog)\ **Replies:** 12\ **Last updated:** [October 26, 2017, 3:20pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274 "2017-10-26T15:20:45Z")
Hi i need use millisecond into syslog file, i have commented out the "RSYSLOG\_TraditionalFileFormat" template fron rsyslog.conf and now i have timestamp in RFC3339 format, i need parse this timestamp but I do not know wh… --- ## [Trying to drop Metricbeat events in Logstash](https://discuss.elastic.co/t/trying-to-drop-metricbeat-events-in-logstash/105095)
**Author:** [@fozboz](https://discuss.elastic.co/u/fozboz)\ **Replies:** 1\ **Last updated:** [October 26, 2017, 2:36pm UTC](https://discuss.elastic.co/t/trying-to-drop-metricbeat-events-in-logstash/105095 "2017-10-26T14:36:32Z")
This Metricbeat event is not indexable in ElasticSearch due to the "18446744073709551615" value. { "@timestamp" =\> 2017-10-24T14:48:53.950Z, "system" =\> { "process" =\> { "memory" =\> { "share" =\> 0… --- ## [Logstash Load Error - Java](https://discuss.elastic.co/t/logstash-load-error-java/104688)
**Author:** [@perkstvr](https://discuss.elastic.co/u/perkstvr)\ **Replies:** 1\ **Last updated:** [October 26, 2017, 1:57pm UTC](https://discuss.elastic.co/t/logstash-load-error-java/104688 "2017-10-26T13:57:44Z")
Can anyone help. I'm running elastic search 5.6.2 and log stash 5.6.3 (and tried 6.0.0) but I'm getting an error when trying to run log stash. Java HotSpot(TM) 64-Bit Server VM warning: Option UseParNewGC was deprecate… --- ## [Kafka Input Logging (And Not Consuming)](https://discuss.elastic.co/t/kafka-input-logging-and-not-consuming/105419)
**Author:** [@directRoot](https://discuss.elastic.co/u/directRoot)\ **Replies:** 0\ **Last updated:** [October 26, 2017, 11:44am UTC](https://discuss.elastic.co/t/kafka-input-logging-and-not-consuming/105419 "2017-10-26T11:44:49Z")
Hi all, I'm attempting to get the Kafka input working, with no success so far! I'd like to see some more logging from it and noticed on its GitHub page... Kafka logs do not respect the Log4J2 root logger level and de… --- ## [Logstash Jmeter Date calculation Epoch to Humandate](https://discuss.elastic.co/t/logstash-jmeter-date-calculation-epoch-to-humandate/105407)
**Author:** [@sri535](https://discuss.elastic.co/u/sri535)\ **Replies:** 0\ **Last updated:** [October 26, 2017, 10:57am UTC](https://discuss.elastic.co/t/logstash-jmeter-date-calculation-epoch-to-humandate/105407 "2017-10-26T10:57:35Z")
HI by using elk preparing Dashboard for Jmeter Lt results.Jmeter writes result to file by using Flexible file writer . i configured That textfile /csv to Logstash but the problem is it had startTime and EndTime variabl… --- ## [Ruby exception occurred](https://discuss.elastic.co/t/ruby-exception-occurred/105073)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 1\ **Last updated:** [October 26, 2017, 10:28am UTC](https://discuss.elastic.co/t/ruby-exception-occurred/105073 "2017-10-26T10:28:19Z")
I am getting the error \[2017-10-24T17:20:25,702\]\[ERROR\]\[logstash.filters.ruby \] Ruby exception occurred: invalid byte sequence in UTF-8 My ruby filter is: ruby { # isolate the server name from the… --- ## [How to update new record only](https://discuss.elastic.co/t/how-to-update-new-record-only/105359)
**Author:** [@Shekhar\_Pandey](https://discuss.elastic.co/u/Shekhar_Pandey)\ **Replies:** 3\ **Last updated:** [October 26, 2017, 10:07am UTC](https://discuss.elastic.co/t/how-to-update-new-record-only/105359 "2017-10-26T10:07:46Z")
I used logstash and try to update change data in mysql to elasticsearch.....but it reindex again the whole documents instead of chnage the version number...bcz when we update anything it should has to chnage ther version… --- ## [How to remove a filed](https://discuss.elastic.co/t/how-to-remove-a-filed/105173)
**Author:** [@raghu1](https://discuss.elastic.co/u/raghu1)\ **Replies:** 7\ **Last updated:** [October 26, 2017, 10:05am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173 "2017-10-26T10:05:25Z")
Hi Team, Please can you suggest how to remove field "payload" from below pattern: I am trying using remove\_field =\> \[ "payload" \]. Please suggest. {match =\> { "message" =\> "\[%{GREEDYDATA:BEGIN}\]|%{GREEDYDATA:service\_i… --- ## [Logstash email output plugin issue](https://discuss.elastic.co/t/logstash-email-output-plugin-issue/105248)
**Author:** [@RohanKumbhar](https://discuss.elastic.co/u/RohanKumbhar)\ **Replies:** 4\ **Last updated:** [October 26, 2017, 9:46am UTC](https://discuss.elastic.co/t/logstash-email-output-plugin-issue/105248 "2017-10-26T09:46:34Z")
Getting below error while using email output plugin \[ERROR\]\[logstash.outputs.email \] Something happen while delivering an email {:exception=\>#\} Kindly hel… --- ## [Prevent Logstash logging to /var/log/messages](https://discuss.elastic.co/t/prevent-logstash-logging-to-var-log-messages/105372)
**Author:** [@pavuk](https://discuss.elastic.co/u/pavuk)\ **Replies:** 8\ **Last updated:** [October 26, 2017, 9:32am UTC](https://discuss.elastic.co/t/prevent-logstash-logging-to-var-log-messages/105372 "2017-10-26T09:32:27Z")
Hi, Could you please guide me how to prevent Logstash logging to /var/log/messages? Thanks. --- ## [Use throttle plugin to detect brute force attempt](https://discuss.elastic.co/t/use-throttle-plugin-to-detect-brute-force-attempt/105379)
**Author:** [@Gwen\_P](https://discuss.elastic.co/u/Gwen_P)\ **Replies:** 0\ **Last updated:** [October 26, 2017, 9:10am UTC](https://discuss.elastic.co/t/use-throttle-plugin-to-detect-brute-force-attempt/105379 "2017-10-26T09:10:52Z")
Hi everyone, I would like to detect bruteforce by using the throttle plugin but I'm not really sure that it is possible. For the input I'm using a csv file for the moment that contains logs with timestamp. Have someone … --- ## [How to stop main pipeline anter data is inserted in elasticsearch](https://discuss.elastic.co/t/how-to-stop-main-pipeline-anter-data-is-inserted-in-elasticsearch/105257)
**Author:** [@virahul29](https://discuss.elastic.co/u/virahul29)\ **Replies:** 5\ **Last updated:** [October 26, 2017, 8:40am UTC](https://discuss.elastic.co/t/how-to-stop-main-pipeline-anter-data-is-inserted-in-elasticsearch/105257 "2017-10-26T08:40:46Z")
hi How can i stop the main pipeline from the logstash script after data get inserted from the text file to the elastic index so that i can again run the logstash script --- ## [Schedule to run periodically](https://discuss.elastic.co/t/schedule-to-run-periodically/104554)
**Author:** [@Newuser](https://discuss.elastic.co/u/Newuser)\ **Replies:** 0\ **Last updated:** [October 19, 2017, 12:57pm UTC](https://discuss.elastic.co/t/schedule-to-run-periodically/104554 "2017-10-19T12:57:16Z")
No response --- ## [Timestamp stdout is correct but to es is not](https://discuss.elastic.co/t/timestamp-stdout-is-correct-but-to-es-is-not/105351)
**Author:** [@imqishi](https://discuss.elastic.co/u/imqishi)\ **Replies:** 2\ **Last updated:** [October 26, 2017, 8:21am UTC](https://discuss.elastic.co/t/timestamp-stdout-is-correct-but-to-es-is-not/105351 "2017-10-26T08:21:52Z")
I have a log file which is customized by myself, I want to use timestamp by my log and write config as following shows: input { file { path =\> "/var/log/testwrd.log" start\_position =\> "beginning" sincedb\_pat… --- ## [Error while starting Logstash](https://discuss.elastic.co/t/error-while-starting-logstash/105268)
**Author:** [@yoga-sara](https://discuss.elastic.co/u/yoga-sara)\ **Replies:** 3\ **Last updated:** [October 26, 2017, 7:55am UTC](https://discuss.elastic.co/t/error-while-starting-logstash/105268 "2017-10-26T07:55:07Z")
Hello, I'm having ELK 5.6.2.My environment like below, FileBeat(Machine A) =\> Logstash (Machine B) =\> Elasticsearch (Machine B) =\> Kibana (Machine B) I have configured Logstash as output. #--------------------------… --- ## [Logstash is not parsing one-day date format](https://discuss.elastic.co/t/logstash-is-not-parsing-one-day-date-format/105360)
**Author:** [@Igor\_Kozyrev](https://discuss.elastic.co/u/Igor_Kozyrev)\ **Replies:** 0\ **Last updated:** [October 26, 2017, 7:41am UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-one-day-date-format/105360 "2017-10-26T07:41:52Z")
Hi all, I have a problem with parsing one day format. I have this format of date in a log: Sat Oct 7 17:24:32 2017 and have those lines in config file: \[image\] But nothing is happened. When I change the format to … --- ## [Data Filter issue](https://discuss.elastic.co/t/data-filter-issue/105331)
**Author:** [@cdra](https://discuss.elastic.co/u/cdra)\ **Replies:** 3\ **Last updated:** [October 26, 2017, 7:00am UTC](https://discuss.elastic.co/t/data-filter-issue/105331 "2017-10-26T07:00:11Z")
We have some switches that send syslog messages like: Oct 25 16:35:05 2017 wsscc000 %%10SHELL/6/SHELL\_CMD(l): -Task=vt0-IPAddr=10.61.7.10-User=afg; Command is sys In this case, logstash set syslog\_hostname = 2017 not w… --- ## [Logstash UI](https://discuss.elastic.co/t/logstash-ui/105285)
**Author:** [@Ricigliano](https://discuss.elastic.co/u/Ricigliano)\ **Replies:** 1\ **Last updated:** [October 26, 2017, 5:22am UTC](https://discuss.elastic.co/t/logstash-ui/105285 "2017-10-26T05:22:28Z")
Is there a UI for Logstash configuration and input parms , it seems like something that would be really useful. If we decide to use this we may end up having to create one, otherwise everyone needs to know all this synt… --- ## [Keep getting failed to install template undefined method '\[\]' for nil:NilClass?](https://discuss.elastic.co/t/keep-getting-failed-to-install-template-undefined-method-for-nil-nilclass/105297)
**Author:** [@magdieldominguez](https://discuss.elastic.co/u/magdieldominguez)\ **Replies:** 0\ **Last updated:** [October 25, 2017, 8:43pm UTC](https://discuss.elastic.co/t/keep-getting-failed-to-install-template-undefined-method-for-nil-nilclass/105297 "2017-10-25T20:43:56Z")
I am trying to send information from a logfile to elasticsearch using logstash. I cant seem to find the solution to this error. Any help would be greatly appreciated. The following is my config file: input { file { p… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=884) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=886)