# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=482 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 483 --- ## [Logstash Kafka Output Plugin loss of events or delay in delivering events](https://discuss.elastic.co/t/logstash-kafka-output-plugin-loss-of-events-or-delay-in-delivering-events/198035)
**Author:** [@giorgiogale](https://discuss.elastic.co/u/giorgiogale)\ **Replies:** 2\ **Last updated:** [September 5, 2019, 1:00pm UTC](https://discuss.elastic.co/t/logstash-kafka-output-plugin-loss-of-events-or-delay-in-delivering-events/198035 "2019-09-05T13:00:34Z")
Hi, I have a problem sending events to cluster kafka. The events read in input from a file are not always received on the kafka cluster ( it would seem only the first after the logstash process started). From the log… --- ## [Configuration if \[path\] =~ "access" not working](https://discuss.elastic.co/t/configuration-if-path-access-not-working/197570)
**Author:** [@Senthil\_ak](https://discuss.elastic.co/u/Senthil_ak)\ **Replies:** 6\ **Last updated:** [September 5, 2019, 12:59pm UTC](https://discuss.elastic.co/t/configuration-if-path-access-not-working/197570 "2019-09-05T12:59:44Z")
Hi Team, I am trying to filter apache logs messages, but its always goes to random\_logs than the apache\_access or apache\_error. When i tried to run as adhoc its parsing properly. # Logstash configuration # Beats -\> Log… --- ## [Log parsing by grok filter](https://discuss.elastic.co/t/log-parsing-by-grok-filter/198248)
**Author:** [@emilio](https://discuss.elastic.co/u/emilio)\ **Replies:** 0\ **Last updated:** [September 5, 2019, 12:47pm UTC](https://discuss.elastic.co/t/log-parsing-by-grok-filter/198248 "2019-09-05T12:47:28Z")
Dear all, Please, help me parse it with grok filter: 03-09-2019 00:00:01 \[DEBUG\] {SomeLoging.java:36} \[SOMEUSER:1:9999xxx888yyy-1234567\] - recieved \[Some\_Command, Some\_Command\_Name, SOMEUSER, 2099-01-02 00:00:01, SOME… --- ## [Drop the complete message containing specific strings](https://discuss.elastic.co/t/drop-the-complete-message-containing-specific-strings/197799)
**Author:** [@Klaus.Lucas](https://discuss.elastic.co/u/Klaus.Lucas)\ **Replies:** 8\ **Last updated:** [September 5, 2019, 12:40pm UTC](https://discuss.elastic.co/t/drop-the-complete-message-containing-specific-strings/197799 "2019-09-05T12:40:43Z")
Hello, I have the following definition to drop messages from a log file containing strings and text: input { file { path =\> "/opt/mapr/logs/cldb.log" tags =\> "mapr\_cldb" codec =\> plain {charset =\> "ISO-8859-1"} } … --- ## [Forwarding logs from filebeat to logstash via a proxy server (or something similar)](https://discuss.elastic.co/t/forwarding-logs-from-filebeat-to-logstash-via-a-proxy-server-or-something-similar/198245)
**Author:** [@yotam\_h](https://discuss.elastic.co/u/yotam_h)\ **Replies:** 0\ **Last updated:** [September 5, 2019, 12:20pm UTC](https://discuss.elastic.co/t/forwarding-logs-from-filebeat-to-logstash-via-a-proxy-server-or-something-similar/198245 "2019-09-05T12:20:46Z")
Hi I'm looking for a way to forward my logs with filebeat from one environment to a remote logstash that sits in a different location than my environment. The issue is that the logstash server is closed to outside conn… --- ## [How to split array field in json and send it as separate event](https://discuss.elastic.co/t/how-to-split-array-field-in-json-and-send-it-as-separate-event/198227)
**Author:** [@uma\_rengasamy](https://discuss.elastic.co/u/uma_rengasamy)\ **Replies:** 2\ **Last updated:** [September 5, 2019, 12:13pm UTC](https://discuss.elastic.co/t/how-to-split-array-field-in-json-and-send-it-as-separate-event/198227 "2019-09-05T12:13:08Z")
Please find the debug o/p , We want each logEvents as separate event in kibana, since logEvents will be huge in count and it takes time to load in kibana { "logEvents" =\> \[ \[ 0\] { "message" =\> "id" =\> "timestamp"… --- ## [Confusing error logs when using pipelines.yml](https://discuss.elastic.co/t/confusing-error-logs-when-using-pipelines-yml/198187)
**Author:** [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\ **Replies:** 1\ **Last updated:** [September 5, 2019, 12:01pm UTC](https://discuss.elastic.co/t/confusing-error-logs-when-using-pipelines-yml/198187 "2019-09-05T12:01:27Z")
We have a bunch of pipelines defined in pipelines.yml, and while chasing down some exceptions we're noticing that there's not good way to find out which pipeline is failing, as there's one common logfile for all pipeline… --- ## [Is it possible to have different logstash parse patterns for diffrent logs files linked to the same index?](https://discuss.elastic.co/t/is-it-possible-to-have-different-logstash-parse-patterns-for-diffrent-logs-files-linked-to-the-same-index/198181)
**Author:** [@nick78](https://discuss.elastic.co/u/nick78)\ **Replies:** 3\ **Last updated:** [September 5, 2019, 11:35am UTC](https://discuss.elastic.co/t/is-it-possible-to-have-different-logstash-parse-patterns-for-diffrent-logs-files-linked-to-the-same-index/198181 "2019-09-05T11:35:46Z")
I would like to ask if it is possible to have different logstash parse patterns for different log files to the same index,? What i want to achieve is to have different parse patterns mapping to the same attributes and st… --- ## [How to parse data in message into separate fields](https://discuss.elastic.co/t/how-to-parse-data-in-message-into-separate-fields/198209)
**Author:** [@Nurbek](https://discuss.elastic.co/u/Nurbek)\ **Replies:** 0\ **Last updated:** [September 5, 2019, 9:29am UTC](https://discuss.elastic.co/t/how-to-parse-data-in-message-into-separate-fields/198209 "2019-09-05T09:29:21Z")
This is my data in KIbana input { http\_poller { urls =\> { soap\_request =\> { method =\> post url =\> "URL deleted" headers =\> { "Content-Type" =\> "text/xml; charset=utf-8" "SOAPAction" =\>… --- ## [Logstash ERROR: Unable to retrieve license information from license server {:message=\>"Unrecognized SSL message, plaintext connection?"}](https://discuss.elastic.co/t/logstash-error-unable-to-retrieve-license-information-from-license-server-message-unrecognized-ssl-message-plaintext-connection/195219)
**Author:** [@daniele.saccon](https://discuss.elastic.co/u/daniele.saccon)\ **Replies:** 2\ **Last updated:** [September 5, 2019, 8:50am UTC](https://discuss.elastic.co/t/logstash-error-unable-to-retrieve-license-information-from-license-server-message-unrecognized-ssl-message-plaintext-connection/195219 "2019-09-05T08:50:22Z")
Hi, I have a cluster consisting of 3 nodes and an installation of kibana and logstash, all to version 6.8.1. I followed the procedure to configure SSL, TLS and HTTPS certificates: The Elasticsearch nodes are workin… --- ## [Am facing a problem](https://discuss.elastic.co/t/am-facing-a-problem/198168)
**Author:** [@Chandunadh](https://discuss.elastic.co/u/Chandunadh)\ **Replies:** 1\ **Last updated:** [September 5, 2019, 8:32am UTC](https://discuss.elastic.co/t/am-facing-a-problem/198168 "2019-09-05T08:32:57Z")
Microsoft Windows \[Version 10.0.17763.615\] (c) 2018 Microsoft Corporation. All rights reserved. C:\\chandu ibm\\softwares\\logstash-6.4.2\\bin\>logstash -f logstash-sample.conf Error: Could not find or load main class ibm\\… --- ## [Parsing JSON? output from web service](https://discuss.elastic.co/t/parsing-json-output-from-web-service/198170)
**Author:** [@vspies](https://discuss.elastic.co/u/vspies)\ **Replies:** 1\ **Last updated:** [September 5, 2019, 8:29am UTC](https://discuss.elastic.co/t/parsing-json-output-from-web-service/198170 "2019-09-05T08:29:47Z")
We are polling a security web service to get the scanning and detection logs. This is what I get when I poll the service with logstash: { "current\_link" =\> "https://xxx/v1/siem/security\_events?service=… --- ## [Custom applog filtering using logtash](https://discuss.elastic.co/t/custom-applog-filtering-using-logtash/197972)
**Author:** [@ravi\_varma](https://discuss.elastic.co/u/ravi_varma)\ **Replies:** 2\ **Last updated:** [September 5, 2019, 3:33am UTC](https://discuss.elastic.co/t/custom-applog-filtering-using-logtash/197972 "2019-09-05T03:33:46Z")
I am quite new to elk stack, and need your kind help on how can we do format the below log using logstash filters 2019-08-06 09:13:39,921BUYER-AR : FPX Info FPXTRANSID:1908060 - m\_sNewFPXXml in verifyFPXCheckSum: \ **Author:** [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\ **Replies:** 0\ **Last updated:** [September 5, 2019, 1:25am UTC](https://discuss.elastic.co/t/logstash-output-email-test/198133 "2019-09-05T01:25:45Z") We have an on-premises exchange server that I would like to use to send email notifications from logstash Does anyone have a sample config I could use? I know the ip of the exchange server, port 25 is open and if SMTP … --- ## [Ok, Conf issue, i am missing something stupid](https://discuss.elastic.co/t/ok-conf-issue-i-am-missing-something-stupid/198127)
**Author:** [@Philip\_Thomson1](https://discuss.elastic.co/u/Philip_Thomson1)\ **Replies:** 7\ **Last updated:** [September 5, 2019, 12:37am UTC](https://discuss.elastic.co/t/ok-conf-issue-i-am-missing-something-stupid/198127 "2019-09-05T00:37:33Z")
What's wrong with this conf?: (free beer for anyone who can help, in Guelph Ontario) #Input from Log03 servers for bellMTS input { elasticsearch { host =\> "192.168.76.249" query =\> '{"query":{ "match": {"Web\_Host": … --- ## [Logstash se apaga después de estar ejecutándose unos segundos](https://discuss.elastic.co/t/logstash-se-apaga-despues-de-estar-ejecutandose-unos-segundos/197529)
**Author:** [@auzin](https://discuss.elastic.co/u/auzin)\ **Replies:** 7\ **Last updated:** [September 5, 2019, 12:19am UTC](https://discuss.elastic.co/t/logstash-se-apaga-despues-de-estar-ejecutandose-unos-segundos/197529 "2019-09-05T00:19:38Z")
Hi, Every time I try to run logstash with my config file, it shuts down a few seconds after de service is started. This is my config file: input{ file { path =\> "E:\\DATA\\test\\\*log" start\_position =\> "beginning"… --- ## [Pipeline Code Issue](https://discuss.elastic.co/t/pipeline-code-issue/197930)
**Author:** [@Tony\_Chirillo](https://discuss.elastic.co/u/Tony_Chirillo)\ **Replies:** 2\ **Last updated:** [September 4, 2019, 7:47pm UTC](https://discuss.elastic.co/t/pipeline-code-issue/197930 "2019-09-04T19:47:45Z")
Hello, I have some trouble with my Logstash Pipeline code. I just added the following to my pipeline filter section: if \[beat\]\[hostname\] == ”qa-exchg16” { mutate { replace =\> { "\[fields\]\[app\_id\]" =\> "exchangeqa"… --- ## [Unexpected end-of-input within/between Object entries](https://discuss.elastic.co/t/unexpected-end-of-input-within-between-object-entries/198077)
**Author:** [@S\_Chase](https://discuss.elastic.co/u/S_Chase)\ **Replies:** 4\ **Last updated:** [September 4, 2019, 6:50pm UTC](https://discuss.elastic.co/t/unexpected-end-of-input-within-between-object-entries/198077 "2019-09-04T18:50:30Z")
This is the message i'm getting in the logstash log. JSON parse error, original data now in message field {:error=\>#\ **Author:** [@SteveSmith](https://discuss.elastic.co/u/SteveSmith)\ **Replies:** 5\ **Last updated:** [September 4, 2019, 6:17pm UTC](https://discuss.elastic.co/t/geo-point-and-logstash/198066 "2019-09-04T18:17:51Z") Hi Everyone, I am having a hard time lately understanding how the Geo\_Point type is created. I found that when I create the pipeline (using this link: https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-… --- ## [Preety print json in output](https://discuss.elastic.co/t/preety-print-json-in-output/198078)
**Author:** [@Ulka](https://discuss.elastic.co/u/Ulka)\ **Replies:** 0\ **Last updated:** [September 4, 2019, 4:40pm UTC](https://discuss.elastic.co/t/preety-print-json-in-output/198078 "2019-09-04T16:40:27Z")
I am using filbeat as input and Elastic search as output. using json filter to parse json msgs. But I dont see output coming in proper json format (preety print) in Kibana . I tried with Ruby filter to print preety js… --- ## [Logstash RabbitMQ Input Plugin not recognize host and port](https://discuss.elastic.co/t/logstash-rabbitmq-input-plugin-not-recognize-host-and-port/198070)
**Author:** [@ThiagoBarradas](https://discuss.elastic.co/u/ThiagoBarradas)\ **Replies:** 1\ **Last updated:** [September 4, 2019, 3:50pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-input-plugin-not-recognize-host-and-port/198070 "2019-09-04T15:50:31Z")
Hello Guys! I'm trying to use Logstash to receive messages from RabbitMQ but it not recognize my configuration, cosidering only localhost:5672 and failing when tries to connect. I have already tested the connection wit… --- ## [Update/rename nested field](https://discuss.elastic.co/t/update-rename-nested-field/198057)
**Author:** [@bvoros](https://discuss.elastic.co/u/bvoros)\ **Replies:** 3\ **Last updated:** [September 4, 2019, 3:08pm UTC](https://discuss.elastic.co/t/update-rename-nested-field/198057 "2019-09-04T15:08:14Z")
Hello All, What is the best way to rename a nested field? I would like to do the following: if \[filed\]\[subfield\] rename it to \[field\] Would it make sense to add a temporary field with the value of the original, drop… --- ## [How to autoscale Logstash](https://discuss.elastic.co/t/how-to-autoscale-logstash/198061)
**Author:** [@bala2](https://discuss.elastic.co/u/bala2)\ **Replies:** 0\ **Last updated:** [September 4, 2019, 2:05pm UTC](https://discuss.elastic.co/t/how-to-autoscale-logstash/198061 "2019-09-04T14:05:22Z")
Hi Team, I have a requirement, where i have a database pushing events to one Logstash. If one logstash is down another logstash should automatically start. Could you please help me with some reference link. Here i dont … --- ## [Importing Logstash configuration from another file](https://discuss.elastic.co/t/importing-logstash-configuration-from-another-file/198056)
**Author:** [@corramatteo](https://discuss.elastic.co/u/corramatteo)\ **Replies:** 1\ **Last updated:** [September 4, 2019, 1:47pm UTC](https://discuss.elastic.co/t/importing-logstash-configuration-from-another-file/198056 "2019-09-04T13:47:54Z")
Hi guys, I was wondering if it is possible to include a Logstash configuration from a file to another one. And if it is possible, what is the best way? Below I will make an example about what I want to achieve. In a … --- ## [Output writing wrong event data](https://discuss.elastic.co/t/output-writing-wrong-event-data/198002)
**Author:** [@Pablo\_Gutierrez](https://discuss.elastic.co/u/Pablo_Gutierrez)\ **Replies:** 2\ **Last updated:** [September 4, 2019, 1:36pm UTC](https://discuss.elastic.co/t/output-writing-wrong-event-data/198002 "2019-09-04T13:36:15Z")
Heyy! I'm newbie with Logstash so I need some help here with a filter done in .rb file. Situation: I'm reading from csv file a code, for example 1500. Then, in filter I use a ruby file to look for that 1500 as a ke… --- ## [Syslog input character adjustment](https://discuss.elastic.co/t/syslog-input-character-adjustment/197988)
**Author:** [@shenshensinian](https://discuss.elastic.co/u/shenshensinian)\ **Replies:** 3\ **Last updated:** [September 4, 2019, 1:31pm UTC](https://discuss.elastic.co/t/syslog-input-character-adjustment/197988 "2019-09-04T13:31:28Z")
In the case of push logs using syslog, how do I adjust only the character encoding of individual IPs in the input phase. --- ## [Grok to extract info from non KV formatted data?](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901)
**Author:** [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Replies:** 6\ **Last updated:** [September 4, 2019, 1:23pm UTC](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901 "2019-09-04T13:23:53Z")
Ok.. I'm back with another head scratcher.. I'm so damn new... but I'm learning.:smiley: I'm sending my entire message through kv{} to have it break out the KV pairs for me.. but I need to make a KV pair for items lis… --- ## [Logstash not process large XML with multiple xpath](https://discuss.elastic.co/t/logstash-not-process-large-xml-with-multiple-xpath/197758)
**Author:** [@Rodrigo\_Tornis](https://discuss.elastic.co/u/Rodrigo_Tornis)\ **Replies:** 3\ **Last updated:** [September 4, 2019, 1:14pm UTC](https://discuss.elastic.co/t/logstash-not-process-large-xml-with-multiple-xpath/197758 "2019-09-04T13:14:06Z")
Hello people! I have the following problem: 1 - I have an XMLs with estimated size 69MB and my configuration is as follows: input { file { path =\> "/tmp/ACOR\_2019.XML" start\_position =\> "beginning" sincedb\_pa… --- ## [Custom Multiline Parsing](https://discuss.elastic.co/t/custom-multiline-parsing/197946)
**Author:** [@Naveesh\_Doolhur](https://discuss.elastic.co/u/Naveesh_Doolhur)\ **Replies:** 1\ **Last updated:** [September 4, 2019, 12:02pm UTC](https://discuss.elastic.co/t/custom-multiline-parsing/197946 "2019-09-04T12:02:26Z")
I have a Log file as follows: ===================================== REQUEST 03-09-2019 01:12:25.808599 REF:5001462 ========================================= \ \ **Author:** [@fernando974](https://discuss.elastic.co/u/fernando974)\ **Replies:** 0\ **Last updated:** [September 4, 2019, 11:17am UTC](https://discuss.elastic.co/t/parse-syslog-message/198034 "2019-09-04T11:17:48Z") Hello, I know i'm using EFK and not ELK but i'm still trying maybe someone know how to do it with fluentd on this forum I have the following syslog message : "54:07+04:00 test-ls100.pi.com - - - - 1,2019/09/04 09:54:0… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=481) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=483)