# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=590 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 591 --- ## [Logstash-6.6 input Kafka Client Version 0.8](https://discuss.elastic.co/t/logstash-6-6-input-kafka-client-version-0-8/170777)
**Author:** [@rpuserh](https://discuss.elastic.co/u/rpuserh)\ **Replies:** 0\ **Last updated:** [March 4, 2019, 6:55pm UTC](https://discuss.elastic.co/t/logstash-6-6-input-kafka-client-version-0-8/170777 "2019-03-04T18:55:17Z")
Hi, I have logstash-2.4 which one works fine with Kafka Client Version 0.8 input I need to upgrade ES to version 6.6 and also need to upgrade logstash as logstash2.4 cannot write to ES-6.6 How do I make kafka Client V… --- ## [Logstash filter: copying values after split](https://discuss.elastic.co/t/logstash-filter-copying-values-after-split/170754)
**Author:** [@fringeadmin](https://discuss.elastic.co/u/fringeadmin)\ **Replies:** 2\ **Last updated:** [March 4, 2019, 5:14pm UTC](https://discuss.elastic.co/t/logstash-filter-copying-values-after-split/170754 "2019-03-04T17:14:11Z")
I have azure event hub data coming into logstash via the eponymous plugin. I get huge junks of json that contain multiple events which I then have to split into distinct events. Here is a simplified example: {"records":… --- ## [How to calculate Date difference using scripted fields](https://discuss.elastic.co/t/how-to-calculate-date-difference-using-scripted-fields/170763)
**Author:** [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\ **Replies:** 0\ **Last updated:** [March 4, 2019, 4:23pm UTC](https://discuss.elastic.co/t/how-to-calculate-date-difference-using-scripted-fields/170763 "2019-03-04T16:23:01Z")
Hello Team, I have a use case to take the difference between two dates in the form of year,month,day,hour,minute,second format. for example 10-mar-2019 10:00:00 - 09-mar-2019 10:00:00 would be = 0 years,0 month,1 day… --- ## [Kafka input - Could not find a 'KafkaClient' entry in the JAAS configuration](https://discuss.elastic.co/t/kafka-input-could-not-find-a-kafkaclient-entry-in-the-jaas-configuration/170505)
**Author:** [@plundgren](https://discuss.elastic.co/u/plundgren)\ **Replies:** 1\ **Last updated:** [March 4, 2019, 3:47pm UTC](https://discuss.elastic.co/t/kafka-input-could-not-find-a-kafkaclient-entry-in-the-jaas-configuration/170505 "2019-03-04T15:47:08Z")
Version: Logstash 6.4.1 Operating System: Ubuntu 16.04 Hi, Im having issue with configuring logstash to subscribe to a kafka topic over SASL SSL. From what it looks like Kafka fails to read the kafka client configu… --- ## [Stop Logstash from automatic shutting down when using CPM](https://discuss.elastic.co/t/stop-logstash-from-automatic-shutting-down-when-using-cpm/170741)
**Author:** [@bruno.henriques](https://discuss.elastic.co/u/bruno.henriques)\ **Replies:** 1\ **Last updated:** [March 4, 2019, 3:15pm UTC](https://discuss.elastic.co/t/stop-logstash-from-automatic-shutting-down-when-using-cpm/170741 "2019-03-04T15:15:16Z")
Hello, I want to use Centralized Pipeline Management (https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html) to manage my Logstash pipelines, however Logstash shutdowns automatic… --- ## [Logstash not dropping messages as configured](https://discuss.elastic.co/t/logstash-not-dropping-messages-as-configured/170554)
**Author:** [@LollerAgent](https://discuss.elastic.co/u/LollerAgent)\ **Replies:** 5\ **Last updated:** [March 4, 2019, 2:58pm UTC](https://discuss.elastic.co/t/logstash-not-dropping-messages-as-configured/170554 "2019-03-04T14:58:31Z")
Can someone please help me understand why the following "drop" filter is not working? As you can see, the original \[message\] field is in \[json\]\[log\]\[message\] but is renamed to \[log\]\[message\] in the second filter. I hav… --- ## [Logstash null fields to drop](https://discuss.elastic.co/t/logstash-null-fields-to-drop/170731)
**Author:** [@fedwe](https://discuss.elastic.co/u/fedwe)\ **Replies:** 4\ **Last updated:** [March 4, 2019, 2:52pm UTC](https://discuss.elastic.co/t/logstash-null-fields-to-drop/170731 "2019-03-04T14:52:54Z")
if i have 2 fields i want a condition if \[field1\] == nil and \[field2\] == nil { drop{} } if both fieds are null values i want to drop the event but none of the conditions i tried worked \[field1\] == "" and \[field2\] ==… --- ## [Copying field in from one event to other in logstash](https://discuss.elastic.co/t/copying-field-in-from-one-event-to-other-in-logstash/170621)
**Author:** [@Pranav](https://discuss.elastic.co/u/Pranav)\ **Replies:** 8\ **Last updated:** [March 4, 2019, 2:25pm UTC](https://discuss.elastic.co/t/copying-field-in-from-one-event-to-other-in-logstash/170621 "2019-03-04T14:25:04Z")
Can anyone tell me if I can copy a field from one event to another in logstash based on a unique\_key present in both the events? --- ## [Resend data from beats to Elasticsearch via logstash](https://discuss.elastic.co/t/resend-data-from-beats-to-elasticsearch-via-logstash/170732)
**Author:** [@FoRMaTC](https://discuss.elastic.co/u/FoRMaTC)\ **Replies:** 0\ **Last updated:** [March 4, 2019, 1:53pm UTC](https://discuss.elastic.co/t/resend-data-from-beats-to-elasticsearch-via-logstash/170732 "2019-03-04T13:53:14Z")
Hi, I had an outage on logstash for a couple of days and the data from logstash (file/winlog beats) didn't come in into elastic. Is there a way to resend/ trigger data from between to dates? Cheers --- ## [Reverse field hashed with fingerprint](https://discuss.elastic.co/t/reverse-field-hashed-with-fingerprint/170704)
**Author:** [@stef35340](https://discuss.elastic.co/u/stef35340)\ **Replies:** 2\ **Last updated:** [March 4, 2019, 1:29pm UTC](https://discuss.elastic.co/t/reverse-field-hashed-with-fingerprint/170704 "2019-03-04T13:29:26Z")
Hello, I've some questions about the "fingerprint" module of Logstash. After some tests, I've noticed few things: "key" parameter mandatory, but the documentation says not As per the "key" is mandatory, the hashed fi… --- ## [Org.apache.cxf.services filtering in logstash](https://discuss.elastic.co/t/org-apache-cxf-services-filtering-in-logstash/170690)
**Author:** [@Primeval](https://discuss.elastic.co/u/Primeval)\ **Replies:** 1\ **Last updated:** [March 4, 2019, 1:26pm UTC](https://discuss.elastic.co/t/org-apache-cxf-services-filtering-in-logstash/170690 "2019-03-04T13:26:27Z")
I have logs looking like this: --- ## [Grok regex don't match](https://discuss.elastic.co/t/grok-regex-dont-match/170188)
**Author:** [@elastic\_trg](https://discuss.elastic.co/u/elastic_trg)\ **Replies:** 8\ **Last updated:** [March 4, 2019, 12:07pm UTC](https://discuss.elastic.co/t/grok-regex-dont-match/170188 "2019-03-04T12:07:30Z")
Hi everybody, I'm asking for your help because I can not perform a correct parsing of one of my log. Indeed my original log include regex pattern and I would to delete them. I made a regex whish working fine on https://… --- ## [Unable to get logs from Windows 2012 server to logstash using filebeat](https://discuss.elastic.co/t/unable-to-get-logs-from-windows-2012-server-to-logstash-using-filebeat/170713)
**Author:** [@shub8050](https://discuss.elastic.co/u/shub8050)\ **Replies:** 2\ **Last updated:** [March 4, 2019, 12:03pm UTC](https://discuss.elastic.co/t/unable-to-get-logs-from-windows-2012-server-to-logstash-using-filebeat/170713 "2019-03-04T12:03:51Z")
I have been trying hard to get logs to my elk stack from a windows server using filebeat. I can get the logs from inetpub iis logs but when i specify my logs then i couldnt get it on kibana. Here is my filebeat.yml con… --- ## [How to make the loastash.conf effective](https://discuss.elastic.co/t/how-to-make-the-loastash-conf-effective/170645)
**Author:** [@yinkai\_cheng](https://discuss.elastic.co/u/yinkai_cheng)\ **Replies:** 1\ **Last updated:** [March 4, 2019, 11:54am UTC](https://discuss.elastic.co/t/how-to-make-the-loastash-conf-effective/170645 "2019-03-04T11:54:25Z")
Hi，all i configed the filebeat to send docker log to the logstash，And it does works, i can see the docker log on the logstash container sent by filebeat. then i config the logstash config file "logstash.conf" to send… --- ## [Processing Custom Log Input from Filebeat in Logstash](https://discuss.elastic.co/t/processing-custom-log-input-from-filebeat-in-logstash/170656)
**Author:** [@luke14](https://discuss.elastic.co/u/luke14)\ **Replies:** 3\ **Last updated:** [March 4, 2019, 9:23am UTC](https://discuss.elastic.co/t/processing-custom-log-input-from-filebeat-in-logstash/170656 "2019-03-04T09:23:57Z")
Hi I have setup Filebeat on a server which I would like it to pass a custom application log to my logstash server. On my logstash server I currently have a beats\_input.conf which is listening for all my beats inputs. A… --- ## [Grok: filter options match not create new field](https://discuss.elastic.co/t/grok-filter-options-match-not-create-new-field/170499)
**Author:** [@Ret](https://discuss.elastic.co/u/Ret)\ **Replies:** 2\ **Last updated:** [March 4, 2019, 7:12am UTC](https://discuss.elastic.co/t/grok-filter-options-match-not-create-new-field/170499 "2019-03-04T07:12:51Z")
Hi everyone ! I am collecting Windows Event Logs from winlogbeat with Logstash. And I have some problem grok filter. I created custom pattren for find user account in event log ..... And the pattern working in Kibana G… --- ## [Grok Filter For my Log file](https://discuss.elastic.co/t/grok-filter-for-my-log-file/170276)
**Author:** [@sana1](https://discuss.elastic.co/u/sana1)\ **Replies:** 14\ **Last updated:** [March 4, 2019, 4:34am UTC](https://discuss.elastic.co/t/grok-filter-for-my-log-file/170276 "2019-03-04T04:34:27Z")
\[\] \[1:1000039:0\] updatelocation - Alert \[\] \[Priority: 0\] 02/27-14:29:46.412090 0A:01:01:01:01:01 -\> 0A:02:02:02:02:02 type:0x800 len:0xC6 10.1.1.1 -\> 10.2.2.2 SCTP TTL:255 TOS:0x0 ID:4660 IpLen:20 DgmLen:184 This is … --- ## [I am able to put time in pmmmmm=\> startTime0 how to convert it to time?](https://discuss.elastic.co/t/i-am-able-to-put-time-in-pmmmmm-starttime0-how-to-convert-it-to-time/169171)
**Author:** [@jamespb](https://discuss.elastic.co/u/jamespb)\ **Replies:** 5\ **Last updated:** [March 4, 2019, 3:19am UTC](https://discuss.elastic.co/t/i-am-able-to-put-time-in-pmmmmm-starttime0-how-to-convert-it-to-time/169171 "2019-03-04T03:19:39Z")
I am able to put time in pmmmmm=\> startTime0 how to convert it to time? Thanks a lot "pmmmmm" =\> { "ShPNR" =\> 0.0, "RelayShPNRNoRoute" =\> 0.0, "RelayRcvdShPNR" =\> 0.0, "pmContai… --- ## [Remove head and tail for xml](https://discuss.elastic.co/t/remove-head-and-tail-for-xml/169331)
**Author:** [@jamespb](https://discuss.elastic.co/u/jamespb)\ **Replies:** 7\ **Last updated:** [March 4, 2019, 3:19am UTC](https://discuss.elastic.co/t/remove-head-and-tail-for-xml/169331 "2019-03-04T03:19:22Z")
I am trying to break one xml file into multiple event. I use \ "\) and the last line? or, how can I use mu… --- ## [Archsignt error while set up. rror when executing Kibana client request {:error=\>#\ **Author:** [@rajatdey80](https://discuss.elastic.co/u/rajatdey80)\ **Replies:** 1\ **Last updated:** [March 3, 2019, 9:57pm UTC](https://discuss.elastic.co/t/archsignt-error-while-set-up-rror-when-executing-kibana-client-request-error-mantic/170430 "2019-03-03T21:57:58Z") While setting Up Archsight ./logstash --modules arcsight --setup -M "arcsight.var.input.smartconnector.port=5602" -M "arcsight.var.elasticsearch.hosts=10.1.20.96:9200" -M "arcsight.var.kibana.host=10.1.20.96:5… --- ## [Remove date from syslog message](https://discuss.elastic.co/t/remove-date-from-syslog-message/170650)
**Author:** [@paulohperes](https://discuss.elastic.co/u/paulohperes)\ **Replies:** 1\ **Last updated:** [March 3, 2019, 8:09pm UTC](https://discuss.elastic.co/t/remove-date-from-syslog-message/170650 "2019-03-03T20:09:53Z")
Hello, I'm building a server logs, the server receive logs from network equipments (Juniper, Ericsson, Cisco, Huawei and Nokia). The server does forward the logs to logstash server on port 55514. Analyzing the logs, lo… --- ## [Index pattern from Logstash](https://discuss.elastic.co/t/index-pattern-from-logstash/170613)
**Author:** [@Kathir\_J](https://discuss.elastic.co/u/Kathir_J)\ **Replies:** 0\ **Last updated:** [March 3, 2019, 7:28am UTC](https://discuss.elastic.co/t/index-pattern-from-logstash/170613 "2019-03-03T07:28:39Z")
\#==================== Elasticsearch template setting ========================== setup.template.settings: index.number\_of\_shards: 1 index.codec: best\_compression #\_source.enabled: false setup.template.name: "metric… --- ## [Logstash - Agent Failing to execute Actions](https://discuss.elastic.co/t/logstash-agent-failing-to-execute-actions/170580)
**Author:** [@Elliott954](https://discuss.elastic.co/u/Elliott954)\ **Replies:** 4\ **Last updated:** [March 3, 2019, 7:23am UTC](https://discuss.elastic.co/t/logstash-agent-failing-to-execute-actions/170580 "2019-03-03T07:23:13Z")
Hey Everyone, I'm currently trying to load some .csv data into my Elasticsearch/Kibana by using logstash. At this current time I'm getting the following error when I'm trying to load my data. \[WARN \] 2019-03-02 18:52:4… --- ## [Logstash parsing messages consuming more space in /var/log](https://discuss.elastic.co/t/logstash-parsing-messages-consuming-more-space-in-var-log/170594)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 1\ **Last updated:** [March 2, 2019, 8:27pm UTC](https://discuss.elastic.co/t/logstash-parsing-messages-consuming-more-space-in-var-log/170594 "2019-03-02T20:27:31Z")
HI Team, I have a ELK stack setup on one single server for my my Non prodcution setup, i will upgrade it to 3 node cluster in future,but for the moment i am working on single node. Everything seems to working fine, exc… --- ## [Create array with only one element](https://discuss.elastic.co/t/create-array-with-only-one-element/170572)
**Author:** [@Idar\_Lund](https://discuss.elastic.co/u/Idar_Lund)\ **Replies:** 2\ **Last updated:** [March 2, 2019, 8:33am UTC](https://discuss.elastic.co/t/create-array-with-only-one-element/170572 "2019-03-02T08:33:34Z")
Hi, I'm trying to create an array with only one element, so that I can add more elements to it later with elasticsearch action update and script. When adding an array with two elements like this: filter { mutate { a… --- ## [How to parse json array to get object count](https://discuss.elastic.co/t/how-to-parse-json-array-to-get-object-count/170409)
**Author:** [@pcrai](https://discuss.elastic.co/u/pcrai)\ **Replies:** 4\ **Last updated:** [March 1, 2019, 11:21pm UTC](https://discuss.elastic.co/t/how-to-parse-json-array-to-get-object-count/170409 "2019-03-01T23:21:18Z")
I have given one json log file as input in logstash and getting corresponding mapping in kibana but I want to count element available in json array object. need help to get that. --- ## [How to forward data from ELK to Arcsight](https://discuss.elastic.co/t/how-to-forward-data-from-elk-to-arcsight/170567)
**Author:** [@kamalbeg](https://discuss.elastic.co/u/kamalbeg)\ **Replies:** 0\ **Last updated:** [March 1, 2019, 9:41pm UTC](https://discuss.elastic.co/t/how-to-forward-data-from-elk-to-arcsight/170567 "2019-03-01T21:41:04Z")
Hello, I have been assigned task to forward TAM data from ELK to Arcsight? Can someone please help me guide on how to achieve that? I know I have to use logstash but what should be my output as I do not see any output mo… --- ## [Grok pattern file for log message](https://discuss.elastic.co/t/grok-pattern-file-for-log-message/170564)
**Author:** [@Rajashekar](https://discuss.elastic.co/u/Rajashekar)\ **Replies:** 0\ **Last updated:** [March 1, 2019, 9:05pm UTC](https://discuss.elastic.co/t/grok-pattern-file-for-log-message/170564 "2019-03-01T21:05:53Z")
Hi all, Can anyone create Grok pattern for below log file. \<102\>Feb 10 18:22:27 abc-rbjyyyb5demo2a info logger: \[xxl\_acc\] 120.0.0.1 - - \[10/Feb/2019:18:22:27 +0000\] "/iControl/iDemoPortal.dbi" 100 2014\\n","host":"100.0… --- ## [Remove Empty String Value](https://discuss.elastic.co/t/remove-empty-string-value/170560)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 3\ **Last updated:** [March 1, 2019, 8:59pm UTC](https://discuss.elastic.co/t/remove-empty-string-value/170560 "2019-03-01T20:59:27Z")
Running Logstash 6.4.1 I have a field that contains an array of numbers. I am converting the field to string values so that I can translate them to their text values with gsub. All of this works great, except that for… --- ## [Logstash Config File Error2](https://discuss.elastic.co/t/logstash-config-file-error2/170528)
**Author:** [@brandonstephens922](https://discuss.elastic.co/u/brandonstephens922)\ **Replies:** 12\ **Last updated:** [March 1, 2019, 7:48pm UTC](https://discuss.elastic.co/t/logstash-config-file-error2/170528 "2019-03-01T19:48:08Z")
Can someone please review this file for errors? I have been over it and over it and I keep getting this error: \[2019-03-01T14:46:07,935\]\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::Pip… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=589) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=591)