# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=532 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 533 --- ## [Unknown setting 'jass\_path' for kafka in kafka input](https://discuss.elastic.co/t/unknown-setting-jass-path-for-kafka-in-kafka-input/184953)
**Author:** [@Sharon\_Abu](https://discuss.elastic.co/u/Sharon_Abu)\ **Replies:** 1\ **Last updated:** [June 10, 2019, 12:38pm UTC](https://discuss.elastic.co/t/unknown-setting-jass-path-for-kafka-in-kafka-input/184953 "2019-06-10T12:38:11Z")
Hi, I'm trying to configure logstash to connect to confluent cloud kafka using input configuration a jaas file using docker compose with logstash image version 6.0.1 (also tried with v6.8) my docker service looks like… --- ## [Convert an inner field using regex for the field name](https://discuss.elastic.co/t/convert-an-inner-field-using-regex-for-the-field-name/184980)
**Author:** [@shlomit](https://discuss.elastic.co/u/shlomit)\ **Replies:** 0\ **Last updated:** [June 10, 2019, 11:39am UTC](https://discuss.elastic.co/t/convert-an-inner-field-using-regex-for-the-field-name/184980 "2019-06-10T11:39:17Z")
Hi, For my json events processing, I need the mutate filter with convert, for inner fields with a name format such as: filter { mutate { convert =\> { "\[additionalData\]\[contentCount\_1\]" =\> "integer" "\[additionalData… --- ## [Aggregate filter to remember field accross lines](https://discuss.elastic.co/t/aggregate-filter-to-remember-field-accross-lines/184712)
**Author:** [@msk\_76](https://discuss.elastic.co/u/msk_76)\ **Replies:** 2\ **Last updated:** [June 10, 2019, 10:57am UTC](https://discuss.elastic.co/t/aggregate-filter-to-remember-field-accross-lines/184712 "2019-06-10T10:57:54Z")
I have to remember date from one line and use it while processing event\_timestamp filed on other lines of log file. For this I am using aggregate filter. Please guide what I am doing wrong here or any other way to do it? … --- ## [Problem with timestamp field](https://discuss.elastic.co/t/problem-with-timestamp-field/184967)
**Author:** [@Prabhav](https://discuss.elastic.co/u/Prabhav)\ **Replies:** 1\ **Last updated:** [June 10, 2019, 10:44am UTC](https://discuss.elastic.co/t/problem-with-timestamp-field/184967 "2019-06-10T10:44:16Z")
I want to upload a .csv file to elasticsearch but the main problem that i am facing is with the formatting of timestamp field. I want to be able to add a field in the format of HH.mm.ss.SSS but all i can find is forma… --- ## [Conversion of roman number mixed with text](https://discuss.elastic.co/t/conversion-of-roman-number-mixed-with-text/184172)
**Author:** [@dharani](https://discuss.elastic.co/u/dharani)\ **Replies:** 3\ **Last updated:** [June 10, 2019, 6:08am UTC](https://discuss.elastic.co/t/conversion-of-roman-number-mixed-with-text/184172 "2019-06-10T06:08:17Z")
how to deal with the datatype consist of roman number mixed with text. like quarter type field consist of Roman numbers mixed with text. Quarter Type: III V Gaz V Gaz --- ## [Logstash inflight events](https://discuss.elastic.co/t/logstash-inflight-events/184923)
**Author:** [@Gaurao\_Burghate](https://discuss.elastic.co/u/Gaurao_Burghate)\ **Replies:** 0\ **Last updated:** [June 10, 2019, 6:05am UTC](https://discuss.elastic.co/t/logstash-inflight-events/184923 "2019-06-10T06:05:23Z")
We are using logstash distributed pattern as below main.conf input {kinsis } output { if condition 1 then pipeline 1 else if condition 2 then pipeline 2 } pipeline1.conf input {pipeline1} filter {} output {am… --- ## [Size of indices are too large](https://discuss.elastic.co/t/size-of-indices-are-too-large/181801)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 10\ **Last updated:** [June 10, 2019, 5:26am UTC](https://discuss.elastic.co/t/size-of-indices-are-too-large/181801 "2019-06-10T05:26:13Z")
I am having logs of 6.9GB but the size of index created is more than 20GB. Is there any efficient way via which we can control the size of indices?????? --- ## [Error "lumberjack All host unavailable, sleeping" when Logstash-to-Logstash communication is configured](https://discuss.elastic.co/t/error-lumberjack-all-host-unavailable-sleeping-when-logstash-to-logstash-communication-is-configured/184918)
**Author:** [@garima](https://discuss.elastic.co/u/garima)\ **Replies:** 0\ **Last updated:** [June 10, 2019, 5:15am UTC](https://discuss.elastic.co/t/error-lumberjack-all-host-unavailable-sleeping-when-logstash-to-logstash-communication-is-configured/184918 "2019-06-10T05:15:32Z")
Logstash Version: 6.4.1 Use case: Logstash-to-Logstash Communication Upstream hostname : node-1 (IP 172.30.64.98) Downstream hostname : node-2 (IP 172.30.64.115) In /etc/hosts both hostname are already mentioned and … --- ## [Logstash VirusTotal plugin error](https://discuss.elastic.co/t/logstash-virustotal-plugin-error/184688)
**Author:** [@Jeeth](https://discuss.elastic.co/u/Jeeth)\ **Replies:** 2\ **Last updated:** [June 10, 2019, 5:10am UTC](https://discuss.elastic.co/t/logstash-virustotal-plugin-error/184688 "2019-06-10T05:10:43Z")
Dear team, We are trying out logstash integration with ''logstash-filter-virustotal'' plugin which is a Virustotal Lookup filter for Logstash. https://github.com/coolacid/logstash-filter-virustotal I have installed th… --- ## [Unable to load new Index onLogstash & Elasticsearch](https://discuss.elastic.co/t/unable-to-load-new-index-onlogstash-elasticsearch/184881)
**Author:** [@Mshala](https://discuss.elastic.co/u/Mshala)\ **Replies:** 1\ **Last updated:** [June 9, 2019, 3:20pm UTC](https://discuss.elastic.co/t/unable-to-load-new-index-onlogstash-elasticsearch/184881 "2019-06-09T15:20:17Z")
I have created a new index file as per below but its not showing up on my Elasticsearch index. --- ## [Retrying failed action with response code: 500](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-500/184813)
**Author:** [@Debasish\_Rath](https://discuss.elastic.co/u/Debasish_Rath)\ **Replies:** 3\ **Last updated:** [June 9, 2019, 11:26am UTC](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-500/184813 "2019-06-09T11:26:31Z")
Hi All, My config file was working in logstash before installing XPACK. After installation, I am getting below error in Logstash. retrying failed action with response code: 500 ({"type"=\>"illegal\_state\_exception", "re… --- ## [Jdbc input plugin read data multiple times from database](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 6\ **Last updated:** [June 8, 2019, 8:54pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314 "2019-06-08T20:54:23Z")
Hi All I am using JDBC input plugin, i have 10,000 records in Database I am setting jdbc\_page\_size as 500 It goes in loop 20, to get all the 10,000 records for this, i schedule this as schedule =\> \*/2 \* \* \* \* With … --- ## [Problem to parse geo\_shape from .csv trough logstash](https://discuss.elastic.co/t/problem-to-parse-geo-shape-from-csv-trough-logstash/183780)
**Author:** [@grainsngrubble](https://discuss.elastic.co/u/grainsngrubble)\ **Replies:** 1\ **Last updated:** [June 8, 2019, 10:34am UTC](https://discuss.elastic.co/t/problem-to-parse-geo-shape-from-csv-trough-logstash/183780 "2019-06-08T10:34:38Z")
Hi! I´m trying to ingest a .csv with geo\_shapes as linestrings. Though I keep getting parse exceptions. Have tries to search and to read the documentation, but no progress so I put my faith in you now :man\_technologist:… --- ## [Single grok expressions for different format of logs?](https://discuss.elastic.co/t/single-grok-expressions-for-different-format-of-logs/184805)
**Author:** [@habi](https://discuss.elastic.co/u/habi)\ **Replies:** 1\ **Last updated:** [June 8, 2019, 12:15am UTC](https://discuss.elastic.co/t/single-grok-expressions-for-different-format-of-logs/184805 "2019-06-08T00:15:17Z")
Hey all, I am new to logstash and I am setting up logstash on a server which gets multiple formats of logs in the same type. I was wondering if there is a way i could write just one grok expressions for these logs by add… --- ## [X PACK CONFIGURATION](https://discuss.elastic.co/t/x-pack-configuration/184758)
**Author:** [@Debasish\_Rath](https://discuss.elastic.co/u/Debasish_Rath)\ **Replies:** 4\ **Last updated:** [June 7, 2019, 7:33pm UTC](https://discuss.elastic.co/t/x-pack-configuration/184758 "2019-06-07T19:33:14Z")
I am using Redhat version 7.0 (Maipo). I am trying to install XPack on ELK. I got success in ELK and Kibana. But when I am trying to run config file below error is coming. \[2019-06-07T13:48:05,392\]\[INFO \]\[logstash.outpu… --- ## [Cloudwatch Logs to Logstash Input plugin has no result](https://discuss.elastic.co/t/cloudwatch-logs-to-logstash-input-plugin-has-no-result/183170)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 2\ **Last updated:** [May 29, 2019, 10:42pm UTC](https://discuss.elastic.co/t/cloudwatch-logs-to-logstash-input-plugin-has-no-result/183170 "2019-05-29T22:42:39Z")
Code: input{ cloudwatch { access\_key\_id =\> "...id..." secret\_access\_key =\> "...key..." namespace =\> "AWS/Logs" metrics =\> \["IncomingBytes", "ForwardedBytes", "IncomingLogEvents", "ForwardedLogEvents"\] … --- ## [Help with logstash performance inside Kubernetes (AKS)](https://discuss.elastic.co/t/help-with-logstash-performance-inside-kubernetes-aks/184799)
**Author:** [@kevinjqiu](https://discuss.elastic.co/u/kevinjqiu)\ **Replies:** 0\ **Last updated:** [June 7, 2019, 5:50pm UTC](https://discuss.elastic.co/t/help-with-logstash-performance-inside-kubernetes-aks/184799 "2019-06-07T17:50:10Z")
Hi all, We've deployed the Filebeat, ELK stack in our Kubernetes cluster (using Azure Kubernetes Service). The setup is: Filebeat -\> Kafka -\> Logstash -\> ES \<- Kibana. However, we noticed a very significant lag. It s… --- ## [MULTIPLE FILES IN S3 as INPUT AND CREATE MULITPLE INDEXES IN S3](https://discuss.elastic.co/t/multiple-files-in-s3-as-input-and-create-mulitple-indexes-in-s3/184795)
**Author:** [@Mohit\_Ruke](https://discuss.elastic.co/u/Mohit_Ruke)\ **Replies:** 0\ **Last updated:** [June 7, 2019, 5:32pm UTC](https://discuss.elastic.co/t/multiple-files-in-s3-as-input-and-create-mulitple-indexes-in-s3/184795 "2019-06-07T17:32:28Z")
HI Everyone, My requirement is I have an s3 bucket - it will have a hierarachy FOLDERNAME1 --\> filename1 FOLDERNAME2 --\> Filename2 and so on So logstash should make the filename as an index in elasticsearch.. Th… --- ## [Datadog\_metrics output plugin: Is it possible to send metrics of type "distribution"?](https://discuss.elastic.co/t/datadog-metrics-output-plugin-is-it-possible-to-send-metrics-of-type-distribution/184793)
**Author:** [@opcitymatt](https://discuss.elastic.co/u/opcitymatt)\ **Replies:** 1\ **Last updated:** [June 7, 2019, 5:00pm UTC](https://discuss.elastic.co/t/datadog-metrics-output-plugin-is-it-possible-to-send-metrics-of-type-distribution/184793 "2019-06-07T17:00:52Z")
Hi there, We've been using Logstash to parse nginx logs and send request timing to CloudWatch, and we'd like to start using Datadog as our metric destination. The most useful metric type for this sort of data is "distri… --- ## [Passs Json input from s3 files to elasticsearch](https://discuss.elastic.co/t/passs-json-input-from-s3-files-to-elasticsearch/184663)
**Author:** [@Mohit\_Ruke](https://discuss.elastic.co/u/Mohit_Ruke)\ **Replies:** 3\ **Last updated:** [June 7, 2019, 2:28pm UTC](https://discuss.elastic.co/t/passs-json-input-from-s3-files-to-elasticsearch/184663 "2019-06-07T14:28:32Z")
I am trying to pass my data from s3 to elasticsearch 1 THE FILES ARE IN S3. this is my config file input { s3 { bucket =\> "awsesbucket1rec" access\_key\_id =\> "AKIAVU3QDW2HVZ2NFV6I" secret\_access\_key =\> "LJPyM+IuM7… --- ## [Remove charactes with gstug](https://discuss.elastic.co/t/remove-charactes-with-gstug/184771)
**Author:** [@AntonioSanz](https://discuss.elastic.co/u/AntonioSanz)\ **Replies:** 0\ **Last updated:** [June 7, 2019, 1:59pm UTC](https://discuss.elastic.co/t/remove-charactes-with-gstug/184771 "2019-06-07T13:59:18Z")
Hi all, I would need some help with mutate gsub filter. I have something like ArchivedCanonicalItemEvent CanonicalProductSalesAreaToCatalogMapping SAPCONFIGURATION\_POOL 26472b62-8943-44a4-a88e-6dabee93d825 20\_1001 0989… --- ## [SubDocument from a left outer join in JDBC plugin](https://discuss.elastic.co/t/subdocument-from-a-left-outer-join-in-jdbc-plugin/184769)
**Author:** [@BenjaminD](https://discuss.elastic.co/u/BenjaminD)\ **Replies:** 0\ **Last updated:** [June 7, 2019, 1:53pm UTC](https://discuss.elastic.co/t/subdocument-from-a-left-outer-join-in-jdbc-plugin/184769 "2019-06-07T13:53:06Z")
Hi all, I have a MySQL database with two tables linked : customer and invoice. Of course, there can be many invoices for a given customer. Using Logstash 7.1.1 and the JDBC input plugin, I try to load data in Elastics… --- ## [Unexpected behavior when accessing field data in a pipeline-to-pipeline configuration](https://discuss.elastic.co/t/unexpected-behavior-when-accessing-field-data-in-a-pipeline-to-pipeline-configuration/184765)
**Author:** [@psmolkin](https://discuss.elastic.co/u/psmolkin)\ **Replies:** 0\ **Last updated:** [June 7, 2019, 1:45pm UTC](https://discuss.elastic.co/t/unexpected-behavior-when-accessing-field-data-in-a-pipeline-to-pipeline-configuration/184765 "2019-06-07T13:45:04Z")
Hello I receive events from Filebeat and want to send it to the pipeline whose name matches the data from one of the event fields, for example input { beats { port =\> "5000" } } output { if \[event\]\[module\] { … --- ## [Split an output of a kv filter](https://discuss.elastic.co/t/split-an-output-of-a-kv-filter/184614)
**Author:** [@arjun\_purushotham](https://discuss.elastic.co/u/arjun_purushotham)\ **Replies:** 4\ **Last updated:** [June 7, 2019, 1:25pm UTC](https://discuss.elastic.co/t/split-an-output-of-a-kv-filter/184614 "2019-06-07T13:25:20Z")
Hi Below is my logstash filter configuration. filter { if \[type\] in \[ "apache" , "apache\_access" , "apache-access" \] { grok { match =\> \[ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra\_fields}", "message" , … --- ## [Trying to parse a multine array of json objects with logstash split and json](https://discuss.elastic.co/t/trying-to-parse-a-multine-array-of-json-objects-with-logstash-split-and-json/184564)
**Author:** [@GBJCLE1](https://discuss.elastic.co/u/GBJCLE1)\ **Replies:** 5\ **Last updated:** [June 7, 2019, 12:51pm UTC](https://discuss.elastic.co/t/trying-to-parse-a-multine-array-of-json-objects-with-logstash-split-and-json/184564 "2019-06-07T12:51:55Z")
Hi all, I'll preface this by saying that I'm still very new to ELK so I'm only doing what I think is the right way to do things, if there are any mistakes in my process feel free to point them out. Anyways, here's the j… --- ## [ELK - Unable to see datatype conversion in Kibana](https://discuss.elastic.co/t/elk-unable-to-see-datatype-conversion-in-kibana/184742)
**Author:** [@sukanta007](https://discuss.elastic.co/u/sukanta007)\ **Replies:** 0\ **Last updated:** [June 7, 2019, 10:55am UTC](https://discuss.elastic.co/t/elk-unable-to-see-datatype-conversion-in-kibana/184742 "2019-06-07T10:55:46Z")
Hello team, I am facing one strange issue with ELK v5.6 where I can see that out of 6 integer fields, 4 fields datatype got converted to number successfully whereas 2 fields are still showing datatype as string. I tried… --- ## [Break\_on\_match not working](https://discuss.elastic.co/t/break-on-match-not-working/184715)
**Author:** [@Hyllegaard](https://discuss.elastic.co/u/Hyllegaard)\ **Replies:** 3\ **Last updated:** [June 7, 2019, 10:49am UTC](https://discuss.elastic.co/t/break-on-match-not-working/184715 "2019-06-07T10:49:12Z")
After upgrading to 7.1.0, the logic seems to have changed. I used to have this grok: grok { match =\> { "message" =\> "webapp=%{URIPATH:webapp} path=%{URIPATH:path} params={%{GREEDYDATA:Params}} hits=%{BASE10NUM:hits} s… --- ## [Logstash failed while stream from Logback](https://discuss.elastic.co/t/logstash-failed-while-stream-from-logback/184707)
**Author:** [@manickalai](https://discuss.elastic.co/u/manickalai)\ **Replies:** 0\ **Last updated:** [June 7, 2019, 8:06am UTC](https://discuss.elastic.co/t/logstash-failed-while-stream-from-logback/184707 "2019-06-07T08:06:52Z")
Hi, i'm sending logs from 10 different micro services(Springboot) via Logback.xml to Logstash. Due to some failure, the connection of Logback.xml to Logstash was down, after some time later the connection was enabled,… --- ## [Best practice: single logstash instance with multiple pipelines or multiple logstash instances?](https://discuss.elastic.co/t/best-practice-single-logstash-instance-with-multiple-pipelines-or-multiple-logstash-instances/184700)
**Author:** [@Oistein](https://discuss.elastic.co/u/Oistein)\ **Replies:** 1\ **Last updated:** [June 7, 2019, 9:33am UTC](https://discuss.elastic.co/t/best-practice-single-logstash-instance-with-multiple-pipelines-or-multiple-logstash-instances/184700 "2019-06-07T09:33:29Z")
Hello everyone. We are currently running a single Logstash instance which handles log collection from a number of different applications. There is approximately 10 pipelines, each with different filters for parsing the … --- ## [Elastic search input plugin not working](https://discuss.elastic.co/t/elastic-search-input-plugin-not-working/184689)
**Author:** [@Smack\_Alpha](https://discuss.elastic.co/u/Smack_Alpha)\ **Replies:** 1\ **Last updated:** [June 7, 2019, 7:31am UTC](https://discuss.elastic.co/t/elastic-search-input-plugin-not-working/184689 "2019-06-07T07:31:08Z")
I have some data in the index of Elasticsearch. I want to copy the data from the existing index in to the new index in Elasticsearch using logstash. I have used Elasticsearch input plugin and output plugin to load it to … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=531) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=533)