# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=919 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 920 --- ## [Applying grok filter to Kakfa.key](https://discuss.elastic.co/t/applying-grok-filter-to-kakfa-key/98418)
**Author:** [@ashvita](https://discuss.elastic.co/u/ashvita)\ **Replies:** 1\ **Last updated:** [August 25, 2017, 9:27pm UTC](https://discuss.elastic.co/t/applying-grok-filter-to-kakfa-key/98418 "2017-08-25T21:27:19Z")
Hi, I am trying to apply grok filter on kafka key but it is not getting parsed. grok { match =\> \[‘key’, ‘%{IP:NodeIP}%{DATA:}%{IP:Ip2}’\] } I cant see these fields getting created. Thank you! --- ## [Logstash 5.5.0 Windows - S3 Upload - fails](https://discuss.elastic.co/t/logstash-5-5-0-windows-s3-upload-fails/98139)
**Author:** [@PowerShellMike](https://discuss.elastic.co/u/PowerShellMike)\ **Replies:** 0\ **Last updated:** [August 23, 2017, 10:44pm UTC](https://discuss.elastic.co/t/logstash-5-5-0-windows-s3-upload-fails/98139 "2017-08-23T22:44:59Z")
Does anyone know how to fix this error? Error validating bucket write permissions! {:message=\>":bucket option must not contain a forward-slash (/)", :class=\>“ArgumentError”} LogStash::ConfigurationError: Logstash must … --- ## [Sending JSON file to Elasticsearch via Logstash HTTP input plugin and Docker](https://discuss.elastic.co/t/sending-json-file-to-elasticsearch-via-logstash-http-input-plugin-and-docker/97967)
**Author:** [@patchong](https://discuss.elastic.co/u/patchong)\ **Replies:** 4\ **Last updated:** [August 25, 2017, 8:52pm UTC](https://discuss.elastic.co/t/sending-json-file-to-elasticsearch-via-logstash-http-input-plugin-and-docker/97967 "2017-08-25T20:52:16Z")
I have a JSON file I want to send to Elasticseaerch via Logstash’s Http input plugin. I also have Docker installed. I'm just confused as to why I can't see the data I see the data I've sent in Kibana/Elasticsearch This… --- ## [Aggregate fields based on nested filter with custom separator](https://discuss.elastic.co/t/aggregate-fields-based-on-nested-filter-with-custom-separator/98008)
**Author:** [@malhotras](https://discuss.elastic.co/u/malhotras)\ **Replies:** 15\ **Last updated:** [August 25, 2017, 8:39pm UTC](https://discuss.elastic.co/t/aggregate-fields-based-on-nested-filter-with-custom-separator/98008 "2017-08-25T20:39:16Z")
I would like to aggregate some of the fields based on other nested fields. Here is how my data looks like: { "place" =\> "abc", "m\_id" =\> 5099, "address" =\> "Kurt-Schumacher-Ring 15-17", "name" =\>… --- ## [Creating an array of objects](https://discuss.elastic.co/t/creating-an-array-of-objects/97961)
**Author:** [@AnandDivakaran](https://discuss.elastic.co/u/AnandDivakaran)\ **Replies:** 2\ **Last updated:** [August 25, 2017, 3:52pm UTC](https://discuss.elastic.co/t/creating-an-array-of-objects/97961 "2017-08-25T15:52:11Z")
Lets say in a given event i have 4 fields psn1\_name (holding the value “A”, psn1\_age (holding the value 10, psn2\_name (holding the value “B”, psn2\_age (holding the value 20. I want to create a field person such that… --- ## [Problem with logstash as a service (RHEL)](https://discuss.elastic.co/t/problem-with-logstash-as-a-service-rhel/98351)
**Author:** [@Sam67000](https://discuss.elastic.co/u/Sam67000)\ **Replies:** 1\ **Last updated:** [August 25, 2017, 3:51pm UTC](https://discuss.elastic.co/t/problem-with-logstash-as-a-service-rhel/98351 "2017-08-25T15:51:56Z")
I have a problem with logstash that I don’t understand. It works fine with command but as a service it doesn’t work. I have placed my configuration file in /etc/logstash/conf.d/test.conf I have checked the permission … --- ## [Logstash-filter-yaml seems to depend on old logstash-core](https://discuss.elastic.co/t/logstash-filter-yaml-seems-to-depend-on-old-logstash-core/98391)
**Author:** [@johnd](https://discuss.elastic.co/u/johnd)\ **Replies:** 0\ **Last updated:** [August 25, 2017, 3:21pm UTC](https://discuss.elastic.co/t/logstash-filter-yaml-seems-to-depend-on-old-logstash-core/98391 "2017-08-25T15:21:12Z")
$ sudo -u logstash ./bin/logstash-plugin install logstash-filter-yaml Validating logstash-filter-yaml Installing logstash-filter-yaml Plugin version conflict, aborting ERROR: Installation Aborted, message: Bundler co… --- ## [No enum constant org.logstash.bivalues.BiValues](https://discuss.elastic.co/t/no-enum-constant-org-logstash-bivalues-bivalues/98388)
**Author:** [@Dan\_Markhasin](https://discuss.elastic.co/u/Dan_Markhasin)\ **Replies:** 0\ **Last updated:** [August 25, 2017, 3:13pm UTC](https://discuss.elastic.co/t/no-enum-constant-org-logstash-bivalues-bivalues/98388 "2017-08-25T15:13:58Z")
Following this thread: I encountered the same issue when using the Kafka input plugin with a ByteArrayDeserializer as the kafka key deserializer class. This adds a byte field to the event, causing the exception describ… --- ## [Parsing csv file which contains xml data as a field](https://discuss.elastic.co/t/parsing-csv-file-which-contains-xml-data-as-a-field/98340)
**Author:** [@remziay](https://discuss.elastic.co/u/remziay)\ **Replies:** 1\ **Last updated:** [August 25, 2017, 2:19pm UTC](https://discuss.elastic.co/t/parsing-csv-file-which-contains-xml-data-as-a-field/98340 "2017-08-25T14:19:21Z")
Hello , I have a csv file in which one of the fields contains xml data. I am using csv filter in order to parse the document but logstash is unable to parse it . The error is : CSV::MalformedCSVError: Illegal quoting in … --- ## [Lowercase hashtag twitter field](https://discuss.elastic.co/t/lowercase-hashtag-twitter-field/98290)
**Author:** [@Singa\_Forever](https://discuss.elastic.co/u/Singa_Forever)\ **Replies:** 4\ **Last updated:** [August 25, 2017, 1:28pm UTC](https://discuss.elastic.co/t/lowercase-hashtag-twitter-field/98290 "2017-08-25T13:28:38Z")
Hello, I'm collecting tweets from twitter using ELK 5.X, In order to handle my application, I need to : 1- Convert all collected twitter hashtags to lowercase : I used mutate in logstash file filter{ mutate { … --- ## [Logstash does not start analysing directly](https://discuss.elastic.co/t/logstash-does-not-start-analysing-directly/98247)
**Author:** [@ThaPoox](https://discuss.elastic.co/u/ThaPoox)\ **Replies:** 3\ **Last updated:** [August 25, 2017, 1:15pm UTC](https://discuss.elastic.co/t/logstash-does-not-start-analysing-directly/98247 "2017-08-25T13:15:04Z")
Hi, I have configured Logstash to run on a log file that I have. I have specified the path in the configuration, and I have added some other stuff in the file configuration: input { file { path =\> "C:\\users\\XXX\\Dow… --- ## [How do I parse a tmx file (xml file for translation data) in logstash](https://discuss.elastic.co/t/how-do-i-parse-a-tmx-file-xml-file-for-translation-data-in-logstash/98349)
**Author:** [@vinodMS](https://discuss.elastic.co/u/vinodMS)\ **Replies:** 0\ **Last updated:** [August 25, 2017, 8:39am UTC](https://discuss.elastic.co/t/how-do-i-parse-a-tmx-file-xml-file-for-translation-data-in-logstash/98349 "2017-08-25T08:39:52Z")
I am using TMX files (xml file for translation data) as my source in Logstash to index data in Elasticsearch. A sample TMX file looks like this, \ \ \
**Author:** [@EpiqSty](https://discuss.elastic.co/u/EpiqSty)\ **Replies:** 1\ **Last updated:** [August 25, 2017, 10:18am UTC](https://discuss.elastic.co/t/logstash-jdbc-not-creating-index-silently-debug-also-wrong/98263 "2017-08-25T10:18:05Z") Hello! I use ELK stack for visualisation data from DB2, and have 2 jdbc inputs and outputs like: input{ jdbc{ jdbc\_connection\_string =\> "jdbc:db2://db2:50000/db2" jdbc\_driver\_library =\> … --- ## [Grokparsefailure with a pattern verified by grok debug](https://discuss.elastic.co/t/grokparsefailure-with-a-pattern-verified-by-grok-debug/98251)
**Author:** [@Mehdi\_Mouslih](https://discuss.elastic.co/u/Mehdi_Mouslih)\ **Replies:** 9\ **Last updated:** [August 25, 2017, 10:02am UTC](https://discuss.elastic.co/t/grokparsefailure-with-a-pattern-verified-by-grok-debug/98251 "2017-08-25T10:02:44Z")
Hello i am using logstash to collect cisco logs like the following one :slight\_smile: 2017-08-23T11:03:43.068Z 192.168.1.254 \<189\>79: \*Mar 1 01:17:53.151: %SYS-5-CONFIG\_I: Configured from console by console this is my… --- ## [Mssql jdbc](https://discuss.elastic.co/t/mssql-jdbc/98339)
**Author:** [@rolandbp](https://discuss.elastic.co/u/rolandbp)\ **Replies:** 4\ **Last updated:** [August 25, 2017, 9:24am UTC](https://discuss.elastic.co/t/mssql-jdbc/98339 "2017-08-25T09:24:10Z")
Hi, i want push Data from MSQL into Elasticsearch. In Logstash following Input file: input { jdbc { jdbc\_driver\_library =\> "/etc/logstash/jdbc/mssql/sqljdbc\_6.0/enu/jre8/sqljdbc42.jar" jdbc\_driver\_class =\> "com.mic… --- ## [TIMESTAMP\_ISO8601 issue](https://discuss.elastic.co/t/timestamp-iso8601-issue/98346)
**Author:** [@Peter\_Hendriks](https://discuss.elastic.co/u/Peter_Hendriks)\ **Replies:** 1\ **Last updated:** [August 25, 2017, 8:51am UTC](https://discuss.elastic.co/t/timestamp-iso8601-issue/98346 "2017-08-25T08:51:08Z")
Hi, I’ve recently debugged an issue back to the GROK pattern that logstash uses. TIMESTAMP\_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}\[T \]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601\_TIMEZONE}? The short version is: this o… --- ## [Logstash filtering SOLR events](https://discuss.elastic.co/t/logstash-filtering-solr-events/97033)
**Author:** [@sandeep1](https://discuss.elastic.co/u/sandeep1)\ **Replies:** 2\ **Last updated:** [August 25, 2017, 8:31am UTC](https://discuss.elastic.co/t/logstash-filtering-solr-events/97033 "2017-08-25T08:31:17Z")
I’m trying to parse SOLR logs using logstash. Below i have pasted results from grok debugger but i dont see same results in elastic/kibana. Here is my logstash filter: if \[fields\]\[log\_type\] == “SOLR\_0343”{ grok { mat… --- ## [Filter Multiple Patterns](https://discuss.elastic.co/t/filter-multiple-patterns/98016)
**Author:** [@Bhuvanesh](https://discuss.elastic.co/u/Bhuvanesh)\ **Replies:** 3\ **Last updated:** [August 25, 2017, 7:58am UTC](https://discuss.elastic.co/t/filter-multiple-patterns/98016 "2017-08-25T07:58:01Z")
Hi All, I have an ELK integrated log server. Normally I get all the messages in the format %{SYSLOGTIMESTAMP} %{SYSLOGHOST:sysloghost} %{SYSLOGPROG:syslogprog}: %{GREEDYDATA:Message} I have two level filters where I … --- ## [Logstash fails to open file: Permission denied](https://discuss.elastic.co/t/logstash-fails-to-open-file-permission-denied/98170)
**Author:** [@Hisushi](https://discuss.elastic.co/u/Hisushi)\ **Replies:** 5\ **Last updated:** [August 25, 2017, 7:44am UTC](https://discuss.elastic.co/t/logstash-fails-to-open-file-permission-denied/98170 "2017-08-25T07:44:46Z")
Hi everybody, I’m currently encountering the error that Logstash is not able to open several files due to a “Permission denied” error \[2017-08-24T08:25:43,619\]\[WARN \]\[logstash.inputs.file \] failed to open /media/Lo… --- ## [Convert string to date and change date format](https://discuss.elastic.co/t/convert-string-to-date-and-change-date-format/98333)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 0\ **Last updated:** [August 25, 2017, 6:43am UTC](https://discuss.elastic.co/t/convert-string-to-date-and-change-date-format/98333 "2017-08-25T06:43:12Z")
I am using JDBC input. In my sql query craeteddate is a date field,When i am running logstash without any filter createddate field recognizing as date field. But i don’t want time in craeteddate,So applied ruby filter… --- ## [How to handle '=' in values, splitting on | but KV takes over all '=' not only the first](https://discuss.elastic.co/t/how-to-handle-in-values-splitting-on-but-kv-takes-over-all-not-only-the-first/97851)
**Author:** [@Mojster](https://discuss.elastic.co/u/Mojster)\ **Replies:** 2\ **Last updated:** [August 25, 2017, 6:25am UTC](https://discuss.elastic.co/t/how-to-handle-in-values-splitting-on-but-kv-takes-over-all-not-only-the-first/97851 "2017-08-25T06:25:09Z")
I’m parsing custom logs. Here’s a snippet: 18.7.2012 9:05:57\\t|C3|date=18.07.2012 09:05:57|acronym=BS|… |firstsignUpDate=30.07.2007|bibl001c=m|biblUDK675s=(038)33=111=163.6|… My second (first one ist just mutate/gsub j… --- ## [Grok for values with and without '%' eg: 10, 10%](https://discuss.elastic.co/t/grok-for-values-with-and-without-eg-10-10/98293)
**Author:** [@vreddy](https://discuss.elastic.co/u/vreddy)\ **Replies:** 1\ **Last updated:** [August 25, 2017, 5:13am UTC](https://discuss.elastic.co/t/grok-for-values-with-and-without-eg-10-10/98293 "2017-08-25T05:13:53Z")
Hi, I have tried the following but it did not work: “dedupSavings” =\> “(?m)%{NUMBER:dedup\_savings:float}\[^%\]” For instance, (.\*\[^%\]) -\> regex outputs 10 for 10 or 10%. But above one is not. It works fine for data with… --- ## [Logstash with java](https://discuss.elastic.co/t/logstash-with-java/98213)
**Author:** [@Lbr1](https://discuss.elastic.co/u/Lbr1)\ **Replies:** 3\ **Last updated:** [August 25, 2017, 5:07am UTC](https://discuss.elastic.co/t/logstash-with-java/98213 "2017-08-25T05:07:47Z")
Hello, I would like to ask two questions regarding logstash. I’m developing a project to parse text, and I’m also using Logstash and elasticSearch. My questions are: 1- Is it possible to use Logstash from the java app… --- ## [Logstash kafka output works but kafka input doesn't](https://discuss.elastic.co/t/logstash-kafka-output-works-but-kafka-input-doesnt/98235)
**Author:** [@foresightyj](https://discuss.elastic.co/u/foresightyj)\ **Replies:** 0\ **Last updated:** [August 24, 2017, 12:52pm UTC](https://discuss.elastic.co/t/logstash-kafka-output-works-but-kafka-input-doesnt/98235 "2017-08-24T12:52:13Z")
I have been using logstash for the past two years and consider myself to be fairly familiar with configurations. Actually this configuration worked before and just stopped working a few days ago. To make a minimal case,… --- ## [Logstash output to CloudWatch Logs](https://discuss.elastic.co/t/logstash-output-to-cloudwatch-logs/98299)
**Author:** [@rherr63](https://discuss.elastic.co/u/rherr63)\ **Replies:** 0\ **Last updated:** [August 24, 2017, 10:52pm UTC](https://discuss.elastic.co/t/logstash-output-to-cloudwatch-logs/98299 "2017-08-24T22:52:05Z")
I am aware of the ‘Cloudwatch output plugin’ for Logstash, https://www.elastic.co/guide/en/logstash/5.5/plugins-outputs-cloudwatch.html However, there is a need to send log data from Logstash to Cloudwatch Logs. I ran a… --- ## [Logstash s3 input with iam role](https://discuss.elastic.co/t/logstash-s3-input-with-iam-role/97986)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 4\ **Last updated:** [August 24, 2017, 10:38pm UTC](https://discuss.elastic.co/t/logstash-s3-input-with-iam-role/97986 "2017-08-24T22:38:19Z")
Hi , I am trying to use logstash s3 input plugin to process cloudtrail data. My configuration is all OK but as i am running this logstash on prem, my access to s3 bucket is via a IAM role. I cannot get access and secret… --- ## [Logback TCP Appender Vs Filebeat structured logging](https://discuss.elastic.co/t/logback-tcp-appender-vs-filebeat-structured-logging/97829)
**Author:** [@baldur](https://discuss.elastic.co/u/baldur)\ **Replies:** 2\ **Last updated:** [August 24, 2017, 8:41pm UTC](https://discuss.elastic.co/t/logback-tcp-appender-vs-filebeat-structured-logging/97829 "2017-08-24T20:41:31Z")
Hi all, I use Elastic Stack to centralize logs from Java web applications. Logs are based on Sfl4j+Logback so I’ve used Logback JSON encoder. I’ve setup two possible solutions: LogstashTcpSocketAppender: messages are … --- ## [Exclude\_pattern in logstash s3 input](https://discuss.elastic.co/t/exclude-pattern-in-logstash-s3-input/97993)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 4\ **Last updated:** [August 24, 2017, 8:28pm UTC](https://discuss.elastic.co/t/exclude-pattern-in-logstash-s3-input/97993 "2017-08-24T20:28:15Z")
This is my s3 input config. I am trying to exclude a key that has name called CloudTrail-Digest. input { s3 { type =\> "cloudtrail" bucket =\> "aws" prefix =\> "Accounts/" add\_field =\> { source =\> gzfiles… --- ## [How to capture text from a path or s3 prefix](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 11\ **Last updated:** [August 24, 2017, 8:27pm UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998 "2017-08-24T20:27:49Z")
I have a slightly different requirement where i have a directory structure something like this /data/aws-test1 /data/aws-test2 … and so on Is there a way when using the file input template to capture aws-test1 and aws… --- ## [Winlogbeat logging.level not working properly](https://discuss.elastic.co/t/winlogbeat-logging-level-not-working-properly/98024)
**Author:** [@g4nj4wizard](https://discuss.elastic.co/u/g4nj4wizard)\ **Replies:** 1\ **Last updated:** [August 24, 2017, 8:17pm UTC](https://discuss.elastic.co/t/winlogbeat-logging-level-not-working-properly/98024 "2017-08-24T20:17:52Z")
Hello everyone, i’ve set logging.level to warning. So this should only collect events with level = critical, error, warning. But i am still receiving Information messages. I already restarted the service, but its not s… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=918) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=920)