# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1081 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1082 --- ## [Agrregate Filter For each](https://discuss.elastic.co/t/agrregate-filter-for-each/65474)
**Author:** [@AndrePires](https://discuss.elastic.co/u/AndrePires)\ **Replies:** 0\ **Last updated:** [November 9, 2016, 11:37am UTC](https://discuss.elastic.co/t/agrregate-filter-for-each/65474 "2016-11-09T11:37:13Z")
So I have this code: filter { aggregate { task\_id =\> "%{id}" code =\> " map\['Author'\] ||= \[\] event.to\_hash.each do |key,value| map\[key\] … --- ## [Please ignore](https://discuss.elastic.co/t/please-ignore/65454)
**Author:** [@Mahdy\_S](https://discuss.elastic.co/u/Mahdy_S)\ **Replies:** 0\ **Last updated:** [November 9, 2016, 8:53am UTC](https://discuss.elastic.co/t/please-ignore/65454 "2016-11-09T08:53:42Z")
It was a problem of wrong configurations. Please ignore this question --- ## [Having a cron job scheduler for file type input in logstash 5.0?](https://discuss.elastic.co/t/having-a-cron-job-scheduler-for-file-type-input-in-logstash-5-0/65441)
**Author:** [@Kulasangar\_Gowrisang](https://discuss.elastic.co/u/Kulasangar_Gowrisang)\ **Replies:** 3\ **Last updated:** [November 9, 2016, 10:16am UTC](https://discuss.elastic.co/t/having-a-cron-job-scheduler-for-file-type-input-in-logstash-5-0/65441 "2016-11-09T10:16:47Z")
I'm having the logstash configuration where it picks up the log files from a folder and pushing it to the index. This is how my input looks like: input { file { path =\> "/home/chamith/work/ELK/s… --- ## [How to parse Syslog messages](https://discuss.elastic.co/t/how-to-parse-syslog-messages/65428)
**Author:** [@blueheart](https://discuss.elastic.co/u/blueheart)\ **Replies:** 2\ **Last updated:** [November 9, 2016, 9:24am UTC](https://discuss.elastic.co/t/how-to-parse-syslog-messages/65428 "2016-11-09T09:24:39Z")
Need support in parsing Syslog messages I want to parse the below syslog message. \<13\> 172.0.0.2 Nov 9 09:53:53 172.0.0.2 1 2016-11-09T09:57:14.018719Z \[interface\] \[172.0.0.2\] timestamp=2016-11-09 09:57:14 machine=… --- ## [Logstash Parsing Key Value Pair without quotes](https://discuss.elastic.co/t/logstash-parsing-key-value-pair-without-quotes/65457)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 0\ **Last updated:** [November 9, 2016, 9:08am UTC](https://discuss.elastic.co/t/logstash-parsing-key-value-pair-without-quotes/65457 "2016-11-09T09:08:29Z")
I have a situation where - extension="field1= value1\_1 value1\_2 |field2=value3\_1 value\_3\_2" The intend output is field1 = Value1\_1 value2\_2 field2 = Value2\_1 value2\_2 I have intention to use the kv filter The ch… --- ## [How to improve io performance in elasticsearch?](https://discuss.elastic.co/t/how-to-improve-io-performance-in-elasticsearch/65451)
**Author:** [@Evgeny\_Rachlenko](https://discuss.elastic.co/u/Evgeny_Rachlenko)\ **Replies:** 2\ **Last updated:** [November 9, 2016, 8:36am UTC](https://discuss.elastic.co/t/how-to-improve-io-performance-in-elasticsearch/65451 "2016-11-09T08:36:01Z")
Hello , I've followed document explain well how to improve the io performance : https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html but cluster still works too hards. The question… --- ## [How to make custom Logstash start parameters / arguments](https://discuss.elastic.co/t/how-to-make-custom-logstash-start-parameters-arguments/65165)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 5\ **Last updated:** [November 9, 2016, 6:41am UTC](https://discuss.elastic.co/t/how-to-make-custom-logstash-start-parameters-arguments/65165 "2016-11-09T06:41:56Z")
Hi, Logstash takes a lot of file parameters while starting like; --path.settings=/logstash-5.0.0/config/ I want to pass something like, for example, I have a jdbc input in logstash and how can I pass it through comm… --- ## [Logstash webhdfs output plugin creates folder and subfolders with user "alpha" even after mentioning user "beta" in webhdfs](https://discuss.elastic.co/t/logstash-webhdfs-output-plugin-creates-folder-and-subfolders-with-user-alpha-even-after-mentioning-user-beta-in-webhdfs/65433)
**Author:** [@krpatel19](https://discuss.elastic.co/u/krpatel19)\ **Replies:** 0\ **Last updated:** [November 9, 2016, 3:48am UTC](https://discuss.elastic.co/t/logstash-webhdfs-output-plugin-creates-folder-and-subfolders-with-user-alpha-even-after-mentioning-user-beta-in-webhdfs/65433 "2016-11-09T03:48:05Z")
Logstash webhdfs output plugin creates folder and subfolders with user "alpha" even after mentioning user "beta" in webhdfs. Although both user alpha and beta are present in hadoop cluster, and at logstash server as well… --- ## [Logstash Configuration Error with BlueLiv](https://discuss.elastic.co/t/logstash-configuration-error-with-blueliv/65416)
**Author:** [@ossec.junkie](https://discuss.elastic.co/u/ossec.junkie)\ **Replies:** 3\ **Last updated:** [November 9, 2016, 12:44am UTC](https://discuss.elastic.co/t/logstash-configuration-error-with-blueliv/65416 "2016-11-09T00:44:28Z")
Hi there- I'm pretty new to the ELK stack and have been tasked with using ELK Stack and BlueLiv Threat Intelligence Community. I have my basic setup configured for all ELK components, they start and what not however w… --- ## [Share variable between events](https://discuss.elastic.co/t/share-variable-between-events/65122)
**Author:** [@ze.gitan](https://discuss.elastic.co/u/ze.gitan)\ **Replies:** 1\ **Last updated:** [November 9, 2016, 12:09am UTC](https://discuss.elastic.co/t/share-variable-between-events/65122 "2016-11-09T00:09:35Z")
I'm currently working on a plugin that requires a piece of information to be shared between events. I've been looking over some articles, but don't see an appropriate way to do this. Essentially I have a connection objec… --- ## [Logstash 2.4 - S3 output strange ${message} in file](https://discuss.elastic.co/t/logstash-2-4-s3-output-strange-message-in-file/65059)
**Author:** [@rbgeno7](https://discuss.elastic.co/u/rbgeno7)\ **Replies:** 6\ **Last updated:** [November 9, 2016, 12:08am UTC](https://discuss.elastic.co/t/logstash-2-4-s3-output-strange-message-in-file/65059 "2016-11-09T00:08:26Z")
Hi - The S3 output seems to be working in that I'm getting messages, but instead of the parsed output I'm getting a message variable ${message}. This happens in both the temp location and in S3. Output - root@ip-xx… --- ## [Collectd index name](https://discuss.elastic.co/t/collectd-index-name/65386)
**Author:** [@Kernel\_Panic](https://discuss.elastic.co/u/Kernel_Panic)\ **Replies:** 5\ **Last updated:** [November 8, 2016, 11:35pm UTC](https://discuss.elastic.co/t/collectd-index-name/65386 "2016-11-08T23:35:51Z")
Hi there guys I'm configuring collectd to gather info from my docker containers, so far so good, but I'd like the to have a descriptive index name, now it's called like bellow, which was created automatically, I'd li… --- ## [Update Document without doc\_ID](https://discuss.elastic.co/t/update-document-without-doc-id/64996)
**Author:** [@TWalter](https://discuss.elastic.co/u/TWalter)\ **Replies:** 2\ **Last updated:** [November 8, 2016, 9:33pm UTC](https://discuss.elastic.co/t/update-document-without-doc-id/64996 "2016-11-08T21:33:55Z")
Hello there, i import a lot of data in my ES Indexes with Logstash. If i import the same data twice (maybe a error or overlapping with the importer) , the documents are stored twice in ES. Is there any possibility t… --- ## [Logstash input jmx does'nt work in logstash 5.0.0](https://discuss.elastic.co/t/logstash-input-jmx-doesnt-work-in-logstash-5-0-0/65269)
**Author:** [@anasoid](https://discuss.elastic.co/u/anasoid)\ **Replies:** 4\ **Last updated:** [November 8, 2016, 8:07pm UTC](https://discuss.elastic.co/t/logstash-input-jmx-doesnt-work-in-logstash-5-0-0/65269 "2016-11-08T20:07:10Z")
when installing plugin (/logstash-plugin install logstash-input-jmx) i have the folowing error : . . . Running bundle update will rebuild your snapshot from scratch, using only the gems in your Gemfile, which may res… --- ## [Creating a custom pattern(regx problem)](https://discuss.elastic.co/t/creating-a-custom-pattern-regx-problem/65245)
**Author:** [@w0lverine](https://discuss.elastic.co/u/w0lverine)\ **Replies:** 1\ **Last updated:** [November 8, 2016, 7:58pm UTC](https://discuss.elastic.co/t/creating-a-custom-pattern-regx-problem/65245 "2016-11-08T19:58:28Z")
I am trying to parse a log to only recongize nubmers but the regex, though, it is a valid regex I am still getting a compile error. For example: Custom pattern: (?(\[0,1,2,3\]) for: \[Priority: 0\] output: Compile ER… --- ## [Parse complex JSON with filter](https://discuss.elastic.co/t/parse-complex-json-with-filter/65251)
**Author:** [@vmcano](https://discuss.elastic.co/u/vmcano)\ **Replies:** 1\ **Last updated:** [November 8, 2016, 7:56pm UTC](https://discuss.elastic.co/t/parse-complex-json-with-filter/65251 "2016-11-08T19:56:15Z")
Hi all, I am trying to parse a complex JSON with nested JSONs and JSON filter is not working properly. It parses correctly the "first level" in the JSON structure, but it doesn't do the same with the nested JSONs. I … --- ## [Metrics filter + graphite output blocks pipeline during output failure](https://discuss.elastic.co/t/metrics-filter-graphite-output-blocks-pipeline-during-output-failure/65397)
**Author:** [@AlexClineBB](https://discuss.elastic.co/u/AlexClineBB)\ **Replies:** 0\ **Last updated:** [November 8, 2016, 7:48pm UTC](https://discuss.elastic.co/t/metrics-filter-graphite-output-blocks-pipeline-during-output-failure/65397 "2016-11-08T19:48:12Z")
Our logstash is configured to read HTTP logs from redis, filter them, and output to elasticsearch. During filtering, we're using the metrics plugin to generate a count of events being processed by logstash. Here's a simp… --- ## [ETA for v5.0.1](https://discuss.elastic.co/t/eta-for-v5-0-1/65252)
**Author:** [@sjivan](https://discuss.elastic.co/u/sjivan)\ **Replies:** 2\ **Last updated:** [November 8, 2016, 7:16pm UTC](https://discuss.elastic.co/t/eta-for-v5-0-1/65252 "2016-11-08T19:16:45Z")
Hi, There are a couple of plugin fixes that I need to pick up that are part of LS v5.0.1. One of them causes a LS crash so I'm unable to use v5.0. I'm not familiar with the release schedule and was wondering it v5.0.1 is… --- ## [Unable to install plugin logstash-filter-grok from git](https://discuss.elastic.co/t/unable-to-install-plugin-logstash-filter-grok-from-git/65387)
**Author:** [@sjivan](https://discuss.elastic.co/u/sjivan)\ **Replies:** 0\ **Last updated:** [November 8, 2016, 6:23pm UTC](https://discuss.elastic.co/t/unable-to-install-plugin-logstash-filter-grok-from-git/65387 "2016-11-08T18:23:49Z")
Hi, I'm trying to install the logstash-filter-grok plugin based on the latest code in git. I'm on a windows machine. I downloaded the source from git and ran gem build logstash-filter-grok.gemspec This produced the ou… --- ## [Is it possible to use a varchar/string for jdbc sql\_last\_value?](https://discuss.elastic.co/t/is-it-possible-to-use-a-varchar-string-for-jdbc-sql-last-value/64845)
**Author:** [@Kulasangar\_Gowrisang](https://discuss.elastic.co/u/Kulasangar_Gowrisang)\ **Replies:** 6\ **Last updated:** [November 8, 2016, 4:30pm UTC](https://discuss.elastic.co/t/is-it-possible-to-use-a-varchar-string-for-jdbc-sql-last-value/64845 "2016-11-08T16:30:18Z")
I've already tested the sql\_last\_value with integer and datetime type of fields in my jdbc plugin and it's working perfectly. I'm just curios, is there any possibility of using a varchar/string type of field, in orde… --- ## [Can't get filebeat to talk to logstash 5.0 via TLS/SSL - PrivateKeyConverter.generatePkcs8](https://discuss.elastic.co/t/cant-get-filebeat-to-talk-to-logstash-5-0-via-tls-ssl-privatekeyconverter-generatepkcs8/64427)
**Author:** [@robrant](https://discuss.elastic.co/u/robrant)\ **Replies:** 7\ **Last updated:** [November 8, 2016, 4:17pm UTC](https://discuss.elastic.co/t/cant-get-filebeat-to-talk-to-logstash-5-0-via-tls-ssl-privatekeyconverter-generatepkcs8/64427 "2016-11-08T16:17:30Z")
The question: Can someone help me figure out why I can't get filebeats to talk to logstash over TLS/SSL? ##The Error: Here's the error observed in logstash.log (on github gist because I'd exceeded the 5000 chr limit): … --- ## [How to Differentiate Logs from a Host](https://discuss.elastic.co/t/how-to-differentiate-logs-from-a-host/65132)
**Author:** [@Shashi\_Lanka](https://discuss.elastic.co/u/Shashi_Lanka)\ **Replies:** 3\ **Last updated:** [November 8, 2016, 1:42pm UTC](https://discuss.elastic.co/t/how-to-differentiate-logs-from-a-host/65132 "2016-11-08T13:42:55Z")
Usecase: I have a test service instance on a host where a log-stash agent is configured. This logs are stashed to a common elastic server. Now this will be test service instance will be cleaned up and another test serv… --- ## [How to add logstash gork filter and field](https://discuss.elastic.co/t/how-to-add-logstash-gork-filter-and-field/64582)
**Author:** [@zoram](https://discuss.elastic.co/u/zoram)\ **Replies:** 4\ **Last updated:** [November 8, 2016, 1:42pm UTC](https://discuss.elastic.co/t/how-to-add-logstash-gork-filter-and-field/64582 "2016-11-08T13:42:10Z")
I'm using Filebeat -\> Logstash -\> Elasticsearch -\> Kibana to have an overview of my glassfish log file. My log message look like this. \[#|2016-11- 01T11:29:33.347+0100|INFO|glassfish3.1.2|com.MachineProxy|\_ThreadID… --- ## [Work with Environment Variables in logstash.conf / if condition](https://discuss.elastic.co/t/work-with-environment-variables-in-logstash-conf-if-condition/65322)
**Author:** [@Flo\_B](https://discuss.elastic.co/u/Flo_B)\ **Replies:** 2\ **Last updated:** [November 8, 2016, 1:07pm UTC](https://discuss.elastic.co/t/work-with-environment-variables-in-logstash-conf-if-condition/65322 "2016-11-08T13:07:53Z")
Hey, i try to configure a 2nd output for "special" Data in logstash. First a tagged the special data in a filter: grok { match =\> { "source" =\> "%{GREEDYDATA}/specialservice/%{GREEDYDATA}\_access.log" } … --- ## [HTTPS input to Logstash](https://discuss.elastic.co/t/https-input-to-logstash/65331)
**Author:** [@shakticouple](https://discuss.elastic.co/u/shakticouple)\ **Replies:** 7\ **Last updated:** [November 8, 2016, 10:42am UTC](https://discuss.elastic.co/t/https-input-to-logstash/65331 "2016-11-08T10:42:10Z")
Hi, I am new to ELK. I am using Logstash Version: 2.4.0 I need to configure my Logstash to accept HTTPS Input only. I don't need any certificate Validation. I have tried some configuration with HTTP Input. But, was una… --- ## [Logstash Crash when SQS plugin fails](https://discuss.elastic.co/t/logstash-crash-when-sqs-plugin-fails/65332)
**Author:** [@chriscundill](https://discuss.elastic.co/u/chriscundill)\ **Replies:** 0\ **Last updated:** [November 8, 2016, 10:04am UTC](https://discuss.elastic.co/t/logstash-crash-when-sqs-plugin-fails/65332 "2016-11-08T10:04:33Z")
The logstash process is exiting when the SQS plugin receives a networking error. It's frustrating enough that there are no specifics provided around this networking error but the fact that it makes logstash fall over ma… --- ## ["Failed to install template... contact Elasticsearch"](https://discuss.elastic.co/t/failed-to-install-template-contact-elasticsearch/64687)
**Author:** [@bme](https://discuss.elastic.co/u/bme)\ **Replies:** 7\ **Last updated:** [November 8, 2016, 9:50am UTC](https://discuss.elastic.co/t/failed-to-install-template-contact-elasticsearch/64687 "2016-11-08T09:50:17Z")
I am testing elastic search with logstash and filebeats, trying to import our IIS log for analysing and viewing in Kibana. All latest version (5.0). I have run into a problem during the startup of logstash where it say… --- ## [How to filter uwsgi logs on the basis of date specified inside logs](https://discuss.elastic.co/t/how-to-filter-uwsgi-logs-on-the-basis-of-date-specified-inside-logs/65308)
**Author:** [@sumitgupta0001](https://discuss.elastic.co/u/sumitgupta0001)\ **Replies:** 3\ **Last updated:** [November 8, 2016, 9:31am UTC](https://discuss.elastic.co/t/how-to-filter-uwsgi-logs-on-the-basis-of-date-specified-inside-logs/65308 "2016-11-08T09:31:24Z")
\[pid: 5413|app: 0|req: 374528/2711349\] 107.167.105.4 () {82 vars in 1747 bytes} \[Tue Nov 8 11:47:28 2016\] GET /api/v1/core/capturedata/?tag\_id=a1aa493710fc4584aafd4c066119e6a0&cid=a762b82eba66493694058cf4602f6e5a =\> ge… --- ## [Logstash 5 not working with multiline filter](https://discuss.elastic.co/t/logstash-5-not-working-with-multiline-filter/65143)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 15\ **Last updated:** [November 8, 2016, 7:21am UTC](https://discuss.elastic.co/t/logstash-5-not-working-with-multiline-filter/65143 "2016-11-08T07:21:42Z")
Hello, I have a working configuration with Logstash 2.4.0 like this: input { file { path =\> "/path/\*" type =\> "type" start\_position =\> "beginning" } } filter { multiline { pattern =\> "^\ **Author:** [@w0lverine](https://discuss.elastic.co/u/w0lverine)\ **Replies:** 1\ **Last updated:** [November 8, 2016, 6:35am UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-simple-conf/65291 "2016-11-08T06:35:08Z") Following the COMBINEDAPACHELOG example I am trying to get logstash output the apache log from a specific file location but logstash just runs and nothing is parsed. Commands: ./logstash -f modbus.conf --config.test… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1080) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1082)