# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=913 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 914 --- ## [Logstash not creating (some?) indexes in ES, no error in logs](https://discuss.elastic.co/t/logstash-not-creating-some-indexes-in-es-no-error-in-logs/99450)
**Author:** [@unreal4u](https://discuss.elastic.co/u/unreal4u)\ **Replies:** 3\ **Last updated:** [September 7, 2017, 8:25am UTC](https://discuss.elastic.co/t/logstash-not-creating-some-indexes-in-es-no-error-in-logs/99450 "2017-09-07T08:25:15Z")
Hi all :slight\_smile: I have the following configuration: input { beats { port =\> "5044" host =\> "0.0.0.0" } rabbitmq { host =\> "localhost" port =\> 5672 user =\> "yyyy… --- ## [What does Aggregate flush call with {:final=\>false} mean?](https://discuss.elastic.co/t/what-does-aggregate-flush-call-with-final-false-mean/99686)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 0\ **Last updated:** [September 7, 2017, 8:22am UTC](https://discuss.elastic.co/t/what-does-aggregate-flush-call-with-final-false-mean/99686 "2017-09-07T08:22:29Z")
Hi, I am trying to aggregate and I don’t see the aggregated logs in Kibana/ES. When I put logstash logs in debug mode I get the following: “Aggregate flush call with {:final=\>false}” While googling this I can’t find a… --- ## [Getting grokparsefailure with Bro logs, even though everything parses....?](https://discuss.elastic.co/t/getting-grokparsefailure-with-bro-logs-even-though-everything-parses/98969)
**Author:** [@Blaxican707](https://discuss.elastic.co/u/Blaxican707)\ **Replies:** 11\ **Last updated:** [September 7, 2017, 8:00am UTC](https://discuss.elastic.co/t/getting-grokparsefailure-with-bro-logs-even-though-everything-parses/98969 "2017-09-07T08:00:54Z")
I’m parsing Bro Logs and currently everything is working perfectly, EXCEPT that I am receiving grokparsefailures. Everything is moved over into logstash, I confirmed by looking into the stdout log and then i match it wit… --- ## [Trying to aggregate, but I can't see any aggregated logs](https://discuss.elastic.co/t/trying-to-aggregate-but-i-cant-see-any-aggregated-logs/99575)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 0\ **Last updated:** [September 6, 2017, 11:54am UTC](https://discuss.elastic.co/t/trying-to-aggregate-but-i-cant-see-any-aggregated-logs/99575 "2017-09-06T11:54:59Z")
Hi, I am trying to clone and aggregate, but I can’t see any cloned or aggregated logs. input { beats { port =\> 5044 } } filter { json{ source =\> "message" } clone { clones =\> \["clone"… --- ## [Logstash keep creating field despite the dynamic\_mapping being deactivated](https://discuss.elastic.co/t/logstash-keep-creating-field-despite-the-dynamic-mapping-being-deactivated/99391)
**Author:** [@Melvyn\_Peignon](https://discuss.elastic.co/u/Melvyn_Peignon)\ **Replies:** 5\ **Last updated:** [September 7, 2017, 6:27am UTC](https://discuss.elastic.co/t/logstash-keep-creating-field-despite-the-dynamic-mapping-being-deactivated/99391 "2017-09-07T06:27:48Z")
I have defined my own template to be used by logstash where I have deactivate the dynamic mapping: { "my\_index": { "order": 0, "template": "my\_index", "settings": { "index": { "mapper": { "dynamic": "… --- ## [Does XPATH of xml-filter support XPATH 2.0?](https://discuss.elastic.co/t/does-xpath-of-xml-filter-support-xpath-2-0/99518)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [September 7, 2017, 6:12am UTC](https://discuss.elastic.co/t/does-xpath-of-xml-filter-support-xpath-2-0/99518 "2017-09-07T06:12:32Z")
Does XPATH of xml-filter support XPATH 2.0? I am using, a XPATH with for condition and it is not working xpath =\> \[ "for $r in //row return string-join($r//testing//value, ',')","Testing" \] this XPATH doesnt work, bu… --- ## [Parsing a JSON array of objects with field-value](https://discuss.elastic.co/t/parsing-a-json-array-of-objects-with-field-value/99656)
**Author:** [@tatdat](https://discuss.elastic.co/u/tatdat)\ **Replies:** 1\ **Last updated:** [September 7, 2017, 5:37am UTC](https://discuss.elastic.co/t/parsing-a-json-array-of-objects-with-field-value/99656 "2017-09-07T05:37:41Z")
I have some log look like { "ExtendedProperties": \[ { "Value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393", "Name": "UserAg… --- ## [How can I create a new term by subtracting to other terms?](https://discuss.elastic.co/t/how-can-i-create-a-new-term-by-subtracting-to-other-terms/99401)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 1\ **Last updated:** [September 7, 2017, 5:31am UTC](https://discuss.elastic.co/t/how-can-i-create-a-new-term-by-subtracting-to-other-terms/99401 "2017-09-07T05:31:47Z")
Hi, I have 2 terms X and Y. How can I create a third term Z that is X-Y in my logs through logstash? --- ## [Concatenation in Ruby Code](https://discuss.elastic.co/t/concatenation-in-ruby-code/99614)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 1\ **Last updated:** [September 7, 2017, 5:29am UTC](https://discuss.elastic.co/t/concatenation-in-ruby-code/99614 "2017-09-07T05:29:55Z")
Hi - Could someone please let me know how to concatenate a couple of strings in Ruby code over Logstash. Below is the snippet of code: for i in 0…arr.size - 1 cpu\_hash = \[cpu\_hash, {‘CPU#{i}’ =\> arr\[i\].to\_i}\].reduce &… --- ## [How to load big table to es with logstash](https://discuss.elastic.co/t/how-to-load-big-table-to-es-with-logstash/99502)
**Author:** [@gaobo\_yu](https://discuss.elastic.co/u/gaobo_yu)\ **Replies:** 5\ **Last updated:** [September 7, 2017, 5:21am UTC](https://discuss.elastic.co/t/how-to-load-big-table-to-es-with-logstash/99502 "2017-09-07T05:21:26Z")
Hi : I hava a mysql table named “orders” with ten million records ,I am trying to import all the data to Es with the plugin logstash-input-jdbc, but it seems cost a long time to finish this task, I have optimized all th… --- ## [Logstash-plugin and JVM options](https://discuss.elastic.co/t/logstash-plugin-and-jvm-options/99646)
**Author:** [@jakub.krauz](https://discuss.elastic.co/u/jakub.krauz)\ **Replies:** 2\ **Last updated:** [September 7, 2017, 3:18am UTC](https://discuss.elastic.co/t/logstash-plugin-and-jvm-options/99646 "2017-09-07T03:18:26Z")
Hi, I’m having issues invoking logstash-plugin command while logstash is running. It fails with ‘Cannot allocate memory’ error. I believe the reason is it uses the same jvm.options file as logstash which defines 3GB jav… --- ## [Unable to send lof files on disk to Elastic](https://discuss.elastic.co/t/unable-to-send-lof-files-on-disk-to-elastic/99501)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 12\ **Last updated:** [September 7, 2017, 2:23am UTC](https://discuss.elastic.co/t/unable-to-send-lof-files-on-disk-to-elastic/99501 "2017-09-07T02:23:38Z")
Hi, I’m having some problems with ingesting log files I have on disk to Elastic. For testing purposes I have 1 plain text file that I want to push to Elastic however nothing is happening. I don’t see anything in the log… --- ## [Logstash 6.0 beta-1 can't work with IBM JRE 8 due to old JRuby OpenSSL](https://discuss.elastic.co/t/logstash-6-0-beta-1-cant-work-with-ibm-jre-8-due-to-old-jruby-openssl/98547)
**Author:** [@AllenShi](https://discuss.elastic.co/u/AllenShi)\ **Replies:** 2\ **Last updated:** [September 7, 2017, 2:20am UTC](https://discuss.elastic.co/t/logstash-6-0-beta-1-cant-work-with-ibm-jre-8-due-to-old-jruby-openssl/98547 "2017-09-07T02:20:43Z")
I installed Logstash 5.5.1, 5.5.2 and even 6.0-beta1 bundled with another third party plugin to work with IBM JRE 8. The error “no cipher match” is thrown into logstash-plain.log. Per some deep investigation, this issue… --- ## [Autodetect\_column\_names & different header names mixup](https://discuss.elastic.co/t/autodetect-column-names-different-header-names-mixup/99650)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [September 7, 2017, 2:18am UTC](https://discuss.elastic.co/t/autodetect-column-names-different-header-names-mixup/99650 "2017-09-07T02:18:57Z")
Hi, I have a piece of hardware that is outputting 4 .csv log files. The headers are mostly the same but not always and in same cases the same header exists in all 4 files but in a different column (e.g. CN0 exists in al… --- ## [Logstash-input-redis fails to install due to ruby version 2.2.2](https://discuss.elastic.co/t/logstash-input-redis-fails-to-install-due-to-ruby-version-2-2-2/99641)
**Author:** [@alfmatos](https://discuss.elastic.co/u/alfmatos)\ **Replies:** 0\ **Last updated:** [September 7, 2017, 12:05am UTC](https://discuss.elastic.co/t/logstash-input-redis-fails-to-install-due-to-ruby-version-2-2-2/99641 "2017-09-07T00:05:45Z")
When trying to install logstash-input-redis, the plugin install fails due to the redis gem requiring ruby version \>= 2.2.2. Running: JARS\_DEBUG=true JARS\_VERBOSE=true DEBUG=1 /usr/share/logstash/bin/logstash-plugin upd… --- ## [Data logging via uart](https://discuss.elastic.co/t/data-logging-via-uart/99281)
**Author:** [@Nivetha\_Sukumar](https://discuss.elastic.co/u/Nivetha_Sukumar)\ **Replies:** 1\ **Last updated:** [September 6, 2017, 7:33pm UTC](https://discuss.elastic.co/t/data-logging-via-uart/99281 "2017-09-06T19:33:01Z")
I am new to Logstash. I would like to log the real time Bluetooth packets using Logstash and analyse it using elastic search. As of now, I use a serial terminal program called Real term to log the data. Can I use logstas… --- ## [Logstash could not find any input plugin named 'journald', NameError](https://discuss.elastic.co/t/logstash-could-not-find-any-input-plugin-named-journald-nameerror/97471)
**Author:** [@f9atanzh](https://discuss.elastic.co/u/f9atanzh)\ **Replies:** 7\ **Last updated:** [September 6, 2017, 5:23pm UTC](https://discuss.elastic.co/t/logstash-could-not-find-any-input-plugin-named-journald-nameerror/97471 "2017-09-06T17:23:48Z")
Hello Logstash Community, I’m looking for some help resolving a NameError problem when trying to run the journald input plugin. More specifically - I’m building a docker image for Logstash 5.5.1 and installing the jour… --- ## [Are input and output plugins required](https://discuss.elastic.co/t/are-input-and-output-plugins-required/99583)
**Author:** [@marinoc](https://discuss.elastic.co/u/marinoc)\ **Replies:** 5\ **Last updated:** [September 6, 2017, 2:21pm UTC](https://discuss.elastic.co/t/are-input-and-output-plugins-required/99583 "2017-09-06T14:21:05Z")
In doing some debugging for a project, I happened to run a configtest (-t) on a config file that only had a filter plugin section, but NO input and NO output. To my surprised, it passed the config test. If I start Logst… --- ## [Parsing my Zyxel USG110 CEF syslog file](https://discuss.elastic.co/t/parsing-my-zyxel-usg110-cef-syslog-file/98714)
**Author:** [@dlex51](https://discuss.elastic.co/u/dlex51)\ **Replies:** 7\ **Last updated:** [September 6, 2017, 1:51pm UTC](https://discuss.elastic.co/t/parsing-my-zyxel-usg110-cef-syslog-file/98714 "2017-09-06T13:51:30Z")
Hello everybody, i’m new to ELK Stack, but with some exceptions everything worked (installation) like a charme. Now I’m trying to visualize my router logs with Kibana. Therefore i receive the syslogs from my router via… --- ## [Logstash problem with config regex](https://discuss.elastic.co/t/logstash-problem-with-config-regex/99437)
**Author:** [@Fonten](https://discuss.elastic.co/u/Fonten)\ **Replies:** 0\ **Last updated:** [September 5, 2017, 1:41pm UTC](https://discuss.elastic.co/t/logstash-problem-with-config-regex/99437 "2017-09-05T13:41:39Z")
Hello all, I have the following log entry: 2017-08-29 01:10:11.111 \[http-noo-111-exe-1\] TRACE com.javasystemsolutions.xml.gateway.Actions - The XML Gateway encountered an error. The message was Server with id OPA is no… --- ## [Logstash 5.5.1 input kafka 0.9 creates a massive duplication of events](https://discuss.elastic.co/t/logstash-5-5-1-input-kafka-0-9-creates-a-massive-duplication-of-events/99568)
**Author:** [@JMNDev](https://discuss.elastic.co/u/JMNDev)\ **Replies:** 0\ **Last updated:** [September 6, 2017, 11:20am UTC](https://discuss.elastic.co/t/logstash-5-5-1-input-kafka-0-9-creates-a-massive-duplication-of-events/99568 "2017-09-06T11:20:29Z")
I have the following case. 1 filebeat sending events to kafka then 1 logstash reading from kafka and sending to elasticsearch What is happenning is a massive duplication of events in some random periodicity in Elastics… --- ## [Manually adding geoip data fails](https://discuss.elastic.co/t/manually-adding-geoip-data-fails/99562)
**Author:** [@admlko](https://discuss.elastic.co/u/admlko)\ **Replies:** 0\ **Last updated:** [September 6, 2017, 10:53am UTC](https://discuss.elastic.co/t/manually-adding-geoip-data-fails/99562 "2017-09-06T10:53:24Z")
Hi, I would like to rewrite/add geoip data manually to events to/from 172.40.40.0/24 and 172.40.41.0/24 subnets. The filter is configured like this: # Geo-IP lookup if \[src\_ip\] { # Fix public IP subnets if \[src\_ip… --- ## [For join I need split and aggregation](https://discuss.elastic.co/t/for-join-i-need-split-and-aggregation/99539)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 0\ **Last updated:** [September 6, 2017, 8:52am UTC](https://discuss.elastic.co/t/for-join-i-need-split-and-aggregation/99539 "2017-09-06T08:52:54Z")
Hi in a different question I asked how to do “Join” and got an answer to do: “to put in LS a split filter that will duplicate the logs. and on the duplicated part I will use the aggregation” Now I started with the spli… --- ## [Unavailable\_shards\_exception reason=\> filebeat primary shard is not active](https://discuss.elastic.co/t/unavailable-shards-exception-reason-filebeat-primary-shard-is-not-active/99526)
**Author:** [@Swaroop\_Chandre](https://discuss.elastic.co/u/Swaroop_Chandre)\ **Replies:** 3\ **Last updated:** [September 6, 2017, 8:32am UTC](https://discuss.elastic.co/t/unavailable-shards-exception-reason-filebeat-primary-shard-is-not-active/99526 "2017-09-06T08:32:02Z")
Hello can anyone help me with this. Getting following error in logstash logs \[2017-09-05T23:47:53,660\]\[INFO \]\[logstash.outputs.elasticsearch\] retrying failed action with response code: 503 ({"type"=\>"unavailable\_shards\_e… --- ## [\_geoip\_parse\_failure on logstash config](https://discuss.elastic.co/t/-geoip-parse-failure-on-logstash-config/99314)
**Author:** [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\ **Replies:** 9\ **Last updated:** [September 6, 2017, 8:16am UTC](https://discuss.elastic.co/t/-geoip-parse-failure-on-logstash-config/99314 "2017-09-06T08:16:39Z")
Hi all, i have configured packet beat, filebeat and metricbeat. i want to use it’s geoIp feature and therefore i have used filter in logtsash config file. but for each and evry configuration i am getting \_geoip\_parse\_… --- ## [File rotation duplicate data parsed by LS. duplicate data in ES](https://discuss.elastic.co/t/file-rotation-duplicate-data-parsed-by-ls-duplicate-data-in-es/98083)
**Author:** [@sezanawa](https://discuss.elastic.co/u/sezanawa)\ **Replies:** 0\ **Last updated:** [August 23, 2017, 1:31pm UTC](https://discuss.elastic.co/t/file-rotation-duplicate-data-parsed-by-ls-duplicate-data-in-es/98083 "2017-08-23T13:31:16Z")
Hallo Eveybody I am using ELK for Log parsing. I have a Job which dowload Logfiles after every 15 mins to a central log directory. Logstash is configured to parse log files from central directory and send to ES. as i n… --- ## [From logstash i need to send email to gmail](https://discuss.elastic.co/t/from-logstash-i-need-to-send-email-to-gmail/99419)
**Author:** [@suganth](https://discuss.elastic.co/u/suganth)\ **Replies:** 7\ **Last updated:** [September 6, 2017, 8:08am UTC](https://discuss.elastic.co/t/from-logstash-i-need-to-send-email-to-gmail/99419 "2017-09-06T08:08:19Z")
Hi, I have to send the email from my logstash. my configuration is , What are the prerequisites to achieve the requirement apart from the below configuration. email { to =\> 'xyz@gmail.com' via =\> 'smtp' address =\>… --- ## [Logstash unable to listen to Beats Error: event executor terminated](https://discuss.elastic.co/t/logstash-unable-to-listen-to-beats-error-event-executor-terminated/99506)
**Author:** [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Replies:** 2\ **Last updated:** [September 6, 2017, 7:37am UTC](https://discuss.elastic.co/t/logstash-unable-to-listen-to-beats-error-event-executor-terminated/99506 "2017-09-06T07:37:43Z")
I attempting to push logs from PacketBeat (Windows) to Logstash to Elastic Cloud. However above error keep looping in /var/log/logstash/logstash-plain.log. Thanks for assisting \[ERROR\]\[logstash.pipeline \] A plu… --- ## [Custom Logstash docker image creation](https://discuss.elastic.co/t/custom-logstash-docker-image-creation/99496)
**Author:** [@sayeedch](https://discuss.elastic.co/u/sayeedch)\ **Replies:** 2\ **Last updated:** [September 6, 2017, 7:23am UTC](https://discuss.elastic.co/t/custom-logstash-docker-image-creation/99496 "2017-09-06T07:23:06Z")
Is there a guide that describes how to create your own Logstash image, especially for Kubernetes/Openshift? I’m looking for something similar to this: --- ## [PeopleSoft \*.tracesql files](https://discuss.elastic.co/t/peoplesoft-tracesql-files/99525)
**Author:** [@talial](https://discuss.elastic.co/u/talial)\ **Replies:** 0\ **Last updated:** [September 6, 2017, 7:04am UTC](https://discuss.elastic.co/t/peoplesoft-tracesql-files/99525 "2017-09-06T07:04:33Z")
Hello, Is there a way to insert PeopleSoft \*.tracesql files into elasticsearch to analyze them? Maybe using logstash? If there is a step by step documentation it will be very helpfull PeopleCode Trace: The PeopleCod… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=912) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=914)