# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1055 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1056 --- ## [Comprehensive parent-child Logstash/Kibana solution from separate events](https://discuss.elastic.co/t/comprehensive-parent-child-logstash-kibana-solution-from-separate-events/42741)
**Author:** [@kpdude77](https://discuss.elastic.co/u/kpdude77)\ **Replies:** 3\ **Last updated:** [January 11, 2017, 11:44am UTC](https://discuss.elastic.co/t/comprehensive-parent-child-logstash-kibana-solution-from-separate-events/42741 "2017-01-11T11:44:38Z")
We have a db table that represents sessions, and are using the jdbc plugin to import into LS. We also have various logs related to the session (whose lines contain the same session id as in the db table) that we are usin… --- ## [How to send data to elasticsearch using logstash?](https://discuss.elastic.co/t/how-to-send-data-to-elasticsearch-using-logstash/71193)
**Author:** [@Yaswanth](https://discuss.elastic.co/u/Yaswanth)\ **Replies:** 2\ **Last updated:** [January 11, 2017, 9:34am UTC](https://discuss.elastic.co/t/how-to-send-data-to-elasticsearch-using-logstash/71193 "2017-01-11T09:34:27Z")
HI , I want to send a txt.file which contains some documents into elasticsearch using logstash . So, how can i configure my .config file in such a way that it sends the data to elasticsearch.. Thanks --- ## [How to use custom log4j2.properties](https://discuss.elastic.co/t/how-to-use-custom-log4j2-properties/70518)
**Author:** [@wenyao](https://discuss.elastic.co/u/wenyao)\ **Replies:** 3\ **Last updated:** [January 11, 2017, 9:34am UTC](https://discuss.elastic.co/t/how-to-use-custom-log4j2-properties/70518 "2017-01-11T09:34:24Z")
Hi, i need to run two separate instances of logstash with different configurations of log4j2.properties. Is it possible to specify a custom file on execution? If so, please share some examples. Thank you. --- ## [Database table update best practices](https://discuss.elastic.co/t/database-table-update-best-practices/71099)
**Author:** [@PasqualeC](https://discuss.elastic.co/u/PasqualeC)\ **Replies:** 0\ **Last updated:** [January 10, 2017, 2:29pm UTC](https://discuss.elastic.co/t/database-table-update-best-practices/71099 "2017-01-10T14:29:42Z")
Hello, I'm using ELK stack 5.1.1 on a linux machine. I would like to build an aggregation of information from different Databases through ELK and query it to extract some statistics. I started importing one table fro… --- ## [Logstash input plugin for cloudwatch logstreams](https://discuss.elastic.co/t/logstash-input-plugin-for-cloudwatch-logstreams/71184)
**Author:** [@Neera\_Vats](https://discuss.elastic.co/u/Neera_Vats)\ **Replies:** 0\ **Last updated:** [January 11, 2017, 7:46am UTC](https://discuss.elastic.co/t/logstash-input-plugin-for-cloudwatch-logstreams/71184 "2017-01-11T07:46:29Z")
Hi All, I need to process log streams that are being uploaded to cloudwatch using "put\_log\_events" API. Is there a logstash plugin available that uses "get\_log\_events" API to read log streams? https://boto3.readt… --- ## [Logstash output Performance](https://discuss.elastic.co/t/logstash-output-performance/71181)
**Author:** [@ElsayedAwd](https://discuss.elastic.co/u/ElsayedAwd)\ **Replies:** 0\ **Last updated:** [January 11, 2017, 7:20am UTC](https://discuss.elastic.co/t/logstash-output-performance/71181 "2017-01-11T07:20:42Z")
I 'am using Elasticsearch 5.1.1 , logstash 5.1.1 ,I imported 3 millions rows from sqlserver into elastic via logstash in 2 hours I have 1 single windows machine with 4GB Ram , core I 3 :slight\_smile: is there … --- ## [Malformed Error](https://discuss.elastic.co/t/malformed-error/71151)
**Author:** [@romspacenut](https://discuss.elastic.co/u/romspacenut)\ **Replies:** 6\ **Last updated:** [January 11, 2017, 6:39am UTC](https://discuss.elastic.co/t/malformed-error/71151 "2017-01-11T06:39:26Z")
Can't figure out these errors I've been seeing. Been seeing a lot of these kind errors in my logstash.log file: "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Invalid format: \\"2017-01-10T12:36:22-08:00… --- ## [How to install logstash plugin from source](https://discuss.elastic.co/t/how-to-install-logstash-plugin-from-source/71167)
**Author:** [@sdaruna](https://discuss.elastic.co/u/sdaruna)\ **Replies:** 1\ **Last updated:** [January 11, 2017, 5:02am UTC](https://discuss.elastic.co/t/how-to-install-logstash-plugin-from-source/71167 "2017-01-11T05:02:08Z")
Hi, We need to install datadog\_metrics plugin in logstash. Due to our network settings, logstash defaul install plugin command cannot connect outside world and get the required libraries it seems. We end up receiving … --- ## [Elasticsearch output - index name pattern](https://discuss.elastic.co/t/elasticsearch-output-index-name-pattern/36428)
**Author:** [@zombie101](https://discuss.elastic.co/u/zombie101)\ **Replies:** 9\ **Last updated:** [January 11, 2017, 4:34am UTC](https://discuss.elastic.co/t/elasticsearch-output-index-name-pattern/36428 "2017-01-11T04:34:22Z")
The elasticsearch output takes a setting for the index name, with a default of 'logstash-YYYY.MM.dd'. Where does the 'YYYY.MM.dd' come from? Are there other options, such as time (e.g. HHmmss), week of year, day of yea… --- ## [Check for existence of a field / sub-field](https://discuss.elastic.co/t/check-for-existence-of-a-field-sub-field/71160)
**Author:** [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Replies:** 1\ **Last updated:** [January 11, 2017, 12:15am UTC](https://discuss.elastic.co/t/check-for-existence-of-a-field-sub-field/71160 "2017-01-11T00:15:52Z")
For my use case, I want to check the existence of a specific field translogid in log message and if it is not present, I want to discard it (Say raise \_grokparsefailure). I followed the steps at Event Dependent Configura… --- ## [Problem with gathering logs from syslog with logstash](https://discuss.elastic.co/t/problem-with-gathering-logs-from-syslog-with-logstash/71107)
**Author:** [@Wojter](https://discuss.elastic.co/u/Wojter)\ **Replies:** 8\ **Last updated:** [January 10, 2017, 9:04pm UTC](https://discuss.elastic.co/t/problem-with-gathering-logs-from-syslog-with-logstash/71107 "2017-01-10T21:04:51Z")
Hello there, I'm new here! What I want to achieve I want to set ELK stack on docker containers and configure it to listen and process logs coming from various applications in JSON form already, over tcp/udp/syslog (w… --- ## [Delete attributes by value in Logstash 5.1.1](https://discuss.elastic.co/t/delete-attributes-by-value-in-logstash-5-1-1/71149)
**Author:** [@rcanzanese](https://discuss.elastic.co/u/rcanzanese)\ **Replies:** 2\ **Last updated:** [January 10, 2017, 9:09pm UTC](https://discuss.elastic.co/t/delete-attributes-by-value-in-logstash-5-1-1/71149 "2017-01-10T21:09:17Z")
We have recently upgraded from Logstash 2.2.4 to Logstash 5.1.1. Previously, we used the following to remove "-" valued attributes: code =\> "event.to\_hash.keep\_if{|k,v| v != '-'}" The new Event API breaks this code:… --- ## [Logstash pull date out of json event](https://discuss.elastic.co/t/logstash-pull-date-out-of-json-event/71010)
**Author:** [@Chris\_Clifton](https://discuss.elastic.co/u/Chris_Clifton)\ **Replies:** 2\ **Last updated:** [January 10, 2017, 9:08pm UTC](https://discuss.elastic.co/t/logstash-pull-date-out-of-json-event/71010 "2017-01-10T21:08:15Z")
Wondering what's best way to pull date and timezone of a structured json event that comes into logstash, event looks like : { "datetime" =\> { "date" =\> "2017-01-10 00:05:25.768826", … --- ## [Jdbc input plugin with scheduler opens 96 extra connections to Oracle](https://discuss.elastic.co/t/jdbc-input-plugin-with-scheduler-opens-96-extra-connections-to-oracle/71009)
**Author:** [@iti](https://discuss.elastic.co/u/iti)\ **Replies:** 1\ **Last updated:** [January 10, 2017, 8:59pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-with-scheduler-opens-96-extra-connections-to-oracle/71009 "2017-01-10T20:59:22Z")
I am using ES 5.1.1 LS 5.1.1 and logstash-input-jdbc (4.1.3) With the following input config: input { jdbc { # Oracle jdbc connection string to Oracle database jdbc\_connection\_string =\> "jdbc:oracle… --- ## [Logstash crashes when using conditional on nested json field](https://discuss.elastic.co/t/logstash-crashes-when-using-conditional-on-nested-json-field/71141)
**Author:** [@choyuri](https://discuss.elastic.co/u/choyuri)\ **Replies:** 0\ **Last updated:** [January 10, 2017, 7:29pm UTC](https://discuss.elastic.co/t/logstash-crashes-when-using-conditional-on-nested-json-field/71141 "2017-01-10T19:29:03Z")
logstash Version : 5.1.1 Plugins installed: \[\] JVM version: 1.8.0 OS version: centos 7.3.1611 Description of the problem including expected versus actual behavior: Steps to reproduce: I'm using rabbitmq input p… --- ## [Add Field example in Logstash Date Filter should contain date-specific relevant values](https://discuss.elastic.co/t/add-field-example-in-logstash-date-filter-should-contain-date-specific-relevant-values/71123)
**Author:** [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Replies:** 2\ **Last updated:** [January 10, 2017, 7:20pm UTC](https://discuss.elastic.co/t/add-field-example-in-logstash-date-filter-should-contain-date-specific-relevant-values/71123 "2017-01-10T19:20:08Z")
I believe it would be more helpful if the logstash date filter documentation for add\_field contains values that are pertaining to date rather than "Hello World, from %{host}". Thoughts? --- ## [Logstash is waiting for new line even if in content is XML and multiline codec is used](https://discuss.elastic.co/t/logstash-is-waiting-for-new-line-even-if-in-content-is-xml-and-multiline-codec-is-used/71103)
**Author:** [@mich32](https://discuss.elastic.co/u/mich32)\ **Replies:** 1\ **Last updated:** [January 10, 2017, 4:35pm UTC](https://discuss.elastic.co/t/logstash-is-waiting-for-new-line-even-if-in-content-is-xml-and-multiline-codec-is-used/71103 "2017-01-10T16:35:44Z")
Logstash version: 5.1 I have not formatted xml files (each xml is in one line) and there is no new line. Im using multiline codec: file { path =\> "/data/xmls/\*\*/\*.xml" codec =\> multiline { … --- ## [Adding tags after executing ruby code filter](https://discuss.elastic.co/t/adding-tags-after-executing-ruby-code-filter/37347)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 6\ **Last updated:** [January 10, 2017, 3:45pm UTC](https://discuss.elastic.co/t/adding-tags-after-executing-ruby-code-filter/37347 "2017-01-10T15:45:54Z")
Hi, I have a filter which check all inbound fields for leading underscores and if it finds any changes the underscore to other text. I'd like to tag messages that have been modified with a tag. How do I do this? Atm, t… --- ## [How to remplace my bash script by logstash?](https://discuss.elastic.co/t/how-to-remplace-my-bash-script-by-logstash/70797)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 7\ **Last updated:** [January 10, 2017, 12:54pm UTC](https://discuss.elastic.co/t/how-to-remplace-my-bash-script-by-logstash/70797 "2017-01-10T12:54:59Z")
Hello, (Ah before all, excuse me for my bad english !) Currently : I developed many many many BASH scripts to check text files, manipulate with zcat, awk, grep, sum... Results are send into a stats file. My Cacti… --- ## [Grouping several words in one pattern](https://discuss.elastic.co/t/grouping-several-words-in-one-pattern/70956)
**Author:** [@DannyBiker](https://discuss.elastic.co/u/DannyBiker)\ **Replies:** 4\ **Last updated:** [January 10, 2017, 11:12am UTC](https://discuss.elastic.co/t/grouping-several-words-in-one-pattern/70956 "2017-01-10T11:12:09Z")
Hello all, I just discovered gork today and I'm wondering how one can group a sequence of words into one syntax. Here's an example. The input : \> 83.149.9.216 - - \[04/Jan/2015:05:13:42 +0000\] "GET /presentations/logs… --- ## [Sqlserver logstash configration](https://discuss.elastic.co/t/sqlserver-logstash-configration/71029)
**Author:** [@ElsayedAwd](https://discuss.elastic.co/u/ElsayedAwd)\ **Replies:** 7\ **Last updated:** [January 10, 2017, 10:01am UTC](https://discuss.elastic.co/t/sqlserver-logstash-configration/71029 "2017-01-10T10:01:16Z")
I have windows as well as Elasticsearch 5.1.1 , I made these configuration with sqlserver and logstash . please what is the error of these configurations . input { jdbc { jdbc\_driver\_library =\> "D:\\Usefull\_Jars\\sq… --- ## [Plug-in install problem](https://discuss.elastic.co/t/plug-in-install-problem/71050)
**Author:** [@atillademirel](https://discuss.elastic.co/u/atillademirel)\ **Replies:** 0\ **Last updated:** [January 10, 2017, 9:56am UTC](https://discuss.elastic.co/t/plug-in-install-problem/71050 "2017-01-10T09:56:31Z")
i try to install a gem file, but i am getting connection refused. i cannot use proxy, so i want to install as a offline solution. could you please help me ? ES 5.1 /data/install/logstash/logstash-5.1.1/bin\]# .… --- ## [Log parsing using Logstash 2.1](https://discuss.elastic.co/t/log-parsing-using-logstash-2-1/71042)
**Author:** [@aviral\_srivastava](https://discuss.elastic.co/u/aviral_srivastava)\ **Replies:** 0\ **Last updated:** [January 10, 2017, 9:38am UTC](https://discuss.elastic.co/t/log-parsing-using-logstash-2-1/71042 "2017-01-10T09:38:23Z")
Hi, I have the following log:- 11/24/2016 06:56:22 - Process(6416.3) User(f\_wmq\_prod) Program(amqzmur0.exe) Host(ST-W4005) Installation(WMQ75) VRMF(7.5.0.1) QMgr(P0401) AMQ… --- ## [No suitable driver found for jdbc](https://discuss.elastic.co/t/no-suitable-driver-found-for-jdbc/70957)
**Author:** [@JohnG](https://discuss.elastic.co/u/JohnG)\ **Replies:** 2\ **Last updated:** [January 10, 2017, 8:47am UTC](https://discuss.elastic.co/t/no-suitable-driver-found-for-jdbc/70957 "2017-01-10T08:47:35Z")
Hi, Hope someone can help. I have the following jar files in a directory named /etc/logstash/jars/ -rw-r--r-- 1 logstash logstash 987191 Jan 9 15:08 mysql-connector-java-5.1.38.jar -rw-r--r-- 1 logstash logstash 9… --- ## [Windows File Server](https://discuss.elastic.co/t/windows-file-server/71014)
**Author:** [@Hassan\_Nashrah](https://discuss.elastic.co/u/Hassan_Nashrah)\ **Replies:** 3\ **Last updated:** [January 10, 2017, 8:45am UTC](https://discuss.elastic.co/t/windows-file-server/71014 "2017-01-10T08:45:22Z")
I need to monitor which user accessed which files in network share. at offline times. from windows file server is there a way i can get this information and send to ELK to monitor. i need to grab their IP address/AD u… --- ## [Error: illegal\_argument\_exception", "reason"=\> FIELD is not an IP string literal.?](https://discuss.elastic.co/t/error-illegal-argument-exception-reason-field-is-not-an-ip-string-literal/70664)
**Author:** [@sliddjur](https://discuss.elastic.co/u/sliddjur)\ **Replies:** 3\ **Last updated:** [January 10, 2017, 8:19am UTC](https://discuss.elastic.co/t/error-illegal-argument-exception-reason-field-is-not-an-ip-string-literal/70664 "2017-01-10T08:19:20Z")
I have netflow and ipfix data coming in from logstash. When logstash tries to send this to elasticsearch I get these errors. I tried to tidy up my code using the alter filter instead of all the ifs' and mutate. I am gett… --- ## [How to override logstash @timestamp value with my date time field from the logs](https://discuss.elastic.co/t/how-to-override-logstash-timestamp-value-with-my-date-time-field-from-the-logs/71026)
**Author:** [@ankkha](https://discuss.elastic.co/u/ankkha)\ **Replies:** 6\ **Last updated:** [January 10, 2017, 7:58am UTC](https://discuss.elastic.co/t/how-to-override-logstash-timestamp-value-with-my-date-time-field-from-the-logs/71026 "2017-01-10T07:58:37Z")
My logs Dec 12 13:45:43 \[0x80000001\]\[xsltmsg\]\[notice\]|6583|3018|0x00000000-0x00000000|235| Dec 12 14:45:43 \[0x80000001\]\[xsltmsg\]\[notice\]|6583|3018|0x00000000-0x00000000|205| My logstash config filter { grok { mat… --- ## [Pid(/var/run/logstash.pid) file is not being created](https://discuss.elastic.co/t/pid-var-run-logstash-pid-file-is-not-being-created/70771)
**Author:** [@blnprasad](https://discuss.elastic.co/u/blnprasad)\ **Replies:** 3\ **Last updated:** [January 10, 2017, 7:46am UTC](https://discuss.elastic.co/t/pid-var-run-logstash-pid-file-is-not-being-created/70771 "2017-01-10T07:46:49Z")
Hi, I started using latest logstash(5.0) on ubuntu16.0 version and finding that /var/run/logstash.pid is not being created. I'm able run log stash successfully using sysctrl start logstash.service but pid file is not… --- ## [How to convert gc logs from mb kb to bytes](https://discuss.elastic.co/t/how-to-convert-gc-logs-from-mb-kb-to-bytes/70659)
**Author:** [@jagadees](https://discuss.elastic.co/u/jagadees)\ **Replies:** 1\ **Last updated:** [January 10, 2017, 6:41am UTC](https://discuss.elastic.co/t/how-to-convert-gc-logs-from-mb-kb-to-bytes/70659 "2017-01-10T06:41:28Z")
filter { match =\> { "store" =\> "^%{\[@metadata\]\[store\_number\]:float}(?)\[kKmMgGtT\])(?\[kKmMgGtT\])(?\[kKmMgGtT\])" } } mutate { add\_field =\> { "\[@metadata\]\[before\_memory\]\[INT:after\_memory\]\[NOTSPA… --- ## [Struggling with mutate - not uploading after conversion](https://discuss.elastic.co/t/struggling-with-mutate-not-uploading-after-conversion/70329)
**Author:** [@Msaxton](https://discuss.elastic.co/u/Msaxton)\ **Replies:** 3\ **Last updated:** [January 10, 2017, 2:32am UTC](https://discuss.elastic.co/t/struggling-with-mutate-not-uploading-after-conversion/70329 "2017-01-10T02:32:46Z")
Hello all, I am brand new to Elastic and working with a CSV. After my initial upload, everything converted to strings so I used mutate to convert. However, my file no longer uploads. I've gone through all the manua… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1054) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1056)