# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=497 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 498 --- ## [Handle timeouts with the syslog output](https://discuss.elastic.co/t/handle-timeouts-with-the-syslog-output/194194)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 1\ **Last updated:** [August 7, 2019, 11:33am UTC](https://discuss.elastic.co/t/handle-timeouts-with-the-syslog-output/194194 "2019-08-07T11:33:46Z")
Hi, We have a use case of sending logs to two ELK stacks and letting the first ELK stack forward some events to a second ELK stack. So messages arrive at ELK-Stack1. Logstash receives the messages and outputs to its El… --- ## [Filter based on input (Filebeat/Winlogbeat) in logstash](https://discuss.elastic.co/t/filter-based-on-input-filebeat-winlogbeat-in-logstash/194189)
**Author:** [@rplus](https://discuss.elastic.co/u/rplus)\ **Replies:** 2\ **Last updated:** [August 7, 2019, 11:15am UTC](https://discuss.elastic.co/t/filter-based-on-input-filebeat-winlogbeat-in-logstash/194189 "2019-08-07T11:15:30Z")
Hello! I've configured winlogbeat and filebeat on a windows host. Both filebeat and winlogbeat have the same logstash output. What I would like to do, and I what I am having trouble with is that I would like to apply d… --- ## [MYSQL to ELASTICSEARCH multiple tables at once](https://discuss.elastic.co/t/mysql-to-elasticsearch-multiple-tables-at-once/194040)
**Author:** [@Karnish\_Master](https://discuss.elastic.co/u/Karnish_Master)\ **Replies:** 3\ **Last updated:** [August 7, 2019, 10:59am UTC](https://discuss.elastic.co/t/mysql-to-elasticsearch-multiple-tables-at-once/194040 "2019-08-07T10:59:02Z")
I am porting data from mysql to elastisearch , I am using type =\> "table\_type" in input and output to bifurcate into different tables ,data is going proper for each index however i am getting extra field named type with… --- ## [Logstash how to solve the problem about jackson-databind CVE-2019-12384](https://discuss.elastic.co/t/logstash-how-to-solve-the-problem-about-jackson-databind-cve-2019-12384/194134)
**Author:** [@muyuqj](https://discuss.elastic.co/u/muyuqj)\ **Replies:** 2\ **Last updated:** [August 7, 2019, 10:33am UTC](https://discuss.elastic.co/t/logstash-how-to-solve-the-problem-about-jackson-databind-cve-2019-12384/194134 "2019-08-07T10:33:54Z")
logstash how to solve the problem about jackson-databind CVE-2019-12384 my path is: /usr/local/logstash/logstash-core/lib/jars/jackson-annotations-2.9.5.jar it is seem the jackson lib version is 2.9.5, This bug … --- ## [Issues with kv filtering/ElasticSearch filtering](https://discuss.elastic.co/t/issues-with-kv-filtering-elasticsearch-filtering/194019)
**Author:** [@jshepheard](https://discuss.elastic.co/u/jshepheard)\ **Replies:** 3\ **Last updated:** [August 7, 2019, 9:18am UTC](https://discuss.elastic.co/t/issues-with-kv-filtering-elasticsearch-filtering/194019 "2019-08-07T09:18:06Z")
Hello. I am having a strange issue with ELK stack which I am struggling to get my head around. (ElasticSearch 6.5). My logstash config contains: kv { field\_split =\> ", " value\_split =\> "=" } which is working fine, … --- ## [Problem with Imap input plugin and Office 365 Public Folders](https://discuss.elastic.co/t/problem-with-imap-input-plugin-and-office-365-public-folders/194182)
**Author:** [@SaschaR](https://discuss.elastic.co/u/SaschaR)\ **Replies:** 1\ **Last updated:** [August 7, 2019, 9:08am UTC](https://discuss.elastic.co/t/problem-with-imap-input-plugin-and-office-365-public-folders/194182 "2019-08-07T09:08:43Z")
Hey there, We have migrated from Kerio Mailserver to Office 365. On both Servers we have used Public Folders to get Mails with the imap input Plugin into Kibana. Since this Migration, i'am not able to get Mails with th… --- ## [\_grokparsefailure problem](https://discuss.elastic.co/t/grokparsefailure-problem/194038)
**Author:** [@Manal\_Sadgal](https://discuss.elastic.co/u/Manal_Sadgal)\ **Replies:** 2\ **Last updated:** [August 7, 2019, 7:42am UTC](https://discuss.elastic.co/t/grokparsefailure-problem/194038 "2019-08-07T07:42:22Z")
Hello, I have a problem with my filter, i get the "\_grokparsefailure" tag when the concerned logs are processed. Here is my filter file: filter { if \[source\] == "/var/log/auth.log"{ grok { match =\> \[ "message", "%… --- ## [Postgres Logstash JSON parsing with JSON and Ruby](https://discuss.elastic.co/t/postgres-logstash-json-parsing-with-json-and-ruby/194146)
**Author:** [@milanbgd011](https://discuss.elastic.co/u/milanbgd011)\ **Replies:** 0\ **Last updated:** [August 7, 2019, 5:44am UTC](https://discuss.elastic.co/t/postgres-logstash-json-parsing-with-json-and-ruby/194146 "2019-08-07T05:44:03Z")
Spent a lot of time trying to debug this simple scenario, so here it is for others to use. I did try to find an answer here but everything was partial or not working, besides a ruby plugin implementation which is overkil… --- ## [Kafka input plugin serialization](https://discuss.elastic.co/t/kafka-input-plugin-serialization/194138)
**Author:** [@real\_legend](https://discuss.elastic.co/u/real_legend)\ **Replies:** 1\ **Last updated:** [August 7, 2019, 5:24am UTC](https://discuss.elastic.co/t/kafka-input-plugin-serialization/194138 "2019-08-07T05:24:27Z")
Hi there in docs i was readed that serialization method of kafka pulgin is org.apache.kafka.common.serialization.Stringserializer and org.apache.kafka.common.serialization.StringDeserializer i want to use some other m… --- ## [Log lines appearing twice](https://discuss.elastic.co/t/log-lines-appearing-twice/193532)
**Author:** [@Eva](https://discuss.elastic.co/u/Eva)\ **Replies:** 4\ **Last updated:** [August 7, 2019, 5:02am UTC](https://discuss.elastic.co/t/log-lines-appearing-twice/193532 "2019-08-07T05:02:26Z")
Hello I'm shipping json logs to Logstash and have used below config: input { beats { port =\> 5044 } beats { port =\> 5045 } } filter{ if (\[fields\]\[app\] == "abc") { json { source =\> "message" } split { fiel… --- ## [Issue converting number field to date](https://discuss.elastic.co/t/issue-converting-number-field-to-date/189497)
**Author:** [@moti.mor](https://discuss.elastic.co/u/moti.mor)\ **Replies:** 3\ **Last updated:** [July 10, 2019, 4:39am UTC](https://discuss.elastic.co/t/issue-converting-number-field-to-date/189497 "2019-07-10T04:39:42Z")
Hi there, I'm trying to convert the following number field to date. Please see pic attached. and it's not working... please see my logstash configuration attahced on another photo. this is urgent issue for me , h… --- ## [Parse xml with logstash](https://discuss.elastic.co/t/parse-xml-with-logstash/193910)
**Author:** [@wzxmt\_l](https://discuss.elastic.co/u/wzxmt_l)\ **Replies:** 2\ **Last updated:** [August 7, 2019, 1:29am UTC](https://discuss.elastic.co/t/parse-xml-with-logstash/193910 "2019-08-07T01:29:40Z")
Hi All,Here is my xml,I want to parse this xml, how can I parse it? \ \ \10028603\ \*\*\*\*\*\*\*\* \0\ \1000\ **Author:** [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Replies:** 7\ **Last updated:** [August 6, 2019, 11:20pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074 "2019-08-06T23:20:24Z") Hi all, I am using version 7.2. I encountered some trouble while using the filter elasticsearch plugin. The behavior is very random, some of the data uploaded correctly but I also get some error Here is the log from … --- ## [Manually send json object to beats listener](https://discuss.elastic.co/t/manually-send-json-object-to-beats-listener/193908)
**Author:** [@nnet](https://discuss.elastic.co/u/nnet)\ **Replies:** 4\ **Last updated:** [August 6, 2019, 6:57pm UTC](https://discuss.elastic.co/t/manually-send-json-object-to-beats-listener/193908 "2019-08-06T18:57:40Z")
Hi, I have a working beats listener configured in logstash, and a filebeats prospector shipping json data from a log file. input { beats { port =\> 5047 include\_codec\_tag =\> false } } I want to test arbitrar… --- ## [Conditional statements in logstash.conf](https://discuss.elastic.co/t/conditional-statements-in-logstash-conf/192108)
**Author:** [@griffer98](https://discuss.elastic.co/u/griffer98)\ **Replies:** 10\ **Last updated:** [August 6, 2019, 6:35pm UTC](https://discuss.elastic.co/t/conditional-statements-in-logstash-conf/192108 "2019-08-06T18:35:22Z")
I am parsing xml documents but there are different types with different names. Each type of document has different tags so each document will require different xpath expressions. What I want to do is write a conditional … --- ## [Grok until first period](https://discuss.elastic.co/t/grok-until-first-period/193909)
**Author:** [@uklipse](https://discuss.elastic.co/u/uklipse)\ **Replies:** 4\ **Last updated:** [August 6, 2019, 6:05pm UTC](https://discuss.elastic.co/t/grok-until-first-period/193909 "2019-08-06T18:05:10Z")
Hi, new to creating parsing rules so bear with me. I need to parse up until the first period but can't seem to make it work. Below is what I'm trying to parse. GlobalProtect gateway user authentication succeeded. Login … --- ## [How to grok parse a pipe | delimited string of key-value pairs (sorta like FIX)](https://discuss.elastic.co/t/how-to-grok-parse-a-pipe-delimited-string-of-key-value-pairs-sorta-like-fix/194073)
**Author:** [@kmiklas](https://discuss.elastic.co/u/kmiklas)\ **Replies:** 1\ **Last updated:** [August 6, 2019, 5:49pm UTC](https://discuss.elastic.co/t/how-to-grok-parse-a-pipe-delimited-string-of-key-value-pairs-sorta-like-fix/194073 "2019-08-06T17:49:59Z")
Hello All, I need to grok a pipe-delimited string of values in a grok line; for example: |NAME=keith|DAY=wednesday|TIME=09:27:423227|DATE=08/06/2019|amount=68.23|currency=USD|etc... What is the easiest way to do this?… --- ## [Custom field to process DOB field calculating 'age'](https://discuss.elastic.co/t/custom-field-to-process-dob-field-calculating-age/194018)
**Author:** [@asad\_ali](https://discuss.elastic.co/u/asad_ali)\ **Replies:** 12\ **Last updated:** [August 6, 2019, 5:44pm UTC](https://discuss.elastic.co/t/custom-field-to-process-dob-field-calculating-age/194018 "2019-08-06T17:44:46Z")
I have sample data as:- Sr No,CNIC,Cell Phone,Date Of Birth,Gender 1,3710144892657,3215510968,06/06/1961,male 2,3710155658463,3238547712,6/17/1952,male 3,3710153121295,3015357676,01/01/1956,male 4,3710116643365,32157019… --- ## [Kafka input - Type missing](https://discuss.elastic.co/t/kafka-input-type-missing/194064)
**Author:** [@asacr](https://discuss.elastic.co/u/asacr)\ **Replies:** 3\ **Last updated:** [August 6, 2019, 5:37pm UTC](https://discuss.elastic.co/t/kafka-input-type-missing/194064 "2019-08-06T17:37:00Z")
Hello, We've got an existing ELK environment using filebeat primarily. Everything is working fine there, but I'm trying to add some data using the Kafka input plugin and running into an issue. Logstash Version: 5.4.0 … --- ## [I want to take Lat Long from MYSQL database and map as geopoint in elasticsearch index](https://discuss.elastic.co/t/i-want-to-take-lat-long-from-mysql-database-and-map-as-geopoint-in-elasticsearch-index/194060)
**Author:** [@Sandeep\_Bind](https://discuss.elastic.co/u/Sandeep_Bind)\ **Replies:** 2\ **Last updated:** [August 6, 2019, 4:48pm UTC](https://discuss.elastic.co/t/i-want-to-take-lat-long-from-mysql-database-and-map-as-geopoint-in-elasticsearch-index/194060 "2019-08-06T16:48:58Z")
Below is my logstash configuration file. input { jdbc { jdbc\_driver\_library =\> "mysql-connector-java-8.0.16/mysql-connector-java-8.0.16.jar" jdbc\_driver\_class =\> "com.mysql.cj.jdbc.Driver" jdbc\_connection\_string =\> … --- ## [Jdbc input plugin and a null value to a output jdbc plugin](https://discuss.elastic.co/t/jdbc-input-plugin-and-a-null-value-to-a-output-jdbc-plugin/192962)
**Author:** [@Henrique\_Santiago](https://discuss.elastic.co/u/Henrique_Santiago)\ **Replies:** 1\ **Last updated:** [August 6, 2019, 4:16pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-and-a-null-value-to-a-output-jdbc-plugin/192962 "2019-08-06T16:16:45Z")
I'm using a jdbc input(sqlserver) to input data in a postgresql with jdbc output. input { jdbc { jdbc\_connection\_string =\> "\*\*\*" jdbc\_driver\_library =\> "\*\*\*" jdbc\_user =\> "\*\*\*" jdbc\_p… --- ## [Centralised rufus scheduling for all configuration files ( JDBC plugin)](https://discuss.elastic.co/t/centralised-rufus-scheduling-for-all-configuration-files-jdbc-plugin/194047)
**Author:** [@Duella](https://discuss.elastic.co/u/Duella)\ **Replies:** 0\ **Last updated:** [August 6, 2019, 3:55pm UTC](https://discuss.elastic.co/t/centralised-rufus-scheduling-for-all-configuration-files-jdbc-plugin/194047 "2019-08-06T15:55:22Z")
I would like to centralise my Rufus schedule for my Logstash configuration files, all of them are currently using the JDBC plugin to pull information out of my database. I was wondering if anyone has done this before? --- ## [How to remove duplicate values in logstash](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-logstash/193769)
**Author:** [@Sajal](https://discuss.elastic.co/u/Sajal)\ **Replies:** 1\ **Last updated:** [August 6, 2019, 3:45pm UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-values-in-logstash/193769 "2019-08-06T15:45:24Z")
Hi . Unable to remove duplicate values from logstash . "src" =\> \[ \[0\] "10.207.73.1", \[1\] "10.207.73.1" \], "agentName" =\> \[ \[0\] "sl73tdsdb1q01 ", \[1\] "sl73tdsdb1q01 " \], "StoredProc" =\> \[ \[0\] "False", \[1\] "Fal… --- ## [Logstash shutsdown without error](https://discuss.elastic.co/t/logstash-shutsdown-without-error/193873)
**Author:** [@Kevin\_Juliano](https://discuss.elastic.co/u/Kevin_Juliano)\ **Replies:** 3\ **Last updated:** [August 5, 2019, 8:28pm UTC](https://discuss.elastic.co/t/logstash-shutsdown-without-error/193873 "2019-08-05T20:28:41Z")
I am still learning about elasticsearch and logstash, I am trying to index a table from postgresDB but every time I run it, it keeps shutting down here's my logstash config: # file: index1.conf input { jdbc { … --- ## [Gathering RabbitMQ payload when it has a filed message](https://discuss.elastic.co/t/gathering-rabbitmq-payload-when-it-has-a-filed-message/194009)
**Author:** [@pszemesy](https://discuss.elastic.co/u/pszemesy)\ **Replies:** 3\ **Last updated:** [August 6, 2019, 2:44pm UTC](https://discuss.elastic.co/t/gathering-rabbitmq-payload-when-it-has-a-filed-message/194009 "2019-08-06T14:44:11Z")
Hi All, I'm trying to store the original RabbitMQ payload to orig\_msg field with Logstash, but the payload itself contains a filed called message. Logstash add\_field mutate { add\_field =\> { "orig\_msg" =\> "%{message}" } … --- ## [Sql last value can only store timestamp?](https://discuss.elastic.co/t/sql-last-value-can-only-store-timestamp/193926)
**Author:** [@pyerunka](https://discuss.elastic.co/u/pyerunka)\ **Replies:** 1\ **Last updated:** [August 6, 2019, 12:27pm UTC](https://discuss.elastic.co/t/sql-last-value-can-only-store-timestamp/193926 "2019-08-06T12:27:24Z")
Hello, Can anyone help me in this? can sql last value stores only timestamp? I am trying to use one number column using jdbc plugin and comparing it with sql last value. I am getting an error, SQLSyntaxErrorException… --- ## [Logstash input kafka 7.1.1](https://discuss.elastic.co/t/logstash-input-kafka-7-1-1/194007)
**Author:** [@Erik\_Parienty1](https://discuss.elastic.co/u/Erik_Parienty1)\ **Replies:** 0\ **Last updated:** [August 6, 2019, 12:26pm UTC](https://discuss.elastic.co/t/logstash-input-kafka-7-1-1/194007 "2019-08-06T12:26:07Z")
kafka Client Not Following Broker IP Change if i will kill a broker ( 10.20.60.45 and bring up the same broker with different ip ( 10.20.43.170 ) i see in the logs Initiating connection to node 10.20.43.170:9092 (id: … --- ## [Kafka ingest performance issues](https://discuss.elastic.co/t/kafka-ingest-performance-issues/193639)
**Author:** [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Replies:** 5\ **Last updated:** [August 6, 2019, 12:19pm UTC](https://discuss.elastic.co/t/kafka-ingest-performance-issues/193639 "2019-08-06T12:19:04Z")
Im sending all my beat\* data into kafka and reading it from there via logstash (running on kubernetes). I use kafka for lots of other purposes and I know we can read 1M records a second using a simple consumer. When I lo… --- ## [Logstash not processing data](https://discuss.elastic.co/t/logstash-not-processing-data/193726)
**Author:** [@mzidy](https://discuss.elastic.co/u/mzidy)\ **Replies:** 5\ **Last updated:** [August 6, 2019, 10:14am UTC](https://discuss.elastic.co/t/logstash-not-processing-data/193726 "2019-08-06T10:14:36Z")
Hi guys, Trying to send data Logstash -\> Elasticsearch (WIN 10, both are running on localhost, configs are tested and are ok) and I'm experiencing issue that actual data is not sent (this is the only warning I get): \[2… --- ## [Use elastic for laravel logs](https://discuss.elastic.co/t/use-elastic-for-laravel-logs/193961)
**Author:** [@farshadf](https://discuss.elastic.co/u/farshadf)\ **Replies:** 0\ **Last updated:** [August 6, 2019, 8:05am UTC](https://discuss.elastic.co/t/use-elastic-for-laravel-logs/193961 "2019-08-06T08:05:20Z")
hi and good day , i want to know if there is any good source or plugin to use the laravel log system in elk i know there are some tutorials out there like using graylog but none working with elk now unfortunately . th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=496) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=498)