# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=562 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 563 --- ## [How to filter multiple content types in Logstash](https://discuss.elastic.co/t/how-to-filter-multiple-content-types-in-logstash/177577)
**Author:** [@jagadeeshkoka](https://discuss.elastic.co/u/jagadeeshkoka)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 2:48pm UTC](https://discuss.elastic.co/t/how-to-filter-multiple-content-types-in-logstash/177577 "2019-04-19T14:48:35Z")
Hi, I need to design a system to read different types of Payloads and a Json Log file. I am able to read both of them and create Indexes in ElS (Filebeat--\> Logstash--\>ES--\>Kiabana) individually. But now I have two p… --- ## [Logstash-filter-verifier @timestamp issue](https://discuss.elastic.co/t/logstash-filter-verifier-timestamp-issue/177563)
**Author:** [@SManorathna](https://discuss.elastic.co/u/SManorathna)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 2:10pm UTC](https://discuss.elastic.co/t/logstash-filter-verifier-timestamp-issue/177563 "2019-04-19T14:10:18Z")
Hi, I am trying to use logstash-filter-verifier (https://github.com/magnusbaeck/logstash-filter-verifier) for verifying my logstash filter. There is a simple issue and I think that I am doing a simple mistake here, but … --- ## [Date parsing issue in ELK Logstash with the custom Java timestamp format of logs](https://discuss.elastic.co/t/date-parsing-issue-in-elk-logstash-with-the-custom-java-timestamp-format-of-logs/177282)
**Author:** [@kcmenon](https://discuss.elastic.co/u/kcmenon)\ **Replies:** 4\ **Last updated:** [April 19, 2019, 2:07pm UTC](https://discuss.elastic.co/t/date-parsing-issue-in-elk-logstash-with-the-custom-java-timestamp-format-of-logs/177282 "2019-04-19T14:07:39Z")
Following is the sample logs recieved from java application 2019-04-11 9:08:22:562 Log 1 2019-04-11 9:08:22:660 Log 2 2019-04-11 9:08:43:79 Log 3 2019-04-11 9:08:43:156 Log 4 From above logs, I'm facing issue with … --- ## [Grokparsefailure, Geoip lookup failure error](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure-error/177568)
**Author:** [@khergner](https://discuss.elastic.co/u/khergner)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 2:03pm UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure-error/177568 "2019-04-19T14:03:54Z")
Hi, I searched but could not solution. Why don't know error get \[Grokparsefailure, Geoip lookup failure\] I don't understansd where the error is. I think grok pattern because used online grok tester so correct data.I i… --- ## [Logstash not uploading the entire json file](https://discuss.elastic.co/t/logstash-not-uploading-the-entire-json-file/177628)
**Author:** [@Prabhav](https://discuss.elastic.co/u/Prabhav)\ **Replies:** 0\ **Last updated:** [April 19, 2019, 2:03pm UTC](https://discuss.elastic.co/t/logstash-not-uploading-the-entire-json-file/177628 "2019-04-19T14:03:30Z")
I am using logstash to upload a json file to elasticsearch and it always fails to upload the data after certain size. I am not able to figure out reason for this. At the end it just shows unresponsive script slowing … --- ## [Logstash does not create fields using grok](https://discuss.elastic.co/t/logstash-does-not-create-fields-using-grok/177615)
**Author:** [@Sam\_Kisada](https://discuss.elastic.co/u/Sam_Kisada)\ **Replies:** 5\ **Last updated:** [April 19, 2019, 1:10pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-fields-using-grok/177615 "2019-04-19T13:10:32Z")
Hi! This is another "help me with grok" topic, so please help... :blush: Source Log (simple f2b): 2019-04-18 16:39:03,454 fail2ban.actions \[25673\]: NOTICE \[postfix-sasl\] Ban 185.234.218.237 2019-04-18 19:19:20… --- ## [Getting \_split\_type\_failure while splitting xml, not sure why but records are processed](https://discuss.elastic.co/t/getting-split-type-failure-while-splitting-xml-not-sure-why-but-records-are-processed/177608)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 12:59pm UTC](https://discuss.elastic.co/t/getting-split-type-failure-while-splitting-xml-not-sure-why-but-records-are-processed/177608 "2019-04-19T12:59:28Z")
Hello All, Based on an advice from earlier post, I created a config responsible for splitting my xml into multiple events. It looks like this: input { file { path =\> "/mnt/abc/\*.xml" start\_position =\> "beginning" … --- ## [JDBC Input parameters adds a N'' within the query](https://discuss.elastic.co/t/jdbc-input-parameters-adds-a-n-within-the-query/176880)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 12:36pm UTC](https://discuss.elastic.co/t/jdbc-input-parameters-adds-a-n-within-the-query/176880 "2019-04-19T12:36:40Z")
Hi Guys, I'm trying to exact resultset from Microsoft SQL Server using JDBC but i have stumbled upon an issue. Here's the config: input { jdbc { parameters =\> { database =\> ABC } jdbc\_driver\_library =\> "/usr/sha… --- ## [Protobuf codec error "undefined method \`parse'"](https://discuss.elastic.co/t/protobuf-codec-error-undefined-method-parse/177596)
**Author:** [@vaisov](https://discuss.elastic.co/u/vaisov)\ **Replies:** 0\ **Last updated:** [April 19, 2019, 10:02am UTC](https://discuss.elastic.co/t/protobuf-codec-error-undefined-method-parse/177596 "2019-04-19T10:02:48Z")
Hello, we want to use protobuf codec to get messages from Kafka, for that I converted following proto scheme to ruby using 3.5.0 google converter. Logstash starts the pipeline without errors, but when I publish messages… --- ## [Upgrade dilemma](https://discuss.elastic.co/t/upgrade-dilemma/177603)
**Author:** [@agiorgi](https://discuss.elastic.co/u/agiorgi)\ **Replies:** 0\ **Last updated:** [April 19, 2019, 10:21am UTC](https://discuss.elastic.co/t/upgrade-dilemma/177603 "2019-04-19T10:21:51Z")
Hello there! I'd like to update my ELK stack to the 7.0 version. Still, i don't understand whether i have to turn off beats, forwarders etc.. When upgrading logstash. Assuming that i have 6 instances of logstash, behi… --- ## [I have to concatenate below two fields](https://discuss.elastic.co/t/i-have-to-concatenate-below-two-fields/177228)
**Author:** [@hemantkhokhar](https://discuss.elastic.co/u/hemantkhokhar)\ **Replies:** 2\ **Last updated:** [April 19, 2019, 9:49am UTC](https://discuss.elastic.co/t/i-have-to-concatenate-below-two-fields/177228 "2019-04-19T09:49:05Z")
%{URIPATH:url}%{DATA:\[status\]} I am using this but not success add\_field =\> \["%{DATA:actual-url}", "%{URIPATH:url} %{DATA:\[status\]}"\] --- ## [Unknown setting '"index"' for elasticsearch](https://discuss.elastic.co/t/unknown-setting-index-for-elasticsearch/177497)
**Author:** [@joostdb](https://discuss.elastic.co/u/joostdb)\ **Replies:** 2\ **Last updated:** [April 19, 2019, 9:46am UTC](https://discuss.elastic.co/t/unknown-setting-index-for-elasticsearch/177497 "2019-04-19T09:46:32Z")
I'm new to the elastic search concept. I followed multiple tutorials to install but I'm getting these errors: Unknown setting '"index"', '"host'" and "'document\_type'" for elasticsearch The output of sudo -Hu root /us… --- ## [Logstash 7 documentation with filebeat 7](https://discuss.elastic.co/t/logstash-7-documentation-with-filebeat-7/175960)
**Author:** [@willouuu](https://discuss.elastic.co/u/willouuu)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 9:16am UTC](https://discuss.elastic.co/t/logstash-7-documentation-with-filebeat-7/175960 "2019-04-19T09:16:54Z")
Hello I prepare the future migration in 7.0 :nerd\_face: Filebeat renamed fields in 7.0 ( https://www.elastic.co/guide/en/beats/libbeat/7.0/breaking-changes-7.0.html ) suddenly the filters here do not match anymore : … --- ## [Can't create indices](https://discuss.elastic.co/t/cant-create-indices/174492)
**Author:** [@HuanW](https://discuss.elastic.co/u/HuanW)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 8:57am UTC](https://discuss.elastic.co/t/cant-create-indices/174492 "2019-04-19T08:57:16Z")
Hi, The question is one of my input setting doesn't work. The way I check is I go to Kibana and check on the index management page, and I didn't see the indice. The input that have problem is type=\>"anotherApplog" co… --- ## [Logstash config issue](https://discuss.elastic.co/t/logstash-config-issue/177541)
**Author:** [@udayradhika](https://discuss.elastic.co/u/udayradhika)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 2:22am UTC](https://discuss.elastic.co/t/logstash-config-issue/177541 "2019-04-19T02:22:35Z")
input { jdbc { jdbc\_connection\_string =\> "jdbc:mysql://localhost:3306/flask" jdbc\_user =\> "root" jdbc\_password =\> "coppergate51" jdbc\_driver\_library =\> "/Users/upakalapati/Downloads/mysql-connector-java-5.1.46-bin.j… --- ## [Logstash 7.0 Error](https://discuss.elastic.co/t/logstash-7-0-error/177553)
**Author:** [@mweisler](https://discuss.elastic.co/u/mweisler)\ **Replies:** 0\ **Last updated:** [April 19, 2019, 1:40am UTC](https://discuss.elastic.co/t/logstash-7-0-error/177553 "2019-04-19T01:40:20Z")
I keep getting this error after just updating from logstash version 6.6.2 \[logstash.log\]\[ERROR\] A plugin had an unrecoverable error. Will restart this plugin. Pipeline\_id:main Plugin: \ **Author:** [@Tim\_Flowers](https://discuss.elastic.co/u/Tim_Flowers)\ **Replies:** 6\ **Last updated:** [April 18, 2019, 8:46pm UTC](https://discuss.elastic.co/t/filter-help-please/177532 "2019-04-18T20:46:59Z") I am banging my head on something and I am hoping that someone can help me out. I am also open to better ways to do what I am doing if anyone has any ideas. Currently I get logs from multiple systems that I am trying t… --- ## [Suggestion: improve documentation on Logstash-to-logstash page, certs expire in one month](https://discuss.elastic.co/t/suggestion-improve-documentation-on-logstash-to-logstash-page-certs-expire-in-one-month/177535)
**Author:** [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Replies:** 0\ **Last updated:** [April 18, 2019, 8:00pm UTC](https://discuss.elastic.co/t/suggestion-improve-documentation-on-logstash-to-logstash-page-certs-expire-in-one-month/177535 "2019-04-18T20:00:10Z")
the page https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html explains, step by step, how to setup your infrastructure to use Lumberjack for LogStash to LogStash communication. However, a key point is missing.… --- ## [Logstash - how to configure CSV filter for joining 2 CSV files](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files/177531)
**Author:** [@teju](https://discuss.elastic.co/u/teju)\ **Replies:** 2\ **Last updated:** [April 18, 2019, 7:44pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files/177531 "2019-04-18T19:44:17Z")
I have 2 CSV Files 1 files have ip address and vsad id, the other CVS File has on ip address. may be or may not be both files has same ip address. based on file1 we should map to file 2 and we need to asign the vsad id u… --- ## [Logstash Stops if Grok Patterns file/pattern does not exist - Logstash 6.3.2](https://discuss.elastic.co/t/logstash-stops-if-grok-patterns-file-pattern-does-not-exist-logstash-6-3-2/177534)
**Author:** [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Replies:** 0\ **Last updated:** [April 18, 2019, 7:39pm UTC](https://discuss.elastic.co/t/logstash-stops-if-grok-patterns-file-pattern-does-not-exist-logstash-6-3-2/177534 "2019-04-18T19:39:00Z")
Hi All, I have noticed that Logstash will stop and not recover if a grok pattern is reference from a file that does not exist. Currently we use an git repo to manage our grok patterns in a set of folders & files. Once … --- ## [Converting a date input as a string into a new format](https://discuss.elastic.co/t/converting-a-date-input-as-a-string-into-a-new-format/177498)
**Author:** [@thadc](https://discuss.elastic.co/u/thadc)\ **Replies:** 1\ **Last updated:** [April 18, 2019, 4:29pm UTC](https://discuss.elastic.co/t/converting-a-date-input-as-a-string-into-a-new-format/177498 "2019-04-18T16:29:21Z")
Hello, I have a need to convert a field value for a log time stamp from its current ISO8601 into a format of yyyy.mm.dd hh.mm.ss on output. I have a filter similar to the following: filter { . . (grok section to pars… --- ## [\[closed\]Logstash output with "else-if" not working](https://discuss.elastic.co/t/closed-logstash-output-with-else-if-not-working/177509)
**Author:** [@lgwapnitsky](https://discuss.elastic.co/u/lgwapnitsky)\ **Replies:** 1\ **Last updated:** [April 18, 2019, 4:26pm UTC](https://discuss.elastic.co/t/closed-logstash-output-with-else-if-not-working/177509 "2019-04-18T16:26:55Z")
I'm tagging my cloudflare data in the http\_poller input in logstash. I have the following config in my output file, but the cf section is getting skipped and going right to the last stanza, writing the wrong index name … --- ## [Insert first lines as fields to each line](https://discuss.elastic.co/t/insert-first-lines-as-fields-to-each-line/177385)
**Author:** [@charles.liu](https://discuss.elastic.co/u/charles.liu)\ **Replies:** 1\ **Last updated:** [April 18, 2019, 2:01pm UTC](https://discuss.elastic.co/t/insert-first-lines-as-fields-to-each-line/177385 "2019-04-18T14:01:53Z")
Hi all I am new to ELK and Logstash. Here is my question: I have raw log file, content like below: AA:aa BB:bb a1,b1,c1,d1 a2,b2,c2,d2 a3,b3,c3,d3 a4,b4,c4,d4 My question is how to setup the configure so my fina… --- ## [Problem using timeout\_timestamp\_field in the aggregate filter](https://discuss.elastic.co/t/problem-using-timeout-timestamp-field-in-the-aggregate-filter/177379)
**Author:** [@lens](https://discuss.elastic.co/u/lens)\ **Replies:** 2\ **Last updated:** [April 18, 2019, 2:00pm UTC](https://discuss.elastic.co/t/problem-using-timeout-timestamp-field-in-the-aggregate-filter/177379 "2019-04-18T14:00:21Z")
Hi, I'm trying to use the aggregate filter with a timeout based on a timestamp in the event. In the filter, I have set timeout\_timestamp\_field =\> "start" When I run logstash, I get the error undefined method 'time' … --- ## [Logstash elastic search filter plugin](https://discuss.elastic.co/t/logstash-elastic-search-filter-plugin/177457)
**Author:** [@Gomathi\_Meena](https://discuss.elastic.co/u/Gomathi_Meena)\ **Replies:** 1\ **Last updated:** [April 18, 2019, 1:55pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-filter-plugin/177457 "2019-04-18T13:55:46Z")
Hi , I'm quite new to ELK stack . What is the purpose of options like aggregation\_fields,sort and enable\_sort in elasticsearch filter plugin ? how do they work ? and i have even tried with the following configuration ,t… --- ## [Turning one .xml into multiple events where not all values are always filled](https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 4\ **Last updated:** [April 18, 2019, 1:09pm UTC](https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384 "2019-04-18T13:09:06Z")
Hello All, I'm learning Logstash basics and I'm trying to find a way of converting an .xml file into multiple events (separate json files) which will be sent out to elasticsearch. The thing is, specific elements of xml … --- ## [Logstash Filter not being applied correctly](https://discuss.elastic.co/t/logstash-filter-not-being-applied-correctly/177470)
**Author:** [@Urs\_Bronk](https://discuss.elastic.co/u/Urs_Bronk)\ **Replies:** 0\ **Last updated:** [April 18, 2019, 1:17pm UTC](https://discuss.elastic.co/t/logstash-filter-not-being-applied-correctly/177470 "2019-04-18T13:17:33Z")
Hi Guys, I've recently updated my entire ELK cluster to the latest version. I basically started from scratch. I have the following config file setup for my logstash output to elasticsearch. https://pastebin.com/VCBhga… --- ## [Need to use bind variables for Oracle query in jdbc-input (using sql\_last\_value) in Logstash](https://discuss.elastic.co/t/need-to-use-bind-variables-for-oracle-query-in-jdbc-input-using-sql-last-value-in-logstash/163716)
**Author:** [@scotta](https://discuss.elastic.co/u/scotta)\ **Replies:** 2\ **Last updated:** [April 18, 2019, 1:12pm UTC](https://discuss.elastic.co/t/need-to-use-bind-variables-for-oracle-query-in-jdbc-input-using-sql-last-value-in-logstash/163716 "2019-04-18T13:12:59Z")
As per the jdbc-input-plugin documentation, we're using the 'sql\_last\_value' to only load data since the query was last run e.g. input { jdbc{ ... statement =\> "SELECT \* from schema.table WHERE updated\_date \>=… --- ## [Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at line 6,](https://discuss.elastic.co/t/failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror-message-expected-one-of-at-line-6/177387)
**Author:** [@Apurva\_Ganthi](https://discuss.elastic.co/u/Apurva_Ganthi)\ **Replies:** 3\ **Last updated:** [April 18, 2019, 1:03pm UTC](https://discuss.elastic.co/t/failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror-message-expected-one-of-at-line-6/177387 "2019-04-18T13:03:43Z")
Hello, My code for the logstash-simple.conf file is as follows:- input { file { type =\> "mylog" path =\> "C:\\Softwares\\ELK\\custom-\*.log" mode =\> "tail" start\_ position =\> "beginning" } } filter { if (\[t… --- ## [Logstash is reindexing lastline and newline](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120)
**Author:** [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Replies:** 20\ **Last updated:** [April 18, 2019, 9:29am UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120 "2019-04-18T09:29:55Z")
Hello community. I am using logstash to parse a logfile and index to elastic I have seen that if I enter a new line in the file, logstash is indexing the newline but also the last indexed line Looks like a bug an it c… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=561) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=563)