# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=846 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 847 --- ## [Logstash 6.x / Elasticsearch 6.x error](https://discuss.elastic.co/t/logstash-6-x-elasticsearch-6-x-error/114320)
**Author:** [@mdk999](https://discuss.elastic.co/u/mdk999)\ **Replies:** 0\ **Last updated:** [January 5, 2018, 5:39pm UTC](https://discuss.elastic.co/t/logstash-6-x-elasticsearch-6-x-error/114320 "2018-01-05T17:39:25Z")
I recently upgraded logstash, kibana and elasticsearch to the newest version. I read the breaking changes and looked at the 'configuring logstash 6' link. I see that Elasticsearch now requires headers to be sent but I do… --- ## [Logstash Error When Running - Not Sure Of Error](https://discuss.elastic.co/t/logstash-error-when-running-not-sure-of-error/114290)
**Author:** [@OpSecMonkey](https://discuss.elastic.co/u/OpSecMonkey)\ **Replies:** 2\ **Last updated:** [January 5, 2018, 2:21pm UTC](https://discuss.elastic.co/t/logstash-error-when-running-not-sure-of-error/114290 "2018-01-05T14:21:02Z")
Good morning everyone. I am running a custom config from a class on Udemy for ElasticSearch, LogStash and Kibana. For whatever reason I get errors and I cant get it to run. I have been trying over and over and over. Can… --- ## [Tuning logstash performance](https://discuss.elastic.co/t/tuning-logstash-performance/114245)
**Author:** [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Replies:** 1\ **Last updated:** [January 5, 2018, 2:00pm UTC](https://discuss.elastic.co/t/tuning-logstash-performance/114245 "2018-01-05T14:00:03Z")
Hi, I use logstash to connect between elasticsearch and ntopng(a flow collector). but there are many drop flows, so I think the bottle neck is on logstash because my RAM is 20G and CPU 8 cores. But I am not sure which … --- ## [Can't find error "\_grokparsefailure" IIS Advanced Logs](https://discuss.elastic.co/t/cant-find-error-grokparsefailure-iis-advanced-logs/113773)
**Author:** [@cassiodeveloper](https://discuss.elastic.co/u/cassiodeveloper)\ **Replies:** 2\ **Last updated:** [January 5, 2018, 11:46am UTC](https://discuss.elastic.co/t/cant-find-error-grokparsefailure-iis-advanced-logs/113773 "2018-01-05T11:46:53Z")
Hello everyone, I can't find the error on logstash using grok filter. My config file: input { file { type =\> "iis" path =\> "C:/inetpub/logs/LogFiles/AdvancedLogs/\*.log" } } filter { if \[message\] =~ "^#" { d… --- ## [Merge indices with timestamp in epoch](https://discuss.elastic.co/t/merge-indices-with-timestamp-in-epoch/114276)
**Author:** [@oskars](https://discuss.elastic.co/u/oskars)\ **Replies:** 0\ **Last updated:** [January 5, 2018, 11:08am UTC](https://discuss.elastic.co/t/merge-indices-with-timestamp-in-epoch/114276 "2018-01-05T11:08:14Z")
Im having problems merging indices with "@timestamp" field with epoch time. so i got indice (es.2.4) : { "\_source": { "@timestamp": "1514851163146", . . . } } im using logstash (tested with 2.4v and lates… --- ## [Logstash pipeline with MongoDB input jdbc](https://discuss.elastic.co/t/logstash-pipeline-with-mongodb-input-jdbc/114273)
**Author:** [@mahajan344](https://discuss.elastic.co/u/mahajan344)\ **Replies:** 0\ **Last updated:** [January 5, 2018, 10:47am UTC](https://discuss.elastic.co/t/logstash-pipeline-with-mongodb-input-jdbc/114273 "2018-01-05T10:47:00Z")
Hi, I wanted to create a logstash pipeline to have input from MongoDB and output to Elasticsearch. Since MongoDB don't have any offcial JDBC driver. So I am using unityjdbc driver for MongoDB. I have create a simple p… --- ## [How to split the records on data with only value and without key](https://discuss.elastic.co/t/how-to-split-the-records-on-data-with-only-value-and-without-key/114134)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 2\ **Last updated:** [January 5, 2018, 9:57am UTC](https://discuss.elastic.co/t/how-to-split-the-records-on-data-with-only-value-and-without-key/114134 "2018-01-05T09:57:43Z")
Hi All Currently, i am getting data in the below format "rows": \[ \[ "New Visitor", "(not set)", "(direct)", "(none)", "desktop", "123456… --- ## [Logstash 6.1.1 isn't executing \[logging issue\]](https://discuss.elastic.co/t/logstash-6-1-1-isnt-executing-logging-issue/113745)
**Author:** [@vigneshaj](https://discuss.elastic.co/u/vigneshaj)\ **Replies:** 2\ **Last updated:** [January 5, 2018, 9:52am UTC](https://discuss.elastic.co/t/logstash-6-1-1-isnt-executing-logging-issue/113745 "2018-01-05T09:52:23Z")
Hi - I have a 3 node cluster that i am trying to setup and run the ELK stack. Elastic search is configured and runs ok in all 3 nodes. Have 2 issues: #1: In Node1: while running the conf file i get the following erro… --- ## [Pipeline startup/shutdown ordering when using multiple pipeline](https://discuss.elastic.co/t/pipeline-startup-shutdown-ordering-when-using-multiple-pipeline/113912)
**Author:** [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Replies:** 2\ **Last updated:** [January 5, 2018, 9:49am UTC](https://discuss.elastic.co/t/pipeline-startup-shutdown-ordering-when-using-multiple-pipeline/113912 "2018-01-05T09:49:49Z")
Hi, I am using Logstash-6.1.1 and its multiple pipeline feature. I need to send data to elasticsearch and hadoop with the caveat that hadoop service may become unavailable (it is managed by a second team and is located … --- ## [Logstash json restructuring](https://discuss.elastic.co/t/logstash-json-restructuring/114200)
**Author:** [@Raj2](https://discuss.elastic.co/u/Raj2)\ **Replies:** 1\ **Last updated:** [January 5, 2018, 9:37am UTC](https://discuss.elastic.co/t/logstash-json-restructuring/114200 "2018-01-05T09:37:59Z")
I have a structure like { f1: 12, f2 : { nf1: {........} } } I want to convert it to { f1:12, nf1: {......} } When I try add\_field { "nf1"=\> "\[f2\]\[nf1\]" }, the RHS is not evaluated. If I use %{}, the value is str… --- ## [Failed multiple date fields parsing](https://discuss.elastic.co/t/failed-multiple-date-fields-parsing/114101)
**Author:** [@cilzzz](https://discuss.elastic.co/u/cilzzz)\ **Replies:** 2\ **Last updated:** [January 5, 2018, 8:54am UTC](https://discuss.elastic.co/t/failed-multiple-date-fields-parsing/114101 "2018-01-05T08:54:09Z")
Hi i have a logfile which contains mutiple date fields and i need to parse 3 date fields in my logstash config file i've wrote : date { match =\> \["Starting\_Time", "yy-MM-dd HH:mm:ss"\] … --- ## [Logstash hangs using dns filter](https://discuss.elastic.co/t/logstash-hangs-using-dns-filter/114235)
**Author:** [@qlanusse](https://discuss.elastic.co/u/qlanusse)\ **Replies:** 0\ **Last updated:** [January 5, 2018, 8:37am UTC](https://discuss.elastic.co/t/logstash-hangs-using-dns-filter/114235 "2018-01-05T08:37:13Z")
Hi there, -Hope this is the right place to put my problem- I am using Logstash 6.0.0 to parse logs from Filebeat 6.0.0 instances. I tried to change some IP addresses with server names using the DNS filter but after a wh… --- ## [Throttle filter per value?](https://discuss.elastic.co/t/throttle-filter-per-value/114190)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 1\ **Last updated:** [January 5, 2018, 7:35am UTC](https://discuss.elastic.co/t/throttle-filter-per-value/114190 "2018-01-05T07:35:40Z")
Happy new year to everybody. I have a question regarding the throttle filter. I need to rate limit some of my data. Situation: I've got a application log with a GPS field and a error field. For the GPS field I only ne… --- ## [Fatal error when processing CSV file](https://discuss.elastic.co/t/fatal-error-when-processing-csv-file/114184)
**Author:** [@code\_diva](https://discuss.elastic.co/u/code_diva)\ **Replies:** 2\ **Last updated:** [January 4, 2018, 11:05pm UTC](https://discuss.elastic.co/t/fatal-error-when-processing-csv-file/114184 "2018-01-04T23:05:40Z")
I am processing a CSV file with a conf file I have used before together and have not gotten this error in the past on a different server. Passes config test and about 1/4 of the data makes it into Elastic without any iss… --- ## [Trying to avoid manual changes to files pre-logstash](https://discuss.elastic.co/t/trying-to-avoid-manual-changes-to-files-pre-logstash/113824)
**Author:** [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Replies:** 9\ **Last updated:** [January 4, 2018, 9:20pm UTC](https://discuss.elastic.co/t/trying-to-avoid-manual-changes-to-files-pre-logstash/113824 "2018-01-04T21:20:41Z")
Hello, I have csv files that include bad output from SQL output that logstash fails to parse because of quote marks that stay open. Example line: "1";"2";"3";"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N I am fixing them … --- ## [Slow processing from s3 for larger gzip file](https://discuss.elastic.co/t/slow-processing-from-s3-for-larger-gzip-file/114143)
**Author:** [@elastico\_imp](https://discuss.elastic.co/u/elastico_imp)\ **Replies:** 0\ **Last updated:** [January 4, 2018, 6:15pm UTC](https://discuss.elastic.co/t/slow-processing-from-s3-for-larger-gzip-file/114143 "2018-01-04T18:15:31Z")
we are reading from S3 buckets - cloudtrail logs packed by Account under a single bucket. Each file varies from 5kb to 500kb or more.. 2 scenerios we tested with 4cpu,8gb docker. Larger file takes about 35mins to process… --- ## [Sending batch to Kafka failed. Will retry after a delay Logstash error](https://discuss.elastic.co/t/sending-batch-to-kafka-failed-will-retry-after-a-delay-logstash-error/114133)
**Author:** [@joaoantoniopereira](https://discuss.elastic.co/u/joaoantoniopereira)\ **Replies:** 0\ **Last updated:** [January 4, 2018, 5:18pm UTC](https://discuss.elastic.co/t/sending-batch-to-kafka-failed-will-retry-after-a-delay-logstash-error/114133 "2018-01-04T17:18:50Z")
Hi, I have a Logstash agent sending lines to a multi broker kafka setup. Currently I am getting the following error in Logstash log: \[2017-12-27T15:06:15,369\]\[INFO \]\[logstash.outputs.kafka \] Sending batch to Kafka f… --- ## [Kafka message frequency](https://discuss.elastic.co/t/kafka-message-frequency/113935)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 3\ **Last updated:** [January 4, 2018, 3:48pm UTC](https://discuss.elastic.co/t/kafka-message-frequency/113935 "2018-01-04T15:48:47Z")
I would like to consume messages from Kafka, with Logstash. Is it possible for Logstash, to always read the last message that exists on Kafka topic, even if it was sent a little while ago? So far I see I can configure … --- ## [Logstash input error](https://discuss.elastic.co/t/logstash-input-error/113847)
**Author:** [@agonex](https://discuss.elastic.co/u/agonex)\ **Replies:** 2\ **Last updated:** [January 4, 2018, 3:43pm UTC](https://discuss.elastic.co/t/logstash-input-error/113847 "2018-01-04T15:43:10Z")
Hi I need help with a issue in logstash... I am testing a version 6 ELK and i have problems with inputs, i triying add another input type multiline and logs say me that: 2018-01-02T21:09:28.628326533Z \[2018-01-02T21:09… --- ## [Write ruby filter method in class](https://discuss.elastic.co/t/write-ruby-filter-method-in-class/113873)
**Author:** [@pradeep\_reddy](https://discuss.elastic.co/u/pradeep_reddy)\ **Replies:** 1\ **Last updated:** [January 4, 2018, 3:07pm UTC](https://discuss.elastic.co/t/write-ruby-filter-method-in-class/113873 "2018-01-04T15:07:19Z")
Hi, I have to map the keys in event to other fields which comes from external configuration . my event: "id" =\> 10, "updated\_on" =\> 2018-01-03T03:17:56.000Z, "name" =\> "ggd", "type" =\> "mysql\_read" f… --- ## [Ruby Code](https://discuss.elastic.co/t/ruby-code/112586)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 2\ **Last updated:** [January 4, 2018, 2:59pm UTC](https://discuss.elastic.co/t/ruby-code/112586 "2018-01-04T14:59:28Z")
Hi - I built a gem locally, and I would need to install it on the logstash to consume those methods in my logstash ruby block. Would you please help me in installing it ? And also could you please let me know how do I … --- ## [Grok pattern working fine in grok debugger but the same pattern is not working when running with logstash](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150)
**Author:** [@govinda.rao](https://discuss.elastic.co/u/govinda.rao)\ **Replies:** 13\ **Last updated:** [January 4, 2018, 2:30pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150 "2018-01-04T14:30:43Z")
Hi, I am using below pattern to parse below Log, the pattern is working fine grok debugger but failing when running in logstash. Please help Grok Pattern: %{DATA:hour}:%{DATA:minute}:%{DATA:second},%{DATA:milisecond} %{… --- ## [Close\_older not working as expected](https://discuss.elastic.co/t/close-older-not-working-as-expected/113993)
**Author:** [@pgervais](https://discuss.elastic.co/u/pgervais)\ **Replies:** 5\ **Last updated:** [January 4, 2018, 2:22pm UTC](https://discuss.elastic.co/t/close-older-not-working-as-expected/113993 "2018-01-04T14:22:37Z")
I'm running logstash from a bash script. I have the following input section : input { file { path =\> "/home/pxg110/logstash/mach/test.csv" start\_position =\> "end" sincedb\_path =\> "/dev/null" close\_older =\> 1 } } … --- ## [Installing Mongodb input plugin in Logstash](https://discuss.elastic.co/t/installing-mongodb-input-plugin-in-logstash/113886)
**Author:** [@mahajan344](https://discuss.elastic.co/u/mahajan344)\ **Replies:** 2\ **Last updated:** [January 4, 2018, 1:57pm UTC](https://discuss.elastic.co/t/installing-mongodb-input-plugin-in-logstash/113886 "2018-01-04T13:57:25Z")
Hello, I have a MongoDB database and I want to create a pipeline in Logstash to transfer data from MongoDb to Elasticsearch using Logstash. I have installed Elasticsearch latest version 6.1 and I checked official websi… --- ## [Many workers whereas only 1 expected](https://discuss.elastic.co/t/many-workers-whereas-only-1-expected/113942)
**Author:** [@Nicolas\_Guyomard](https://discuss.elastic.co/u/Nicolas_Guyomard)\ **Replies:** 1\ **Last updated:** [January 4, 2018, 1:45pm UTC](https://discuss.elastic.co/t/many-workers-whereas-only-1-expected/113942 "2018-01-04T13:45:57Z")
I use aggregate filter plugin to extract metrics from the log of my application using the following config: input { file { path =\> "application.log" start\_position =\> "beginning" sincedb\_path =\> "/de… --- ## [How to index inconstant log delimted with tabs](https://discuss.elastic.co/t/how-to-index-inconstant-log-delimted-with-tabs/114050)
**Author:** [@d4n](https://discuss.elastic.co/u/d4n)\ **Replies:** 1\ **Last updated:** [January 4, 2018, 1:03pm UTC](https://discuss.elastic.co/t/how-to-index-inconstant-log-delimted-with-tabs/114050 "2018-01-04T13:03:33Z")
Hi All, Im doing POC in my company of ELK stack i dont have issue with parsing iis logs for example but one of our apps have really poor inconstant txt log without headers. Example: 10:49:37 exwhh1s2ofneggxlpuvgg1fc … --- ## [Logstash: unrecognized service error](https://discuss.elastic.co/t/logstash-unrecognized-service-error/114077)
**Author:** [@alpha\_gamma](https://discuss.elastic.co/u/alpha_gamma)\ **Replies:** 0\ **Last updated:** [January 4, 2018, 11:39am UTC](https://discuss.elastic.co/t/logstash-unrecognized-service-error/114077 "2018-01-04T11:39:10Z")
Hi Everbody , I am geeting this error in logstash ,if I try to watch logs it gives the below error require at org/jruby/RubyKernel.java:955 \ at /usr/share/logstash/logstash-core/lib/logstash/filters/ba… --- ## [Determining the cause of data loss](https://discuss.elastic.co/t/determining-the-cause-of-data-loss/113554)
**Author:** [@oguz](https://discuss.elastic.co/u/oguz)\ **Replies:** 6\ **Last updated:** [January 4, 2018, 10:07am UTC](https://discuss.elastic.co/t/determining-the-cause-of-data-loss/113554 "2018-01-04T10:07:44Z")
Hi All, i'm using jdbc plugin to transfer data from postgres to es. My sample data is 6028 records on postgres. I get different number of records transferred to es each time i try. Sometimes all 6028 records gets transf… --- ## [Error While Installing Logstash](https://discuss.elastic.co/t/error-while-installing-logstash/114008)
**Author:** [@kartheek91](https://discuss.elastic.co/u/kartheek91)\ **Replies:** 2\ **Last updated:** [January 4, 2018, 9:59am UTC](https://discuss.elastic.co/t/error-while-installing-logstash/114008 "2018-01-04T09:59:21Z")
Error while running logstash 5.6.5 in windows 10 .It is throwing me \*\* The system cannot find the path specified. \*\* \*\* "could not find jruby in C:\\Program Files\\logstash\\vendor\\jruby" \*\* Please help for the a… --- ## [Dissect filter is not dissecting properly](https://discuss.elastic.co/t/dissect-filter-is-not-dissecting-properly/114044)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 1\ **Last updated:** [January 4, 2018, 9:46am UTC](https://discuss.elastic.co/t/dissect-filter-is-not-dissecting-properly/114044 "2018-01-04T09:46:08Z")
Hi, I am using ELK GA 6.0.0. I am trying out dissect filter to chop my log file entry, but some field is not getting dissected properly. I am pushing messages into Kafka using file beat, and using my Logstash to fetch t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=845) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=847)