# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=721 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 722 --- ## [How elasticsearch will fetch data from MySQL database?](https://discuss.elastic.co/t/how-elasticsearch-will-fetch-data-from-mysql-database/139951)
**Author:** [@mamta](https://discuss.elastic.co/u/mamta)\ **Replies:** 7\ **Last updated:** [July 16, 2018, 6:34pm UTC](https://discuss.elastic.co/t/how-elasticsearch-will-fetch-data-from-mysql-database/139951 "2018-07-16T18:34:33Z")
Hi, Suppose I have MySQL database and I connected with this database through ELK. In MySQL database have 1 lac records and that all data is shown in kibana. After sometimes mysql updated with 50k records. So my question… --- ## [Ruby does not execute require inside logstash](https://discuss.elastic.co/t/ruby-does-not-execute-require-inside-logstash/140012)
**Author:** [@mpsa76](https://discuss.elastic.co/u/mpsa76)\ **Replies:** 3\ **Last updated:** [July 16, 2018, 6:27pm UTC](https://discuss.elastic.co/t/ruby-does-not-execute-require-inside-logstash/140012 "2018-07-16T18:27:00Z")
I have installed logstash the moment to execute require 'jdbc / mysql' the the following error appears: \[ERROR\] \[logstash.filters.ruby\] Ruby exception occurred: no such file to load - jdbc / mysql Please your help I do… --- ## [Dealing with logstash conf file for json string array](https://discuss.elastic.co/t/dealing-with-logstash-conf-file-for-json-string-array/140156)
**Author:** [@emailshak](https://discuss.elastic.co/u/emailshak)\ **Replies:** 4\ **Last updated:** [July 16, 2018, 4:13pm UTC](https://discuss.elastic.co/t/dealing-with-logstash-conf-file-for-json-string-array/140156 "2018-07-16T16:13:11Z")
I am trying to move the following json in a text file to elasticsearch using logstash. {"vehicles":{"brand":"honda", "type":\["suv","sedan","convertible"\]}} . The logstash processs runs fine and inserts this record to e… --- ## [Logstash not stashing logs even after starting successfully](https://discuss.elastic.co/t/logstash-not-stashing-logs-even-after-starting-successfully/140010)
**Author:** [@APJ](https://discuss.elastic.co/u/APJ)\ **Replies:** 9\ **Last updated:** [July 16, 2018, 3:59pm UTC](https://discuss.elastic.co/t/logstash-not-stashing-logs-even-after-starting-successfully/140010 "2018-07-16T15:59:04Z")
Hi all, I have an apache log file that I want to ingest into logstash and then send the stashed data to elasticsearch. I run the logstash command like this: .\\logstash -f logtstash.conf After running the command th… --- ## [Grok pattern to match different log formats in same log file](https://discuss.elastic.co/t/grok-pattern-to-match-different-log-formats-in-same-log-file/140155)
**Author:** [@Shweta\_Priyadarshani](https://discuss.elastic.co/u/Shweta_Priyadarshani)\ **Replies:** 1\ **Last updated:** [July 16, 2018, 2:31pm UTC](https://discuss.elastic.co/t/grok-pattern-to-match-different-log-formats-in-same-log-file/140155 "2018-07-16T14:31:14Z")
Hi Everyone, I am trying to apply filter in logstash conf file by giving grok pattern . But the challenge here is that my log file thats the input for logstash , has different formats. Ex: 1.) 27.06.2018 00:00:00.009 I… --- ## [How can I check ruby version using in Logstash](https://discuss.elastic.co/t/how-can-i-check-ruby-version-using-in-logstash/139081)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 3\ **Last updated:** [July 16, 2018, 2:20pm UTC](https://discuss.elastic.co/t/how-can-i-check-ruby-version-using-in-logstash/139081 "2018-07-16T14:20:35Z")
How can I check ruby version using in Logstash?? --- ## [No Logstash init script - RHEL 6.10](https://discuss.elastic.co/t/no-logstash-init-script-rhel-6-10/139726)
**Author:** [@andrew\_michael](https://discuss.elastic.co/u/andrew_michael)\ **Replies:** 5\ **Last updated:** [July 16, 2018, 2:05pm UTC](https://discuss.elastic.co/t/no-logstash-init-script-rhel-6-10/139726 "2018-07-16T14:05:43Z")
Hi. We are running logstash version 5.5.0 on a RHEL 6.10 fully patched host. Oddly when I deploy the ELK stack all the other applications get init scripts put in /etc/init.d however logstash seems to put it's script in… --- ## [Need advise on an Kafka-ELK cluster and Logstash settings to improve indexing rate](https://discuss.elastic.co/t/need-advise-on-an-kafka-elk-cluster-and-logstash-settings-to-improve-indexing-rate/140114)
**Author:** [@anon24174512](https://discuss.elastic.co/u/anon24174512)\ **Replies:** 11\ **Last updated:** [July 16, 2018, 1:55pm UTC](https://discuss.elastic.co/t/need-advise-on-an-kafka-elk-cluster-and-logstash-settings-to-improve-indexing-rate/140114 "2018-07-16T13:55:52Z")
Dear all, I am trying to build an ELK cluster together with Kafka to get the best indexing rate in Elasticsearch. Following the description of my cluster hardware and configuration below : is the architecture correct … --- ## [Unable to read .gz files in logstash](https://discuss.elastic.co/t/unable-to-read-gz-files-in-logstash/136174)
**Author:** [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\ **Replies:** 12\ **Last updated:** [July 16, 2018, 1:11pm UTC](https://discuss.elastic.co/t/unable-to-read-gz-files-in-logstash/136174 "2018-07-16T13:11:40Z")
Hi, I am trying to read .gz files in logstash input plugin but I can't read gunzip files in logstash. When I try to read those log files it throws error message like this, A plugin had an unrecoverable error. Will rest… --- ## [Logstash Error while uploading](https://discuss.elastic.co/t/logstash-error-while-uploading/140102)
**Author:** [@djk1](https://discuss.elastic.co/u/djk1)\ **Replies:** 2\ **Last updated:** [July 16, 2018, 9:14am UTC](https://discuss.elastic.co/t/logstash-error-while-uploading/140102 "2018-07-16T09:14:49Z")
I am getting error like this. Error registering pl ugin {:pipeline\_id=\>"main", :plugin=\>"#LogStash::OutputDelegator:0x20ee3e0a", :error=\>"Unrecognized SSL message, plaintext connection?", :thread=\>"#\ **Author:** [@Loudone](https://discuss.elastic.co/u/Loudone)\ **Replies:** 1\ **Last updated:** [July 16, 2018, 11:11am UTC](https://discuss.elastic.co/t/problem-with-index-created/140115 "2018-07-16T11:11:42Z") Hello, I encounter some problems with Elk suite, and try to solve it. For a project, we are tryring to parses with logstash this kind of log: \`2018-07-16T11:11:53 coll\_bddf \[3\] \[0\] \[11\] \[61\] \[9\] \[24\]\` so, with logsta… --- ## [Grok is succesfull in various debuggers but fails in practice with specific keyword](https://discuss.elastic.co/t/grok-is-succesfull-in-various-debuggers-but-fails-in-practice-with-specific-keyword/140112)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 2\ **Last updated:** [July 16, 2018, 10:38am UTC](https://discuss.elastic.co/t/grok-is-succesfull-in-various-debuggers-but-fails-in-practice-with-specific-keyword/140112 "2018-07-16T10:38:02Z")
Hi, I have the following logstash config which parses syslog messages: filter { if "syslog" in \[tags\] and "pre-processed" not in \[tags\] { if "%ASA-" in \[message\] { mutate { gsub =\> \[ "mes… --- ## [Parse Log File To Logstash (Nested Optional)](https://discuss.elastic.co/t/parse-log-file-to-logstash-nested-optional/140064)
**Author:** [@d332ed223b56b81e16df](https://discuss.elastic.co/u/d332ed223b56b81e16df)\ **Replies:** 3\ **Last updated:** [July 16, 2018, 10:13am UTC](https://discuss.elastic.co/t/parse-log-file-to-logstash-nested-optional/140064 "2018-07-16T10:13:45Z")
I need your help. I'm working at company and now we need to analyse data using ELK. In general, we have few questions and we save each question data such as: cookie id of the client,the answer chosen,how much time it to… --- ## [Pushing flush onto pipeline error](https://discuss.elastic.co/t/pushing-flush-onto-pipeline-error/139564)
**Author:** [@rihab\_feki](https://discuss.elastic.co/u/rihab_feki)\ **Replies:** 2\ **Last updated:** [July 16, 2018, 9:35am UTC](https://discuss.elastic.co/t/pushing-flush-onto-pipeline-error/139564 "2018-07-16T09:35:04Z")
i want to start logstash with a custom configuration and showing the result in stdout but I've alwyas this logs ..Ididn't understand the probllem .. can anyone help me thanks \[2018-07-11T11:46:32,944\]\[INFO \]\[logstash.pi… --- ## [How to get prefix of a string](https://discuss.elastic.co/t/how-to-get-prefix-of-a-string/139915)
**Author:** [@vschakka](https://discuss.elastic.co/u/vschakka)\ **Replies:** 2\ **Last updated:** [July 16, 2018, 9:02am UTC](https://discuss.elastic.co/t/how-to-get-prefix-of-a-string/139915 "2018-07-16T09:02:14Z")
Hi, I have a field with value "sv4axmaic04". I need to extract value before "axmaic". But substring "axmaic" is same/constant across all the field values. So I just need to extract value "sv4" from the given example. … --- ## [Elasticsearch and logstash giving exception for long and float data type casting](https://discuss.elastic.co/t/elasticsearch-and-logstash-giving-exception-for-long-and-float-data-type-casting/140093)
**Author:** [@ajit](https://discuss.elastic.co/u/ajit)\ **Replies:** 1\ **Last updated:** [July 16, 2018, 7:42am UTC](https://discuss.elastic.co/t/elasticsearch-and-logstash-giving-exception-for-long-and-float-data-type-casting/140093 "2018-07-16T07:42:58Z")
Hi Team, I have to create index dynamically using logstash jdbc. I have used sql query in logstash. And my few column contains some decimal and some long values mixed. After starting logstash I am getting exception can… --- ## [Date filter is not working in Logstash](https://discuss.elastic.co/t/date-filter-is-not-working-in-logstash/140038)
**Author:** [@Harsh\_Sharma](https://discuss.elastic.co/u/Harsh_Sharma)\ **Replies:** 5\ **Last updated:** [July 16, 2018, 7:32am UTC](https://discuss.elastic.co/t/date-filter-is-not-working-in-logstash/140038 "2018-07-16T07:32:03Z")
Hi There, Below is my log. 2018-07-09 08:08:04,689 INFO c.t.t.r.s.i.BookingPaxReportServiceImpl.lambda$generateAndSavePNRReport$1 55d49113-006e-4543-a7e3-2aaa3e757bc1 127.0.0.1 zoomair.uat.i-tare.com http-nio-8080-exec… --- ## [Elasticsearch and logstash giving exception for long and float type casting](https://discuss.elastic.co/t/elasticsearch-and-logstash-giving-exception-for-long-and-float-type-casting/140091)
**Author:** [@ajit](https://discuss.elastic.co/u/ajit)\ **Replies:** 0\ **Last updated:** [July 16, 2018, 7:00am UTC](https://discuss.elastic.co/t/elasticsearch-and-logstash-giving-exception-for-long-and-float-type-casting/140091 "2018-07-16T07:00:21Z")
Hi Team, I have to create index dynamically using logstash jdbc. I have used sql query in logstash. And my few column contains some decimal and some long values mixed. After starting logstash I am getting exception can… --- ## [Dynamic index name when using multiple filebeat instances](https://discuss.elastic.co/t/dynamic-index-name-when-using-multiple-filebeat-instances/139928)
**Author:** [@Sakshi\_Aggarwal](https://discuss.elastic.co/u/Sakshi_Aggarwal)\ **Replies:** 2\ **Last updated:** [July 16, 2018, 6:18am UTC](https://discuss.elastic.co/t/dynamic-index-name-when-using-multiple-filebeat-instances/139928 "2018-07-16T06:18:58Z")
i have a working model of elk but i want a dynamic index name, all i want is to assign a based on my beat.name field. output { elasticsearch { hosts =\> \[ "localhost:9200" \] index =\> "%{\[beat.name\]\[\_id\]}" } } --- ## [Problem with parsing in logstash](https://discuss.elastic.co/t/problem-with-parsing-in-logstash/139319)
**Author:** [@NIloufarVafaei](https://discuss.elastic.co/u/NIloufarVafaei)\ **Replies:** 8\ **Last updated:** [July 16, 2018, 3:30am UTC](https://discuss.elastic.co/t/problem-with-parsing-in-logstash/139319 "2018-07-16T03:30:59Z")
hello i try to pars json data that has string data which one of these json data just has one string line like : 021965425836 and another one has string line that separate with "," look like this: 0201923165000,0202000… --- ## [Logstash can't process all JSON logs](https://discuss.elastic.co/t/logstash-cant-process-all-json-logs/140003)
**Author:** [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\ **Replies:** 4\ **Last updated:** [July 16, 2018, 3:18am UTC](https://discuss.elastic.co/t/logstash-cant-process-all-json-logs/140003 "2018-07-16T03:18:55Z")
Hi, We are sending logs directly from django based python application to logstash over TCP/9111 port. We found that some logs are missing like {"@timestamp": "2018-07-13T14:38:23.741Z", "remote\_addr": "10.0.2.2", "hos… --- ## [Carriage return needed to continue](https://discuss.elastic.co/t/carriage-return-needed-to-continue/140073)
**Author:** [@safhw91](https://discuss.elastic.co/u/safhw91)\ **Replies:** 0\ **Last updated:** [July 16, 2018, 2:55am UTC](https://discuss.elastic.co/t/carriage-return-needed-to-continue/140073 "2018-07-16T02:55:38Z")
Hello, I've been testing Logstash for a few months now, and mostly I've been able to figure out solutions to most of the problems I've been having. But there is one thing really bugging me, not extremely critical but s… --- ## [Send logs to logstash and elasticsearch but miss some logs](https://discuss.elastic.co/t/send-logs-to-logstash-and-elasticsearch-but-miss-some-logs/139937)
**Author:** [@TLthirtyeight](https://discuss.elastic.co/u/TLthirtyeight)\ **Replies:** 1\ **Last updated:** [July 16, 2018, 1:11am UTC](https://discuss.elastic.co/t/send-logs-to-logstash-and-elasticsearch-but-miss-some-logs/139937 "2018-07-16T01:11:28Z")
Here is my situation.I write a java app which read my logs from file.Then the java app send logs to logstash.The logstash filter the logs and then send them to elasticsearch.However I found that some logs are missed and … --- ## [Dead letter queue isn't cleaned up](https://discuss.elastic.co/t/dead-letter-queue-isnt-cleaned-up/139653)
**Author:** [@jbeck](https://discuss.elastic.co/u/jbeck)\ **Replies:** 1\ **Last updated:** [July 16, 2018, 12:07am UTC](https://discuss.elastic.co/t/dead-letter-queue-isnt-cleaned-up/139653 "2018-07-16T00:07:34Z")
We are using the dead letter queue with the Elasticsearch output plugin, and the dead letter queue input to reprocess events. It seems to work quite well, except that the old entries in the dead letter queue are never re… --- ## [Grok filter customize](https://discuss.elastic.co/t/grok-filter-customize/140048)
**Author:** [@Mojtaba\_Motamed](https://discuss.elastic.co/u/Mojtaba_Motamed)\ **Replies:** 0\ **Last updated:** [July 15, 2018, 6:28am UTC](https://discuss.elastic.co/t/grok-filter-customize/140048 "2018-07-15T06:28:23Z")
how can customize user agent all field in this row 212.16.95.74 - - \[27/May/2018:03:07:23 +0430\] "GET /VIDEO/IMAGE?id=aHR0cDovL2huMTQuYXNzZXQuYXBhcmF0LmNvbS9hcGFyYXQtdmlkZW8vMmRmOWRkMDIzNjJiMmVkNzk4OWI5MDE3MGNhYzRmMTAyN… --- ## [Can logstash tagging help me here?](https://discuss.elastic.co/t/can-logstash-tagging-help-me-here/139459)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 3\ **Last updated:** [July 14, 2018, 3:50pm UTC](https://discuss.elastic.co/t/can-logstash-tagging-help-me-here/139459 "2018-07-14T15:50:17Z")
Hi there, I have file something like this; a dict file probably. Lets say doms.yaml artput.date: Emotet astheir.cricket: Emotet callcar.stream: Emotet catflys.online: Phishing datatax.trade: Phishing eastbus.racin… --- ## [Date filter showing error](https://discuss.elastic.co/t/date-filter-showing-error/139905)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 3\ **Last updated:** [July 14, 2018, 2:09pm UTC](https://discuss.elastic.co/t/date-filter-showing-error/139905 "2018-07-14T14:09:03Z")
date{ match=\>{"date","yyyy-mm-dd hh:mm:ss:sss","ISO8601"} target="updated\_date" } My input timestamp will be:2018-07-09T06:36:44.254Z" exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at l… --- ## [Convert timestamp to normal date structure](https://discuss.elastic.co/t/convert-timestamp-to-normal-date-structure/139872)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 3\ **Last updated:** [July 14, 2018, 1:59pm UTC](https://discuss.elastic.co/t/convert-timestamp-to-normal-date-structure/139872 "2018-07-14T13:59:01Z")
convert timestamp to normal date structure... eg: 2018-07-13T04:48:02.025Z=\>2018-july-13 hh:mm:ss --- ## [How can i solve this logstash error?](https://discuss.elastic.co/t/how-can-i-solve-this-logstash-error/138115)
**Author:** [@yaser](https://discuss.elastic.co/u/yaser)\ **Replies:** 2\ **Last updated:** [July 14, 2018, 8:05am UTC](https://discuss.elastic.co/t/how-can-i-solve-this-logstash-error/138115 "2018-07-14T08:05:56Z")
\[2018-07-02T08:17:30,242\]\[WARN \]\[logstash.shutdownwatcher \] {"inflight\_count"=\>1, "stalling\_thread\_info"=\>{"other"=\>\[{"thread\_id"=\>201, "name"=\>nil, "current\_call"=\>"\[...\]/vendor/bundle/jruby/2.3.0/gems/logstash-output-h… --- ## [Global tag from XML header](https://discuss.elastic.co/t/global-tag-from-xml-header/138937)
**Author:** [@FatePhD](https://discuss.elastic.co/u/FatePhD)\ **Replies:** 7\ **Last updated:** [July 14, 2018, 12:57am UTC](https://discuss.elastic.co/t/global-tag-from-xml-header/138937 "2018-07-14T00:57:13Z")
I'd like to apply a global tag to every message imported from the same log file, based on information found in the first five lines. Background: I'm parsing XML files recorded on a remote system, but imported on a local… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=720) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=722)