# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=714 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 715 --- ## [Help with Grok patterns to parse date and name index from path](https://discuss.elastic.co/t/help-with-grok-patterns-to-parse-date-and-name-index-from-path/141721)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 2\ **Last updated:** [July 26, 2018, 1:35pm UTC](https://discuss.elastic.co/t/help-with-grok-patterns-to-parse-date-and-name-index-from-path/141721 "2018-07-26T13:35:04Z")
Hi, I kindly ask you some help with these Grok patterns. I have two requirements: the first is to extract the date from path and use it to name the Elasticsearch index. The second is to parse a date with a strange forma… --- ## [Reading and using elasticsearch metada with logstash configuration](https://discuss.elastic.co/t/reading-and-using-elasticsearch-metada-with-logstash-configuration/139336)
**Author:** [@msukmana](https://discuss.elastic.co/u/msukmana)\ **Replies:** 1\ **Last updated:** [July 26, 2018, 1:25pm UTC](https://discuss.elastic.co/t/reading-and-using-elasticsearch-metada-with-logstash-configuration/139336 "2018-07-26T13:25:56Z")
I am having a problem to read a dump file from elasticsearch in another system and push it to my elasticsearch using Logstash via file input plugin. My dump file looks like this: {"\_index":"logstash-2018.06.14","\_type":… --- ## [Json message not available in the ELK-stack](https://discuss.elastic.co/t/json-message-not-available-in-the-elk-stack/140754)
**Author:** [@twan](https://discuss.elastic.co/u/twan)\ **Replies:** 3\ **Last updated:** [July 26, 2018, 1:06pm UTC](https://discuss.elastic.co/t/json-message-not-available-in-the-elk-stack/140754 "2018-07-26T13:06:57Z")
Hi there, We've ran into a problem within our ELK-stack. We have an application that logs in json format. When there is an error the application adds an field called "stack\_trace". In this field is the stack trace pars… --- ## [Trying to grok a watchguard](https://discuss.elastic.co/t/trying-to-grok-a-watchguard/141733)
**Author:** [@L33T](https://discuss.elastic.co/u/L33T)\ **Replies:** 2\ **Last updated:** [July 26, 2018, 1:04pm UTC](https://discuss.elastic.co/t/trying-to-grok-a-watchguard/141733 "2018-07-26T13:04:03Z")
Hi, Im trying to make sense when grokking syslog output from a watchguard. Id like it to split into more fields than im actually getting e.g. port, source ip, destination ip etc etc etc... At present in my logstash conf… --- ## [Wrong argument type NilClass (expected IO)](https://discuss.elastic.co/t/wrong-argument-type-nilclass-expected-io/141732)
**Author:** [@helloworld](https://discuss.elastic.co/u/helloworld)\ **Replies:** 1\ **Last updated:** [July 26, 2018, 12:49pm UTC](https://discuss.elastic.co/t/wrong-argument-type-nilclass-expected-io/141732 "2018-07-26T12:49:34Z")
The following error seems to be occurring in random in logstash. I am not able to understand the issue. Could you please explain what does it mean? logstash version: 5.2.0 \[ERROR\]\[logstash.filters.ruby \] Ruby excep… --- ## [Using Ruby C Extensions with Logstash Ruby Plugin](https://discuss.elastic.co/t/using-ruby-c-extensions-with-logstash-ruby-plugin/141766)
**Author:** [@env070](https://discuss.elastic.co/u/env070)\ **Replies:** 1\ **Last updated:** [July 26, 2018, 12:49pm UTC](https://discuss.elastic.co/t/using-ruby-c-extensions-with-logstash-ruby-plugin/141766 "2018-07-26T12:49:14Z")
Hello, I have a Ruby C extension which I've created with the following tutorial: https://en.wikibooks.org/wiki/Ruby\_Programming/C\_Extensions It works perfectly when I create the Makefile with the Ruby installed on my m… --- ## [Grok add\_field from another field](https://discuss.elastic.co/t/grok-add-field-from-another-field/141705)
**Author:** [@igemon](https://discuss.elastic.co/u/igemon)\ **Replies:** 1\ **Last updated:** [July 26, 2018, 12:46pm UTC](https://discuss.elastic.co/t/grok-add-field-from-another-field/141705 "2018-07-26T12:46:44Z")
I have a field "host" that looks like this: "aaa-b1.site.com" I want to add a new field "env" containing only "b1". My filter: grok { match =\> { "message" =\> "url:%{DATA:host}" } add\_field =\> { "env" =\> ".\*%{DATA:env}… --- ## [Selecting values from JSON](https://discuss.elastic.co/t/selecting-values-from-json/141400)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 4\ **Last updated:** [July 26, 2018, 9:04am UTC](https://discuss.elastic.co/t/selecting-values-from-json/141400 "2018-07-26T09:04:47Z")
Hello. I am parsing twitter with following settings: input { twitter { consumer\_key =\> "XX" consumer\_secret =\> "XX" oauth\_token =\> "XX" oauth\_token\_secret =\> "XX" full\_tweet =\> true use\_samples =\> true language… --- ## [Loading MYSQL database into Elasticsearch using Logstash](https://discuss.elastic.co/t/loading-mysql-database-into-elasticsearch-using-logstash/141451)
**Author:** [@anon98066099](https://discuss.elastic.co/u/anon98066099)\ **Replies:** 6\ **Last updated:** [July 26, 2018, 8:14am UTC](https://discuss.elastic.co/t/loading-mysql-database-into-elasticsearch-using-logstash/141451 "2018-07-26T08:14:23Z")
I am trying to load a simple table using logstash into elasticsearch. But i am unable to do so as i am getting the below error: C:\\Users\\IYE8288\\Downloads\\logstash-6.3.1\\logstash-6.3.1\\bin\>logstash -f mysql.conf Sendin… --- ## [Filebeat logstash index not creating](https://discuss.elastic.co/t/filebeat-logstash-index-not-creating/141702)
**Author:** [@jamesarems](https://discuss.elastic.co/u/jamesarems)\ **Replies:** 1\ **Last updated:** [July 26, 2018, 7:22am UTC](https://discuss.elastic.co/t/filebeat-logstash-index-not-creating/141702 "2018-07-26T07:22:52Z")
Hi, Good day..! I have configured ELK on a CentOS 7 machine. Initially i have got index in elastic search and accessed logs from kibana. Once i removed all index data from elastic search , problem started. I have ch… --- ## [Problem with logstash-input-file plugin](https://discuss.elastic.co/t/problem-with-logstash-input-file-plugin/141579)
**Author:** [@Pierre2](https://discuss.elastic.co/u/Pierre2)\ **Replies:** 2\ **Last updated:** [July 26, 2018, 6:59am UTC](https://discuss.elastic.co/t/problem-with-logstash-input-file-plugin/141579 "2018-07-26T06:59:56Z")
Hello ! Since the update of input-file plugin to version 4.1.5 when I start logstash I obtain this error : \[2018-07-25T15:04:51,634\]\[INFO \]\[filewatch.observingtail \] START, creating Disc overer, Watch with file and … --- ## [Logstash multiline codec makes logstash unresponsive!](https://discuss.elastic.co/t/logstash-multiline-codec-makes-logstash-unresponsive/141703)
**Author:** [@shahrukhkhan](https://discuss.elastic.co/u/shahrukhkhan)\ **Replies:** 0\ **Last updated:** [July 26, 2018, 6:45am UTC](https://discuss.elastic.co/t/logstash-multiline-codec-makes-logstash-unresponsive/141703 "2018-07-26T06:45:10Z")
Hey, so i had written an input configuration for reading GC log files(which are actually on size based rolling) and used multiline codec with auto\_flush\_interval set to 1.I am using logstash version 6.1.1. It works and e… --- ## [Elasticsearch index response in logstash](https://discuss.elastic.co/t/elasticsearch-index-response-in-logstash/141285)
**Author:** [@hambrise\_luke\_alan\_j](https://discuss.elastic.co/u/hambrise_luke_alan_j)\ **Replies:** 2\ **Last updated:** [July 25, 2018, 10:38pm UTC](https://discuss.elastic.co/t/elasticsearch-index-response-in-logstash/141285 "2018-07-25T22:38:25Z")
Hi, I am trying to index data to elasticsearch through logstash. I want to get the output like how we get the result using curl commands or apis(like below). { "\_index": "testindex", "\_type": "\_doc", "\_id": "1", "\_v… --- ## [Index not created by Logstash](https://discuss.elastic.co/t/index-not-created-by-logstash/141231)
**Author:** [@Albert](https://discuss.elastic.co/u/Albert)\ **Replies:** 2\ **Last updated:** [July 25, 2018, 10:30pm UTC](https://discuss.elastic.co/t/index-not-created-by-logstash/141231 "2018-07-25T22:30:38Z")
Hi all, I'm following this blog (https://www.elastic.co/blog/monitoring-the-search-queries) to capture search queries of Elastic Search that is installed on a MediaWiki site with a CirrusSearch extension using the follo… --- ## [Logstash to LogRhythm SIEM](https://discuss.elastic.co/t/logstash-to-logrhythm-siem/141666)
**Author:** [@valbizures](https://discuss.elastic.co/u/valbizures)\ **Replies:** 0\ **Last updated:** [July 25, 2018, 9:50pm UTC](https://discuss.elastic.co/t/logstash-to-logrhythm-siem/141666 "2018-07-25T21:50:03Z")
Has anyone had any success setting up Logstash to output directly to a LogRhythm SIEM? If so can you provide some information as to the types of outputs you used for: HTTP(S) - I'm currently using packetbeat DNS - I'm… --- ## [Cannot remove "host" field](https://discuss.elastic.co/t/cannot-remove-host-field/141232)
**Author:** [@MartinCorrado](https://discuss.elastic.co/u/MartinCorrado)\ **Replies:** 1\ **Last updated:** [July 25, 2018, 8:13pm UTC](https://discuss.elastic.co/t/cannot-remove-host-field/141232 "2018-07-25T20:13:38Z")
Hi! I'm trying to remove the "host" field but it seems that it's not working for me and every time I recreate the logstash instance, the second time that my queries are executed the following error appears over and over … --- ## [Jdbc input plugin not indexing all data to elasticsearch](https://discuss.elastic.co/t/jdbc-input-plugin-not-indexing-all-data-to-elasticsearch/141651)
**Author:** [@snigdharao](https://discuss.elastic.co/u/snigdharao)\ **Replies:** 1\ **Last updated:** [July 25, 2018, 8:06pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-not-indexing-all-data-to-elasticsearch/141651 "2018-07-25T20:06:21Z")
Hi , I am trying to index 80 Million data by using postgres jdbc plugin. I have tried pagination and it doesnot index all the records.It is indexing only 60 Million records. I have also used jdbc\_fetch\_size .It doesnt… --- ## [Logstash std / tcp output has a duplicate entry field logentry](https://discuss.elastic.co/t/logstash-std-tcp-output-has-a-duplicate-entry-field-logentry/141149)
**Author:** [@naggappan](https://discuss.elastic.co/u/naggappan)\ **Replies:** 1\ **Last updated:** [July 25, 2018, 7:15pm UTC](https://discuss.elastic.co/t/logstash-std-tcp-output-has-a-duplicate-entry-field-logentry/141149 "2018-07-25T19:15:35Z")
I use mongodb input plugin to get latest document from mongodb and send it to logstash as input, And output as STDOUT & a TCP port and in both the output I see an extra field added in my json data as "log\_entry" and it c… --- ## [Run Logstash in the background on Ubuntu](https://discuss.elastic.co/t/run-logstash-in-the-background-on-ubuntu/141422)
**Author:** [@hHelen](https://discuss.elastic.co/u/hHelen)\ **Replies:** 9\ **Last updated:** [July 25, 2018, 7:10pm UTC](https://discuss.elastic.co/t/run-logstash-in-the-background-on-ubuntu/141422 "2018-07-25T19:10:49Z")
I already have logstash running on the termial with this command: sudo -Hu logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -f logstash-simple.conf I can see files scrolling past on the terminal,… --- ## [Tokenizing incoming stream](https://discuss.elastic.co/t/tokenizing-incoming-stream/141619)
**Author:** [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Replies:** 3\ **Last updated:** [July 25, 2018, 5:45pm UTC](https://discuss.elastic.co/t/tokenizing-incoming-stream/141619 "2018-07-25T17:45:08Z")
I'd like to tokenize a field as soon as I receive it in Logstash. I understand I can make it lowercase but it is possible to tokenize it, too? --- ## [Character '-' (dash) not recognizing by plugin path](https://discuss.elastic.co/t/character-dash-not-recognizing-by-plugin-path/141630)
**Author:** [@Denis\_Neres](https://discuss.elastic.co/u/Denis_Neres)\ **Replies:** 0\ **Last updated:** [July 25, 2018, 4:57pm UTC](https://discuss.elastic.co/t/character-dash-not-recognizing-by-plugin-path/141630 "2018-07-25T16:57:30Z")
Hi there, I'm try to put a filename in input of logstash as this: //path =\> "path/to/my/dir/filename\*.log but it isn't working. I made some tests, and I figure out that when the file has a '-' (dash) character in its … --- ## [Grok skipping on \[ when supplied \\s in a custom filter](https://discuss.elastic.co/t/grok-skipping-on-when-supplied-s-in-a-custom-filter/141603)
**Author:** [@Neeraj\_Jadhav](https://discuss.elastic.co/u/Neeraj_Jadhav)\ **Replies:** 2\ **Last updated:** [July 25, 2018, 3:18pm UTC](https://discuss.elastic.co/t/grok-skipping-on-when-supplied-s-in-a-custom-filter/141603 "2018-07-25T15:18:47Z")
This is the log that I am trying to parse ==\> DESTINATION UNREACHABLE: PORT UNREACHABLE \*\* ORIGINAL DATAGRAM DUMP: 10.8.10.21:28184 -\> 10.8.10.19:69 UDP TTL:64 TOS:0x0 ID:40693 IpLen:20 DgmLen:88 Len: 60 Csum: 19270 (60… --- ## [After upgrade to 6.3.2, nothing emitted until 2nd retart](https://discuss.elastic.co/t/after-upgrade-to-6-3-2-nothing-emitted-until-2nd-retart/141609)
**Author:** [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Replies:** 0\ **Last updated:** [July 25, 2018, 3:06pm UTC](https://discuss.elastic.co/t/after-upgrade-to-6-3-2-nothing-emitted-until-2nd-retart/141609 "2018-07-25T15:06:53Z")
I upgraded from logstash 6.2.2 to 6.3.2 (deleting queue directories). After first start, events were received but none showed emitted, queue files were growing in size. I couldn't find any errors, After restart, it lo… --- ## [Conditional checks in mutate gsub](https://discuss.elastic.co/t/conditional-checks-in-mutate-gsub/141564)
**Author:** [@Honda\_fred\_elk](https://discuss.elastic.co/u/Honda_fred_elk)\ **Replies:** 4\ **Last updated:** [July 25, 2018, 2:44pm UTC](https://discuss.elastic.co/t/conditional-checks-in-mutate-gsub/141564 "2018-07-25T14:44:44Z")
Hello, When parsing WAS System out logs, I want to check the EventType and replace the abbreviation with full Description. like EventType (Replace with) Full Description "I" --\> "Informational" "F" --\> "Fatal" … --- ## [Logstash is stuck?](https://discuss.elastic.co/t/logstash-is-stuck/141571)
**Author:** [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Replies:** 4\ **Last updated:** [July 25, 2018, 2:41pm UTC](https://discuss.elastic.co/t/logstash-is-stuck/141571 "2018-07-25T14:41:26Z")
I'd like to load a csv into Elastic using Logstash. This is the output of my run: / C:\\Users...\\Documents\\logstash-6.3.1\\logstash-6.3.1\\bin\>logstash -f logstash\_rplim3.config Sending Logstash's logs to C:/Users/.../Doc… --- ## [How to collect data periodically from multiple HTTP endpoints, and indexing the result in Elasticsearch](https://discuss.elastic.co/t/how-to-collect-data-periodically-from-multiple-http-endpoints-and-indexing-the-result-in-elasticsearch/141503)
**Author:** [@m.sereda](https://discuss.elastic.co/u/m.sereda)\ **Replies:** 19\ **Last updated:** [July 25, 2018, 2:33pm UTC](https://discuss.elastic.co/t/how-to-collect-data-periodically-from-multiple-http-endpoints-and-indexing-the-result-in-elasticsearch/141503 "2018-07-25T14:33:31Z")
Hi, could anyone explain me how to collect data periodically from multiple HTTP endpoints, and indexing the result in Elasticsearch --- ## [Cloudwatch input plugin- No Metrics to Query Error in 6.3.0](https://discuss.elastic.co/t/cloudwatch-input-plugin-no-metrics-to-query-error-in-6-3-0/137032)
**Author:** [@rahulsaini](https://discuss.elastic.co/u/rahulsaini)\ **Replies:** 3\ **Last updated:** [July 25, 2018, 1:25pm UTC](https://discuss.elastic.co/t/cloudwatch-input-plugin-no-metrics-to-query-error-in-6-3-0/137032 "2018-07-25T13:25:34Z")
I'm new to ELK working on new version 6.3.0 I get this error but everything work fine on 6.2.1 Stack: C:/Elastic/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-input-cloudwatch-2.1.1/lib/logstash/inputs/cloudwat… --- ## [How to filter Errors only from logs? using logstash](https://discuss.elastic.co/t/how-to-filter-errors-only-from-logs-using-logstash/141563)
**Author:** [@naga\_kunchala](https://discuss.elastic.co/u/naga_kunchala)\ **Replies:** 6\ **Last updated:** [July 25, 2018, 1:00pm UTC](https://discuss.elastic.co/t/how-to-filter-errors-only-from-logs-using-logstash/141563 "2018-07-25T13:00:26Z")
Hi, I am using logstash in our project, How to filter debug and info log from logs ? and saved separate file? i am using below configuration: input { file { path =\> "/home/kunchala/TESTING/mywork/trap\_testing/consu… --- ## [Error while parsing csv file using logstash](https://discuss.elastic.co/t/error-while-parsing-csv-file-using-logstash/141552)
**Author:** [@jibinoosae](https://discuss.elastic.co/u/jibinoosae)\ **Replies:** 1\ **Last updated:** [July 25, 2018, 1:00pm UTC](https://discuss.elastic.co/t/error-while-parsing-csv-file-using-logstash/141552 "2018-07-25T13:00:07Z")
Hi Below is the error , \[WARN \] 2018-07-25 11:26:36.878 \[Ruby-0-Thread-6@\[main\]\>worker0: :1\] csv - Error parsing csv {:field=\>"message", :source=\>"\\"Siemens AG\\",\\"INC1358413\\",\\"OATWIEN\_5023\\",\\"\\",\\"Closed\\",\\"4-Plan… --- ## [Start\_position =\> "end"](https://discuss.elastic.co/t/start-position-end/141536)
**Author:** [@Pierre2](https://discuss.elastic.co/u/Pierre2)\ **Replies:** 5\ **Last updated:** [July 25, 2018, 10:31am UTC](https://discuss.elastic.co/t/start-position-end/141536 "2018-07-25T10:31:28Z")
Hello, I used to set the start position to "beginning" but now I want to set the value to "end". The problem is that after setting the start\_position to "end" logstash does not parse my logs anymore. I already tried to… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=713) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=715)