# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1140 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1141 --- ## [Unable to Parse logs using filter{} in Logstash](https://discuss.elastic.co/t/unable-to-parse-logs-using-filter-in-logstash/52763)
**Author:** [@nishant\_goel](https://discuss.elastic.co/u/nishant_goel)\ **Replies:** 0\ **Last updated:** [June 14, 2016, 5:12pm UTC](https://discuss.elastic.co/t/unable-to-parse-logs-using-filter-in-logstash/52763 "2016-06-14T17:12:29Z")
Hi, I want to parse a log but I am not able to achieve this. I am getting no output on my console. Whenever I run logstash using the command: bin/logstash -f firstInput.conf --debug I get this: Starting pipeline {:… --- ## [Is there a way to tag for different grok matches?](https://discuss.elastic.co/t/is-there-a-way-to-tag-for-different-grok-matches/52785)
**Author:** [@Josh\_Reichardt](https://discuss.elastic.co/u/Josh_Reichardt)\ **Replies:** 4\ **Last updated:** [June 16, 2016, 7:06pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-tag-for-different-grok-matches/52785 "2016-06-16T19:06:24Z")
I am using the GELF Docker driver to send logs to Logstash. It has been working well so far but I am having trouble finding a good way to separate nginx access logs and nginx error logs. I was thinking that I would jus… --- ## [ElasticSearch Delayed Indexing](https://discuss.elastic.co/t/elasticsearch-delayed-indexing/52875)
**Author:** [@Eric\_Luellen](https://discuss.elastic.co/u/Eric_Luellen)\ **Replies:** 2\ **Last updated:** [June 16, 2016, 3:56pm UTC](https://discuss.elastic.co/t/elasticsearch-delayed-indexing/52875 "2016-06-16T15:56:58Z")
I currently have the following setup: syslog-ng servers --\> Logstash --\> ElasticSearch The syslog-ng servers are load balanced and write to a SAN location where Logstash just tails the files and sends them to ES. I'm… --- ## [Logstash shuts down while events are still being processed](https://discuss.elastic.co/t/logstash-shuts-down-while-events-are-still-being-processed/52824)
**Author:** [@val](https://discuss.elastic.co/u/val)\ **Replies:** 1\ **Last updated:** [June 16, 2016, 3:02pm UTC](https://discuss.elastic.co/t/logstash-shuts-down-while-events-are-still-being-processed/52824 "2016-06-16T15:02:54Z")
I've encountered a weird case while trying to load events from a DB and forward them to ActiveMQ via the Stomp protocol. Basically, my config is very simple and looks like the one below. The SELECT returns approximatel… --- ## [Cannot build custom plugin due to logstash-core-plugin-api version issue](https://discuss.elastic.co/t/cannot-build-custom-plugin-due-to-logstash-core-plugin-api-version-issue/52991)
**Author:** [@chrstphrhrt](https://discuss.elastic.co/u/chrstphrhrt)\ **Replies:** 0\ **Last updated:** [June 16, 2016, 12:23pm UTC](https://discuss.elastic.co/t/cannot-build-custom-plugin-due-to-logstash-core-plugin-api-version-issue/52991 "2016-06-16T12:23:34Z")
Hi, I've created a custom plugin derived from logstash-input-redis that adds support for additional processing of output that redis emits but which is not parsed by the upstream plugin. I have successfully built the ge… --- ## [Logstash - Problem when index first line of a csv file with "?"](https://discuss.elastic.co/t/logstash-problem-when-index-first-line-of-a-csv-file-with/52098)
**Author:** [@pittoch](https://discuss.elastic.co/u/pittoch)\ **Replies:** 4\ **Last updated:** [June 16, 2016, 12:05pm UTC](https://discuss.elastic.co/t/logstash-problem-when-index-first-line-of-a-csv-file-with/52098 "2016-06-16T12:05:57Z")
Hi, Whenever I index a CSV file , the first line is preceded by a question mark "?". So my date is not properly indexed. "Date": "?2015-01-01 01:00:00", So the date is not correctly read: ttags \_dateparsefailu… --- ## [How to escape row separator is there in the data](https://discuss.elastic.co/t/how-to-escape-row-separator-is-there-in-the-data/52953)
**Author:** [@baneshbanna](https://discuss.elastic.co/u/baneshbanna)\ **Replies:** 2\ **Last updated:** [June 16, 2016, 8:36am UTC](https://discuss.elastic.co/t/how-to-escape-row-separator-is-there-in-the-data/52953 "2016-06-16T08:36:35Z")
How to escape if row separator is there in the data.Field is enclosed. 245544007,247794,6/15/2016 11:46,Your Account is verified with Pan Card. Now you can make payments and do more with your account. Thank you! for ve… --- ## [Logstash: convert date string to timestamp in millisecond](https://discuss.elastic.co/t/logstash-convert-date-string-to-timestamp-in-millisecond/52845)
**Author:** [@sen](https://discuss.elastic.co/u/sen)\ **Replies:** 6\ **Last updated:** [June 16, 2016, 7:50am UTC](https://discuss.elastic.co/t/logstash-convert-date-string-to-timestamp-in-millisecond/52845 "2016-06-16T07:50:40Z")
Hello everyone, I have a filed date where my date is in the following format: 2016-03-30T12:37:27.424709000Z and I would like to convert it in millisecond. Do you how to do ? I didn't find a solution Thank you for… --- ## [Template is not getting reflected after changing the elasticsearch vesion](https://discuss.elastic.co/t/template-is-not-getting-reflected-after-changing-the-elasticsearch-vesion/51916)
**Author:** [@gayathri](https://discuss.elastic.co/u/gayathri)\ **Replies:** 2\ **Last updated:** [June 16, 2016, 7:30am UTC](https://discuss.elastic.co/t/template-is-not-getting-reflected-after-changing-the-elasticsearch-vesion/51916 "2016-06-16T07:30:19Z")
Hi Team, My logstash version: 1.5.3 elasticsearch version:1.5.2 kibana vesion: 4.1.3 I have added new template, and it was working...... When I changed my elasticsearch vesion to 2.3.3 and kibana version … --- ## [Geoip help](https://discuss.elastic.co/t/geoip-help/50476)
**Author:** [@anoopmk007](https://discuss.elastic.co/u/anoopmk007)\ **Replies:** 8\ **Last updated:** [June 16, 2016, 6:53am UTC](https://discuss.elastic.co/t/geoip-help/50476 "2016-06-16T06:53:16Z")
Hello , I am trying to implement ELK for my customized logs. Almost done with my limited knowledge. Now logs are successfully parsing , storing in elasticsearch and showing in kibana. But like to include geo l… --- ## [Server suitability for ELK install](https://discuss.elastic.co/t/server-suitability-for-elk-install/52459)
**Author:** [@smilin\_stan](https://discuss.elastic.co/u/smilin_stan)\ **Replies:** 3\ **Last updated:** [June 16, 2016, 6:35am UTC](https://discuss.elastic.co/t/server-suitability-for-elk-install/52459 "2016-06-16T06:35:01Z")
I have a single ELK server with the following hardware configuration: Dual core Xeon 3.2Ghz 20GB RAM 500GB Hard disk space (RAID 1) I have 10 remote hosts, generating a combined total of: 16 GB of logs per day (N… --- ## [Unable to write matching date pattern in log stash](https://discuss.elastic.co/t/unable-to-write-matching-date-pattern-in-log-stash/52544)
**Author:** [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\ **Replies:** 7\ **Last updated:** [June 16, 2016, 6:27am UTC](https://discuss.elastic.co/t/unable-to-write-matching-date-pattern-in-log-stash/52544 "2016-06-16T06:27:10Z")
Hi, I would like to match the date \[Mon Apr 4 12:23:30 CDT 2016\] in my log file. I tried: filter { grok { match =\> \[ "message", "(?%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})" \] } date { … --- ## [Logstash: Calling webservice in filter plugin for preprocesssing](https://discuss.elastic.co/t/logstash-calling-webservice-in-filter-plugin-for-preprocesssing/52947)
**Author:** [@preehana](https://discuss.elastic.co/u/preehana)\ **Replies:** 1\ **Last updated:** [June 16, 2016, 6:05am UTC](https://discuss.elastic.co/t/logstash-calling-webservice-in-filter-plugin-for-preprocesssing/52947 "2016-06-16T06:05:54Z")
Hello All, I am actually taking beats as an input and I need to pre-process the data in the filter section of logstash before sending to elastic search.The data should be pre-processed by calling a rest service in the … --- ## [Failed to parse \[timestamp\] invalid formate](https://discuss.elastic.co/t/failed-to-parse-timestamp-invalid-formate/52926)
**Author:** [@Josh\_Reichardt](https://discuss.elastic.co/u/Josh_Reichardt)\ **Replies:** 1\ **Last updated:** [June 16, 2016, 12:48am UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp-invalid-formate/52926 "2016-06-16T00:48:15Z")
I am attempting to parse some syslog messages and I'm seeing the following message in the Logstash logs whenever I try to match some parsed syslog field: :response=\>{"create"=\>{"\_index"=\>"logstash-2016.06.15", "\_type"… --- ## [Problem on Logstash setup](https://discuss.elastic.co/t/problem-on-logstash-setup/52405)
**Author:** [@ybritto](https://discuss.elastic.co/u/ybritto)\ **Replies:** 4\ **Last updated:** [June 16, 2016, 12:27am UTC](https://discuss.elastic.co/t/problem-on-logstash-setup/52405 "2016-06-16T00:27:53Z")
Hi, I'm new on ELK stack. In fact, I'm in the first steps of. I was following this tutorial https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html Everything was doing good until the step where I'… --- ## [Logstash + sincedb\_path and start\_position](https://discuss.elastic.co/t/logstash-sincedb-path-and-start-position/52890)
**Author:** [@sarthak2016](https://discuss.elastic.co/u/sarthak2016)\ **Replies:** 1\ **Last updated:** [June 15, 2016, 10:40pm UTC](https://discuss.elastic.co/t/logstash-sincedb-path-and-start-position/52890 "2016-06-15T22:40:30Z")
Hello All, If possible, can someone please help me to undestand sincedb\_path =\> '/dev/null' and start\_position =\> beginning or start\_position =\> end, In which scenario we need to use sincedb\_path or start\_position in… --- ## [Escape JSON Keys in Logstash](https://discuss.elastic.co/t/escape-json-keys-in-logstash/52883)
**Author:** [@Daniel\_Tebernum](https://discuss.elastic.co/u/Daniel_Tebernum)\ **Replies:** 1\ **Last updated:** [June 15, 2016, 10:25pm UTC](https://discuss.elastic.co/t/escape-json-keys-in-logstash/52883 "2016-06-15T22:25:13Z")
Hello, I'm new to the logstash topic and want to log JSON data I'm getting from systems I have no control over. They send me JSON strings which can include characters like a '.' or '$' in their keys because they are some… --- ## [Logstash isn't passing any information to along](https://discuss.elastic.co/t/logstash-isnt-passing-any-information-to-along/52867)
**Author:** [@brandonmcgrath1](https://discuss.elastic.co/u/brandonmcgrath1)\ **Replies:** 1\ **Last updated:** [June 15, 2016, 10:22pm UTC](https://discuss.elastic.co/t/logstash-isnt-passing-any-information-to-along/52867 "2016-06-15T22:22:02Z")
I was trying to use nxlog to transfer logs from windows events logs to kibana but that didnt work. Now im trying to use SNMPTRAPS but it still won't work. Any ideas?! logstash.conf: input { tcp { codec =\> json\_l… --- ## [Taking in netflow records](https://discuss.elastic.co/t/taking-in-netflow-records/52904)
**Author:** [@Fidel\_Netflow](https://discuss.elastic.co/u/Fidel_Netflow)\ **Replies:** 0\ **Last updated:** [June 15, 2016, 5:54pm UTC](https://discuss.elastic.co/t/taking-in-netflow-records/52904 "2016-06-15T17:54:33Z")
Hey, I'm a pretty new to the ELK stack, and I am trying to configure it to take in and make use of netflow records. After some fiddling around, I've got the records making it through the stack, but not all of the fields… --- ## [Mutate only works at the top level?](https://discuss.elastic.co/t/mutate-only-works-at-the-top-level/52893)
**Author:** [@iamthealex](https://discuss.elastic.co/u/iamthealex)\ **Replies:** 0\ **Last updated:** [June 15, 2016, 4:27pm UTC](https://discuss.elastic.co/t/mutate-only-works-at-the-top-level/52893 "2016-06-15T16:27:40Z")
Here is rename.conf input { stdin {} } filter { mutate { add\_field =\> { "\[b\]" =\> "bozo" } } mutate { rename =\> \[ "\[b\]", "\[fancy\]\[pants\]" \] } } output { stdout { codec =\> rubydebug { metadata =\> true } } } H… --- ## [Logstash http output plugin stop working: header issue?](https://discuss.elastic.co/t/logstash-http-output-plugin-stop-working-header-issue/38436)
**Author:** [@Jeffrey\_Skilton](https://discuss.elastic.co/u/Jeffrey_Skilton)\ **Replies:** 2\ **Last updated:** [June 15, 2016, 4:13pm UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-stop-working-header-issue/38436 "2016-06-15T16:13:53Z")
this was working great. Then after a standard yum update on a CENTOS 6 box, my logstash outputs for http all stopped working. NameError: no method 'setHeader' for arguments (org.jruby.RubyString,org.jruby.RubyFixnum)… --- ## [Varnish grok pattern optimization](https://discuss.elastic.co/t/varnish-grok-pattern-optimization/52100)
**Author:** [@michbsd](https://discuss.elastic.co/u/michbsd)\ **Replies:** 2\ **Last updated:** [June 15, 2016, 2:02pm UTC](https://discuss.elastic.co/t/varnish-grok-pattern-optimization/52100 "2016-06-15T14:02:01Z")
Hi, I am looking for help in optimizing my grok pattern for my varnishncsa logs. This is what I have now: %{IP:ip1}, %{IP:ip2} - \\\[%{HTTPDATE:timestamp}\\\] %{WORD:method} '%{URIHOST:host}' '%{PATH:path}' '(?:%{URIP… --- ## [Logstash fails to start when passed -l option](https://discuss.elastic.co/t/logstash-fails-to-start-when-passed-l-option/52863)
**Author:** [@Daniel\_Doyle](https://discuss.elastic.co/u/Daniel_Doyle)\ **Replies:** 1\ **Last updated:** [June 15, 2016, 1:36pm UTC](https://discuss.elastic.co/t/logstash-fails-to-start-when-passed-l-option/52863 "2016-06-15T13:36:14Z")
Hi there, I'm experiencing some issues getting logstash to start as a system service on CentOS 7.2. My configuration is correct and I can start logstash from the bin folder, passing my conf.d directory as -f, however w… --- ## [How to correctly parse and index dates into timestamp that have no offset? (Post title renamed)](https://discuss.elastic.co/t/how-to-correctly-parse-and-index-dates-into-timestamp-that-have-no-offset-post-title-renamed/52535)
**Author:** [@niceguybob](https://discuss.elastic.co/u/niceguybob)\ **Replies:** 6\ **Last updated:** [June 15, 2016, 11:49am UTC](https://discuss.elastic.co/t/how-to-correctly-parse-and-index-dates-into-timestamp-that-have-no-offset-post-title-renamed/52535 "2016-06-15T11:49:49Z")
Hi, Hopefully this is a simple one. I have a rather basic Logstash config to read nginx apache and error logs and push them to an index. It's all working OK except the date is incorrect for my error log file. access … --- ## [How can I get the logstash hostname](https://discuss.elastic.co/t/how-can-i-get-the-logstash-hostname/51771)
**Author:** [@ericpietro](https://discuss.elastic.co/u/ericpietro)\ **Replies:** 3\ **Last updated:** [June 15, 2016, 10:06am UTC](https://discuss.elastic.co/t/how-can-i-get-the-logstash-hostname/51771 "2016-06-15T10:06:09Z")
Hi, logstash add a field named 'host' that contains the source hostname in the input section. but how can I get the hostname where logstash is curently running ? ty --- ## [Date filter usage to avoid \_dateparsefailure](https://discuss.elastic.co/t/date-filter-usage-to-avoid--dateparsefailure/52831)
**Author:** [@sumeshms](https://discuss.elastic.co/u/sumeshms)\ **Replies:** 3\ **Last updated:** [June 15, 2016, 10:03am UTC](https://discuss.elastic.co/t/date-filter-usage-to-avoid--dateparsefailure/52831 "2016-06-15T10:03:48Z")
Hello I am getting \_dateparsefailure This is my actual log: "message": "protocol=6, source-ip=192.168.1.81, source-port=64737, destination-ip=1.1.1.1, destination-port=443, time=2016/06/15 10:34:17, interzone-trust… --- ## [How to group all the values for a column into an array jdbc mysql](https://discuss.elastic.co/t/how-to-group-all-the-values-for-a-column-into-an-array-jdbc-mysql/33433)
**Author:** [@clp](https://discuss.elastic.co/u/clp)\ **Replies:** 1\ **Last updated:** [June 15, 2016, 9:32am UTC](https://discuss.elastic.co/t/how-to-group-all-the-values-for-a-column-into-an-array-jdbc-mysql/33433 "2016-06-15T09:32:18Z")
Hi, I am trying to import data from mysql to elasticsearch via logstash and i would like to know if it's possible to group all the values for a column into an array per id. for example in mysql, results of query i… --- ## [Configure Logstash -\> Kafka -\> Logstash](https://discuss.elastic.co/t/configure-logstash-kafka-logstash/52839)
**Author:** [@Tom-Kun](https://discuss.elastic.co/u/Tom-Kun)\ **Replies:** 1\ **Last updated:** [June 15, 2016, 8:46am UTC](https://discuss.elastic.co/t/configure-logstash-kafka-logstash/52839 "2016-06-15T08:46:54Z")
Hello everyone, I'm actually working on a centralize logs solution for applicative logs and apache logs. But i'm novice with Kafka and i don't understand how to connect Kafka input with the first Logstash and Kafka out… --- ## [Logstash put files in jms queue](https://discuss.elastic.co/t/logstash-put-files-in-jms-queue/52837)
**Author:** [@Florian\_H](https://discuss.elastic.co/u/Florian_H)\ **Replies:** 0\ **Last updated:** [June 15, 2016, 8:29am UTC](https://discuss.elastic.co/t/logstash-put-files-in-jms-queue/52837 "2016-06-15T08:29:20Z")
Hi, sorry for my english. For my ending study projet, i must create a report with streamsets. I create my jms queue, i do my file conf but when i lauch my command logstash launched but no files are put in my queue. … --- ## [How to extract the schema from a LogStach Event](https://discuss.elastic.co/t/how-to-extract-the-schema-from-a-logstach-event/52834)
**Author:** [@Nir\_Sharony](https://discuss.elastic.co/u/Nir_Sharony)\ **Replies:** 2\ **Last updated:** [June 15, 2016, 8:20am UTC](https://discuss.elastic.co/t/how-to-extract-the-schema-from-a-logstach-event/52834 "2016-06-15T08:20:36Z")
Hello, I am using LogStash with the logstash-codec-avro\_schema\_registry codec. However the encoding segment is not implemented yet and I need to implement it myself. Here is my LogStash configuration (POC): input { … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1139) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1141)