# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1103 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1104 --- ## [Target of repeat operator is not specified vCenter pattner](https://discuss.elastic.co/t/target-of-repeat-operator-is-not-specified-vcenter-pattner/60296)
**Author:** [@Felix\_Kutscher](https://discuss.elastic.co/u/Felix_Kutscher)\ **Replies:** 4\ **Last updated:** [September 13, 2016, 2:30pm UTC](https://discuss.elastic.co/t/target-of-repeat-operator-is-not-specified-vcenter-pattner/60296 "2016-09-13T14:30:38Z")
Hello, I use the the pattners from http://everythingshouldbevirtual.com/logstash-vcsa-6-0 Now I get a Syntax error with this pattner. logstash-2.3.4-1 Syntax error in the given pattern \<%{INT:syslog\_pri}\>%{SYSLOGTIM… --- ## [Logstash does not send data to elastic search](https://discuss.elastic.co/t/logstash-does-not-send-data-to-elastic-search/60321)
**Author:** [@manishsh](https://discuss.elastic.co/u/manishsh)\ **Replies:** 3\ **Last updated:** [September 13, 2016, 2:12pm UTC](https://discuss.elastic.co/t/logstash-does-not-send-data-to-elastic-search/60321 "2016-09-13T14:12:58Z")
I am trying to run elastic search to send data to Amazon ES. Have configured everything including Amazon Plugin but it is not sending anything to Amazon ES. After Pipeline Main Started nothing happpen --- ## [Flushing buffer at interval](https://discuss.elastic.co/t/flushing-buffer-at-interval/56527)
**Author:** [@Stormdizzle](https://discuss.elastic.co/u/Stormdizzle)\ **Replies:** 4\ **Last updated:** [September 13, 2016, 1:41pm UTC](https://discuss.elastic.co/t/flushing-buffer-at-interval/56527 "2016-09-13T13:41:02Z")
I am curious if someone can assist me. It appears there's an issue with either the latest Logstash or perhaps Elasticsearch can't handle the traffic, which I don't find to be true. I am seeing the logs being fed into E… --- ## [How to set @timestamp timezone?](https://discuss.elastic.co/t/how-to-set-timestamp-timezone/28401)
**Author:** [@z\_w\_w1981](https://discuss.elastic.co/u/z_w_w1981)\ **Replies:** 19\ **Last updated:** [September 13, 2016, 11:41am UTC](https://discuss.elastic.co/t/how-to-set-timestamp-timezone/28401 "2016-09-13T11:41:29Z")
Hi, I found that the @timestamp always show UTC time, My date filter is: date { match =\> \[ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" \] timezone =\> "Asia/Shanghai" } after process the timestamp is: "@… --- ## [Logstash forwarder re-harvesting a particular file](https://discuss.elastic.co/t/logstash-forwarder-re-harvesting-a-particular-file/60285)
**Author:** [@shivam\_singh](https://discuss.elastic.co/u/shivam_singh)\ **Replies:** 1\ **Last updated:** [September 13, 2016, 7:08am UTC](https://discuss.elastic.co/t/logstash-forwarder-re-harvesting-a-particular-file/60285 "2016-09-13T07:08:12Z")
Hey, Logstash forwarder is re-harvesting a particular file for each log event. I have checked the .logstash-forwarder file and found that it has not updated the offset of this file which I did manually still it does no… --- ## [Output not working](https://discuss.elastic.co/t/output-not-working/60261)
**Author:** [@Lorenzo\_Mainardi](https://discuss.elastic.co/u/Lorenzo_Mainardi)\ **Replies:** 1\ **Last updated:** [September 13, 2016, 5:52am UTC](https://discuss.elastic.co/t/output-not-working/60261 "2016-09-13T05:52:31Z")
Hello to all, I'm configuring logstash for parsing a custom log with logstash and sending it to Elasticsearch. This is my output session: output { stdout { codec =\> rubydebug } elasticsearch { hosts =\> … --- ## [How make logstash run faster?](https://discuss.elastic.co/t/how-make-logstash-run-faster/60341)
**Author:** [@vincent403](https://discuss.elastic.co/u/vincent403)\ **Replies:** 1\ **Last updated:** [September 13, 2016, 5:51am UTC](https://discuss.elastic.co/t/how-make-logstash-run-faster/60341 "2016-09-13T05:51:19Z")
now I had a problem with logstash, for my clients,I use logstash for bind dns's log collecting,but some clients part of date the ES server can't collectd, I think it's because the logstash can't handle these so many and … --- ## [Analyze and extract from url?](https://discuss.elastic.co/t/analyze-and-extract-from-url/60204)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 3\ **Last updated:** [September 13, 2016, 5:46am UTC](https://discuss.elastic.co/t/analyze-and-extract-from-url/60204 "2016-09-13T05:46:54Z")
Hi I am thinking of analyzing the urls and need to extract some information. For example, for below url, https://it.fotolia.com/id/102342882" : i will need to extract id = 102342882 https://us.fotolia.com/search?\*\*… --- ## ["UNIX\_MS" Syntax in Date Plugin](https://discuss.elastic.co/t/unix-ms-syntax-in-date-plugin/59840)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 6\ **Last updated:** [September 13, 2016, 1:02am UTC](https://discuss.elastic.co/t/unix-ms-syntax-in-date-plugin/59840 "2016-09-13T01:02:18Z")
How should I use the "UNIX\_MS" Syntax in Date Plugin? Below is what I've tried and it does not work. I cannot find any examples of this online. date { match =\> \[ "my\_field", "MMM dd YYY HH:mm:ss", "MMM… --- ## [How to reference a new codec](https://discuss.elastic.co/t/how-to-reference-a-new-codec/60333)
**Author:** [@stecino](https://discuss.elastic.co/u/stecino)\ **Replies:** 1\ **Last updated:** [September 12, 2016, 10:53pm UTC](https://discuss.elastic.co/t/how-to-reference-a-new-codec/60333 "2016-09-12T22:53:44Z")
Hello, I have installed a new codec logstash-codec-avro\_schema\_registry-0.9.0.gem But when I look at the installed plugin list it's not showing up. Is there a way for me in the kafka input explicitly point to this… --- ## [Quote escaping bug in logstash configs](https://discuss.elastic.co/t/quote-escaping-bug-in-logstash-configs/60112)
**Author:** [@kaung](https://discuss.elastic.co/u/kaung)\ **Replies:** 0\ **Last updated:** [September 8, 2016, 8:05pm UTC](https://discuss.elastic.co/t/quote-escaping-bug-in-logstash-configs/60112 "2016-09-08T20:05:04Z")
One of my configuration, which is actually a sql query, contains both single and double quote chars. So, the usually preferred way to do single quote wrapping double quotes (or the other way round) cannot be used. I need… --- ## [Logstash jdbc input filter broken for tracking\_column](https://discuss.elastic.co/t/logstash-jdbc-input-filter-broken-for-tracking-column/59832)
**Author:** [@biswajit86](https://discuss.elastic.co/u/biswajit86)\ **Replies:** 1\ **Last updated:** [September 12, 2016, 6:49pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-filter-broken-for-tracking-column/59832 "2016-09-12T18:49:16Z")
Hi, I am using the tracking\_column functionality of logstash jdbc input plugin to fetch database values from the db. Right now, the functionality to write the last :sql\_last\_value" into the metadata path seems to brok… --- ## [Mappings are "not\_analyzed" Can it be set with manage\_template](https://discuss.elastic.co/t/mappings-are-not-analyzed-can-it-be-set-with-manage-template/60172)
**Author:** [@Tyler\_Siemers](https://discuss.elastic.co/u/Tyler_Siemers)\ **Replies:** 4\ **Last updated:** [September 12, 2016, 4:46pm UTC](https://discuss.elastic.co/t/mappings-are-not-analyzed-can-it-be-set-with-manage-template/60172 "2016-09-12T16:46:32Z")
I am importing logs from FireSight(cisco) into Logstash and outputting to Elasticsearch. Here is my conf file from Logstash. input { udp { port =\> 5545 type =\> "firesight" } } filter { grok { patterns\_dir =\> … --- ## [Check for more than one text from the grok filtered message?](https://discuss.elastic.co/t/check-for-more-than-one-text-from-the-grok-filtered-message/60286)
**Author:** [@Kulasangar\_Gowrisang](https://discuss.elastic.co/u/Kulasangar_Gowrisang)\ **Replies:** 2\ **Last updated:** [September 12, 2016, 3:19pm UTC](https://discuss.elastic.co/t/check-for-more-than-one-text-from-the-grok-filtered-message/60286 "2016-09-12T15:19:04Z")
I'm trying to find two log types from every single line of the log file, and if it match i'm assigning a log type to it by adding a log\_type field. This is a piece of snippet from the logstash config: filter { … --- ## [TCP Input line breaker not detected](https://discuss.elastic.co/t/tcp-input-line-breaker-not-detected/60305)
**Author:** [@srpatatas](https://discuss.elastic.co/u/srpatatas)\ **Replies:** 0\ **Last updated:** [September 12, 2016, 3:11pm UTC](https://discuss.elastic.co/t/tcp-input-line-breaker-not-detected/60305 "2016-09-12T15:11:55Z")
Hi all, I was wondering if you could help me with this issue. I'm using the TCP Raw input on Logstash and the message I need to capture and parse is JSON encoded, the problem is that the TCP message ends with "0d 0d", i.… --- ## [The kafka and elasticsearch output plugins won't block each other in one pipeline](https://discuss.elastic.co/t/the-kafka-and-elasticsearch-output-plugins-wont-block-each-other-in-one-pipeline/60214)
**Author:** [@xiaozhuqq48](https://discuss.elastic.co/u/xiaozhuqq48)\ **Replies:** 2\ **Last updated:** [September 12, 2016, 2:34pm UTC](https://discuss.elastic.co/t/the-kafka-and-elasticsearch-output-plugins-wont-block-each-other-in-one-pipeline/60214 "2016-09-12T14:34:08Z")
When I upgrade logstash from version2.3.3 to 2.4.0, I find out the logstash outputs won't block each other. In my enviroment, the output plugins are kafka 0.10 and elasticsearch 2.3.4. When kafka is down , the elasticsea… --- ## [Catch 404 error from output plugin and then write back to log file](https://discuss.elastic.co/t/catch-404-error-from-output-plugin-and-then-write-back-to-log-file/27640)
**Author:** [@crickeys](https://discuss.elastic.co/u/crickeys)\ **Replies:** 1\ **Last updated:** [September 12, 2016, 1:25pm UTC](https://discuss.elastic.co/t/catch-404-error-from-output-plugin-and-then-write-back-to-log-file/27640 "2016-09-12T13:25:47Z")
Is there any way to catch this failed action with response of 404, dropping action: from an output plugin and then re-write it into the log or retry it again in a bit instead of dropping it? --- ## [Some events are lost when using Kafka automatic topic creation](https://discuss.elastic.co/t/some-events-are-lost-when-using-kafka-automatic-topic-creation/60270)
**Author:** [@Or\_Weinberger](https://discuss.elastic.co/u/Or_Weinberger)\ **Replies:** 1\ **Last updated:** [September 12, 2016, 1:11pm UTC](https://discuss.elastic.co/t/some-events-are-lost-when-using-kafka-automatic-topic-creation/60270 "2016-09-12T13:11:14Z")
I'm running the latest stable Logstash version and Kafka v0.10.0.1 (however I've observed the same behavior on 0.8.x and 0.9.x versions). My data flow is: Syslog input -\> Kafka output (topic: raw) Kafka input (topi… --- ## [Logstash doesn't read config from /etc/logstash/conf.d/](https://discuss.elastic.co/t/logstash-doesnt-read-config-from-etc-logstash-conf-d/60161)
**Author:** [@sgm](https://discuss.elastic.co/u/sgm)\ **Replies:** 3\ **Last updated:** [September 12, 2016, 11:47am UTC](https://discuss.elastic.co/t/logstash-doesnt-read-config-from-etc-logstash-conf-d/60161 "2016-09-12T11:47:12Z")
Hello. Sorry for my english. I installed logstash-2.4.0\_all.deb on ubuntu server 16.04.1. I wrote in /etc/logstash/conf.d/ex.conf: input { tcp { type =\> "eventlog" port =\> 3515 codec =\> … --- ## [Logstash not reading files](https://discuss.elastic.co/t/logstash-not-reading-files/60273)
**Author:** [@debarun](https://discuss.elastic.co/u/debarun)\ **Replies:** 2\ **Last updated:** [September 12, 2016, 11:39am UTC](https://discuss.elastic.co/t/logstash-not-reading-files/60273 "2016-09-12T11:39:25Z")
Hi, I have a logstash installation, and it suddenly has stopped reading from files. The sincedb\_path is created but files are not read. I have tried only stdin and stdout, and that works Below is the config file:- in… --- ## [Grok filter for cyberoam syslogs](https://discuss.elastic.co/t/grok-filter-for-cyberoam-syslogs/25618)
**Author:** [@Andreas](https://discuss.elastic.co/u/Andreas)\ **Replies:** 2\ **Last updated:** [September 12, 2016, 9:17am UTC](https://discuss.elastic.co/t/grok-filter-for-cyberoam-syslogs/25618 "2016-09-12T09:17:28Z")
Hi all, i have been experementing with ELK the last 2 weeks. And it looks very good! But my main goal is to analyze our own firewall logs. But it dosnt look like my syslog filter can match any fields. I didnt get any d… --- ## [Logstash mutate filter](https://discuss.elastic.co/t/logstash-mutate-filter/59978)
**Author:** [@TRSRaphael](https://discuss.elastic.co/u/TRSRaphael)\ **Replies:** 8\ **Last updated:** [September 12, 2016, 8:34am UTC](https://discuss.elastic.co/t/logstash-mutate-filter/59978 "2016-09-12T08:34:15Z")
Hello, logstash is getting the events coming from collectd and put it in ElasticSearch. from kibana (plugged on ElasticSearch) I see the logstash index fields : geoip.ip, geoip.location, majflt, ... In the input-co… --- ## [Loagstash with Load Runner](https://discuss.elastic.co/t/loagstash-with-load-runner/60253)
**Author:** [@Senz79](https://discuss.elastic.co/u/Senz79)\ **Replies:** 0\ **Last updated:** [September 12, 2016, 5:49am UTC](https://discuss.elastic.co/t/loagstash-with-load-runner/60253 "2016-09-12T05:49:54Z")
Hello I have successfully integrated JMETER with ELK stack. The same I need to do with Load Runner. Is there any way to do? can anyone share some reference link to start with. Thanks Shantanu --- ## [Duplicate Field with grok parsing iis logs](https://discuss.elastic.co/t/duplicate-field-with-grok-parsing-iis-logs/60218)
**Author:** [@asafi31](https://discuss.elastic.co/u/asafi31)\ **Replies:** 1\ **Last updated:** [September 11, 2016, 8:06pm UTC](https://discuss.elastic.co/t/duplicate-field-with-grok-parsing-iis-logs/60218 "2016-09-11T20:06:02Z")
hello all i am mount my iis logs to ELK centos machine i am using logstash for parsing the iislogs but when i looking iis logs in kibana i saw grok field as duplicate message is o.k kindly your adive thx as… --- ## [Logstash regex option for match settings!](https://discuss.elastic.co/t/logstash-regex-option-for-match-settings/60209)
**Author:** [@gugansankar](https://discuss.elastic.co/u/gugansankar)\ **Replies:** 1\ **Last updated:** [September 11, 2016, 8:04pm UTC](https://discuss.elastic.co/t/logstash-regex-option-for-match-settings/60209 "2016-09-11T20:04:19Z")
In my settings, JSON is an input data. I tried to use log timestamp as index @timestamp value. Its working for normal json too. But regex not working for nested json timestamp field match. Here is my config. date… --- ## [How are metrics calculated?](https://discuss.elastic.co/t/how-are-metrics-calculated/56982)
**Author:** [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\ **Replies:** 2\ **Last updated:** [September 10, 2016, 1:53pm UTC](https://discuss.elastic.co/t/how-are-metrics-calculated/56982 "2016-09-10T13:53:57Z")
I'm trying to figure out how the "metrics" filter calculates its value. We're using metrics in conjunction with the datadog\_metrics plugin to send messagerate data to datadog. However, the rate we're seeing doesn't add u… --- ## [Elasticsearch input plugin not working over ssl](https://discuss.elastic.co/t/elasticsearch-input-plugin-not-working-over-ssl/59051)
**Author:** [@marke72](https://discuss.elastic.co/u/marke72)\ **Replies:** 2\ **Last updated:** [September 9, 2016, 2:34pm UTC](https://discuss.elastic.co/t/elasticsearch-input-plugin-not-working-over-ssl/59051 "2016-09-09T14:34:15Z")
So I have Logstash outputting to Elasticsearch through SSL. It's working great. Haven't had an issue with that. I am currently trying to redo an index (reindex) using Logstash where you use both the elasticsearch input p… --- ## [Logstash init script - how to change default variables values without modifying logstash script?](https://discuss.elastic.co/t/logstash-init-script-how-to-change-default-variables-values-without-modifying-logstash-script/60140)
**Author:** [@Saravanan](https://discuss.elastic.co/u/Saravanan)\ **Replies:** 2\ **Last updated:** [September 9, 2016, 1:03pm UTC](https://discuss.elastic.co/t/logstash-init-script-how-to-change-default-variables-values-without-modifying-logstash-script/60140 "2016-09-09T13:03:15Z")
Hi, I am using logstash software (2.2.4 version) on centos linux, and i want to change the default logstash log (LS\_LOG\_DIR) location to custom location (ex: /var/log/test) without modifying default init script (i.e /e… --- ## [Logstash Config works fine but data not populated in elasticsearch](https://discuss.elastic.co/t/logstash-config-works-fine-but-data-not-populated-in-elasticsearch/60148)
**Author:** [@karthikvalluri85](https://discuss.elastic.co/u/karthikvalluri85)\ **Replies:** 8\ **Last updated:** [September 9, 2016, 11:52am UTC](https://discuss.elastic.co/t/logstash-config-works-fine-but-data-not-populated-in-elasticsearch/60148 "2016-09-09T11:52:23Z")
I'm configuring a csv file via logstash and need the ouptut loaded in elasticsearch. Please see below config file details also the commands executed so far.. The terminal gets stuck after "Logsatsh startup completed".. r… --- ## [Removing fields using mutate (JSON)](https://discuss.elastic.co/t/removing-fields-using-mutate-json/60144)
**Author:** [@dsever](https://discuss.elastic.co/u/dsever)\ **Replies:** 4\ **Last updated:** [September 9, 2016, 10:56am UTC](https://discuss.elastic.co/t/removing-fields-using-mutate-json/60144 "2016-09-09T10:56:54Z")
Hi, While parsing some json logs I'm getting into problem with unavailability to remove one subfield, for instance this is my JSON "alert": \[ { "src": { "url": "xxxx", "domain": "xxxx", "smtp-mail-from": "xxxxxxxx" … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1102) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1104)