# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=992 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 993 --- ## [S3 input plugin behind proxy](https://discuss.elastic.co/t/s3-input-plugin-behind-proxy/32272)
**Author:** [@scaarup](https://discuss.elastic.co/u/scaarup)\ **Replies:** 2\ **Last updated:** [April 29, 2017, 10:43pm UTC](https://discuss.elastic.co/t/s3-input-plugin-behind-proxy/32272 "2017-04-29T22:43:03Z")
Has anybody gotten the s3 input plugin working behind a HTTP proxy? All I get is the below? {:timestamp=\>"2015-10-15T15:06:03.343000+0200", :message=\>"A plugin had an unrecoverable error. Will restart this plugin.\\n … --- ## [Google pub sub high cpu](https://discuss.elastic.co/t/google-pub-sub-high-cpu/84095)
**Author:** [@simonl](https://discuss.elastic.co/u/simonl)\ **Replies:** 0\ **Last updated:** [April 29, 2017, 8:16pm UTC](https://discuss.elastic.co/t/google-pub-sub-high-cpu/84095 "2017-04-29T20:16:05Z")
I'm using the Google Pub Sub input, it works nicely but compared to RabbitMQ it uses a large amount of CPU with only one topic, pushing my load averages very high. Using the RabbitMQ plugin, multiple queues uses a fra… --- ## [Avro input to Json output](https://discuss.elastic.co/t/avro-input-to-json-output/84083)
**Author:** [@no\_jihun](https://discuss.elastic.co/u/no_jihun)\ **Replies:** 0\ **Last updated:** [April 29, 2017, 9:55am UTC](https://discuss.elastic.co/t/avro-input-to-json-output/84083 "2017-04-29T09:55:07Z")
Hello. When events created I send it to kafka topic. now I want to consume that same topic then write it to both of elasticsearch and hdfs directory for elasticsearsh it simply can be done with logstash ( kafka in… --- ## [Check if field value is present at certain file](https://discuss.elastic.co/t/check-if-field-value-is-present-at-certain-file/84066)
**Author:** [@prodrg](https://discuss.elastic.co/u/prodrg)\ **Replies:** 0\ **Last updated:** [April 28, 2017, 10:04pm UTC](https://discuss.elastic.co/t/check-if-field-value-is-present-at-certain-file/84066 "2017-04-28T22:04:23Z")
Hello everybody, I have the following case to solve: There is a list of elements that are on a file. There is an specific input from Logstash that I need to correlate with that file, by checking if a value X on a field… --- ## [S3 Input Plugin - S3-compatible REST endpoint configuration](https://discuss.elastic.co/t/s3-input-plugin-s3-compatible-rest-endpoint-configuration/84061)
**Author:** [@dunnj515](https://discuss.elastic.co/u/dunnj515)\ **Replies:** 2\ **Last updated:** [April 28, 2017, 9:42pm UTC](https://discuss.elastic.co/t/s3-input-plugin-s3-compatible-rest-endpoint-configuration/84061 "2017-04-28T21:42:46Z")
I'm new to Logstash and this forum, so I apologize if this has been answered previously. I'm using an EMC S3 instance, not an AWS S3 instance. The S3 documentation shows how to configure the Access Key ID and Secret Ac… --- ## [Trouble matching timestamp](https://discuss.elastic.co/t/trouble-matching-timestamp/83768)
**Author:** [@s58smith](https://discuss.elastic.co/u/s58smith)\ **Replies:** 4\ **Last updated:** [April 28, 2017, 7:58pm UTC](https://discuss.elastic.co/t/trouble-matching-timestamp/83768 "2017-04-28T19:58:35Z")
I'm having trouble getting the ISO8601 to match timestamp in "Date" filter. Below is my filter and output config. I have tried using the commented out lines for "match" but not working. filter { if "CollectorStatus" … --- ## [Logstash adds fields to JSON by parsing another field](https://discuss.elastic.co/t/logstash-adds-fields-to-json-by-parsing-another-field/84032)
**Author:** [@Li\_Cao](https://discuss.elastic.co/u/Li_Cao)\ **Replies:** 2\ **Last updated:** [April 28, 2017, 5:28pm UTC](https://discuss.elastic.co/t/logstash-adds-fields-to-json-by-parsing-another-field/84032 "2017-04-28T17:28:41Z")
Hi! I am trying to set up a pipeline to ingest some logs from Kafka cluster using LogStash and into elasticsearch. The message on Kafka is a JSON string like: {"a"="123", "b"="456", msg="this is a test"} Sometim… --- ## [Locked Threads on Logstash 2.4](https://discuss.elastic.co/t/locked-threads-on-logstash-2-4/83094)
**Author:** [@lonangel](https://discuss.elastic.co/u/lonangel)\ **Replies:** 1\ **Last updated:** [April 28, 2017, 3:29pm UTC](https://discuss.elastic.co/t/locked-threads-on-logstash-2-4/83094 "2017-04-28T15:29:55Z")
Hi there, I have a workflow with two Logstash's and a rabbitmq in the middle to store messages temporarily. On the second logstash, the one behind the rabbitmq, I'm using the following code to set a field with the time… --- ## [KV with new lines](https://discuss.elastic.co/t/kv-with-new-lines/82989)
**Author:** [@rabbit](https://discuss.elastic.co/u/rabbit)\ **Replies:** 1\ **Last updated:** [April 28, 2017, 3:22pm UTC](https://discuss.elastic.co/t/kv-with-new-lines/82989 "2017-04-28T15:22:56Z")
I have the following log : 2017-04-03 07:53:49,576 DEBUG \[TESTING\_DETAILS\] - ###ERROR DETAILS ### TestingError CD Number: 12525 ID Testing: 20 NK 6969N5L Message :Testing Failed TZE \[L00141D\] URL Path: /TestingURL … --- ## [How can i handle nil values in logstash](https://discuss.elastic.co/t/how-can-i-handle-nil-values-in-logstash/83074)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 1\ **Last updated:** [April 28, 2017, 2:59pm UTC](https://discuss.elastic.co/t/how-can-i-handle-nil-values-in-logstash/83074 "2017-04-28T14:59:49Z")
I am using csv as input as shown below firstname lastname dinesh angari rajesh mahesh rangu and i am using http as output and below is my filter mutate {add\_field… --- ## [Problem date parsing](https://discuss.elastic.co/t/problem-date-parsing/84020)
**Author:** [@Sam67000](https://discuss.elastic.co/u/Sam67000)\ **Replies:** 4\ **Last updated:** [April 28, 2017, 2:38pm UTC](https://discuss.elastic.co/t/problem-date-parsing/84020 "2017-04-28T14:38:30Z")
Hello everybody. I have a small problem with some type of log annd I need your help. I couldn't extract the date. The logs are as follows : \[2017/04/25 15:39:46.255\]\[Info\] Block \[\[Main\]\]\[1\]\[ALLOCALL\] \[2017/04/25 15… --- ## [Elasticsearch unreachable Error in Logstash after installing X-Pack](https://discuss.elastic.co/t/elasticsearch-unreachable-error-in-logstash-after-installing-x-pack/83681)
**Author:** [@jasdf](https://discuss.elastic.co/u/jasdf)\ **Replies:** 1\ **Last updated:** [April 28, 2017, 2:08pm UTC](https://discuss.elastic.co/t/elasticsearch-unreachable-error-in-logstash-after-installing-x-pack/83681 "2017-04-28T14:08:44Z")
Hi, I've set up Kibana 5.3.0, Logstash 5.3.0, Elasticsearch 5.3.0 and Filebeat 5.3.1. I had further issues with X-Pack, so I removed it completely and installed it again. I disabled security, monitoring, graph and wa… --- ## [Logstash-output-s3](https://discuss.elastic.co/t/logstash-output-s3/83545)
**Author:** [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\ **Replies:** 1\ **Last updated:** [April 28, 2017, 2:02pm UTC](https://discuss.elastic.co/t/logstash-output-s3/83545 "2017-04-28T14:02:44Z")
I'm seeing an issue with the above plugin, version 5.2. I'm running it in a docker container on kubernetes. It works for about 5 hours before crashing the container with errors like this: 09:51:44.250 \[\[main\]-pipeline-… --- ## [Filter a string date format on logstash](https://discuss.elastic.co/t/filter-a-string-date-format-on-logstash/84016)
**Author:** [@Djelouah\_Laala](https://discuss.elastic.co/u/Djelouah_Laala)\ **Replies:** 4\ **Last updated:** [April 28, 2017, 1:57pm UTC](https://discuss.elastic.co/t/filter-a-string-date-format-on-logstash/84016 "2017-04-28T13:57:16Z")
Hi guys, I get data from an sql server database and i have a date on a string format like this : "2017-04-25 00:00:00 +0200" and I want to use grok filter to transform this to a similar format to @timestamp I have … --- ## [Java.lang.OutMemoryError](https://discuss.elastic.co/t/java-lang-outmemoryerror/84001)
**Author:** [@vineets928](https://discuss.elastic.co/u/vineets928)\ **Replies:** 8\ **Last updated:** [April 28, 2017, 1:52pm UTC](https://discuss.elastic.co/t/java-lang-outmemoryerror/84001 "2017-04-28T13:52:20Z")
Hi, I have to migrate mysql data to elasticsearch. My logstash configuration file is Getting error when starting logstash In logstash configuration file, changed the JVM configuration to 512m and 8g. But after resta… --- ## [Logstash output elasticsearch - document\_id double document produced](https://discuss.elastic.co/t/logstash-output-elasticsearch-document-id-double-document-produced/83995)
**Author:** [@Nico-DF](https://discuss.elastic.co/u/Nico-DF)\ **Replies:** 1\ **Last updated:** [April 28, 2017, 12:42pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-document-id-double-document-produced/83995 "2017-04-28T12:42:43Z")
Hi, I have the following thing I'm trying to do: I a log file in .csv that contains info about a server (localisation, etc.) And along with this .csv, I got several folder (typically, 1/day) that contains a .txt file w… --- ## [JSON Filter target doesn't work](https://discuss.elastic.co/t/json-filter-target-doesnt-work/84008)
**Author:** [@AndreAga](https://discuss.elastic.co/u/AndreAga)\ **Replies:** 0\ **Last updated:** [April 28, 2017, 12:02pm UTC](https://discuss.elastic.co/t/json-filter-target-doesnt-work/84008 "2017-04-28T12:02:23Z")
Hi, this is my Grok + JSON Filter: grok { match =\> { "message" =\> ".\* \\| URL: (?\\[^ \]\*) - Request: (?\(.|\\r|\\n)\*) - Response: (?\(.|\\r|\\n)\*)" } } if \[request\_json\] != "---" { json { … --- ## [Additional Ruby Gems to use in Logstash's ruby filter](https://discuss.elastic.co/t/additional-ruby-gems-to-use-in-logstashs-ruby-filter/84004)
**Author:** [@davidS](https://discuss.elastic.co/u/davidS)\ **Replies:** 1\ **Last updated:** [April 28, 2017, 11:57am UTC](https://discuss.elastic.co/t/additional-ruby-gems-to-use-in-logstashs-ruby-filter/84004 "2017-04-28T11:57:18Z")
Hi all, how can I install additional ruby gems in Logstash 5.3 to use it by its ruby filter plugin? In Ruby i normally could do ./bundle with a new entry in the Gemfile or by type 'gem install GEM' I tried also this … --- ## [Timestamp not getting my log value using date filter logstash](https://discuss.elastic.co/t/timestamp-not-getting-my-log-value-using-date-filter-logstash/83993)
**Author:** [@virendra\_oswal](https://discuss.elastic.co/u/virendra_oswal)\ **Replies:** 3\ **Last updated:** [April 28, 2017, 11:46am UTC](https://discuss.elastic.co/t/timestamp-not-getting-my-log-value-using-date-filter-logstash/83993 "2017-04-28T11:46:26Z")
I have following config to fetch Http call and log date at time call was made, but @timestamp shows when log was send from FileBeat server to logstash one. input { beats { port =\> 5044 } } filter { if \[message\]… --- ## [Processing hashlist/Json attributes with fingerprint plugin](https://discuss.elastic.co/t/processing-hashlist-json-attributes-with-fingerprint-plugin/83363)
**Author:** [@davidS](https://discuss.elastic.co/u/davidS)\ **Replies:** 2\ **Last updated:** [April 28, 2017, 11:20am UTC](https://discuss.elastic.co/t/processing-hashlist-json-attributes-with-fingerprint-plugin/83363 "2017-04-28T11:20:19Z")
Hi all, I try to get the values of a kv-plugin produces Key-Value Hashlist parameter to process it by the fingerprint-plugin. So here are my filter code and the result. filter { kv { source =\> "uriparam" fi… --- ## [Parsing the date and sending to ElasticSearch](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883)
**Author:** [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Replies:** 12\ **Last updated:** [April 28, 2017, 11:06am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883 "2017-04-28T11:06:22Z")
Hello everyone, Great to be here :slight\_smile: For hours am trying to parse a file from filebeat and send it to ES. It works, but the date field that is created in ES is set to string, instead of date. I am using (Ma… --- ## [How to parse particular text string from logs and send to index?](https://discuss.elastic.co/t/how-to-parse-particular-text-string-from-logs-and-send-to-index/83842)
**Author:** [@virendra\_oswal](https://discuss.elastic.co/u/virendra_oswal)\ **Replies:** 11\ **Last updated:** [April 28, 2017, 9:52am UTC](https://discuss.elastic.co/t/how-to-parse-particular-text-string-from-logs-and-send-to-index/83842 "2017-04-28T09:52:12Z")
This is my current configuration in logstash configuration: Logstash Configuration: input { beats { port =\> 5044 } } output { elasticsearch { hosts =\> \["http://localhost:9200/"\] } stdout { codec =\> rubydeb… --- ## [How to ignore old logs](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954)
**Author:** [@Raja1](https://discuss.elastic.co/u/Raja1)\ **Replies:** 6\ **Last updated:** [April 28, 2017, 9:27am UTC](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954 "2017-04-28T09:27:40Z")
Hi All, I have scenario like below. Gatewaylogs --\> Logstash --\> Kafka --\> Logstash --\>Elasticsearch --\> Kibana I am able to push the logs my machine to Kibana. Now the issue is whenever i am doing some changes in … --- ## [Logstash is not connecting with ElasticSearch and I don't find any error](https://discuss.elastic.co/t/logstash-is-not-connecting-with-elasticsearch-and-i-dont-find-any-error/83818)
**Author:** [@Alvaro\_Sanz\_Garrigue](https://discuss.elastic.co/u/Alvaro_Sanz_Garrigue)\ **Replies:** 5\ **Last updated:** [April 28, 2017, 9:01am UTC](https://discuss.elastic.co/t/logstash-is-not-connecting-with-elasticsearch-and-i-dont-find-any-error/83818 "2017-04-28T09:01:46Z")
Hell I'm new with elasticSearch and logstahs and I'm having some problems. I'm Trying to connect Logstash and ES with a simple conf file but nothing happens in ES and I can get the error from the logs. input { fil… --- ## [Please help Importing csv with logstash and create an index in elasticsearch](https://discuss.elastic.co/t/please-help-importing-csv-with-logstash-and-create-an-index-in-elasticsearch/83689)
**Author:** [@Alvaro\_Sanz\_Garrigue](https://discuss.elastic.co/u/Alvaro_Sanz_Garrigue)\ **Replies:** 5\ **Last updated:** [April 28, 2017, 8:49am UTC](https://discuss.elastic.co/t/please-help-importing-csv-with-logstash-and-create-an-index-in-elasticsearch/83689 "2017-04-28T08:49:33Z")
Hello, I'm new with elasticsearch and I'm trying to import csv data with logstash in Windows 10. The configuration file i'm using for logstash is "csv-config.conf": input { file { path =\> "/Users/asanzgarrigues/Des… --- ## [Couldn't find any filter plugin named 'multiline' Error: NameError Name](https://discuss.elastic.co/t/couldnt-find-any-filter-plugin-named-multiline-error-nameerror-name/83843)
**Author:** [@manopmk](https://discuss.elastic.co/u/manopmk)\ **Replies:** 6\ **Last updated:** [April 28, 2017, 7:22am UTC](https://discuss.elastic.co/t/couldnt-find-any-filter-plugin-named-multiline-error-nameerror-name/83843 "2017-04-28T07:22:22Z")
Hi Buddy im using logstash 5.3.1 , im unable to use multiline in logstash of my version, help me out my conf input { file { path =\> \["/Users/Downloads/logs/\*.log"\] start\_position =\> "beginning" } } f… --- ## [Grab specific fields from multiline log](https://discuss.elastic.co/t/grab-specific-fields-from-multiline-log/83765)
**Author:** [@David\_McWhinnie](https://discuss.elastic.co/u/David_McWhinnie)\ **Replies:** 4\ **Last updated:** [April 28, 2017, 6:45am UTC](https://discuss.elastic.co/t/grab-specific-fields-from-multiline-log/83765 "2017-04-28T06:45:20Z")
Looking for some guidance on how to pull some specific pieces of data out of a multiline log. Have been using a logstash filter, but it is taking too long, and I am hoping there is a better way. The logfile: 2017-04-… --- ## [Update @timestamp as per log data](https://discuss.elastic.co/t/update-timestamp-as-per-log-data/83927)
**Author:** [@sumitbiswas](https://discuss.elastic.co/u/sumitbiswas)\ **Replies:** 8\ **Last updated:** [April 28, 2017, 6:20am UTC](https://discuss.elastic.co/t/update-timestamp-as-per-log-data/83927 "2017-04-28T06:20:42Z")
My log looks like below Apr 27 10:35:44.001204 xxxClient(106809) NOTICE: {970020} ABC CLIENT\_MIN\_LATENCY in the last period for host abc4.example.com was 4906 Can you please help me to know how do I update event's @t… --- ## [How to split a fluentd log data?](https://discuss.elastic.co/t/how-to-split-a-fluentd-log-data/83798)
**Author:** [@LmYjQ](https://discuss.elastic.co/u/LmYjQ)\ **Replies:** 5\ **Last updated:** [April 28, 2017, 1:33am UTC](https://discuss.elastic.co/t/how-to-split-a-fluentd-log-data/83798 "2017-04-28T01:33:51Z")
here is one of my fluentd log data 2017-04-23T16:20:31+08:00 cv.product.access.mobile {"race":"album","video\_id":43633036,"ip":"117.177.78.48","cdn":"cdn-web-qn.colorv.cn","act":"update","ad\_type":"AdExchange","agent":"M… --- ## [Http\_poller authentication problem](https://discuss.elastic.co/t/http-poller-authentication-problem/83930)
**Author:** [@Fan\_Dong](https://discuss.elastic.co/u/Fan_Dong)\ **Replies:** 0\ **Last updated:** [April 27, 2017, 7:00pm UTC](https://discuss.elastic.co/t/http-poller-authentication-problem/83930 "2017-04-27T19:00:23Z")
Logstash 5.1 Problem: I cannot seem to get http\_poller authentication working. For example I can use curl -u user:pass my\_rest\_api to get the output but the same always fail with the following logstash conf. The error… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=991) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=993)