# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1083 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1084 --- ## [S3 output plugin timeout error](https://discuss.elastic.co/t/s3-output-plugin-timeout-error/65045)
**Author:** [@greatsid1](https://discuss.elastic.co/u/greatsid1)\ **Replies:** 0\ **Last updated:** [November 4, 2016, 6:44pm UTC](https://discuss.elastic.co/t/s3-output-plugin-timeout-error/65045 "2016-11-04T18:44:11Z")
I am new to Logstash. I am using version 1:5.0.0-1 on a Amazon Linux EC2 instance and trying to upload a file to S3 using the example config: input { file { path =\> "/tmp/log.txt" … --- ## [Replace field with regex group match](https://discuss.elastic.co/t/replace-field-with-regex-group-match/65025)
**Author:** [@irom77](https://discuss.elastic.co/u/irom77)\ **Replies:** 1\ **Last updated:** [November 4, 2016, 6:24pm UTC](https://discuss.elastic.co/t/replace-field-with-regex-group-match/65025 "2016-11-04T18:24:10Z")
Hey, How do I replace field with regex group match , see example below. mutate { gsub =\> \[ "message",",\\\\\\"(.+)\\\\\\",",\ \] } My message… --- ## [Timestamp operation in http\_poller input](https://discuss.elastic.co/t/timestamp-operation-in-http-poller-input/65033)
**Author:** [@eclionast](https://discuss.elastic.co/u/eclionast)\ **Replies:** 0\ **Last updated:** [November 4, 2016, 5:24pm UTC](https://discuss.elastic.co/t/timestamp-operation-in-http-poller-input/65033 "2016-11-04T17:24:11Z")
Hello, We are trying to pull timed data from a server with logstash's http\_poller input plugin with an interval of one hour. The server need a header named "start" which should be the time when the request is sent min… --- ## [Grok Custom pattern](https://discuss.elastic.co/t/grok-custom-pattern/65005)
**Author:** [@brunopsitech](https://discuss.elastic.co/u/brunopsitech)\ **Replies:** 1\ **Last updated:** [November 4, 2016, 3:50pm UTC](https://discuss.elastic.co/t/grok-custom-pattern/65005 "2016-11-04T15:50:45Z")
Hi, i am trying to pass a field and i cant understand . The word can be ( \*DENIED\* or \*EXCEPTION\* ). I try use ${WORD:action} but dont work. Someone can help me with this ? --- ## [LogStash: how many events per second do they handle?](https://discuss.elastic.co/t/logstash-how-many-events-per-second-do-they-handle/65017)
**Author:** [@sen](https://discuss.elastic.co/u/sen)\ **Replies:** 2\ **Last updated:** [November 4, 2016, 3:37pm UTC](https://discuss.elastic.co/t/logstash-how-many-events-per-second-do-they-handle/65017 "2016-11-04T15:37:39Z")
Hello Folks, I already found a lot of documents about Logstash. However, I did not find the information of how many events per second does logstash handle ? Obviously, it depends of the cluster and the size of the e… --- ## [Auditd](https://discuss.elastic.co/t/auditd/64506)
**Author:** [@brunopsitech](https://discuss.elastic.co/u/brunopsitech)\ **Replies:** 5\ **Last updated:** [November 4, 2016, 3:27pm UTC](https://discuss.elastic.co/t/auditd/64506 "2016-11-04T15:27:03Z")
Hi, someone know how to load audit.log of linux in Logstash ? I try use this but not work: input { file { path =\> \["/var/log/audit/audit.log"\] type =\> "auditd" … --- ## [Logstash http output to Kafka-REST-Proxy](https://discuss.elastic.co/t/logstash-http-output-to-kafka-rest-proxy/64930)
**Author:** [@Olivier\_Girardot](https://discuss.elastic.co/u/Olivier_Girardot)\ **Replies:** 1\ **Last updated:** [November 4, 2016, 2:32pm UTC](https://discuss.elastic.co/t/logstash-http-output-to-kafka-rest-proxy/64930 "2016-11-04T14:32:54Z")
Hi everyone, I know there's a kafka output plugin for logstash but I'd like to use Confluent's REST proxy to push it to kafka. But according to the documentation http://docs.confluent.io/3.0.0/kafka-rest/docs/intro.html… --- ## [Shipping CSV file using Filebeat to Logstash and then from logstash to Elasticsearch](https://discuss.elastic.co/t/shipping-csv-file-using-filebeat-to-logstash-and-then-from-logstash-to-elasticsearch/48921)
**Author:** [@gauravkb](https://discuss.elastic.co/u/gauravkb)\ **Replies:** 2\ **Last updated:** [November 4, 2016, 2:01pm UTC](https://discuss.elastic.co/t/shipping-csv-file-using-filebeat-to-logstash-and-then-from-logstash-to-elasticsearch/48921 "2016-11-04T14:01:25Z")
hi , Im trying to ship csv file using filebeat to logstash and then to elasticsearch but nothing is feeding into logstash or elasticsearch This is how my filebeat.yml looks like.. filebeat: # List of prospectors … --- ## [Removing header in log file using logstash](https://discuss.elastic.co/t/removing-header-in-log-file-using-logstash/64675)
**Author:** [@VijayKarthikeyan](https://discuss.elastic.co/u/VijayKarthikeyan)\ **Replies:** 2\ **Last updated:** [November 4, 2016, 12:08pm UTC](https://discuss.elastic.co/t/removing-header-in-log-file-using-logstash/64675 "2016-11-04T12:08:01Z")
Hi, I have a log file which has a header as the first line to represent the log line's type. While parsing with logstash, it happened to parse the header along with the other log lines also but I need to remove the hea… --- ## [Does logstash update the .sincedb file after a log file is completely read or during the reading process?](https://discuss.elastic.co/t/does-logstash-update-the-sincedb-file-after-a-log-file-is-completely-read-or-during-the-reading-process/64997)
**Author:** [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)\ **Replies:** 0\ **Last updated:** [November 4, 2016, 9:43am UTC](https://discuss.elastic.co/t/does-logstash-update-the-sincedb-file-after-a-log-file-is-completely-read-or-during-the-reading-process/64997 "2016-11-04T09:43:02Z")
Does Logstash update the .sincedb file after a log file is read till the end or during the reading process ? For example: Let's say there is directory which is being monitored by Logstash. A file \[say file1.log with ma… --- ## [Process to output is very slow](https://discuss.elastic.co/t/process-to-output-is-very-slow/64993)
**Author:** [@k.irfee](https://discuss.elastic.co/u/k.irfee)\ **Replies:** 0\ **Last updated:** [November 4, 2016, 9:21am UTC](https://discuss.elastic.co/t/process-to-output-is-very-slow/64993 "2016-11-04T09:21:37Z")
HelloWorld text in log file processing is too slow. takes 10 seconds to output. here is the details logstash --path.settings=/etc/logstash -f test1.conf input { file { path =\> \["/root/input.log"\] start\_positio… --- ## [Twitter users info through logstash](https://discuss.elastic.co/t/twitter-users-info-through-logstash/64889)
**Author:** [@Bhavana](https://discuss.elastic.co/u/Bhavana)\ **Replies:** 1\ **Last updated:** [November 4, 2016, 8:50am UTC](https://discuss.elastic.co/t/twitter-users-info-through-logstash/64889 "2016-11-04T08:50:45Z")
i have a few twitter users celebrities names list , can i get the information about them present in twitter using logstash ? --- ## [How to remove configuration option from document \[SOLVED\]](https://discuss.elastic.co/t/how-to-remove-configuration-option-from-document-solved/64832)
**Author:** [@Archer](https://discuss.elastic.co/u/Archer)\ **Replies:** 6\ **Last updated:** [November 4, 2016, 7:44am UTC](https://discuss.elastic.co/t/how-to-remove-configuration-option-from-document-solved/64832 "2016-11-04T07:44:24Z")
\## when i config like this input { redis { host =\> "127.0.0.1" port =\> 6379 key =\> "logstash:test" data\_type =\> "list" codec =\> "json" type =\> "%{\[type\]}" } } output { elasticsearch { … --- ## [Managing complex XML with Logstash](https://discuss.elastic.co/t/managing-complex-xml-with-logstash/64822)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 1\ **Last updated:** [November 4, 2016, 6:28am UTC](https://discuss.elastic.co/t/managing-complex-xml-with-logstash/64822 "2016-11-04T06:28:14Z")
Hello, Just to receive your opinions/suggestions. In case one is facing a XML formatted like this: \ \ \ \ \… --- ## [Parsing XML managing arrays and multilines](https://discuss.elastic.co/t/parsing-xml-managing-arrays-and-multilines/64540)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 2\ **Last updated:** [November 4, 2016, 6:25am UTC](https://discuss.elastic.co/t/parsing-xml-managing-arrays-and-multilines/64540 "2016-11-04T06:25:03Z")
Hi everyone, I'm dealing with a huge XML and I'm trying to proceed step-by-step. For the moment I'm experiencing difficulties with multiline and arrays management by Logstash. This is the simplified XML I'm trying to… --- ## [Logstash grok matching entire filebeat message instead of message inside](https://discuss.elastic.co/t/logstash-grok-matching-entire-filebeat-message-instead-of-message-inside/64976)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 2\ **Last updated:** [November 4, 2016, 5:59am UTC](https://discuss.elastic.co/t/logstash-grok-matching-entire-filebeat-message-instead-of-message-inside/64976 "2016-11-04T05:59:21Z")
I have setup Filebeat -\> Kafka -\> Logstash -\> Elasticseatch system. I have filebeat output like; {"@timestamp":"2016-11-04T05:16:16.108Z","beat":{"hostname":"localhost","name":"localhost","version":"5.0.0"},"fields":… --- ## [Is Prune Supported on 5.0?](https://discuss.elastic.co/t/is-prune-supported-on-5-0/61751)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 7\ **Last updated:** [November 4, 2016, 5:47am UTC](https://discuss.elastic.co/t/is-prune-supported-on-5-0/61751 "2016-11-04T05:47:52Z")
Wondering if Prune filter is still supported on logstash 5.0? Looks like it requires lower version of logstash. Error: In Gemfile: logstash-filter-prune (\>= 0) java depends on logstash (\< 2.0.0, \>= 1.4.0) jav… --- ## [Index records with JSON type fields from Postgres to Elasticsearch using Logstash](https://discuss.elastic.co/t/index-records-with-json-type-fields-from-postgres-to-elasticsearch-using-logstash/64941)
**Author:** [@sumankar123](https://discuss.elastic.co/u/sumankar123)\ **Replies:** 2\ **Last updated:** [November 3, 2016, 8:18pm UTC](https://discuss.elastic.co/t/index-records-with-json-type-fields-from-postgres-to-elasticsearch-using-logstash/64941 "2016-11-03T20:18:37Z")
Hi, I have a SQL file, that generates records from a Postgres db. I would like to index these records in Elasticsearch using Logstash. Here is a sample record from the database. 1, 'Test Dataset 1', 'Insights', 'suma… --- ## [Filtering mistake for spt field in logstash-codec-cef codec](https://discuss.elastic.co/t/filtering-mistake-for-spt-field-in-logstash-codec-cef-codec/64610)
**Author:** [@emegen](https://discuss.elastic.co/u/emegen)\ **Replies:** 2\ **Last updated:** [November 3, 2016, 7:42pm UTC](https://discuss.elastic.co/t/filtering-mistake-for-spt-field-in-logstash-codec-cef-codec/64610 "2016-11-03T19:42:21Z")
i noticed that occasionally the logstash-codec-cef version 3.0.0 will put additional info in the cef\_ext.spt field. i'm not sure if this was an issue with the previous version of the codec. After the port it will add "ex… --- ## [LS pipeline start but no ES index created \[SOLVED\]](https://discuss.elastic.co/t/ls-pipeline-start-but-no-es-index-created-solved/64783)
**Author:** [@mmac](https://discuss.elastic.co/u/mmac)\ **Replies:** 3\ **Last updated:** [November 3, 2016, 6:15pm UTC](https://discuss.elastic.co/t/ls-pipeline-start-but-no-es-index-created-solved/64783 "2016-11-03T18:15:33Z")
Newbie here. The first time I ran through the LS guide, I was able to start a pipeline that created an index. However, I have not been able to start a pipeline that created an index since. I'm thinking that maybe the por… --- ## [Clientip field is not identified by logstash (Solved)](https://discuss.elastic.co/t/clientip-field-is-not-identified-by-logstash-solved/64624)
**Author:** [@Lonck](https://discuss.elastic.co/u/Lonck)\ **Replies:** 11\ **Last updated:** [November 3, 2016, 3:24pm UTC](https://discuss.elastic.co/t/clientip-field-is-not-identified-by-logstash-solved/64624 "2016-11-03T15:24:38Z")
Hi for all I set the logstash filter with following command: filter { if \[type\] == "apache-error" { grok { match =\> \["message", "\\\[%{WORD:dayname} %{WORD:month} %{DATA:day} … --- ## [Unable to install Logstash 5.0 - Need help](https://discuss.elastic.co/t/unable-to-install-logstash-5-0-need-help/64489)
**Author:** [@VIJAYP](https://discuss.elastic.co/u/VIJAYP)\ **Replies:** 7\ **Last updated:** [November 3, 2016, 3:03pm UTC](https://discuss.elastic.co/t/unable-to-install-logstash-5-0-need-help/64489 "2016-11-03T15:03:43Z")
Hello, I'm able to install Elasticsearch 5.0 and Kibana 5.0 successfully. Running on Linux 3.10.0-123.el7.x86\_64. Installing Logstash 5.0 through YUM following the steps provided in https://www.elastic.co/guide/e… --- ## [Unable to output in csv](https://discuss.elastic.co/t/unable-to-output-in-csv/64856)
**Author:** [@steve123](https://discuss.elastic.co/u/steve123)\ **Replies:** 5\ **Last updated:** [November 3, 2016, 12:15pm UTC](https://discuss.elastic.co/t/unable-to-output-in-csv/64856 "2016-11-03T12:15:22Z")
I am trying to output CSV of select fields but I am not getting the correct data even with a simplified config My config: input { generator { add\_field =\> \["foo","bar"\] count =\> 1 } } output { stdout { codec =\> … --- ## [Field reference failure on HTTP output plugin (message option) \[SOLVED\]](https://discuss.elastic.co/t/field-reference-failure-on-http-output-plugin-message-option-solved/64724)
**Author:** [@kosfar](https://discuss.elastic.co/u/kosfar)\ **Replies:** 2\ **Last updated:** [November 3, 2016, 11:54am UTC](https://discuss.elastic.co/t/field-reference-failure-on-http-output-plugin-message-option-solved/64724 "2016-11-03T11:54:52Z")
Hello, I am failing to send events to an HTTP endpoint using field reference on the "message" setting. If I use the same configuration to send a hardcoded base64 encoded string (and not reference to a field), everything … --- ## [Output to Elasticsearch - no living connections to connection pool](https://discuss.elastic.co/t/output-to-elasticsearch-no-living-connections-to-connection-pool/64733)
**Author:** [@ldhaynesnc](https://discuss.elastic.co/u/ldhaynesnc)\ **Replies:** 2\ **Last updated:** [November 3, 2016, 11:52am UTC](https://discuss.elastic.co/t/output-to-elasticsearch-no-living-connections-to-connection-pool/64733 "2016-11-03T11:52:48Z")
This is a brand new ELK 5.0 install and I am a newbie. ES, Logstash, Kibana and Metricbeat (localhost metrics) installed on single server with xpack local authentication enabled and working as expected.. Attempting to … --- ## [Grok fields are not visible in kibana](https://discuss.elastic.co/t/grok-fields-are-not-visible-in-kibana/64808)
**Author:** [@bopa](https://discuss.elastic.co/u/bopa)\ **Replies:** 8\ **Last updated:** [November 3, 2016, 11:40am UTC](https://discuss.elastic.co/t/grok-fields-are-not-visible-in-kibana/64808 "2016-11-03T11:40:35Z")
I have used filebeat to capture IIS logs. and I have used following grok to filter IIS logs. match =\> \["message", "%{TIMESTAMP\_ISO8601:timestamp} %{IPORHOST:serverip} %{WORD:verb} %{NOTSPACE:request} %{NOTSPACE:querystr… --- ## [Combine multiple events](https://discuss.elastic.co/t/combine-multiple-events/64846)
**Author:** [@mostolog](https://discuss.elastic.co/u/mostolog)\ **Replies:** 0\ **Last updated:** [November 3, 2016, 11:19am UTC](https://discuss.elastic.co/t/combine-multiple-events/64846 "2016-11-03T11:19:02Z")
Hi Is there any (not community) supported alternative to combine multiple events/log lines into one, like https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html does? eg: moving a mail from i… --- ## [Old vs new configurations of logstash confuse me](https://discuss.elastic.co/t/old-vs-new-configurations-of-logstash-confuse-me/64819)
**Author:** [@DimitrisFilippou](https://discuss.elastic.co/u/DimitrisFilippou)\ **Replies:** 1\ **Last updated:** [November 3, 2016, 10:21am UTC](https://discuss.elastic.co/t/old-vs-new-configurations-of-logstash-confuse-me/64819 "2016-11-03T10:21:33Z")
Following this tutorial -\> https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-16-04#install-elasticsearch once i installed logstash i created 3 files … --- ## [How to parse @timestamp field using HTTPDERROR\_DATE pattern?](https://discuss.elastic.co/t/how-to-parse-timestamp-field-using-httpderror-date-pattern/64754)
**Author:** [@vitorlui](https://discuss.elastic.co/u/vitorlui)\ **Replies:** 1\ **Last updated:** [November 3, 2016, 6:46am UTC](https://discuss.elastic.co/t/how-to-parse-timestamp-field-using-httpderror-date-pattern/64754 "2016-11-03T06:46:05Z")
I am trying to parse the timestamp from my log but my filter doesn't parse properly: My log file has this pattern: \[Sun Oct 30 17:16:09 2016\] \[TRACE\_HIGH\] \[TEST1\] MessageTest1 \[Sun Oct 30 17:16:10 2016\] \[TRACE\_HIGH\] … --- ## [What is actually .logstash\_jdbc\_last\_run file?](https://discuss.elastic.co/t/what-is-actually-logstash-jdbc-last-run-file/64705)
**Author:** [@Kulasangar\_Gowrisang](https://discuss.elastic.co/u/Kulasangar_Gowrisang)\ **Replies:** 1\ **Last updated:** [November 3, 2016, 6:41am UTC](https://discuss.elastic.co/t/what-is-actually-logstash-jdbc-last-run-file/64705 "2016-11-03T06:41:22Z")
I'm pretty confused what this file is and where does it get the timestamp which it has in it? I'm aware that this get generated every time you run the logstash conf file. Is this file related by any chance with the da… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1082) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1084)