# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=744 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 745 --- ## [How To Make Logstash Read Entire File Every Time Which is from File Beat?](https://discuss.elastic.co/t/how-to-make-logstash-read-entire-file-every-time-which-is-from-file-beat/135187)
**Author:** [@Yenike\_Raghu\_Ram](https://discuss.elastic.co/u/Yenike_Raghu_Ram)\ **Replies:** 7\ **Last updated:** [June 9, 2018, 5:34pm UTC](https://discuss.elastic.co/t/how-to-make-logstash-read-entire-file-every-time-which-is-from-file-beat/135187 "2018-06-09T17:34:52Z")
input { beats { port =\> 5048 } } output { stdout { codec =\> rubydebug } file { codec =\> line { format =\> "%{\[message\]}" } path =\> "C:/Program Files/logstash-6.2.3/testPractice/LogsCollected/DebugLog09May2018/… --- ## [Logstash Redshift JDBC input plugin drivers](https://discuss.elastic.co/t/logstash-redshift-jdbc-input-plugin-drivers/134999)
**Author:** [@bharath\_naidu](https://discuss.elastic.co/u/bharath_naidu)\ **Replies:** 2\ **Last updated:** [June 9, 2018, 4:29pm UTC](https://discuss.elastic.co/t/logstash-redshift-jdbc-input-plugin-drivers/134999 "2018-06-09T16:29:40Z")
Hi All, Can any one help me with setting up of Logstash JDBC drivers for Redshift DB? Use case: I need to pipeline the data from AWS Redshift database to Logstash-elasticsearch. Any help on this will be appreciated. R… --- ## [Logstash appending new lines even if there is change only in existing line](https://discuss.elastic.co/t/logstash-appending-new-lines-even-if-there-is-change-only-in-existing-line/135220)
**Author:** [@Yenike\_Raghu\_Ram](https://discuss.elastic.co/u/Yenike_Raghu_Ram)\ **Replies:** 0\ **Last updated:** [June 9, 2018, 10:49am UTC](https://discuss.elastic.co/t/logstash-appending-new-lines-even-if-there-is-change-only-in-existing-line/135220 "2018-06-09T10:49:54Z")
Say like in original file I ve 2 lines a b We've modified one line say 'a' now output file in logstash is like a -- original data b a -- modified data b I am expecting it to be like log file to have only two lin… --- ## [Trying to parse this datefield](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331)
**Author:** [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Replies:** 37\ **Last updated:** [June 9, 2018, 9:30am UTC](https://discuss.elastic.co/t/trying-to-parse-this-datefield/134331 "2018-06-09T09:30:23Z")
I have a CSV file with a date field as follows: May 31, 2018 09:35:45.979371597 EEST I am trying to parse it with logstash without any success (the field is still a string) Here's my logstash filter: filter { if \[t… --- ## [Logstash.yml to collect IIS logs with filebeat error](https://discuss.elastic.co/t/logstash-yml-to-collect-iis-logs-with-filebeat-error/135066)
**Author:** [@alexserd](https://discuss.elastic.co/u/alexserd)\ **Replies:** 3\ **Last updated:** [June 9, 2018, 8:23am UTC](https://discuss.elastic.co/t/logstash-yml-to-collect-iis-logs-with-filebeat-error/135066 "2018-06-09T08:23:51Z")
Getting error trying to start logstash with following config file .\\logstash.bat -f logstash.yml LOGSTASH.YML input { beats { port =\> 5044 type =\> 'iis' } } filter { Ignore the comments that IIS will add to th… --- ## [Issue while parsing date](https://discuss.elastic.co/t/issue-while-parsing-date/135179)
**Author:** [@Vikrant\_Aggarwal](https://discuss.elastic.co/u/Vikrant_Aggarwal)\ **Replies:** 4\ **Last updated:** [June 9, 2018, 3:47am UTC](https://discuss.elastic.co/t/issue-while-parsing-date/135179 "2018-06-09T03:47:46Z")
Created following logstash pipeline file. but I am not able to make the date filter working hence commented out in the following output. input { file { path =\> "/Users/viaggarw/Documents/ELK/logstash-6.2.4/bplogs… --- ## [Logstash reloads api not workig](https://discuss.elastic.co/t/logstash-reloads-api-not-workig/135193)
**Author:** [@Sebastian\_Herrera](https://discuss.elastic.co/u/Sebastian_Herrera)\ **Replies:** 0\ **Last updated:** [June 8, 2018, 7:43pm UTC](https://discuss.elastic.co/t/logstash-reloads-api-not-workig/135193 "2018-06-08T19:43:57Z")
hello i am using the logstasha api in order to know if the reloads that i did are good or not, the issue is that when i excecute curl -XGET 'localhost:9600/\_node/stats/pipelines?pretty' it answer the failures but when … --- ## [Converting time to milliseconds](https://discuss.elastic.co/t/converting-time-to-milliseconds/135183)
**Author:** [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\ **Replies:** 2\ **Last updated:** [June 8, 2018, 6:05pm UTC](https://discuss.elastic.co/t/converting-time-to-milliseconds/135183 "2018-06-08T18:05:52Z")
converting 00:00:00.013 in to milliseconds I used to dissect to get values and having trouble calculating the values dissect { mapping =\> { "ExecuteTime" =\> %{hour}:%{minute}:%{second}.%{millisecond}" } } if \[hour… --- ## [Logstash connection to Elastic search](https://discuss.elastic.co/t/logstash-connection-to-elastic-search/134844)
**Author:** [@ChrisW](https://discuss.elastic.co/u/ChrisW)\ **Replies:** 3\ **Last updated:** [June 8, 2018, 3:06pm UTC](https://discuss.elastic.co/t/logstash-connection-to-elastic-search/134844 "2018-06-08T15:06:24Z")
I installed logstash, elastic search and kibana from debs on my system. Version 6.2.4 I then installed the x-pack plugin on all components. I used pretty much the defaults (exception being labels of cluster and such) … --- ## [GeoIP LookUp failure](https://discuss.elastic.co/t/geoip-lookup-failure/134936)
**Author:** [@asajj](https://discuss.elastic.co/u/asajj)\ **Replies:** 5\ **Last updated:** [June 8, 2018, 3:03pm UTC](https://discuss.elastic.co/t/geoip-lookup-failure/134936 "2018-06-08T15:03:59Z")
Hi Everyone, i'm playing a bit with the ELK-Stack and i have a problem with the GeoIP-Filte of Logstash. Via Winlogbeat I send EventLogs (more exact Network Sysmon Events). If a connection is established in generates a… --- ## [How to add a character to a particular field value?](https://discuss.elastic.co/t/how-to-add-a-character-to-a-particular-field-value/135173)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 1\ **Last updated:** [June 8, 2018, 3:03pm UTC](https://discuss.elastic.co/t/how-to-add-a-character-to-a-particular-field-value/135173 "2018-06-08T15:03:36Z")
Hi, I am using grok to extract fields. Once the field is extracted, I want to add } at the end. Please let me know how it can be done? Thanks --- ## [Logstash config to get multiline messages](https://discuss.elastic.co/t/logstash-config-to-get-multiline-messages/135172)
**Author:** [@dr01](https://discuss.elastic.co/u/dr01)\ **Replies:** 2\ **Last updated:** [June 8, 2018, 2:57pm UTC](https://discuss.elastic.co/t/logstash-config-to-get-multiline-messages/135172 "2018-06-08T14:57:20Z")
I need to parse a multiline event and, as suggested in the docs, I'm leaving to Filebeat the handling of the multiline: filebeat.prospectors: - type: log paths: - /path/to/logfiletoparse.log multiline: patte… --- ## [Parsing error in logstash logs](https://discuss.elastic.co/t/parsing-error-in-logstash-logs/134747)
**Author:** [@cppatel](https://discuss.elastic.co/u/cppatel)\ **Replies:** 3\ **Last updated:** [June 8, 2018, 1:22pm UTC](https://discuss.elastic.co/t/parsing-error-in-logstash-logs/134747 "2018-06-08T13:22:33Z")
logstash gives following error in it's logs: \[2018-06-06T10:55:32,611\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"abcd-2018.06.… --- ## [Replicating inputs to multiple InfluxDB outputs](https://discuss.elastic.co/t/replicating-inputs-to-multiple-influxdb-outputs/135123)
**Author:** [@tbm](https://discuss.elastic.co/u/tbm)\ **Replies:** 2\ **Last updated:** [June 8, 2018, 1:22pm UTC](https://discuss.elastic.co/t/replicating-inputs-to-multiple-influxdb-outputs/135123 "2018-06-08T13:22:07Z")
Hi, We collect metrics that are sent to Logstash and then stored to an InfluxDB. We have a case where we need to be able to send the data to an additional InfluxDB in a different data center. What's the best way to achi… --- ## [Installed and configured ElasticSearch (Multi Node) Cluster on RHEL 6.9](https://discuss.elastic.co/t/installed-and-configured-elasticsearch-multi-node-cluster-on-rhel-6-9/135149)
**Author:** [@amit.patra](https://discuss.elastic.co/u/amit.patra)\ **Replies:** 1\ **Last updated:** [June 8, 2018, 1:18pm UTC](https://discuss.elastic.co/t/installed-and-configured-elasticsearch-multi-node-cluster-on-rhel-6-9/135149 "2018-06-08T13:18:02Z")
Hi , I have installed and configured Elastic-search (Multi Node) Cluster on RHEL 6.9. Master : 192.168.2.79 Data-Node1 : 192.168.2.80 Data-Node2 : 192.168.2.80 Now I want to install logstash and kibana as well. On … --- ## [Concat query is not working in logstash](https://discuss.elastic.co/t/concat-query-is-not-working-in-logstash/135004)
**Author:** [@Deepak\_Dandotiya](https://discuss.elastic.co/u/Deepak_Dandotiya)\ **Replies:** 4\ **Last updated:** [June 8, 2018, 12:58pm UTC](https://discuss.elastic.co/t/concat-query-is-not-working-in-logstash/135004 "2018-06-08T12:58:11Z")
Hi, I am trying to combine 2 fields first name & last name and trying to populate as single field but getting some error. Below is my Logstash config file emp-index-logstash.conf:- input { jdbc { jdbc\_connection\_… --- ## [Extract a timestamp between square brackets \[SOLVED\]](https://discuss.elastic.co/t/extract-a-timestamp-between-square-brackets-solved/135014)
**Author:** [@dr01](https://discuss.elastic.co/u/dr01)\ **Replies:** 3\ **Last updated:** [June 8, 2018, 12:55pm UTC](https://discuss.elastic.co/t/extract-a-timestamp-between-square-brackets-solved/135014 "2018-06-08T12:55:52Z")
I need to extract the timestamp from this line: \[Tue, 07 Jun 2018 15:53:29 +0200\] codedeploy-agent started 1)) I've tried the following filter: mutate { copy =\> { "message" =\> "ts" } gsub =\> \[ "ts", "\\\[", "" \] … --- ## [Logstash as a service in Windows](https://discuss.elastic.co/t/logstash-as-a-service-in-windows/135090)
**Author:** [@Maekee](https://discuss.elastic.co/u/Maekee)\ **Replies:** 1\ **Last updated:** [June 8, 2018, 12:15pm UTC](https://discuss.elastic.co/t/logstash-as-a-service-in-windows/135090 "2018-06-08T12:15:20Z")
Hello everybody, This is a question for all you running Logstash on Windows Servers. What software/procedure do you use to create/run Logstash as a service? Whats you experience with different ways? Need a stable way d… --- ## [Logstash: TCP output with tls is not working](https://discuss.elastic.co/t/logstash-tcp-output-with-tls-is-not-working/134193)
**Author:** [@mahibabar](https://discuss.elastic.co/u/mahibabar)\ **Replies:** 4\ **Last updated:** [June 8, 2018, 11:54am UTC](https://discuss.elastic.co/t/logstash-tcp-output-with-tls-is-not-working/134193 "2018-06-08T11:54:31Z")
Hello, I am trying to send events over secure TCP from logstash but facing issues with ssl settings: Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\ **Author:** [@Severin\_N](https://discuss.elastic.co/u/Severin_N)\ **Replies:** 1\ **Last updated:** [June 8, 2018, 9:03am UTC](https://discuss.elastic.co/t/forwarding-error-fatal-events-into-oracle-database/134533 "2018-06-08T09:03:33Z") Hello there I'm using the ELKX stack with filebeat. I was using the E-Mail-output-plugin until i once generated 90000 e-mails in 10 minutes and my e-mail provider contacted me that he will blacklist my smtp server if t… --- ## ["The shutdown process appears to be stalled due to busy or blocked plugins"](https://discuss.elastic.co/t/the-shutdown-process-appears-to-be-stalled-due-to-busy-or-blocked-plugins/135000)
**Author:** [@dr01](https://discuss.elastic.co/u/dr01)\ **Replies:** 2\ **Last updated:** [June 8, 2018, 8:48am UTC](https://discuss.elastic.co/t/the-shutdown-process-appears-to-be-stalled-due-to-busy-or-blocked-plugins/135000 "2018-06-08T08:48:11Z")
I've set up a testing VM with a full ELK stack. Often I see this error in the Logstash logs: \[2018-06-07T15:34:34,248\]\[ERROR\]\[logstash.shutdownwatcher \] The shutdown process appears to be stalled due to busy or blocked… --- ## [Urgent: reparsing log files with Logstash?](https://discuss.elastic.co/t/urgent-reparsing-log-files-with-logstash/135109)
**Author:** [@Michel99\_7](https://discuss.elastic.co/u/Michel99_7)\ **Replies:** 0\ **Last updated:** [June 8, 2018, 8:12am UTC](https://discuss.elastic.co/t/urgent-reparsing-log-files-with-logstash/135109 "2018-06-08T08:12:56Z")
Dear all, Here is the context OS: Linux Kibana: v6.1.1 Elasticsearch & Logstash: v6.1.1 Browser: Google Chrome Version 64.0.3282.186 Context: we are indexing data by using Logstash and the file input plugin for .lo… --- ## [Service seem to hang](https://discuss.elastic.co/t/service-seem-to-hang/134755)
**Author:** [@yks](https://discuss.elastic.co/u/yks)\ **Replies:** 4\ **Last updated:** [June 8, 2018, 6:52am UTC](https://discuss.elastic.co/t/service-seem-to-hang/134755 "2018-06-08T06:52:46Z")
Hi, I am using the guide here : Parsing Logs with Logstash | Logstash Reference \[8.11\] | Elastic, The service seem to hang at here after i executed the following command logstash -f first-pipeline.conf --config.relo… --- ## [Aggregate different inputs](https://discuss.elastic.co/t/aggregate-different-inputs/131509)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 1\ **Last updated:** [June 8, 2018, 6:38am UTC](https://discuss.elastic.co/t/aggregate-different-inputs/131509 "2018-06-08T06:38:21Z")
It's possible to aggregate different input files content? Example: file1.txt May 1 15:57:11 myhost sshd\[34427\]: pam\_unix(sshd:session): session opened for user hendrenj by (uid=0) May 1 16:16:13 myhost sshd\[34427\]:… --- ## [Using Elasticsearch filter plugin to add information to a index](https://discuss.elastic.co/t/using-elasticsearch-filter-plugin-to-add-information-to-a-index/134314)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 1\ **Last updated:** [June 8, 2018, 6:25am UTC](https://discuss.elastic.co/t/using-elasticsearch-filter-plugin-to-add-information-to-a-index/134314 "2018-06-08T06:25:15Z")
Hi, I'm trying to use elasticsearch filter plugin to add some information to an index which I'm populating ingesting a CSV. The CSV is providing this kind of information: id; value; date And I want to add some other s… --- ## [Index create late date](https://discuss.elastic.co/t/index-create-late-date/135072)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 1\ **Last updated:** [June 8, 2018, 5:42am UTC](https://discuss.elastic.co/t/index-create-late-date/135072 "2018-06-08T05:42:45Z")
Hi there Could you someone figure out this? Let me say my scenario first. I stream app logs using tcp input , then use grok filter , after that use ship to amazon Elasticsearch domain. I stream two app logs actually. … --- ## [Logstash qusetion about template change ?need restart?](https://discuss.elastic.co/t/logstash-qusetion-about-template-change-need-restart/135073)
**Author:** [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Replies:** 1\ **Last updated:** [June 8, 2018, 5:41am UTC](https://discuss.elastic.co/t/logstash-qusetion-about-template-change-need-restart/135073 "2018-06-08T05:41:23Z")
logstash 6.2.3 i have a logstash input data to es with template? as output { stdout { codec =\> dots {} } elasticsearch { hosts =\> \["http://xxx:9201/","http://xxx:9201","http://xxx:9201"\] index =\> "apache… --- ## [Learning Grok](https://discuss.elastic.co/t/learning-grok/134773)
**Author:** [@Jukocross](https://discuss.elastic.co/u/Jukocross)\ **Replies:** 5\ **Last updated:** [June 8, 2018, 2:49am UTC](https://discuss.elastic.co/t/learning-grok/134773 "2018-06-08T02:49:35Z")
Hi, i'm learning grok and trying out to craft my own filter. Below is the example which i have tried using https://grokdebug.herokuapp.com to debug. However, i still received grokparsefailure when output to elasticsear… --- ## [Logstash stops processing when using persistent queues](https://discuss.elastic.co/t/logstash-stops-processing-when-using-persistent-queues/134712)
**Author:** [@ceekay](https://discuss.elastic.co/u/ceekay)\ **Replies:** 1\ **Last updated:** [June 7, 2018, 10:47pm UTC](https://discuss.elastic.co/t/logstash-stops-processing-when-using-persistent-queues/134712 "2018-06-07T22:47:20Z")
I've updated logstash.yml to use persistent queues for a bit of extra redundancy, and have run into a problem where individual nodes will just stop processing messages after about 30+ minutes a random amount of time. Ka… --- ## [Dead Letter Queue - Issue](https://discuss.elastic.co/t/dead-letter-queue-issue/134118)
**Author:** [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Replies:** 5\ **Last updated:** [June 7, 2018, 9:24pm UTC](https://discuss.elastic.co/t/dead-letter-queue-issue/134118 "2018-06-07T21:24:19Z")
Environment: Logstash 6.2.4 in docker, multi-pipeline mode Input-Redis to Output-Logstash I have enabled dead letter queue for my pipelines and do see mapping failure data going against appropriate pipelines in dead-l… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=743) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=745)