# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=514 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 515 --- ## [Logstash won't ingest files if the line break type is windows (CR LF)](https://discuss.elastic.co/t/logstash-wont-ingest-files-if-the-line-break-type-is-windows-cr-lf/189471)
**Author:** [@Kav](https://discuss.elastic.co/u/Kav)\ **Replies:** 0\ **Last updated:** [July 9, 2019, 7:23am UTC](https://discuss.elastic.co/t/logstash-wont-ingest-files-if-the-line-break-type-is-windows-cr-lf/189471 "2019-07-09T07:23:13Z")
Hi there, I've been working with ELK 7.1 and 7.2 for employee testing purposes and there have been numerous instances where people are getting stuck at: \[INFO \]\[logstash.agent \] Successfully started Logstash … --- ## [.monitoring index search via http\_poller](https://discuss.elastic.co/t/monitoring-index-search-via-http-poller/189469)
**Author:** [@jasony](https://discuss.elastic.co/u/jasony)\ **Replies:** 1\ **Last updated:** [July 9, 2019, 7:15am UTC](https://discuss.elastic.co/t/monitoring-index-search-via-http-poller/189469 "2019-07-09T07:15:42Z")
hello, i tried to search monitoring data with below query. GET \<.monitoring-es-6-{now}\>/\_search { "query": { "match": { "type": "node\_stats" } }, "sort": \[ { "timestamp": { "order… --- ## [String to date type](https://discuss.elastic.co/t/string-to-date-type/189094)
**Author:** [@groowy](https://discuss.elastic.co/u/groowy)\ **Replies:** 2\ **Last updated:** [July 9, 2019, 6:49am UTC](https://discuss.elastic.co/t/string-to-date-type/189094 "2019-07-09T06:49:25Z")
Hi, I want to change my test\_log\_time field type from String to date and when I start Logstash it shows it's okay but Kibana still shows String even if I refresh the page. my config file: input { beats { port =\> "5… --- ## [Cluster block exception: Forbidden](https://discuss.elastic.co/t/cluster-block-exception-forbidden/189384)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 6\ **Last updated:** [July 9, 2019, 5:30am UTC](https://discuss.elastic.co/t/cluster-block-exception-forbidden/189384 "2019-07-09T05:30:16Z")
I keep getting messages like the following and the drop in indexing rate correlates with these messages but the watermark is far from being breached. \[2019-07-08T08:50:29,228\]\[INFO \]\[logstash.outputs.elasticsearch\] retr… --- ## [Plugin with in-memory dictionary for Logstash](https://discuss.elastic.co/t/plugin-with-in-memory-dictionary-for-logstash/188863)
**Author:** [@mdobrin](https://discuss.elastic.co/u/mdobrin)\ **Replies:** 6\ **Last updated:** [July 9, 2019, 5:18am UTC](https://discuss.elastic.co/t/plugin-with-in-memory-dictionary-for-logstash/188863 "2019-07-09T05:18:39Z")
Currently, my Logstash application reads data from a Kafka stream. I would like to add data from a different source which is updated once a week. To do so, I would like to have an external (Ruby or Java?) plugin which … --- ## [Merge the varnish logs into a single line](https://discuss.elastic.co/t/merge-the-varnish-logs-into-a-single-line/186610)
**Author:** [@Yuvraj\_S](https://discuss.elastic.co/u/Yuvraj_S)\ **Replies:** 4\ **Last updated:** [July 9, 2019, 5:08am UTC](https://discuss.elastic.co/t/merge-the-varnish-logs-into-a-single-line/186610 "2019-07-09T05:08:43Z")
hi team, i am using varnish application which gives a lots of logs in the messages. The need is to merge these lines into a single line in the message. Can you please suggest any options of merging those lines. actual … --- ## [Understanding the default behaviour of log rotation of file output plugin](https://discuss.elastic.co/t/understanding-the-default-behaviour-of-log-rotation-of-file-output-plugin/189339)
**Author:** [@Satish\_Mundru](https://discuss.elastic.co/u/Satish_Mundru)\ **Replies:** 3\ **Last updated:** [July 9, 2019, 4:00am UTC](https://discuss.elastic.co/t/understanding-the-default-behaviour-of-log-rotation-of-file-output-plugin/189339 "2019-07-09T04:00:02Z")
Hi, As I understood that the log is rotated per day by using this config 'path' parameter - file { path =\> "/var/logstash/server-%{+YYYY-MM-dd}.log" codec =\> line { format =\> "%{\[message\]}"} } A) I'm trying to make… --- ## [Remove netflow module](https://discuss.elastic.co/t/remove-netflow-module/189341)
**Author:** [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)\ **Replies:** 2\ **Last updated:** [July 9, 2019, 2:47am UTC](https://discuss.elastic.co/t/remove-netflow-module/189341 "2019-07-09T02:47:49Z")
I enabled netflow module via command line and my pipeline.yml configurations are ignored. How do i remove netflow module in logstash 7.2? or is there any way to use pipeline.yml config and modules? --- ## [Viewing hostname instead of ip](https://discuss.elastic.co/t/viewing-hostname-instead-of-ip/189023)
**Author:** [@JohnM](https://discuss.elastic.co/u/JohnM)\ **Replies:** 4\ **Last updated:** [July 9, 2019, 2:14am UTC](https://discuss.elastic.co/t/viewing-hostname-instead-of-ip/189023 "2019-07-09T02:14:39Z")
LOGSTASH 7.2 ES 7.2 I am receiving simple Linux syslog. In logstash node input filter, I have; mutate { add\_field =\> { "syslog\_host" =\> "%{host}" } } dns { reverse =\> \[ "host" \] action =\> "replace" } but in Ki… --- ## [Securing the aws credentials in logstash kinesis pluging](https://discuss.elastic.co/t/securing-the-aws-credentials-in-logstash-kinesis-pluging/189445)
**Author:** [@asrinivasan](https://discuss.elastic.co/u/asrinivasan)\ **Replies:** 0\ **Last updated:** [July 9, 2019, 1:35am UTC](https://discuss.elastic.co/t/securing-the-aws-credentials-in-logstash-kinesis-pluging/189445 "2019-07-09T01:35:18Z")
Currently my configuration is as below the aws credentials is in the /usr/share/logstash/.aws The logstash is running on vm in google cloud and accessing the aws stream . I want to secure the aws credentails and do n… --- ## [How to define patterns\_dir as global in logstash...?](https://discuss.elastic.co/t/how-to-define-patterns-dir-as-global-in-logstash/188812)
**Author:** [@rahulkothanath](https://discuss.elastic.co/u/rahulkothanath)\ **Replies:** 6\ **Last updated:** [July 8, 2019, 6:07pm UTC](https://discuss.elastic.co/t/how-to-define-patterns-dir-as-global-in-logstash/188812 "2019-07-08T18:07:16Z")
Hi, I have a pattens file which contains diffrent patterns as line by line. I am matching each pattern as follows. if \[tran\_type\_cd\] in \["EMD","EMA","EMC","EMF","EMU","EMX"\] { grok {patterns\_dir =\> \["logs… --- ## [Logstash Server Diskspace issue](https://discuss.elastic.co/t/logstash-server-diskspace-issue/189397)
**Author:** [@ankamraok](https://discuss.elastic.co/u/ankamraok)\ **Replies:** 0\ **Last updated:** [July 8, 2019, 5:43pm UTC](https://discuss.elastic.co/t/logstash-server-diskspace-issue/189397 "2019-07-08T17:43:18Z")
Hi, I am using 6.2 Versions of ELK stack and recently started getting an issue with Logstash servers(Windows Server 2012 R2).Servers disk space is getting filled up quickly as so many dll files(ex,jffi623241609555826737… --- ## [Include plugin logstash-input-azureblob by default as logstash-input-azure\_event\_hubs](https://discuss.elastic.co/t/include-plugin-logstash-input-azureblob-by-default-as-logstash-input-azure-event-hubs/189266)
**Author:** [@Marusyk](https://discuss.elastic.co/u/Marusyk)\ **Replies:** 2\ **Last updated:** [July 8, 2019, 2:06pm UTC](https://discuss.elastic.co/t/include-plugin-logstash-input-azureblob-by-default-as-logstash-input-azure-event-hubs/189266 "2019-07-08T14:06:32Z")
I'm going to use this plugin and notice that it is not installed by default in logstash: bin/logstash-plugin list logstash-input-azure\_event\_hubs logstash-input-beats logstash-input-couchdb\_changes logstash-input-d… --- ## [Need some help with ruby split](https://discuss.elastic.co/t/need-some-help-with-ruby-split/189368)
**Author:** [@lostsoul352](https://discuss.elastic.co/u/lostsoul352)\ **Replies:** 1\ **Last updated:** [July 8, 2019, 3:19pm UTC](https://discuss.elastic.co/t/need-some-help-with-ruby-split/189368 "2019-07-08T15:19:19Z")
I have a field called 'path' which contains alinux filesystem path, and I'm trying to use ruby split to get the last and 3rd fields. These will be stored in two new fields called 'log\_filename' and 'deployment' like this… --- ## [Can't assign ilm policy to indexes created by logstash](https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035)
**Author:** [@key](https://discuss.elastic.co/u/key)\ **Replies:** 5\ **Last updated:** [July 8, 2019, 2:40pm UTC](https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035 "2019-07-08T14:40:25Z")
Hi, I'm using Filebeat, Logstash, Elasticsearch and Kibana all together to log data from an app. For all these I use versions 7.2.0. My problem is : I can't assign automatically an ilm policy to the indexes created by… --- ## [Durability and Scalibility in logstash](https://discuss.elastic.co/t/durability-and-scalibility-in-logstash/185097)
**Author:** [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\ **Replies:** 6\ **Last updated:** [July 8, 2019, 2:09pm UTC](https://discuss.elastic.co/t/durability-and-scalibility-in-logstash/185097 "2019-07-08T14:09:18Z")
Hello folks! Please help with these quesitons. I am running a load test which is generating 8-10 gb of real time logs in an hour, what settings will be the best to make sure that no data is lost? What's the diff bw "s… --- ## [How to parse this format of log](https://discuss.elastic.co/t/how-to-parse-this-format-of-log/188865)
**Author:** [@deepanshu\_goel](https://discuss.elastic.co/u/deepanshu_goel)\ **Replies:** 5\ **Last updated:** [July 8, 2019, 1:33pm UTC](https://discuss.elastic.co/t/how-to-parse-this-format-of-log/188865 "2019-07-08T13:33:54Z")
LOG WHICH I AM TRYIING TO PARSE:- 2019-07-03 17:04:58.562 DELETE accept UDP 10.133.189.203:56897 -\> 233.79.64.194:654 678.133.200.279:56131 -\> 23.64.33.898:555 0 0 The grok pattern whic… --- ## [Completly lost with how to parse date](https://discuss.elastic.co/t/completly-lost-with-how-to-parse-date/189336)
**Author:** [@Elastoc](https://discuss.elastic.co/u/Elastoc)\ **Replies:** 1\ **Last updated:** [July 8, 2019, 12:24pm UTC](https://discuss.elastic.co/t/completly-lost-with-how-to-parse-date/189336 "2019-07-08T12:24:23Z")
Hello, I'm trying for several hours to parse a logfile and i'm stuck with parsing dates. I have a very hard time understanding how it works. Is there an example somewhere with unusual timestamps? My log is like this : … --- ## [There is already another instance using the configured data directory](https://discuss.elastic.co/t/there-is-already-another-instance-using-the-configured-data-directory/189333)
**Author:** [@Mohammad\_hossein\_Taj](https://discuss.elastic.co/u/Mohammad_hossein_Taj)\ **Replies:** 0\ **Last updated:** [July 8, 2019, 11:24am UTC](https://discuss.elastic.co/t/there-is-already-another-instance-using-the-configured-data-directory/189333 "2019-07-08T11:24:00Z")
hi I'm trying to get logs that filebeat harvest by logstash. but when I try to run logstash with this command bin/logstash -f /etc/logstash/beatsToElasticSearchPipeline.conf --config.reload.automatic I got this: hread… --- ## [Map the lookup fields from the CSV to precreated index](https://discuss.elastic.co/t/map-the-lookup-fields-from-the-csv-to-precreated-index/189079)
**Author:** [@shrikantgulia](https://discuss.elastic.co/u/shrikantgulia)\ **Replies:** 3\ **Last updated:** [July 8, 2019, 10:37am UTC](https://discuss.elastic.co/t/map-the-lookup-fields-from-the-csv-to-precreated-index/189079 "2019-07-08T10:37:43Z")
Hello, I need to map the fields existing in my index to a lookup file. For eg: In index i am having field=value and in a csv i have value=name , so i need to map field=name. How can this be achieved. So basically when… --- ## [Docker - how to switch JRE version from 11 to 8?](https://discuss.elastic.co/t/docker-how-to-switch-jre-version-from-11-to-8/189303)
**Author:** [@ben.wolfpack-digital](https://discuss.elastic.co/u/ben.wolfpack-digital)\ **Replies:** 0\ **Last updated:** [July 8, 2019, 7:48am UTC](https://discuss.elastic.co/t/docker-how-to-switch-jre-version-from-11-to-8/189303 "2019-07-08T07:48:44Z")
Hello, Is it possible to easily switch the java/JRE version of Docker image for Logstash to 8? My other option will be to inherit from official image and downgrade JRE to version 8. Thanks, Ben C. --- ## [Parsing xml message inside json file](https://discuss.elastic.co/t/parsing-xml-message-inside-json-file/189100)
**Author:** [@saisimo02](https://discuss.elastic.co/u/saisimo02)\ **Replies:** 2\ **Last updated:** [July 8, 2019, 7:29am UTC](https://discuss.elastic.co/t/parsing-xml-message-inside-json-file/189100 "2019-07-08T07:29:32Z")
Hello, I am getting an error while filtering a json file. Previously, I was using filebeat to send data directly to logstash, but now I am using kafka between filebeat and logstash. So the output is not the same: Previ… --- ## [Logstash is not working for java 11](https://discuss.elastic.co/t/logstash-is-not-working-for-java-11/189062)
**Author:** [@dhiraj07](https://discuss.elastic.co/u/dhiraj07)\ **Replies:** 1\ **Last updated:** [July 8, 2019, 7:28am UTC](https://discuss.elastic.co/t/logstash-is-not-working-for-java-11/189062 "2019-07-08T07:28:27Z")
Hi team, I just installed ELK stack. But my Logstash is not working. I am using Logstash version:7.2.0 Java version: 11 I just run the command: logstash -e 'input { stdin { } } output { stdout {} }' then Got the E… --- ## [Error translation event\_id](https://discuss.elastic.co/t/error-translation-event-id/189085)
**Author:** [@NauGui](https://discuss.elastic.co/u/NauGui)\ **Replies:** 2\ **Last updated:** [July 8, 2019, 6:34am UTC](https://discuss.elastic.co/t/error-translation-event-id/189085 "2019-07-08T06:34:28Z")
Hi, I have a problem with one of my filter for event\_id. This one work perfectly : filter { mutate { convert =\> { "event\_id" =\> "string" } } translate { field =\> "event\_id"… --- ## [Logstash filter for FTP logs](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896)
**Author:** [@akshay\_singh2](https://discuss.elastic.co/u/akshay_singh2)\ **Replies:** 6\ **Last updated:** [July 8, 2019, 5:08am UTC](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896 "2019-07-08T05:08:21Z")
Hello Team, I need to put filters for logstash for querying data coming from FTP servers. I havent worked on filters much, just have rough idea, so i created one for some below test logs: Thu Jul 4 06:01:45 2019 \[pid … --- ## [Mutate Replace not working properly in ELK STACK 7.2 Looks there a bunch of bug of this verion](https://discuss.elastic.co/t/mutate-replace-not-working-properly-in-elk-stack-7-2-looks-there-a-bunch-of-bug-of-this-verion/188995)
**Author:** [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\ **Replies:** 2\ **Last updated:** [July 8, 2019, 3:37am UTC](https://discuss.elastic.co/t/mutate-replace-not-working-properly-in-elk-stack-7-2-looks-there-a-bunch-of-bug-of-this-verion/188995 "2019-07-08T03:37:12Z")
Hi All, Tried these in my logstash filtering. filter { if \[geoip.region\_name\] == "Guangxi" { mutate { replace =\> \[ "geoip.region\_name", "Guangxi Zhuang Autonomous Region" \] } } if \[geoip.… --- ## [Automatically creates fields by \](https://discuss.elastic.co/t/automatically-creates-fields-by-tags/189278)
**Author:** [@piketrcechillas](https://discuss.elastic.co/u/piketrcechillas)\ **Replies:** 0\ **Last updated:** [July 8, 2019, 3:31am UTC](https://discuss.elastic.co/t/automatically-creates-fields-by-tags/189278 "2019-07-08T03:31:28Z")
Hi everyone, I have a log file generated by LogWriter and it is separater by tags, loike this @@@20190708093430179@@@AppName@@@getListNotify@@@\#@n@##@t@#\foo\#@n@##@t@#\oof\ **Author:** [@markov](https://discuss.elastic.co/u/markov)\ **Replies:** 2\ **Last updated:** [July 7, 2019, 11:13pm UTC](https://discuss.elastic.co/t/logs-are-not-passing-into-the-if-and-else-if-condition/188891 "2019-07-07T23:13:41Z") my logs are not passing into the if and else if condition, when the if is removed it works fine. logstash 7.1.1 \> filter { \> if \[source\] =~ "gid" { \> grok { \> match =\> {"message" =\> "%{TIME… --- ## [Logstash 7.2 Keystore Warnings](https://discuss.elastic.co/t/logstash-7-2-keystore-warnings/189264)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 0\ **Last updated:** [July 7, 2019, 8:40pm UTC](https://discuss.elastic.co/t/logstash-7-2-keystore-warnings/189264 "2019-07-07T20:40:44Z")
Going through the process of securing Logstash and run into some warnings. Are these going to cause problems? C:\\ElasticStack\\logstash\\7.2\\bin\>set LOGSTASH\_KEYSTORE\_PASS=pass1 C:\\ElasticStack\\logstash\\7.2\\bin\>logstash… --- ## [Comparing fields from input](https://discuss.elastic.co/t/comparing-fields-from-input/189246)
**Author:** [@Uchiha\_Kakashi](https://discuss.elastic.co/u/Uchiha_Kakashi)\ **Replies:** 2\ **Last updated:** [July 7, 2019, 2:35pm UTC](https://discuss.elastic.co/t/comparing-fields-from-input/189246 "2019-07-07T14:35:06Z")
I have brought a json document from the index "sequence". I want to access one of the fields "value1" in that doc and compare. I am getting errors. My config file is --\> \< input { elasticsearch { hosts =\> "localho… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=513) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=515)