# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=540 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 541 --- ## [Creating indices with ILM](https://discuss.elastic.co/t/creating-indices-with-ilm/182985)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 0\ **Last updated:** [May 28, 2019, 5:27am UTC](https://discuss.elastic.co/t/creating-indices-with-ilm/182985 "2019-05-28T05:27:13Z")
Is there anyway through which I can directly add ILM to my indices during creation???If i add these lines in my "abc.conf" will it be applied to all new indices created ilm\_enabled =\> "true" ilm\_policy =\> "default-poli… --- ## [Hive, elasticsearch, kibana](https://discuss.elastic.co/t/hive-elasticsearch-kibana/182984)
**Author:** [@krishnakekan](https://discuss.elastic.co/u/krishnakekan)\ **Replies:** 0\ **Last updated:** [May 28, 2019, 5:18am UTC](https://discuss.elastic.co/t/hive-elasticsearch-kibana/182984 "2019-05-28T05:18:26Z")
Hi, I have loaded data in Elasticsearch using hive table. I want to plot route i have two parameters latitude and longitude i am not able to convert that into geo-point for visualization. How can i be able to make geop… --- ## [Syslog input going to "%{\[@metadata\]\[beat\]}-2019.05.28"](https://discuss.elastic.co/t/syslog-input-going-to-metadata-beat-2019-05-28/182980)
**Author:** [@tedder](https://discuss.elastic.co/u/tedder)\ **Replies:** 0\ **Last updated:** [May 28, 2019, 4:17am UTC](https://discuss.elastic.co/t/syslog-input-going-to-metadata-beat-2019-05-28/182980 "2019-05-28T04:17:00Z")
I don't know what information is needed, but I've started sending syslog input into logstash (and therefore into ELK). My input simply specifies a port, the filter contains the following: filter { if \[type\] == "syslo… --- ## [Parse json format logagregator logs in logstash](https://discuss.elastic.co/t/parse-json-format-logagregator-logs-in-logstash/182952)
**Author:** [@Aayush\_Chaudhary](https://discuss.elastic.co/u/Aayush_Chaudhary)\ **Replies:** 3\ **Last updated:** [May 27, 2019, 7:46pm UTC](https://discuss.elastic.co/t/parse-json-format-logagregator-logs-in-logstash/182952 "2019-05-27T19:46:02Z")
\<14\>1 2019-05-27T22:29:00.14+0530 \[APP/PROC/WEB/0\] OUT ,{"timeMillis":1558981507774,"thread":"main","level":"INFO","loggerName":"com.philips.hsdp.log.drain.ref.config.LogDrainerRefApplication","message":"Started LogDrai… --- ## [Error in logstash reading a csv file](https://discuss.elastic.co/t/error-in-logstash-reading-a-csv-file/182928)
**Author:** [@decodeit](https://discuss.elastic.co/u/decodeit)\ **Replies:** 3\ **Last updated:** [May 27, 2019, 4:18pm UTC](https://discuss.elastic.co/t/error-in-logstash-reading-a-csv-file/182928 "2019-05-27T16:18:21Z")
I am trying to read a csv file with logstash . Here is my config file: input { file{ path =\> \["/home/abhinavkumar.gurung/Applications/csv/devian/data/autos.csv"\] type =\> "cars" start\_position =\> "beginning" sinced… --- ## [Logstash notification about output availability](https://discuss.elastic.co/t/logstash-notification-about-output-availability/182904)
**Author:** [@vladys](https://discuss.elastic.co/u/vladys)\ **Replies:** 1\ **Last updated:** [May 27, 2019, 2:05pm UTC](https://discuss.elastic.co/t/logstash-notification-about-output-availability/182904 "2019-05-27T14:05:40Z")
Hello, I've got following problem. we've got Logstash instance running on server which sends output to remote server at the customer. Customer insisted on Splunk so we are sending parsed events from Logstash to Splunk. … --- ## [Parse log4j schema error](https://discuss.elastic.co/t/parse-log4j-schema-error/182914)
**Author:** [@zegdene](https://discuss.elastic.co/u/zegdene)\ **Replies:** 0\ **Last updated:** [May 27, 2019, 12:58pm UTC](https://discuss.elastic.co/t/parse-log4j-schema-error/182914 "2019-05-27T12:58:31Z")
Hello, I would like parse lo4j file and the result of schema is: \> { "logger\_name" =\> "org.apache.spark.SparkContext", "path" =\> "org.apache.spark.SparkContext", "class" =\> "?", "priori… --- ## [How can i add two fields which contains string value and store it in a new field](https://discuss.elastic.co/t/how-can-i-add-two-fields-which-contains-string-value-and-store-it-in-a-new-field/182872)
**Author:** [@Vishnu\_mk](https://discuss.elastic.co/u/Vishnu_mk)\ **Replies:** 3\ **Last updated:** [May 27, 2019, 12:20pm UTC](https://discuss.elastic.co/t/how-can-i-add-two-fields-which-contains-string-value-and-store-it-in-a-new-field/182872 "2019-05-27T12:20:24Z")
For ex: name: Vishnu surname\_filed : mk I want a to combine this to filed and in a new field Name: Vishnu mk --- ## [How to get subfield of cloudtrail logs](https://discuss.elastic.co/t/how-to-get-subfield-of-cloudtrail-logs/182825)
**Author:** [@ajaysheoran2323](https://discuss.elastic.co/u/ajaysheoran2323)\ **Replies:** 1\ **Last updated:** [May 27, 2019, 11:59am UTC](https://discuss.elastic.co/t/how-to-get-subfield-of-cloudtrail-logs/182825 "2019-05-27T11:59:01Z")
Hi People, I am using logstash to move logs from cloudtrail to elasticsearch via logstash. Below is the conf file: input { s3 { bucket =\> "test-cloudtrail" prefix =\> "AWSLogs/" region =\> "us-west-2" access\_key… --- ## [ニューヨークのタクシーデータが取り込めない](https://discuss.elastic.co/t/topic/182809)
**Author:** [@ode.raita](https://discuss.elastic.co/u/ode.raita)\ **Replies:** 2\ **Last updated:** [May 27, 2019, 11:41am UTC](https://discuss.elastic.co/t/topic/182809 "2019-05-27T11:41:27Z")
上記の設定を試していますが、以下のエラーが出ます。 なお、Elasticsearch/logstach/kibanaは6.8 X-packのライセンスはアクティブです \[root@vmi247815 data\]# /usr/share/logstash/bin/logstash -f nyc-taxi-yellow-translate-logstash3.conf WARNING: Could not find logstas… --- ## [Use current time / system time in logstash file output?](https://discuss.elastic.co/t/use-current-time-system-time-in-logstash-file-output/182644)
**Author:** [@bjosve](https://discuss.elastic.co/u/bjosve)\ **Replies:** 7\ **Last updated:** [May 27, 2019, 10:10am UTC](https://discuss.elastic.co/t/use-current-time-system-time-in-logstash-file-output/182644 "2019-05-27T10:10:35Z")
Hi, Today we are using these kind of logstash file outputs, path =\> "/mnt/%{\[project\]}/%{\[instance\]}/%{\[beat\]\[hostname\]}/%{\[logtype\]}/%{+YYYY}/%{+MM}/%{+dd}/%{\[log\_filename\]}" but we can't use the date filter. We need… --- ## [Logstash indexing all logs to the index of first log](https://discuss.elastic.co/t/logstash-indexing-all-logs-to-the-index-of-first-log/182275)
**Author:** [@apr589](https://discuss.elastic.co/u/apr589)\ **Replies:** 1\ **Last updated:** [May 27, 2019, 10:02am UTC](https://discuss.elastic.co/t/logstash-indexing-all-logs-to-the-index-of-first-log/182275 "2019-05-27T10:02:29Z")
Hi ,I have a log file in which first few events have a timestamp of 2019.01.13 and rest all events have a timestamp of 2019.01.14 . Now I am indexing based on @timestamp but all the events of the file are getting indexed… --- ## [Comparing Empty String](https://discuss.elastic.co/t/comparing-empty-string/182846)
**Author:** [@Deepak2](https://discuss.elastic.co/u/Deepak2)\ **Replies:** 1\ **Last updated:** [May 27, 2019, 8:19am UTC](https://discuss.elastic.co/t/comparing-empty-string/182846 "2019-05-27T08:19:22Z")
Hi, I am not able to compare empty string and do the necessary operation in logstasgh. Below is my logstash config file line if \[fields\]\[type\] == "serverlog"{ grok { match =\> {"message"=\> "%{DATE:log\_date}%{SPACE}%{T… --- ## [How can i parse the following in logstash](https://discuss.elastic.co/t/how-can-i-parse-the-following-in-logstash/182797)
**Author:** [@Vishnu\_mk](https://discuss.elastic.co/u/Vishnu_mk)\ **Replies:** 6\ **Last updated:** [May 27, 2019, 8:05am UTC](https://discuss.elastic.co/t/how-can-i-parse-the-following-in-logstash/182797 "2019-05-27T08:05:29Z")
1st candidate: 9892143415 is the ph. no. 2nd candidate: my phone no. is 8976635405 3rd candidate: hii all 8976547545 my phone no. Likewise i have 100 candidates and all have wriiten their phone no. in differ… --- ## [Setter value in logstash](https://discuss.elastic.co/t/setter-value-in-logstash/182701)
**Author:** [@MamBarry](https://discuss.elastic.co/u/MamBarry)\ **Replies:** 7\ **Last updated:** [May 27, 2019, 7:21am UTC](https://discuss.elastic.co/t/setter-value-in-logstash/182701 "2019-05-27T07:21:53Z")
Hi, I try to retrieve one key in my message, if this key existe not add, else add key. I try this code : ruby { code =\> ' size = event.get("262") tata= event.get("50") if event.get("50").nil? event.set("size", … --- ## [Logstash-Syslog Configuration in 7.0](https://discuss.elastic.co/t/logstash-syslog-configuration-in-7-0/180118)
**Author:** [@wcpoon](https://discuss.elastic.co/u/wcpoon)\ **Replies:** 6\ **Last updated:** [May 27, 2019, 6:46am UTC](https://discuss.elastic.co/t/logstash-syslog-configuration-in-7-0/180118 "2019-05-27T06:46:59Z")
My current logstash indexing is not working in 7.0 is it the syntax in 7.0 are changed? btw, all this while, it is working in 6.7 My current configuration as below, input { tcp { port =\> 5514 type =\> syslog … --- ## [Logstash runner unexpected error occured](https://discuss.elastic.co/t/logstash-runner-unexpected-error-occured/182218)
**Author:** [@vivekkarne](https://discuss.elastic.co/u/vivekkarne)\ **Replies:** 4\ **Last updated:** [May 27, 2019, 6:45am UTC](https://discuss.elastic.co/t/logstash-runner-unexpected-error-occured/182218 "2019-05-27T06:45:28Z")
Hi, I am trying to parse really large multiline logs, the file is first ingested using filebeat. There i have specified multiline settings so it seems to be fine over ther. However, while ingesting files in logstash an … --- ## [Parsing of field which have dynamic number of subfield](https://discuss.elastic.co/t/parsing-of-field-which-have-dynamic-number-of-subfield/182194)
**Author:** [@swapnali](https://discuss.elastic.co/u/swapnali)\ **Replies:** 3\ **Last updated:** [May 27, 2019, 5:53am UTC](https://discuss.elastic.co/t/parsing-of-field-which-have-dynamic-number-of-subfield/182194 "2019-05-27T05:53:39Z")
hi, I am trying to parse NAS logs,in that i have field "to" in to there are number of receiver address and my logs are streaming logs. so i cant predict how many receiver address will be there in that field how to pars… --- ## [Rabbitmq output plugin - Connection to localhost:5672 failed](https://discuss.elastic.co/t/rabbitmq-output-plugin-connection-to-localhost-5672-failed/182817)
**Author:** [@Pouria\_Fewzee](https://discuss.elastic.co/u/Pouria_Fewzee)\ **Replies:** 0\ **Last updated:** [May 26, 2019, 10:34pm UTC](https://discuss.elastic.co/t/rabbitmq-output-plugin-connection-to-localhost-5672-failed/182817 "2019-05-26T22:34:22Z")
Hi, I receive the following error: RabbitMQ connection error, will retry. {:error\_message=\>"Connection to localhost:5672 failed: timeout", :exception=\>"MarchHare::ConnectionRefused"} although I'm explicitly defining a… --- ## [Logstash: /usr/share/logstash/config/logstash.yml: Read-only file system](https://discuss.elastic.co/t/logstash-usr-share-logstash-config-logstash-yml-read-only-file-system/182808)
**Author:** [@ariguillegp](https://discuss.elastic.co/u/ariguillegp)\ **Replies:** 0\ **Last updated:** [May 26, 2019, 4:58pm UTC](https://discuss.elastic.co/t/logstash-usr-share-logstash-config-logstash-yml-read-only-file-system/182808 "2019-05-26T16:58:41Z")
Im trying to start a kubernetes pod with a logstash (Alpine as base OS) container inside and I am getting the following error: chown: /usr/share/logstash/config/logstash.yml: Read-only file system Any help will be welc… --- ## [How to push droped events into elastic search](https://discuss.elastic.co/t/how-to-push-droped-events-into-elastic-search/182802)
**Author:** [@rahulkothanath](https://discuss.elastic.co/u/rahulkothanath)\ **Replies:** 1\ **Last updated:** [May 26, 2019, 3:19pm UTC](https://discuss.elastic.co/t/how-to-push-droped-events-into-elastic-search/182802 "2019-05-26T15:19:00Z")
I have to index dropped events from filter plugin into elastic search . below is a sample code. filter{ grok { match =\>{ "message" =\> "(?\(?\<=^.{58}).{3})(?\.{11})(?\.{3})"} … --- ## [Logstash not receiving remote tcp data](https://discuss.elastic.co/t/logstash-not-receiving-remote-tcp-data/182783)
**Author:** [@Barry\_C](https://discuss.elastic.co/u/Barry_C)\ **Replies:** 2\ **Last updated:** [May 26, 2019, 9:32am UTC](https://discuss.elastic.co/t/logstash-not-receiving-remote-tcp-data/182783 "2019-05-26T09:32:52Z")
I am new to ELK and I spent days to troubleshoot and still not able to find out what's wrong with my configuration. The following is my problem and what I have done to troubleshoot I setup a basic ELK 7.1 on a single… --- ## [\[Docker\] Logstash not sending any data to elasticsearch](https://discuss.elastic.co/t/docker-logstash-not-sending-any-data-to-elasticsearch/182776)
**Author:** [@elfinitiy](https://discuss.elastic.co/u/elfinitiy)\ **Replies:** 0\ **Last updated:** [May 25, 2019, 8:35pm UTC](https://discuss.elastic.co/t/docker-logstash-not-sending-any-data-to-elasticsearch/182776 "2019-05-25T20:35:38Z")
Hello, while reading and setting up logstash I got the following result. Filebeats sends data to logstash (port 5044), logstash logging is set to trace so I see that the data comes. Then I get output for stdout { codec… --- ## [Logstash Elasticsearch filter plugin cache](https://discuss.elastic.co/t/logstash-elasticsearch-filter-plugin-cache/182138)
**Author:** [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\ **Replies:** 2\ **Last updated:** [May 25, 2019, 5:09pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-filter-plugin-cache/182138 "2019-05-25T17:09:07Z")
Hello Team, Could you please help me understand if elasticsearch filter plugin uses cache and if there are performance issues with the plugin. I am planning to use this filter plugin forvl data lookup. Thank you for y… --- ## [Grock Parsing high CPU not stopping](https://discuss.elastic.co/t/grock-parsing-high-cpu-not-stopping/182682)
**Author:** [@Antoine.E](https://discuss.elastic.co/u/Antoine.E)\ **Replies:** 4\ **Last updated:** [May 25, 2019, 9:14am UTC](https://discuss.elastic.co/t/grock-parsing-high-cpu-not-stopping/182682 "2019-05-25T09:14:49Z")
SLES 12 SP3 ELK 7.1.0 Hello everybody, I have a problem with my Logstash, whenever i create a grock pattern for a certain part of my log, my CPU keep going up and stopping there. It look like the thread with the grock… --- ## [Logstash-7.0.1 Error](https://discuss.elastic.co/t/logstash-7-0-1-error/182568)
**Author:** [@Rizwan](https://discuss.elastic.co/u/Rizwan)\ **Replies:** 2\ **Last updated:** [May 25, 2019, 4:32am UTC](https://discuss.elastic.co/t/logstash-7-0-1-error/182568 "2019-05-25T04:32:10Z")
I am trying to execute this command logstash.bat -f sql.conf and I am getting below exception.. I am using logstash-7.0.1 Can someone help me!! Thanks in advance!! \[2019-05-24T01:08:42,080\]\[WARN \]\[logstash.config.sou… --- ## [Logs missing in Elastic Search](https://discuss.elastic.co/t/logs-missing-in-elastic-search/182734)
**Author:** [@Haren](https://discuss.elastic.co/u/Haren)\ **Replies:** 1\ **Last updated:** [May 25, 2019, 3:21am UTC](https://discuss.elastic.co/t/logs-missing-in-elastic-search/182734 "2019-05-25T03:21:26Z")
Hello Experts, I have a situation where in many cases I am not able to find my application's complete logs of a particular operation in Kibana however the logs are available in application server log file. When I looke… --- ## [Logstash not parsing json from kafka](https://discuss.elastic.co/t/logstash-not-parsing-json-from-kafka/182728)
**Author:** [@ramzey1981](https://discuss.elastic.co/u/ramzey1981)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 9:24pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-json-from-kafka/182728 "2019-05-24T21:24:42Z")
I have the below json message coming into logstash and entering with format indicated below into elasticsearch but unparsed even though i am using the json code. Please let me know if I'm missing something. thanks com… --- ## [Dynamic Elasticsearch Host for Logstash on Docker](https://discuss.elastic.co/t/dynamic-elasticsearch-host-for-logstash-on-docker/182727)
**Author:** [@jamiejackson](https://discuss.elastic.co/u/jamiejackson)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 9:10pm UTC](https://discuss.elastic.co/t/dynamic-elasticsearch-host-for-logstash-on-docker/182727 "2019-05-24T21:10:08Z")
How do I set the Elasticsearch host without hardcoding it in pipeline.conf? I think https://www.elastic.co/guide/en/logstash/current/docker-config.html#docker-env-config is trying to tell me how, but I can't put the pie… --- ## [Logstash parsing of statistics log file](https://discuss.elastic.co/t/logstash-parsing-of-statistics-log-file/181891)
**Author:** [@ankitachow](https://discuss.elastic.co/u/ankitachow)\ **Replies:** 14\ **Last updated:** [May 24, 2019, 8:26pm UTC](https://discuss.elastic.co/t/logstash-parsing-of-statistics-log-file/181891 "2019-05-24T20:26:07Z")
Below is my input log file format. 19-01-01 Name Succ Fail Timeout Reject Retransmission Duplicate Thrput Response time (ms) Detected Answered (/s) Av… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=539) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=541)