# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=269 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 270 --- ## [Unable to parse the mule logs in logstash](https://discuss.elastic.co/t/unable-to-parse-the-mule-logs-in-logstash/259265)
**Author:** [@vinodkumar](https://discuss.elastic.co/u/vinodkumar)\ **Replies:** 10\ **Last updated:** [December 22, 2020, 7:31am UTC](https://discuss.elastic.co/t/unable-to-parse-the-mule-logs-in-logstash/259265 "2020-12-22T07:31:57Z")
Hi Team, I am trying to parse the mule application logs logstash, tried different options but unable to achieve it. My sample logs: 2020-12-18 10:27:10,666 \[\[hl7v2-ORU-message-processor-api\].hl7v2-fhir-api-flowsFlow.s… --- ## [Path "/usr/share/logstash/data" must be a writable directory -- Most secure way to make it so?](https://discuss.elastic.co/t/path-usr-share-logstash-data-must-be-a-writable-directory-most-secure-way-to-make-it-so/259338)
**Author:** [@danibe](https://discuss.elastic.co/u/danibe)\ **Replies:** 0\ **Last updated:** [December 22, 2020, 7:38am UTC](https://discuss.elastic.co/t/path-usr-share-logstash-data-must-be-a-writable-directory-most-secure-way-to-make-it-so/259338 "2020-12-22T07:38:56Z")
Running '/usr/share/logstash/bin/logstash' as root works perfectly but obviously is not the recommended long term mode of use. So, I did: chown -R logstash:logstash conf.d chmod -R 775 /usr/share/logstash/data sudo use… --- ## [Decode kafka message](https://discuss.elastic.co/t/decode-kafka-message/259355)
**Author:** [@lstoneir](https://discuss.elastic.co/u/lstoneir)\ **Replies:** 0\ **Last updated:** [December 22, 2020, 9:47am UTC](https://discuss.elastic.co/t/decode-kafka-message/259355 "2020-12-22T09:47:13Z")
Hi there i have a logstash config like this: input{ kafka { bootstrap\_servers =\> "localhost:9092" topics =\> \["xx"\] client\_id =\> "v1" group\_id =\> "v1-g" } } out… --- ## [Retrieving keys from logstash-keystore externally](https://discuss.elastic.co/t/retrieving-keys-from-logstash-keystore-externally/259342)
**Author:** [@mahi2](https://discuss.elastic.co/u/mahi2)\ **Replies:** 0\ **Last updated:** [December 22, 2020, 7:51am UTC](https://discuss.elastic.co/t/retrieving-keys-from-logstash-keystore-externally/259342 "2020-12-22T07:51:55Z")
Hi I need to retrieve keys in logstash-keystore externally. Currently it only allows to retrieve it from logstash using config file. I'd like to know if there is a way to get it from cmd line or maybe any way other th… --- ## [Last value is not advancing after manual modification of the file](https://discuss.elastic.co/t/last-value-is-not-advancing-after-manual-modification-of-the-file/259333)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 0\ **Last updated:** [December 22, 2020, 6:21am UTC](https://discuss.elastic.co/t/last-value-is-not-advancing-after-manual-modification-of-the-file/259333 "2020-12-22T06:21:11Z")
Hi, after manipulating the last value file, its not moving forward in time anymore, this is my last value file --- !ruby/object:DateTime '2020-12-18 20:39:46.000000000 Z' and it seems that is writing duplicates every t… --- ## [Make Logstash listen on localhost AND external IPs?](https://discuss.elastic.co/t/make-logstash-listen-on-localhost-and-external-ips/259276)
**Author:** [@riahc3](https://discuss.elastic.co/u/riahc3)\ **Replies:** 1\ **Last updated:** [December 21, 2020, 9:48pm UTC](https://discuss.elastic.co/t/make-logstash-listen-on-localhost-and-external-ips/259276 "2020-12-21T21:48:04Z")
Hello I want to able to make my Elastic Stack (Elasticsearch, Logstash, Kibana, and Beats) all listen on the localhost AND on the external IP (or at least bind it to a IP) All I see is creating a reverse proxy with Ngi… --- ## [Add date{} config make logstash not working](https://discuss.elastic.co/t/add-date-config-make-logstash-not-working/259213)
**Author:** [@azteker](https://discuss.elastic.co/u/azteker)\ **Replies:** 5\ **Last updated:** [December 21, 2020, 7:36pm UTC](https://discuss.elastic.co/t/add-date-config-make-logstash-not-working/259213 "2020-12-21T19:36:13Z")
I add date{} config into filter {}, but then the log is not passed to elasticseatch(cannot see new log in kibana). Logs are back if I removed the date config. This is the date config I added: date { match =\> \["ts"… --- ## [Another dateparse failure](https://discuss.elastic.co/t/another-dateparse-failure/259089)
**Author:** [@calanon](https://discuss.elastic.co/u/calanon)\ **Replies:** 3\ **Last updated:** [December 21, 2020, 4:57pm UTC](https://discuss.elastic.co/t/another-dateparse-failure/259089 "2020-12-21T16:57:41Z")
I cant figure out how to match this: RAW output \[2020-12-17T16:45:45+01:00\] Elasticsearch JSON Dez 18 11:03:58 mon logstash\[23980\]: "log\_timestamp" =\> "2020-12-18T11:03:56+01:00", Dez 18 11:03:58 mon logstash\[239… --- ## [How to connect logstash from logstash?](https://discuss.elastic.co/t/how-to-connect-logstash-from-logstash/259258)
**Author:** [@iooi](https://discuss.elastic.co/u/iooi)\ **Replies:** 0\ **Last updated:** [December 21, 2020, 1:33pm UTC](https://discuss.elastic.co/t/how-to-connect-logstash-from-logstash/259258 "2020-12-21T13:33:22Z")
I want to use one logstash send data to multiple logstash services. If I use this setting in k8s, it can work. service.yml apiVersion: v1 kind: Service metadata: labels: app: logstash name: logstash spec: se… --- ## [Convert graphite field name in to tags](https://discuss.elastic.co/t/convert-graphite-field-name-in-to-tags/259257)
**Author:** [@Mario\_Giammarco](https://discuss.elastic.co/u/Mario_Giammarco)\ **Replies:** 0\ **Last updated:** [December 21, 2020, 1:30pm UTC](https://discuss.elastic.co/t/convert-graphite-field-name-in-to-tags/259257 "2020-12-21T13:30:39Z")
Hello, I am a logstash newbie. I have this problem: I receive data from graphite. Data is in this format: computername.cpu.load.cpu0 = 10 I would like to insert this into elastic search: tags=\> \[ computername, cpu,… --- ## [Logstash data type change in config file](https://discuss.elastic.co/t/logstash-data-type-change-in-config-file/258680)
**Author:** [@anjali28](https://discuss.elastic.co/u/anjali28)\ **Replies:** 3\ **Last updated:** [December 21, 2020, 7:19am UTC](https://discuss.elastic.co/t/logstash-data-type-change-in-config-file/258680 "2020-12-21T07:19:18Z")
Hello i am creating a config file to import the data from a csv file. when i call the config file as it is, its working properly and data is coming in kibana. But when i am changing the data type of a field in logstas… --- ## [Filter Plugin - "Tried to load a plugin's code, but failed."](https://discuss.elastic.co/t/filter-plugin-tried-to-load-a-plugins-code-but-failed/259216)
**Author:** [@smm](https://discuss.elastic.co/u/smm)\ **Replies:** 1\ **Last updated:** [December 21, 2020, 6:34am UTC](https://discuss.elastic.co/t/filter-plugin-tried-to-load-a-plugins-code-but-failed/259216 "2020-12-21T06:34:57Z")
Hi there, there is a logstash filter plugin called elapsed. I put (also) this into a config: elapsed { unique\_id\_field =\> "reqid" start\_tag =\> "TranBegin" end\_tag =\> "TranEnd" new\_event\_on\_match =\> false } a… --- ## [Why can't receive log from the ftp server?](https://discuss.elastic.co/t/why-cant-receive-log-from-the-ftp-server/259217)
**Author:** [@111418](https://discuss.elastic.co/u/111418)\ **Replies:** 0\ **Last updated:** [December 21, 2020, 6:14am UTC](https://discuss.elastic.co/t/why-cant-receive-log-from-the-ftp-server/259217 "2020-12-21T06:14:20Z")
My conf is input { tcp { port =\> 10514 type=\> "leo\_ftp" } tcp { port =\> 10515 type =\> "jim\_ftp" } } filter { } output { if\[type\]=="leo\_ftp"{ elasticsearch { hosts =\> \["ip:9200"\] inde… --- ## [HAProxy + Logstash](https://discuss.elastic.co/t/haproxy-logstash/259215)
**Author:** [@Gigazo1d](https://discuss.elastic.co/u/Gigazo1d)\ **Replies:** 0\ **Last updated:** [December 21, 2020, 6:04am UTC](https://discuss.elastic.co/t/haproxy-logstash/259215 "2020-12-21T06:04:45Z")
Hi! I plan to use HAProxy + Logstash to receive events and send them to another syslog server. Could you please share examples of your HAProxy config for my view. --- ## [Reading environment variables set with exec plugin](https://discuss.elastic.co/t/reading-environment-variables-set-with-exec-plugin/259189)
**Author:** [@Lukasz\_Geras](https://discuss.elastic.co/u/Lukasz_Geras)\ **Replies:** 4\ **Last updated:** [December 21, 2020, 5:56am UTC](https://discuss.elastic.co/t/reading-environment-variables-set-with-exec-plugin/259189 "2020-12-21T05:56:59Z")
Hello, could you please help me with reading environment variables set with exec plugin in logstash. Generally, my input looks like this: input{ exec{ command=\> „export FILENAME=$(ls -Arst /path/to/… --- ## [Logstash is picking messages quiet slowly](https://discuss.elastic.co/t/logstash-is-picking-messages-quiet-slowly/259160)
**Author:** [@rohitarorait82](https://discuss.elastic.co/u/rohitarorait82)\ **Replies:** 2\ **Last updated:** [December 21, 2020, 5:02am UTC](https://discuss.elastic.co/t/logstash-is-picking-messages-quiet-slowly/259160 "2020-12-21T05:02:28Z")
Hi All, We have traffic around 100 million records per day and it was working fine till yesterday , but today I can see 50 million records are piled up at the the source system . i have tried increasing pipeline.workers… --- ## [How to monitor/identify retry result with output elasticsearch](https://discuss.elastic.co/t/how-to-monitor-identify-retry-result-with-output-elasticsearch/258681)
**Author:** [@loker](https://discuss.elastic.co/u/loker)\ **Replies:** 0\ **Last updated:** [December 15, 2020, 8:28am UTC](https://discuss.elastic.co/t/how-to-monitor-identify-retry-result-with-output-elasticsearch/258681 "2020-12-15T08:28:00Z")
Hello there, Recently i meet some warning about es reject caused by queue exceed 200 limit. According to its retry policy for es output plugin, it should retry indefinitely for this kind of error(429). But i don't kno… --- ## [Parsing ip address coming at random places inside logs](https://discuss.elastic.co/t/parsing-ip-address-coming-at-random-places-inside-logs/259177)
**Author:** [@Muhammad\_Faisal](https://discuss.elastic.co/u/Muhammad_Faisal)\ **Replies:** 4\ **Last updated:** [December 20, 2020, 4:05pm UTC](https://discuss.elastic.co/t/parsing-ip-address-coming-at-random-places-inside-logs/259177 "2020-12-20T16:05:12Z")
hi guys, input log message contains IP address but the occurence of ip is random inside message and also some times the message will not contain any ip ...how can we extract this ip address using grok or other filters. … --- ## [Need to parse logs and enrich some values from mysql data at logstats](https://discuss.elastic.co/t/need-to-parse-logs-and-enrich-some-values-from-mysql-data-at-logstats/259176)
**Author:** [@abhi.coerian](https://discuss.elastic.co/u/abhi.coerian)\ **Replies:** 3\ **Last updated:** [December 20, 2020, 3:58pm UTC](https://discuss.elastic.co/t/need-to-parse-logs-and-enrich-some-values-from-mysql-data-at-logstats/259176 "2020-12-20T15:58:07Z")
Need to parse logs on logstash and enrich some values from mysql db based on the log message and generate json message . e.x JOB application\_1603918538928\_3286731 QUEUE default USER xyz Need to fetch details of USER :… --- ## [Logs including emojis](https://discuss.elastic.co/t/logs-including-emojis/258946)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 6\ **Last updated:** [December 20, 2020, 9:29am UTC](https://discuss.elastic.co/t/logs-including-emojis/258946 "2020-12-20T09:29:07Z")
Hello everyone! is there a way to parse this log with emojis? Could it be shown in kibana or logstash output? I do not have a clue how to parse the emojis in logstash pipeline. "emoji\_code":"🌸" Can someone please help … --- ## [Pattern for this log](https://discuss.elastic.co/t/pattern-for-this-log/259184)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 6\ **Last updated:** [December 20, 2020, 9:24am UTC](https://discuss.elastic.co/t/pattern-for-this-log/259184 "2020-12-20T09:24:36Z")
Hello everyone! Can someone please help me with writing a grok pattern for this log line? I am not able to seperate its fields. {"name":"test","test\_arr":\[{"key\_word":"ok","test\_code":"👌","h\_code":"1234.png"}\],"create\_d… --- ## [Removing ongoing duplicates](https://discuss.elastic.co/t/removing-ongoing-duplicates/259157)
**Author:** [@cyberzlo](https://discuss.elastic.co/u/cyberzlo)\ **Replies:** 1\ **Last updated:** [December 19, 2020, 3:36pm UTC](https://discuss.elastic.co/t/removing-ongoing-duplicates/259157 "2020-12-19T15:36:14Z")
This is what I need however I would like do it on input due parsing jsons. What I mean this tutorial show how do it post-factum, but I would like do it every time I recieve new data basing for example on data from last… --- ## [Geoip lookup failure while parsing http json data](https://discuss.elastic.co/t/geoip-lookup-failure-while-parsing-http-json-data/259120)
**Author:** [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\ **Replies:** 2\ **Last updated:** [December 18, 2020, 5:49pm UTC](https://discuss.elastic.co/t/geoip-lookup-failure-while-parsing-http-json-data/259120 "2020-12-18T17:49:56Z")
I am trying to perform Geoip filter while parsing json logs data from clouflare API but it says \_geoip\_lookup\_failure here is my conf file configurations input { http\_poller { urls =\> { test… --- ## [Unable to include runtime log data in 'mutate' filter plugin](https://discuss.elastic.co/t/unable-to-include-runtime-log-data-in-mutate-filter-plugin/259125)
**Author:** [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\ **Replies:** 6\ **Last updated:** [December 18, 2020, 5:18pm UTC](https://discuss.elastic.co/t/unable-to-include-runtime-log-data-in-mutate-filter-plugin/259125 "2020-12-18T17:18:34Z")
I am trying to fetch cloudflare log data via API and want add a new field such as actor.email in index. My end goal is to add a new field in index having data in meaningful sentence like A user someone@email.com(actor.… --- ## [Logstash 7.10 is ignoring custom index template](https://discuss.elastic.co/t/logstash-7-10-is-ignoring-custom-index-template/258615)
**Author:** [@ManuelF](https://discuss.elastic.co/u/ManuelF)\ **Replies:** 26\ **Last updated:** [December 18, 2020, 2:35pm UTC](https://discuss.elastic.co/t/logstash-7-10-is-ignoring-custom-index-template/258615 "2020-12-18T14:35:56Z")
Hi, This is a fresh installation of ELK 7.10.0. I am trying to configure Logstash to process .csv files and index into ES. I created a config file and a custom index template. It looks like the .csv are being processed,… --- ## [Elasticsearch input plugin: range query doesn't match any document](https://discuss.elastic.co/t/elasticsearch-input-plugin-range-query-doesnt-match-any-document/259097)
**Author:** [@Mathias\_Bazin](https://discuss.elastic.co/u/Mathias_Bazin)\ **Replies:** 0\ **Last updated:** [December 18, 2020, 12:04pm UTC](https://discuss.elastic.co/t/elasticsearch-input-plugin-range-query-doesnt-match-any-document/259097 "2020-12-18T12:04:26Z")
Hello, I'm having trouble collecting data periodically from an elasticsearch index. My input config is quite simple input { elasticsearch { hosts =\> "${ELASTICSEARCH\_URL\_INPUT}" index =\> 'logcentral\_pn' d… --- ## [How to use if else in logstash output](https://discuss.elastic.co/t/how-to-use-if-else-in-logstash-output/256793)
**Author:** [@lokeshbabloo](https://discuss.elastic.co/u/lokeshbabloo)\ **Replies:** 6\ **Last updated:** [December 18, 2020, 1:16pm UTC](https://discuss.elastic.co/t/how-to-use-if-else-in-logstash-output/256793 "2020-12-18T13:16:00Z")
Hi i need a help in constructing the logstash output i am having two log files like this, one is \<#\> 20200806 17:04:23.261 280018000 EV.INF \[ MVINB-LSL2.T1G1\_WEBSVR1 main.main TSP1.T1G1\_WEBSVR1 … --- ## [Dateparse failure](https://discuss.elastic.co/t/dateparse-failure/259001)
**Author:** [@calanon](https://discuss.elastic.co/u/calanon)\ **Replies:** 2\ **Last updated:** [December 18, 2020, 10:09am UTC](https://discuss.elastic.co/t/dateparse-failure/259001 "2020-12-18T10:09:16Z")
I cannot find a way to either debug or fix this. I have provided below the log output and the various grok filters that are used. What can I do to fix this? RAW Log output \`logstash\[23980\]: \[2020-12-17T14:19:12,925… --- ## [Refresh data](https://discuss.elastic.co/t/refresh-data/258989)
**Author:** [@mihai.radulescu](https://discuss.elastic.co/u/mihai.radulescu)\ **Replies:** 3\ **Last updated:** [December 18, 2020, 9:03am UTC](https://discuss.elastic.co/t/refresh-data/258989 "2020-12-18T09:03:17Z")
Hello, I have a logstash configuration that is inserting the data from csv files from a specific folder. In the past two weeks I have made numerous changes and the data that I inserted is incomplete. How do I drop ever… --- ## [Mapping error while writing to index](https://discuss.elastic.co/t/mapping-error-while-writing-to-index/259068)
**Author:** [@BoKu](https://discuss.elastic.co/u/BoKu)\ **Replies:** 0\ **Last updated:** [December 18, 2020, 8:13am UTC](https://discuss.elastic.co/t/mapping-error-while-writing-to-index/259068 "2020-12-18T08:13:04Z")
Hello together, i am trying to read, with logstash, from an influxdb via http-poller, that seems to work when writing to an file, but when i write to an index i get the following error: \[2020-12-18T08:59:14,281\]\[WARN \]… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=268) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=270)