# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=220 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 221 --- ## [How use metricbeat for logstash logs files in docker container](https://discuss.elastic.co/t/how-use-metricbeat-for-logstash-logs-files-in-docker-container/275540)
**Author:** [@Atesrin](https://discuss.elastic.co/u/Atesrin)\ **Replies:** 1\ **Last updated:** [June 11, 2021, 9:08pm UTC](https://discuss.elastic.co/t/how-use-metricbeat-for-logstash-logs-files-in-docker-container/275540 "2021-06-11T21:08:37Z")
Hi everyone, My first post, so sorry if I made some mistakes. I'm trying to use metricbeat to monitor Logstash. I have elastic, kibana, metricbeat and logstash running in docker containers with 2 docker-compose files … --- ## [Parsing csv file through Logstash](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 17\ **Last updated:** [June 11, 2021, 9:01pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792 "2021-06-11T21:01:28Z")
Hi All, I am running ELK stack 7.6.2. I need to parse a csv through Logstash. My logstash conf file looks like this: input { file { path =\> "/opt/gtal/ictal/elasticsearch/app/logstash/stage/STATS-01062021.txt" … --- ## [How to replace @timestamp field in logstash aggregate filter map](https://discuss.elastic.co/t/how-to-replace-timestamp-field-in-logstash-aggregate-filter-map/275708)
**Author:** [@jratliff](https://discuss.elastic.co/u/jratliff)\ **Replies:** 1\ **Last updated:** [June 11, 2021, 7:13pm UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-field-in-logstash-aggregate-filter-map/275708 "2021-06-11T19:13:49Z")
I'm using the logstash aggregate filter. When the map is pushed upon timeout, I want to set the @timestamp field to be a specific valu. I've tried using timeout\_code =\> event.set('@timestamp', "2021-06-10T19:00:00.000Z"… --- ## [Kv filter has no support for this type of data {:type=\>Hash, :value=\>{"METHODNAME"=\>](https://discuss.elastic.co/t/kv-filter-has-no-support-for-this-type-of-data-type-hash-value-methodname/275659)
**Author:** [@Chandana](https://discuss.elastic.co/u/Chandana)\ **Replies:** 2\ **Last updated:** [June 11, 2021, 5:56pm UTC](https://discuss.elastic.co/t/kv-filter-has-no-support-for-this-type-of-data-type-hash-value-methodname/275659 "2021-06-11T17:56:38Z")
Can some please help me to parse the below log, I need the key-value pairs in DEBUGMESSAGE as my fields. Here is my sample logs snippet: {"thread":"default task-131","TimeStamp":"06/08/2021:00:58:02,848-08:00","level":… --- ## [Force logstash to recreate/ update the same index](https://discuss.elastic.co/t/force-logstash-to-recreate-update-the-same-index/275705)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 0\ **Last updated:** [June 11, 2021, 5:23pm UTC](https://discuss.elastic.co/t/force-logstash-to-recreate-update-the-same-index/275705 "2021-06-11T17:23:50Z")
Hello. I need to force logstash to recreate (re-read) the "same" index and update it. Is there an option for that? Please guide for ELK stack 7.6.2 Thanks --- ## [Logstash Key store](https://discuss.elastic.co/t/logstash-key-store/275633)
**Author:** [@farciarz121](https://discuss.elastic.co/u/farciarz121)\ **Replies:** 7\ **Last updated:** [June 11, 2021, 12:42pm UTC](https://discuss.elastic.co/t/logstash-key-store/275633 "2021-06-11T12:42:24Z")
I am trying to improve my ELK security with logstash keystore. I am following instruction from this article: https://www.elastic.co/guide/en/logstash/current/keystore.html All instructions seems to be straight forward… --- ## [Getting \_split\_type\_failure and \_jsonparsefailure in my logstash pipeline using only 1 filter (elasticsearch)](https://discuss.elastic.co/t/getting-split-type-failure-and-jsonparsefailure-in-my-logstash-pipeline-using-only-1-filter-elasticsearch/275671)
**Author:** [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Replies:** 1\ **Last updated:** [June 11, 2021, 10:35am UTC](https://discuss.elastic.co/t/getting-split-type-failure-and-jsonparsefailure-in-my-logstash-pipeline-using-only-1-filter-elasticsearch/275671 "2021-06-11T10:35:59Z")
Hi, i have 2 pipelines Pipeline 1: input { elasticsearch {}} filter { split {} json {} } output { pipeline {} } Pipeline 2: input { pipeline {}} filter { elasticsearch {} } output { stdout {} } Pipelin 1 where … --- ## [Watchguard Firebox to Logstash](https://discuss.elastic.co/t/watchguard-firebox-to-logstash/275672)
**Author:** [@gygrill](https://discuss.elastic.co/u/gygrill)\ **Replies:** 0\ **Last updated:** [June 11, 2021, 10:24am UTC](https://discuss.elastic.co/t/watchguard-firebox-to-logstash/275672 "2021-06-11T10:24:08Z")
Hi, total newbie here. How do I send Firebox logs to logstash? Do a specific beat exists or is it another method required? thanks for any help --- ## [Avoid too many useless field on elastic search - kibana](https://discuss.elastic.co/t/avoid-too-many-useless-field-on-elastic-search-kibana/274357)
**Author:** [@rebug](https://discuss.elastic.co/u/rebug)\ **Replies:** 9\ **Last updated:** [June 11, 2021, 7:30am UTC](https://discuss.elastic.co/t/avoid-too-many-useless-field-on-elastic-search-kibana/274357 "2021-06-11T07:30:03Z")
Hi, I am pretty new to this but, I have setup a elastic stack for starting a log centralization. I have just tested with Nginx log and syslog. My problem is the number of fields created (and many of them are useless a… --- ## [How to parse message use ELK+Kafka](https://discuss.elastic.co/t/how-to-parse-message-use-elk-kafka/275642)
**Author:** [@tod-yangyd](https://discuss.elastic.co/u/tod-yangyd)\ **Replies:** 1\ **Last updated:** [June 11, 2021, 5:52am UTC](https://discuss.elastic.co/t/how-to-parse-message-use-elk-kafka/275642 "2021-06-11T05:52:25Z")
Recently I tried to analyzed the log with the stream：filebeat -\> kafka -\>logstash -\> ES The data stored in ES becomes the structure shown below： { “\_source”:{ “@timestamp” : "YYYY-MM-DDTHH:mm:ss.sss" … --- ## [How to do lookup based on Ip ranges](https://discuss.elastic.co/t/how-to-do-lookup-based-on-ip-ranges/275397)
**Author:** [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Replies:** 1\ **Last updated:** [June 11, 2021, 5:38am UTC](https://discuss.elastic.co/t/how-to-do-lookup-based-on-ip-ranges/275397 "2021-06-11T05:38:00Z")
Hi Team, I have requirement where I have to do lookup on ranges like, If Ip 10.10.x.xx to 10.10x.xxx matches then add fileds Firewall 10.12.xx.xx to 10.12.x.xxx matches then add fileds Application. Below is my logst… --- ## [Logstash taking wrong url for Elasticserach](https://discuss.elastic.co/t/logstash-taking-wrong-url-for-elasticserach/275527)
**Author:** [@Shubham\_Dhote](https://discuss.elastic.co/u/Shubham_Dhote)\ **Replies:** 5\ **Last updated:** [June 11, 2021, 4:12am UTC](https://discuss.elastic.co/t/logstash-taking-wrong-url-for-elasticserach/275527 "2021-06-11T04:12:57Z")
Hi, I have deployed elasticserach and logstash in kubernetes cluster. And my logstash is unable to connect to elasticserach. Problem is:- my elasticserach url is:- https://my-server.com/elasticsearch And logstash is ta… --- ## [Logstash communication | configuration](https://discuss.elastic.co/t/logstash-communication-configuration/275331)
**Author:** [@farciarz121](https://discuss.elastic.co/u/farciarz121)\ **Replies:** 7\ **Last updated:** [June 10, 2021, 8:02pm UTC](https://discuss.elastic.co/t/logstash-communication-configuration/275331 "2021-06-10T20:02:57Z")
I am traying to send logs from beats to logs to ES cloud. 5 beats are hosted on IP addresses from 192.168.1.10-15. Logstash is installed on 192.168.1.100 node. Here is example of part of packet beat conf file from one … --- ## [Module for Hardware Monitoring - iDRAC and iLO](https://discuss.elastic.co/t/module-for-hardware-monitoring-idrac-and-ilo/275609)
**Author:** [@albertosoares](https://discuss.elastic.co/u/albertosoares)\ **Replies:** 0\ **Last updated:** [June 10, 2021, 6:23pm UTC](https://discuss.elastic.co/t/module-for-hardware-monitoring-idrac-and-ilo/275609 "2021-06-10T18:23:25Z")
Hello, I would like to collect data from iDRAC and iLO. My intention is to send Syslog messages to Elasticsearch, but before I do it, I would like to know if there are other ways to do it. For example, a module of logst… --- ## [Aggregate filter: How to delete the event when no matching task\_id found within the timeout?](https://discuss.elastic.co/t/aggregate-filter-how-to-delete-the-event-when-no-matching-task-id-found-within-the-timeout/275375)
**Author:** [@subash](https://discuss.elastic.co/u/subash)\ **Replies:** 3\ **Last updated:** [June 10, 2021, 4:39pm UTC](https://discuss.elastic.co/t/aggregate-filter-how-to-delete-the-event-when-no-matching-task-id-found-within-the-timeout/275375 "2021-06-10T16:39:09Z")
Hi, I am using aggregate filter in Logstash to merge the events based on task\_id. The problem is, I don't want to push the event when no matching task\_id found within the given timeout. Please pour you ideas to achiev… --- ## [Failed to parse field \[eAgent.tstEvent\] of type \[date\]](https://discuss.elastic.co/t/failed-to-parse-field-eagent-tstevent-of-type-date/275576)
**Author:** [@vaclav1](https://discuss.elastic.co/u/vaclav1)\ **Replies:** 1\ **Last updated:** [June 10, 2021, 4:35pm UTC](https://discuss.elastic.co/t/failed-to-parse-field-eagent-tstevent-of-type-date/275576 "2021-06-10T16:35:11Z")
Dear team, I have used date filter plugin to parse the date field: date { match =\> \[ "\[message\]\[tstEventPlan\]", "yyyy-MM-dd HH:mm:ss'.'SSS" \] remove\_field =\> \[ "\[message\]\[tstEventPlan\]" \] target =\> "\[eAgent\]\[t… --- ## [Logstash Elasticsearch plugin compare inputs](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152)
**Author:** [@Armen\_Petrosyan](https://discuss.elastic.co/u/Armen_Petrosyan)\ **Replies:** 6\ **Last updated:** [June 10, 2021, 4:07pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152 "2021-06-10T16:07:21Z")
I have logstash configured to have two inputs from different ES clusters input { elasticsearch { hosts =\> \["xxx:111"\] ssl =\> true user =\> "" password =\> "" ca\_file =\> "" … --- ## [Parsing result from custom Beats](https://discuss.elastic.co/t/parsing-result-from-custom-beats/275574)
**Author:** [@blackberrySherbet](https://discuss.elastic.co/u/blackberrySherbet)\ **Replies:** 1\ **Last updated:** [June 10, 2021, 3:05pm UTC](https://discuss.elastic.co/t/parsing-result-from-custom-beats/275574 "2021-06-10T15:05:06Z")
I have created my own Beats and I am passing data from that Beats to logstash. This is what I get when I print to stdout in logstash(stdout { codec =\> "rubydebug"}) { "agent" =\> { "type" =\> "cu… --- ## [Lookup for IP range](https://discuss.elastic.co/t/lookup-for-ip-range/275520)
**Author:** [@abhishek30](https://discuss.elastic.co/u/abhishek30)\ **Replies:** 1\ **Last updated:** [June 10, 2021, 2:57pm UTC](https://discuss.elastic.co/t/lookup-for-ip-range/275520 "2021-06-10T14:57:17Z")
Hi, I would like to perform lookup for IP range in logstash pipeline. IP ranges meaning 10.0.0.0 to 10.255.255.0. Kindly Help Thanks & Regards Abhishek --- ## [How to change number format in Logstash](https://discuss.elastic.co/t/how-to-change-number-format-in-logstash/275587)
**Author:** [@rknd](https://discuss.elastic.co/u/rknd)\ **Replies:** 1\ **Last updated:** [June 10, 2021, 2:39pm UTC](https://discuss.elastic.co/t/how-to-change-number-format-in-logstash/275587 "2021-06-10T14:39:46Z")
I have numbers like 1.553 23 2.383 26 566 When I parse this with logstash, I get string value of these. When I try to use mutate filter plugin like below; mutate { convert =\> { "number\_field" =\> "int… --- ## [Azure Event Hubs plugin problem](https://discuss.elastic.co/t/azure-event-hubs-plugin-problem/275463)
**Author:** [@cRaZyT](https://discuss.elastic.co/u/cRaZyT)\ **Replies:** 2\ **Last updated:** [June 10, 2021, 10:57am UTC](https://discuss.elastic.co/t/azure-event-hubs-plugin-problem/275463 "2021-06-10T10:57:36Z")
Hi there, I've installed logstash 7.13.1 and the logstash-input-azure\_event\_hubs 1.3.0 plugin. I got the following error during the initialization: \[ERROR\]\[logstash.inputs.azureeventhubs\]\[main\]\[07395995e41d84f42b907d… --- ## [Logstash Extract message field .log file](https://discuss.elastic.co/t/logstash-extract-message-field-log-file/274577)
**Author:** [@fredsson](https://discuss.elastic.co/u/fredsson)\ **Replies:** 0\ **Last updated:** [June 1, 2021, 5:48am UTC](https://discuss.elastic.co/t/logstash-extract-message-field-log-file/274577 "2021-06-01T05:48:59Z")
Hey guys, i play a bit with the Elastic Stack and i try to visualize some log files. i allready read so many topics how to filter these files, but i didn't get a solution, hopefully i can get some help here … --- ## [High availability with exec command](https://discuss.elastic.co/t/high-availability-with-exec-command/275437)
**Author:** [@Bilanda](https://discuss.elastic.co/u/Bilanda)\ **Replies:** 2\ **Last updated:** [June 10, 2021, 7:48am UTC](https://discuss.elastic.co/t/high-availability-with-exec-command/275437 "2021-06-10T07:48:06Z")
Hello ! I'm currently configuring a highly availabable logstash cluster with two nodes. I'm also using the exec plugin with logstash in order to pull new events. I was wondering if it was possible to set the exec plug… --- ## [Using fingerprint and document\_id for at-least-once delivery and dedupe with ILM](https://discuss.elastic.co/t/using-fingerprint-and-document-id-for-at-least-once-delivery-and-dedupe-with-ilm/275522)
**Author:** [@tomr](https://discuss.elastic.co/u/tomr)\ **Replies:** 3\ **Last updated:** [June 10, 2021, 7:21am UTC](https://discuss.elastic.co/t/using-fingerprint-and-document-id-for-at-least-once-delivery-and-dedupe-with-ilm/275522 "2021-06-10T07:21:37Z")
TL;DR Is ILM compatible with at-least-once-delivery / deduplication / idempotence? Longer version.. I routinely use a fingerprint filter combined with the elasticsearch output's document\_id setting for at-least-once del… --- ## [Logstash 7.13.1 Windows pipeline.separate\_logs Problem](https://discuss.elastic.co/t/logstash-7-13-1-windows-pipeline-separate-logs-problem/275503)
**Author:** [@KSib](https://discuss.elastic.co/u/KSib)\ **Replies:** 4\ **Last updated:** [June 10, 2021, 3:58am UTC](https://discuss.elastic.co/t/logstash-7-13-1-windows-pipeline-separate-logs-problem/275503 "2021-06-10T03:58:21Z")
I'm not sure what's going on here. I'm using Logstash 7.13.1 on Windows Server 2019. When I run logstash with the following C:\\Elastic\\logstash\\bin\\logstash.bat -f "C:\\Elastic\\logstash\\redacted-config\\pipeline\\\_temp\\cus… --- ## [Logstash is unable to push logs to elasticsearch even ssl verify is set to fals](https://discuss.elastic.co/t/logstash-is-unable-to-push-logs-to-elasticsearch-even-ssl-verify-is-set-to-fals/275510)
**Author:** [@prataprajasekhar](https://discuss.elastic.co/u/prataprajasekhar)\ **Replies:** 1\ **Last updated:** [June 10, 2021, 3:02am UTC](https://discuss.elastic.co/t/logstash-is-unable-to-push-logs-to-elasticsearch-even-ssl-verify-is-set-to-fals/275510 "2021-06-10T03:02:27Z")
Dear Friends, I have enabled TLS/SSL in my elasticsearch latest version and all my filebeat, metricbeats are working perfectly. However, logstash is unable to connect to elasticsearch after enabling TLS and return the … --- ## [@timestamp for time zone issues](https://discuss.elastic.co/t/timestamp-for-time-zone-issues/275507)
**Author:** [@asasas234](https://discuss.elastic.co/u/asasas234)\ **Replies:** 0\ **Last updated:** [June 9, 2021, 11:11pm UTC](https://discuss.elastic.co/t/timestamp-for-time-zone-issues/275507 "2021-06-09T23:11:32Z")
I have an old logstash docker image, its version is 6.2.2 and the time zone is +0800, my new logstash docker image version is 7.12.1 , the time zone is +0000, but there is a surprising result, the final output @timestamp … --- ## [Ingesting JSON Data Samples w/ Logstash](https://discuss.elastic.co/t/ingesting-json-data-samples-w-logstash/275185)
**Author:** [@pkward](https://discuss.elastic.co/u/pkward)\ **Replies:** 33\ **Last updated:** [June 9, 2021, 9:57pm UTC](https://discuss.elastic.co/t/ingesting-json-data-samples-w-logstash/275185 "2021-06-09T21:57:10Z")
Hello, I'm trying to test a machine learning classifier in elasticsearch, but I am having a hard time ingesting my data. I've tried using JQ to stream the data via Bulk API, so now I'm ready to try Logstash which may be… --- ## [Error unable to find valid certification path](https://discuss.elastic.co/t/error-unable-to-find-valid-certification-path/122304)
**Author:** [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Replies:** 9\ **Last updated:** [June 9, 2021, 8:52pm UTC](https://discuss.elastic.co/t/error-unable-to-find-valid-certification-path/122304 "2021-06-09T20:52:05Z")
Hi, I'm trying to configure TLS between Logstash and Elastic. Currently I have configured (correctly, it seems), TLS on my cluster (3 nodes) and I can access to it using Kibana. In my Logstash (installed in localhost … --- ## [Why is the value in the field on kibana repeated?](https://discuss.elastic.co/t/why-is-the-value-in-the-field-on-kibana-repeated/275420)
**Author:** [@Cong\_To](https://discuss.elastic.co/u/Cong_To)\ **Replies:** 2\ **Last updated:** [June 9, 2021, 4:03pm UTC](https://discuss.elastic.co/t/why-is-the-value-in-the-field-on-kibana-repeated/275420 "2021-06-09T16:03:32Z")
Update: I found the cause, the error was because I had two configuration files in the logstash folder and they had the same grok. Thank you for following. Hi I use ELK version 7.13.1 on Ubuntu 18.04. I tried reading t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=219) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=221)