# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=298 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 299 --- ## [Linux auditd grokking without auditbeat (but from files)](https://discuss.elastic.co/t/linux-auditd-grokking-without-auditbeat-but-from-files/250710)
**Author:** [@kelk](https://discuss.elastic.co/u/kelk)\ **Replies:** 0\ **Last updated:** [October 1, 2020, 3:21pm UTC](https://discuss.elastic.co/t/linux-auditd-grokking-without-auditbeat-but-from-files/250710 "2020-10-01T15:21:52Z")
We have quite lot of devices sending auditd dataset via syslog and/or filebeats (where we don't have control to install auditbeat). Does anyone have a auditd module for parsing the dataset and enrichment within logstash?… --- ## [UNABLE TO UPDATE THE DOCUMENT USING MONGODB LOGSTASH PLUGIN](https://discuss.elastic.co/t/unable-to-update-the-document-using-mongodb-logstash-plugin/250695)
**Author:** [@Raja\_Kushwah](https://discuss.elastic.co/u/Raja_Kushwah)\ **Replies:** 1\ **Last updated:** [October 1, 2020, 3:21pm UTC](https://discuss.elastic.co/t/unable-to-update-the-document-using-mongodb-logstash-plugin/250695 "2020-10-01T15:21:05Z")
Hi, I am trying to migrate the data using logstash mongodb plugin .. I am able to migrate the data but it is failing while updating the data in mongo db as it is creating a new document while i am trying to append or up… --- ## [동일 로그에 대한 다중 인덱스 생성 방법](https://discuss.elastic.co/t/topic/250610)
**Author:** [@wwwryuwww](https://discuss.elastic.co/u/wwwryuwww)\ **Replies:** 1\ **Last updated:** [October 1, 2020, 2:38pm UTC](https://discuss.elastic.co/t/topic/250610 "2020-10-01T14:38:09Z")
현재 filebeat -\> logstash -\> elasticsearch 구조로 모두 6.6.2 버전을 사용하고 있는데 동일 서버 내 한 개의 서비스 로그를 내용 패턴과 생성 목적에 따라서 별도의 인덱스로 elasticsearch에 저장하고자 합니다. "customs.log" -\> 1. index "case1" -\> 2. index "case2" 현재는 서버의 filebeat에서 동… --- ## [CEF codec plugin](https://discuss.elastic.co/t/cef-codec-plugin/250644)
**Author:** [@david.preston](https://discuss.elastic.co/u/david.preston)\ **Replies:** 1\ **Last updated:** [October 1, 2020, 2:29pm UTC](https://discuss.elastic.co/t/cef-codec-plugin/250644 "2020-10-01T14:29:43Z")
Hi I'd like to use the CEF codec as a filter plugin - is this possible? In a similar way that you have a json input codec, and a json filter plugin. I would like to check the message is a CEF message, and process appr… --- ## [Logstash env configuration to set the elastic search url](https://discuss.elastic.co/t/logstash-env-configuration-to-set-the-elastic-search-url/250675)
**Author:** [@masber](https://discuss.elastic.co/u/masber)\ **Replies:** 2\ **Last updated:** [October 1, 2020, 2:24pm UTC](https://discuss.elastic.co/t/logstash-env-configuration-to-set-the-elastic-search-url/250675 "2020-10-01T14:24:39Z")
Hi, I am running logstash docker container and need to know which env to use to tell logstash where to find the elasticsearch server. \[2020-10-01T11:37:39,829\]\[WARN \]\[logstash.licensechecker.licensereader\] Attempted to … --- ## [Can't monitor Logstash with Metricbeat](https://discuss.elastic.co/t/cant-monitor-logstash-with-metricbeat/249542)
**Author:** [@robertitox](https://discuss.elastic.co/u/robertitox)\ **Replies:** 3\ **Last updated:** [October 1, 2020, 1:27pm UTC](https://discuss.elastic.co/t/cant-monitor-logstash-with-metricbeat/249542 "2020-10-01T13:27:00Z")
Dear all, I have a 3-node Elasticsearch 7.9.1 cluster. In another server, I have a Logstash 7.9.1 service and Metricbeat with logstash-xpack module enable. The problem is that in my Kibana --\> Stack Monitoring section, … --- ## [Configure logstash to show logs from port 5002](https://discuss.elastic.co/t/configure-logstash-to-show-logs-from-port-5002/250205)
**Author:** [@Lusca\_lusca](https://discuss.elastic.co/u/Lusca_lusca)\ **Replies:** 19\ **Last updated:** [October 1, 2020, 11:44am UTC](https://discuss.elastic.co/t/configure-logstash-to-show-logs-from-port-5002/250205 "2020-10-01T11:44:56Z")
I am knew to ELK and I can't see the logs coming from port 5002 on kibana. Can anyone please help me configure logstash to send the logs to kibana? --- ## [Failed to parse field \[datasize\] of type \[integer\] in document with id](https://discuss.elastic.co/t/failed-to-parse-field-datasize-of-type-integer-in-document-with-id/250650)
**Author:** [@martb](https://discuss.elastic.co/u/martb)\ **Replies:** 0\ **Last updated:** [October 1, 2020, 10:56am UTC](https://discuss.elastic.co/t/failed-to-parse-field-datasize-of-type-integer-in-document-with-id/250650 "2020-10-01T10:56:16Z")
Hi, I'm getting the following error from logstash. The field is mapped as an integer and the number is only 11 digits long. Is this mapped as the correct type? Please advise. \[2020-10-01T10:10:18,893\]\[WARN \]\[logstash.o… --- ## [Manage Logstash using API](https://discuss.elastic.co/t/manage-logstash-using-api/250638)
**Author:** [@pchakour](https://discuss.elastic.co/u/pchakour)\ **Replies:** 0\ **Last updated:** [October 1, 2020, 9:59am UTC](https://discuss.elastic.co/t/manage-logstash-using-api/250638 "2020-10-01T09:59:25Z")
Hello ! I have a little question about Logstash management. I want to know if an API exists on Logstash to create, delete or edit a pipeline ? The ony API that I found in the documentation is dedicated to the Logstash… --- ## [Logstash if \>= causes logstash to crash](https://discuss.elastic.co/t/logstash-if-causes-logstash-to-crash/250547)
**Author:** [@probson](https://discuss.elastic.co/u/probson)\ **Replies:** 3\ **Last updated:** [September 30, 2020, 7:07pm UTC](https://discuss.elastic.co/t/logstash-if-causes-logstash-to-crash/250547 "2020-09-30T19:07:16Z")
Hi, I am on ES 7.9.2, i am writing a logstash parser to add a severity based on the risk level. The risk\_score is between 0-10 and can be to 1 decimal place mutate { convert =\> { "risk\_score" =\> "float" } } if… --- ## [Taking more time to send data to elastic search](https://discuss.elastic.co/t/taking-more-time-to-send-data-to-elastic-search/250602)
**Author:** [@PRASHANTHMATOORI](https://discuss.elastic.co/u/PRASHANTHMATOORI)\ **Replies:** 0\ **Last updated:** [October 1, 2020, 6:41am UTC](https://discuss.elastic.co/t/taking-more-time-to-send-data-to-elastic-search/250602 "2020-10-01T06:41:58Z")
Hi all, Logstash taking more time to send data When I am sending data to elastic search via logstash script. I have 6lakhs records in csv file with 3 fields such as name, date &time, count. Would appreciate if any one h… --- ## [How to Put Elasticsearch data to Oracle Table](https://discuss.elastic.co/t/how-to-put-elasticsearch-data-to-oracle-table/250498)
**Author:** [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Replies:** 7\ **Last updated:** [October 1, 2020, 2:48am UTC](https://discuss.elastic.co/t/how-to-put-elasticsearch-data-to-oracle-table/250498 "2020-10-01T02:48:26Z")
Hi Could you please help me with my requirement of putting Elasticsearch index data into oracle table. Do we have any additional plugin required for logstash to achieve same. ? I am aware of CSV method , but its too t… --- ## [Cluster Server Handling](https://discuss.elastic.co/t/cluster-server-handling/249958)
**Author:** [@Amrita\_Mahule](https://discuss.elastic.co/u/Amrita_Mahule)\ **Replies:** 1\ **Last updated:** [October 1, 2020, 1:45am UTC](https://discuss.elastic.co/t/cluster-server-handling/249958 "2020-10-01T01:45:11Z")
Hi Team, If we have 2 servers which are currently in down state. And we have removed those servers from cluster and added them later. So , data for those servers can be recovered or not?? --- ## [No logs in Kibana](https://discuss.elastic.co/t/no-logs-in-kibana/250555)
**Author:** [@SaraBetty](https://discuss.elastic.co/u/SaraBetty)\ **Replies:** 5\ **Last updated:** [September 30, 2020, 9:37pm UTC](https://discuss.elastic.co/t/no-logs-in-kibana/250555 "2020-09-30T21:37:29Z")
Hello everyone, I performed the beats installation (packetbeat/winlogbeat) on the DCs and the logstash installation on the server. I am not getting the logs in Kibana, though i see the packets (connectivity) between the… --- ## [How to deal with inconsistent dates (day or month or year may be missing)?](https://discuss.elastic.co/t/how-to-deal-with-inconsistent-dates-day-or-month-or-year-may-be-missing/250436)
**Author:** [@wblondel](https://discuss.elastic.co/u/wblondel)\ **Replies:** 4\ **Last updated:** [September 30, 2020, 8:37pm UTC](https://discuss.elastic.co/t/how-to-deal-with-inconsistent-dates-day-or-month-or-year-may-be-missing/250436 "2020-09-30T20:37:57Z")
Hi, First of all, I'm a beginner with the ELK stack, so please don't hurt me if I did stupid things :laughing: I am importing CSV files that have the following structure: "nomprenom";"sexe";"datenaiss";"lieunaiss";"co… --- ## [Duplicaties with Logstash rss plugin](https://discuss.elastic.co/t/duplicaties-with-logstash-rss-plugin/250521)
**Author:** [@Robin020](https://discuss.elastic.co/u/Robin020)\ **Replies:** 2\ **Last updated:** [September 30, 2020, 6:33pm UTC](https://discuss.elastic.co/t/duplicaties-with-logstash-rss-plugin/250521 "2020-09-30T18:33:41Z")
Hello, Currently I am configuring a rss feed with logstash rss plugin. This is my logstash pipeline config: input { rss { url =\> "" interval =\> 7200 tags =\> \["rs… --- ## [SSL with Filebeat not working](https://discuss.elastic.co/t/ssl-with-filebeat-not-working/250520)
**Author:** [@santos1204](https://discuss.elastic.co/u/santos1204)\ **Replies:** 3\ **Last updated:** [September 30, 2020, 5:00pm UTC](https://discuss.elastic.co/t/ssl-with-filebeat-not-working/250520 "2020-09-30T17:00:54Z")
Hi, I'm trying to secure the traffic between filebeats hosts and the logstash server, my logstash config is; input { beats { port =\> 5044 ssl =\> true ssl\_certificate\_authorities =\> \["/etc/filebeat/removed.pem"\] s… --- ## [Defining Source When Using Filebeat](https://discuss.elastic.co/t/defining-source-when-using-filebeat/249877)
**Author:** [@teej](https://discuss.elastic.co/u/teej)\ **Replies:** 2\ **Last updated:** [September 30, 2020, 4:32pm UTC](https://discuss.elastic.co/t/defining-source-when-using-filebeat/249877 "2020-09-30T16:32:08Z")
Hi, I am using filebeat to read xml files from a directory and then feed it to logstash where it is filetered then sent out to elastic. I thought that perhaps when the pipeline gets created in the logstash config, it do… --- ## [Logstash not able to fetch messages from Amazon MSK](https://discuss.elastic.co/t/logstash-not-able-to-fetch-messages-from-amazon-msk/250538)
**Author:** [@Akhil96](https://discuss.elastic.co/u/Akhil96)\ **Replies:** 0\ **Last updated:** [September 30, 2020, 4:20pm UTC](https://discuss.elastic.co/t/logstash-not-able-to-fetch-messages-from-amazon-msk/250538 "2020-09-30T16:20:36Z")
I am trying to consume from an MSK cluster using logstash and push the messages to an Elasticsearch instance. Due to some reason the data is not getting fetched correctly, and getting below message in logs. Skipping fet… --- ## [Logstash parsing in JSON](https://discuss.elastic.co/t/logstash-parsing-in-json/249659)
**Author:** [@Sivajanani](https://discuss.elastic.co/u/Sivajanani)\ **Replies:** 11\ **Last updated:** [September 30, 2020, 3:14pm UTC](https://discuss.elastic.co/t/logstash-parsing-in-json/249659 "2020-09-30T15:14:56Z")
Hello, I have a json in the following format. {"group" : \[{"name":"group1","tag":\[{"name":"tag1"},{"name":"tag2"},{"name":"tag3"},{"name":"tag4"}\]}\],"association" : } Using Logstash config I have to parse tag field an… --- ## [Update indexed docs based on a value from a new doc still does not indexed](https://discuss.elastic.co/t/update-indexed-docs-based-on-a-value-from-a-new-doc-still-does-not-indexed/250478)
**Author:** [@Marcelo\_Lavor](https://discuss.elastic.co/u/Marcelo_Lavor)\ **Replies:** 1\ **Last updated:** [September 30, 2020, 1:11pm UTC](https://discuss.elastic.co/t/update-indexed-docs-based-on-a-value-from-a-new-doc-still-does-not-indexed/250478 "2020-09-30T13:11:56Z")
Hello! I need to update several docs into an old index based on a value of new records still do not indexed? Is there any way to implement it with Logstash? This table below describes my index with old docs. The yellow … --- ## [Iterate parsing json in Logstash](https://discuss.elastic.co/t/iterate-parsing-json-in-logstash/249749)
**Author:** [@Sivajanani](https://discuss.elastic.co/u/Sivajanani)\ **Replies:** 11\ **Last updated:** [September 30, 2020, 1:02pm UTC](https://discuss.elastic.co/t/iterate-parsing-json-in-logstash/249749 "2020-09-30T13:02:16Z")
Hello All, My Json field looks like below, {"group" : \[{"name":"group1","tag":\[{"name":"tag1"},{"name":"tag2"},{"name":"tag3"},{"name":"tag4"}\]}\],"association" : } And my config file having mutate filter is as follows… --- ## [Logstash index format interpreted literally](https://discuss.elastic.co/t/logstash-index-format-interpreted-literally/250461)
**Author:** [@vinci](https://discuss.elastic.co/u/vinci)\ **Replies:** 3\ **Last updated:** [September 30, 2020, 12:41pm UTC](https://discuss.elastic.co/t/logstash-index-format-interpreted-literally/250461 "2020-09-30T12:41:02Z")
Hello, I've created several indices with logstash using the following settings: output { stdout { codec =\> dots } elasticsearch { hosts =\> \["http://192.168.99.18… --- ## [Logstash.conf filter doesn't work](https://discuss.elastic.co/t/logstash-conf-filter-doesnt-work/250091)
**Author:** [@zen.xen](https://discuss.elastic.co/u/zen.xen)\ **Replies:** 14\ **Last updated:** [September 30, 2020, 9:58am UTC](https://discuss.elastic.co/t/logstash-conf-filter-doesnt-work/250091 "2020-09-30T09:58:03Z")
Hello, I want to send events from my .log file with filebeat. I have created such filter but it doesn't work. input { beats { port =\> "5044" } } filter { grok { match =\> { "message" =\> "(%{TIMESTA… --- ## [Is a loadbalancer required before logstash, to maintain high availability?](https://discuss.elastic.co/t/is-a-loadbalancer-required-before-logstash-to-maintain-high-availability/250341)
**Author:** [@Pooja](https://discuss.elastic.co/u/Pooja)\ **Replies:** 2\ **Last updated:** [September 30, 2020, 5:58am UTC](https://discuss.elastic.co/t/is-a-loadbalancer-required-before-logstash-to-maintain-high-availability/250341 "2020-09-30T05:58:43Z")
If I am reading data from multiple OS, and flavours of OS, network devices, around 60 different log sources altogether. Do I need a load balancer or does elastic have a loadbalancer in bulit? FYI, I do not want to use Fi… --- ## [Use kv filter on newline & return split pattern](https://discuss.elastic.co/t/use-kv-filter-on-newline-return-split-pattern/249655)
**Author:** [@ParashB](https://discuss.elastic.co/u/ParashB)\ **Replies:** 2\ **Last updated:** [September 30, 2020, 5:39am UTC](https://discuss.elastic.co/t/use-kv-filter-on-newline-return-split-pattern/249655 "2020-09-30T05:39:56Z")
Hi, I am trying to use kv filter to parse the below logs Name: abc\\r\\nAge: 39\\r\\nGender: Male\\r\\nSalary: 87 My kv filter is as below kv { source =\> "log" field\_split\_pattern =\> "\\\\\\r\\\\\\n" value\_split =\> ":" } I h… --- ## [How can make a cluster with Logstash? Is it possible?!?](https://discuss.elastic.co/t/how-can-make-a-cluster-with-logstash-is-it-possible/250444)
**Author:** [@Farid\_N](https://discuss.elastic.co/u/Farid_N)\ **Replies:** 2\ **Last updated:** [September 30, 2020, 5:15am UTC](https://discuss.elastic.co/t/how-can-make-a-cluster-with-logstash-is-it-possible/250444 "2020-09-30T05:15:22Z")
Hi I have 3 nodes for logstash. each logstash has it own pipeline (for example, one logstash has http module and some metricfiles. another one has two jdbc driver and some filebeats and....) So, I can\`t use file and h… --- ## [Logstash http plugin 'threads'](https://discuss.elastic.co/t/logstash-http-plugin-threads/250310)
**Author:** [@stsm](https://discuss.elastic.co/u/stsm)\ **Replies:** 2\ **Last updated:** [September 30, 2020, 4:59am UTC](https://discuss.elastic.co/t/logstash-http-plugin-threads/250310 "2020-09-30T04:59:50Z")
Hi there. I'm playing around with the "http" Logstash plugin. I ask myself, how should 'threads' be calculate? (As far as i remember, i read somewhere that the plugin depends on the "puma server") Can anyone give me a … --- ## [Logstash 7.9.2 service will not start on Ubuntu 18.04](https://discuss.elastic.co/t/logstash-7-9-2-service-will-not-start-on-ubuntu-18-04/250270)
**Author:** [@Xeny](https://discuss.elastic.co/u/Xeny)\ **Replies:** 4\ **Last updated:** [September 30, 2020, 4:09am UTC](https://discuss.elastic.co/t/logstash-7-9-2-service-will-not-start-on-ubuntu-18-04/250270 "2020-09-30T04:09:20Z")
Hello, I have a new installation on Ubuntu 18.04 where I have successfully installed "elasticsearch and kibana". However, after installing logstach and attempting to start it, it continues to fail with the following m… --- ## [Pass Envt Variables to Logstash config file](https://discuss.elastic.co/t/pass-envt-variables-to-logstash-config-file/250375)
**Author:** [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Replies:** 4\ **Last updated:** [September 30, 2020, 3:46am UTC](https://discuss.elastic.co/t/pass-envt-variables-to-logstash-config-file/250375 "2020-09-30T03:46:18Z")
Hello Team I am having Logstash v7.8.0 installed along with Elasticsearch v7.8.0. My Requirement is to run multiple logstash instances using pipelines via individual "conf" files. These conf files , are having va… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=297) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=299)