# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=902 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 903 --- ## [How to setup Kafka input in Logstash ? i have ELK setup and kafka installed in windows ? How can we create a logstash.config by mentioning kafka as input?](https://discuss.elastic.co/t/how-to-setup-kafka-input-in-logstash-i-have-elk-setup-and-kafka-installed-in-windows-how-can-we-create-a-logstash-config-by-mentioning-kafka-as-input/101937)
**Author:** [@manu\_john](https://discuss.elastic.co/u/manu_john)\ **Replies:** 2\ **Last updated:** [September 27, 2017, 8:56am UTC](https://discuss.elastic.co/t/how-to-setup-kafka-input-in-logstash-i-have-elk-setup-and-kafka-installed-in-windows-how-can-we-create-a-logstash-config-by-mentioning-kafka-as-input/101937 "2017-09-27T08:56:30Z")
How to setup Kafka input in Logstash ? i have ELK setup and kafka installed in windows ? How can we create a logstash.config by mentioning kafka as input? I just installed ELK and kafka in my windows system. My requirem… --- ## [Grok pattern with regex](https://discuss.elastic.co/t/grok-pattern-with-regex/101938)
**Author:** [@lois](https://discuss.elastic.co/u/lois)\ **Replies:** 0\ **Last updated:** [September 27, 2017, 7:21am UTC](https://discuss.elastic.co/t/grok-pattern-with-regex/101938 "2017-09-27T07:21:24Z")
Hi, I am trying to match a string using a grok pattern with regex. An example string is: (1) 12.3 abc Chrome/4.56 OPR/7.89 In this case I want my regex to match OPR. In case the string looks like this: (2) 12.3 abc… --- ## [Issue installing Translate Plugin](https://discuss.elastic.co/t/issue-installing-translate-plugin/101816)
**Author:** [@sreejiths](https://discuss.elastic.co/u/sreejiths)\ **Replies:** 5\ **Last updated:** [September 27, 2017, 6:28am UTC](https://discuss.elastic.co/t/issue-installing-translate-plugin/101816 "2017-09-27T06:28:18Z")
We are using logstash 5.2 ..We are trying to install translate plugin and getting below error ..Can anyone advice \[root@x01gnwatlgs1a support\]# /usr/share/logstash/bin/logstash-plugin install file:////home/support/logst… --- ## [Failed to install template error](https://discuss.elastic.co/t/failed-to-install-template-error/101888)
**Author:** [@Zama\_Las](https://discuss.elastic.co/u/Zama_Las)\ **Replies:** 5\ **Last updated:** [September 27, 2017, 6:06am UTC](https://discuss.elastic.co/t/failed-to-install-template-error/101888 "2017-09-27T06:06:35Z")
Hi , I just started working on setting up the ELK stack using docker. I was able to start all the three components successfully using docker install . But I can see one error message during startup of logstash . The e… --- ## [Netflow Template length doesn't fit cleanly into flowset](https://discuss.elastic.co/t/netflow-template-length-doesnt-fit-cleanly-into-flowset/101698)
**Author:** [@lumi](https://discuss.elastic.co/u/lumi)\ **Replies:** 4\ **Last updated:** [September 27, 2017, 5:59am UTC](https://discuss.elastic.co/t/netflow-template-length-doesnt-fit-cleanly-into-flowset/101698 "2017-09-27T05:59:01Z")
Hello I have got a fortigate. It should send Netflow Data to my elk stack. But i alway get this message: \[2017-09-25T14:02:53,313\]\[WARN \]\[logstash.codecs.netflow \] Template length doesn't fit cleanly into flowset {:t… --- ## [Get notification once data indexing is completed using logstash](https://discuss.elastic.co/t/get-notification-once-data-indexing-is-completed-using-logstash/101893)
**Author:** [@surajbs](https://discuss.elastic.co/u/surajbs)\ **Replies:** 3\ **Last updated:** [September 27, 2017, 5:25am UTC](https://discuss.elastic.co/t/get-notification-once-data-indexing-is-completed-using-logstash/101893 "2017-09-27T05:25:27Z")
HI, I am trying to get notification once data indexing is completed, so that I can add email plugin to send an email that data indexing is completed. So I would like to know if logstash log somewhere that data indexing… --- ## [5.6.1 Netflow not working](https://discuss.elastic.co/t/5-6-1-netflow-not-working/101923)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [September 27, 2017, 4:29am UTC](https://discuss.elastic.co/t/5-6-1-netflow-not-working/101923 "2017-09-27T04:29:05Z")
Hi, I've updated from 5.4 to 5.6.1 so I could use the latest netflow codec with support for Fortigate devices but now I got no netflow data coming in anymore. I did not change the config and logstash log shows no erro… --- ## [Conditionals in Grok filter for more than one log type](https://discuss.elastic.co/t/conditionals-in-grok-filter-for-more-than-one-log-type/101656)
**Author:** [@akarsha](https://discuss.elastic.co/u/akarsha)\ **Replies:** 11\ **Last updated:** [September 27, 2017, 2:31am UTC](https://discuss.elastic.co/t/conditionals-in-grok-filter-for-more-than-one-log-type/101656 "2017-09-27T02:31:40Z")
I am using grok filter for parsing my log file by logstash. I am not getting any error, stll not getting expected results in kibana. my config file looks like below: My config file is: filter { if \[type\] == "sdp" { i… --- ## [Logstash 5.6 Persistent Queue issue (Docker)](https://discuss.elastic.co/t/logstash-5-6-persistent-queue-issue-docker/101162)
**Author:** [@godson2006](https://discuss.elastic.co/u/godson2006)\ **Replies:** 2\ **Last updated:** [September 26, 2017, 11:57pm UTC](https://discuss.elastic.co/t/logstash-5-6-persistent-queue-issue-docker/101162 "2017-09-26T23:57:30Z")
Hello, I currently have logstash running within a docker container and when I have it configured for persistent queue it works as it should. The only issue is when I mount a volume to where the persistent queue director… --- ## [Logstash to exit or stop once the data indexing is completed](https://discuss.elastic.co/t/logstash-to-exit-or-stop-once-the-data-indexing-is-completed/101755)
**Author:** [@surajbs](https://discuss.elastic.co/u/surajbs)\ **Replies:** 3\ **Last updated:** [September 26, 2017, 10:58pm UTC](https://discuss.elastic.co/t/logstash-to-exit-or-stop-once-the-data-indexing-is-completed/101755 "2017-09-26T22:58:22Z")
Hi, I am trying to get a solution for logstash to stop after the data indexing gets completed. Or, logstash should stop reading the file once the job is done (command prompt should get stop and exit). Please share you … --- ## [Logstash on low end ARM devices?](https://discuss.elastic.co/t/logstash-on-low-end-arm-devices/101634)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 3\ **Last updated:** [September 26, 2017, 10:50pm UTC](https://discuss.elastic.co/t/logstash-on-low-end-arm-devices/101634 "2017-09-26T22:50:11Z")
Hi, I've searched for Logstash minimum system requirements but I couldn't find anything. Is it possible to run logstash on something like a Raspberry pi or a 600Mhz 512mb ram ARM device? In my case it would only proce… --- ## [Can Anyone tell each and every step of how to use geoIP with logstash and kibana?](https://discuss.elastic.co/t/can-anyone-tell-each-and-every-step-of-how-to-use-geoip-with-logstash-and-kibana/101420)
**Author:** [@PinkiMondal](https://discuss.elastic.co/u/PinkiMondal)\ **Replies:** 11\ **Last updated:** [September 26, 2017, 10:49pm UTC](https://discuss.elastic.co/t/can-anyone-tell-each-and-every-step-of-how-to-use-geoip-with-logstash-and-kibana/101420 "2017-09-26T22:49:07Z")
Hi All I have GeoLite2-City.mmdb and logstash ver 5.4.0. I am trying to use the db but nothing is working out. Can anyone please explain the steps needed to do the loading of IP address in Logstash and transformed it … --- ## [Using logstash for HTML file?](https://discuss.elastic.co/t/using-logstash-for-html-file/101906)
**Author:** [@magdieldominguez](https://discuss.elastic.co/u/magdieldominguez)\ **Replies:** 0\ **Last updated:** [September 26, 2017, 9:15pm UTC](https://discuss.elastic.co/t/using-logstash-for-html-file/101906 "2017-09-26T21:15:05Z")
Hi im very new to the ELK stack. Ive managed to get kibana, elasticsearch, logstash, and filebeat running together just from the elasticsearch tutorials. There is an html page that I downloaded which contains log data th… --- ## [Logstash elastic search output with very big JSON is not outputting](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632)
**Author:** [@elasticuser1](https://discuss.elastic.co/u/elasticuser1)\ **Replies:** 12\ **Last updated:** [September 26, 2017, 5:39pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632 "2017-09-26T17:39:56Z")
Hi, I have very big nested json (22KB) which I am trying to output to elastic search but it's not outputting. Output to file works as expected. I tried with small json which works. Is there any limitation on json filter… --- ## [How does sql\_last\_value parameter work in jdbc input plugin?](https://discuss.elastic.co/t/how-does-sql-last-value-parameter-work-in-jdbc-input-plugin/101624)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 8\ **Last updated:** [September 26, 2017, 3:55pm UTC](https://discuss.elastic.co/t/how-does-sql-last-value-parameter-work-in-jdbc-input-plugin/101624 "2017-09-26T15:55:30Z")
Hi, I am using JDBC input plugin with use\_column\_value =\> true tracking\_column =\> "id". But the problem is that my id column is not an incremental all the time. For example it may be as 1,2,3,5,6,7,10,9,8. So what do… --- ## [Logstash monitoring not working properly](https://discuss.elastic.co/t/logstash-monitoring-not-working-properly/101874)
**Author:** [@Mr.Flakes](https://discuss.elastic.co/u/Mr.Flakes)\ **Replies:** 8\ **Last updated:** [September 26, 2017, 3:39pm UTC](https://discuss.elastic.co/t/logstash-monitoring-not-working-properly/101874 "2017-09-26T15:39:11Z")
Hello, I am not sure which category I should use for this thread but since it is monitoring related I will use X-Pack. So I have 4 Logstash 5.6.1 instances running and 5 Elasticsearch 5.6.1 nodes all running X-Pack 5.6.… --- ## [File output, disk full: messages lost?](https://discuss.elastic.co/t/file-output-disk-full-messages-lost/101869)
**Author:** [@kunricsi\_dh](https://discuss.elastic.co/u/kunricsi_dh)\ **Replies:** 0\ **Last updated:** [September 26, 2017, 2:43pm UTC](https://discuss.elastic.co/t/file-output-disk-full-messages-lost/101869 "2017-09-26T14:43:44Z")
Hi there, I would like to save every message into a single textfile with file output. (filters will be applied later). I have tested the behaviour of logstash in the case of the disk full, and I was surprised: logstash … --- ## [Logstash-plugin list --group apparently ignoring some plugins](https://discuss.elastic.co/t/logstash-plugin-list-group-apparently-ignoring-some-plugins/101854)
**Author:** [@mrec](https://discuss.elastic.co/u/mrec)\ **Replies:** 0\ **Last updated:** [September 26, 2017, 1:53pm UTC](https://discuss.elastic.co/t/logstash-plugin-list-group-apparently-ignoring-some-plugins/101854 "2017-09-26T13:53:21Z")
Newbie, working through https://www.elastic.co/guide/en/logstash/5.6/working-with-plugins.html The output for bin/logstash-plugin list includes: logstash-input-beats logstash-input-couchdb\_changes logstash-input-dead\_l… --- ## [Regex to break from specific word or character in a log line](https://discuss.elastic.co/t/regex-to-break-from-specific-word-or-character-in-a-log-line/101840)
**Author:** [@tharu85](https://discuss.elastic.co/u/tharu85)\ **Replies:** 1\ **Last updated:** [September 26, 2017, 1:43pm UTC](https://discuss.elastic.co/t/regex-to-break-from-specific-word-or-character-in-a-log-line/101840 "2017-09-26T13:43:33Z")
I want to split below log line "APP-DETECT TeamViewer remote administration tool outbound connection attempt" \[Impact: Potentially Vulnerable\] From ABC As like name : APP-DETECT TeamViewer remote administration tool o… --- ## [Json parse](https://discuss.elastic.co/t/json-parse/101701)
**Author:** [@neo\_levi](https://discuss.elastic.co/u/neo_levi)\ **Replies:** 4\ **Last updated:** [September 26, 2017, 1:40pm UTC](https://discuss.elastic.co/t/json-parse/101701 "2017-09-26T13:40:19Z")
hi all, i want to parse my massage , it look like this below so the columnName value will be the field name and ColumnValue will be the value of the new field. my code looks like :slight\_smile: input { jdbc { jdbc… --- ## [\[X-Pack\] Logstash PKI client authentication not working](https://discuss.elastic.co/t/x-pack-logstash-pki-client-authentication-not-working/101670)
**Author:** [@Bozzerreaneaux](https://discuss.elastic.co/u/Bozzerreaneaux)\ **Replies:** 1\ **Last updated:** [September 26, 2017, 1:11pm UTC](https://discuss.elastic.co/t/x-pack-logstash-pki-client-authentication-not-working/101670 "2017-09-26T13:11:50Z")
Hello, I am struggling with getting Logstash to present a client certificate to ES instance. After couple discussions and misunderstandings from my side I realized that X-Pack supports client certificates through the ke… --- ## [Issue parcing csv file using Input csv plugin - ELK 5.5.0](https://discuss.elastic.co/t/issue-parcing-csv-file-using-input-csv-plugin-elk-5-5-0/101739)
**Author:** [@elkuserh](https://discuss.elastic.co/u/elkuserh)\ **Replies:** 5\ **Last updated:** [September 26, 2017, 1:08pm UTC](https://discuss.elastic.co/t/issue-parcing-csv-file-using-input-csv-plugin-elk-5-5-0/101739 "2017-09-26T13:08:42Z")
I am trying to breakdown a csv file with incident data using the csv input filter and have run into the following error. I had used a multiline codec to combine multiple lines into a single line (multiple lines within a … --- ## [Conditional mutate not working](https://discuss.elastic.co/t/conditional-mutate-not-working/101821)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 2\ **Last updated:** [September 26, 2017, 10:35am UTC](https://discuss.elastic.co/t/conditional-mutate-not-working/101821 "2017-09-26T10:35:15Z")
I am logging to more than one ports using syslog plugin. However the following conditional filter does not work (although logs are received in more than one port as expected. Any suggestions? input { tcp { port … --- ## [Max\_partition\_fetch\_bytes not work](https://discuss.elastic.co/t/max-partition-fetch-bytes-not-work/101814)
**Author:** [@3629decca366f3ccc233](https://discuss.elastic.co/u/3629decca366f3ccc233)\ **Replies:** 0\ **Last updated:** [September 26, 2017, 9:44am UTC](https://discuss.elastic.co/t/max-partition-fetch-bytes-not-work/101814 "2017-09-26T09:44:37Z")
max\_partition\_fetch\_bytes not work ， i have big message in kafka (abount 150000b)。 i want to limit to 500, and print it . i find max\_partition\_fetch\_bytes config ,but it not work, it print is also big message config i… --- ## [Help converting the Date format in Logstash - 5.5.0](https://discuss.elastic.co/t/help-converting-the-date-format-in-logstash-5-5-0/101745)
**Author:** [@elkuserh](https://discuss.elastic.co/u/elkuserh)\ **Replies:** 1\ **Last updated:** [September 26, 2017, 9:30am UTC](https://discuss.elastic.co/t/help-converting-the-date-format-in-logstash-5-5-0/101745 "2017-09-26T09:30:21Z")
My input data has a date filed which has values such as this :slight\_smile: 2017-01-09-11.23.08.000000 2017-04-10-22.10.11.000000 So I went with the following date filter: date{ match =\> \[ "CREATEDTTM", "yyyy-mm-dd-H… --- ## [Interesting logstash behavior](https://discuss.elastic.co/t/interesting-logstash-behavior/101803)
**Author:** [@Marco\_Spadoni](https://discuss.elastic.co/u/Marco_Spadoni)\ **Replies:** 1\ **Last updated:** [September 26, 2017, 8:51am UTC](https://discuss.elastic.co/t/interesting-logstash-behavior/101803 "2017-09-26T08:51:58Z")
I am running version 5.5.2 stack of logstash and elasticsearch in a hot-warm architecture. To free heap space on elastic running on warm nodes, oldest indices are closed. The logstash pipeline reads messages from Kafka,… --- ## [Aggregate filter](https://discuss.elastic.co/t/aggregate-filter/101800)
**Author:** [@Cristi\_Iaroi](https://discuss.elastic.co/u/Cristi_Iaroi)\ **Replies:** 0\ **Last updated:** [September 26, 2017, 8:17am UTC](https://discuss.elastic.co/t/aggregate-filter/101800 "2017-09-26T08:17:32Z")
Hi, I'm using the latest aggregate plugin filter with the latest logstash version and for some reason I'm having some issues with indexing it to elasticsearch (latest version as well). this is the logstash config file … --- ## [\[solved\]Configuration input http\_poller + TCP](https://discuss.elastic.co/t/solved-configuration-input-http-poller-tcp/101118)
**Author:** [@sparis](https://discuss.elastic.co/u/sparis)\ **Replies:** 2\ **Last updated:** [September 26, 2017, 7:04am UTC](https://discuss.elastic.co/t/solved-configuration-input-http-poller-tcp/101118 "2017-09-26T07:04:45Z")
Hello, I have two files .conf. First have 1 input jdbc to taking data from mysql bdd / no filter / output TCP to logstash. With this data, i build url for API. ex : https://nomapi/var1/var2/var3 This .conf is ok. in… --- ## [Using two kafka in the ELK setup(Load balancing)](https://discuss.elastic.co/t/using-two-kafka-in-the-elk-setup-load-balancing/101646)
**Author:** [@Jay\_Dabhi](https://discuss.elastic.co/u/Jay_Dabhi)\ **Replies:** 3\ **Last updated:** [September 26, 2017, 6:37am UTC](https://discuss.elastic.co/t/using-two-kafka-in-the-elk-setup-load-balancing/101646 "2017-09-26T06:37:40Z")
Hi, I am designing the ELK setup.I want the two kafka setup for the load balancing. My setup will be look like below: filebeat-\>kafka(2 servers)-\>logstash(3 servers)-\>ES-\>Kibana I want to kafka server so that the log… --- ## [Variable value is not accesible in nested if condition](https://discuss.elastic.co/t/variable-value-is-not-accesible-in-nested-if-condition/101779)
**Author:** [@raja\_gopal](https://discuss.elastic.co/u/raja_gopal)\ **Replies:** 5\ **Last updated:** [September 26, 2017, 6:06am UTC](https://discuss.elastic.co/t/variable-value-is-not-accesible-in-nested-if-condition/101779 "2017-09-26T06:06:59Z")
In a nested if condition like if \[type\] == "tcp"{ if \[parsed\_message\]\[NotificationType\] == "NOTIFY" The if block inside "tcp" is not working since it is not getting the value of "\[parsed\_message\]\[NotificationType\]" bu… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=901) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=903)