# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=789 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 790 --- ## [The logstash(indexer) can not get logs from kafka](https://discuss.elastic.co/t/the-logstash-indexer-can-not-get-logs-from-kafka/126432)
**Author:** [@CISSY-Lu](https://discuss.elastic.co/u/CISSY-Lu)\ **Replies:** 0\ **Last updated:** [April 2, 2018, 1:00pm UTC](https://discuss.elastic.co/t/the-logstash-indexer-can-not-get-logs-from-kafka/126432 "2018-04-02T13:00:01Z")
logstash(agent)--\>kafka---\>logstash(indexer)--\>ES logstash(agent) collected logs and send them to kafka was good.but the logstash(indexer) can't get the logs from kafka to ES.and there is no error logs.how could i sol… --- ## [Twitter Input: Controlling Fields to Ingest](https://discuss.elastic.co/t/twitter-input-controlling-fields-to-ingest/125844)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 1\ **Last updated:** [April 2, 2018, 9:50am UTC](https://discuss.elastic.co/t/twitter-input-controlling-fields-to-ingest/125844 "2018-04-02T09:50:27Z")
The Twitter input will create well over 1k fields if left to its own devices. I have a mutate in my pipeline to remove a few hundred, unfortunately, it's kind of a resource hog to do it this way. Is there another, more… --- ## [Autodetect column is taking data row as column name instead of header](https://discuss.elastic.co/t/autodetect-column-is-taking-data-row-as-column-name-instead-of-header/126383)
**Author:** [@ysl](https://discuss.elastic.co/u/ysl)\ **Replies:** 0\ **Last updated:** [April 2, 2018, 7:38am UTC](https://discuss.elastic.co/t/autodetect-column-is-taking-data-row-as-column-name-instead-of-header/126383 "2018-04-02T07:38:00Z")
Hello, I am running Logstash from command line with the config file below. I am not getting the first row header but the record (second row data after header) has appeared as column names. Please help. Thanks input { f… --- ## [Grok!](https://discuss.elastic.co/t/grok/126362)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 4\ **Last updated:** [April 2, 2018, 6:11am UTC](https://discuss.elastic.co/t/grok/126362 "2018-04-02T06:11:44Z")
So I understand grok patterns to be regex queries (Like DNS names for IP addresses). I am trying to build some patterns for the following log entries. New Node: 212.55.78.183 Issue: Semalt Project Spam Bot Network: LL… --- ## [Can't install coustom plugins](https://discuss.elastic.co/t/cant-install-coustom-plugins/126180)
**Author:** [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Replies:** 4\ **Last updated:** [April 2, 2018, 4:50am UTC](https://discuss.elastic.co/t/cant-install-coustom-plugins/126180 "2018-04-02T04:50:27Z")
output error. \[root@soc logstash-5.6.8\]# bin/logstash-plugin install logstash-filter-ip2location-1.0.2.gem Validating logstash-filter-ip2location-1.0.2.gem Installing logstash-filter-ip2location Error Bundler::InstallE… --- ## [Evaluate Multiple Strings in IF Statement](https://discuss.elastic.co/t/evaluate-multiple-strings-in-if-statement/126355)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 8\ **Last updated:** [April 2, 2018, 1:24am UTC](https://discuss.elastic.co/t/evaluate-multiple-strings-in-if-statement/126355 "2018-04-02T01:24:10Z")
I have an IF statement like below: if "tobem" in \[user\]\[url\] or "foocrypt" in \[user\]\[url\] or "theautomatski" in \[user\]\[url\] I also imagine that this is going to get a bit larger as time goes on and I identify more user… --- ## [S3 input taking very long time to start](https://discuss.elastic.co/t/s3-input-taking-very-long-time-to-start/126365)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 0\ **Last updated:** [April 2, 2018, 1:22am UTC](https://discuss.elastic.co/t/s3-input-taking-very-long-time-to-start/126365 "2018-04-02T01:22:19Z")
I noticed that when I start Logstash with just an s3 input, it takes a super long time to do anything. I don't get any errors, and I don't see any specific logs that tell me anything like "connected to s3", but I do see … --- ## [Convert Inline Ruby to Ruby Script](https://discuss.elastic.co/t/convert-inline-ruby-to-ruby-script/126357)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 6\ **Last updated:** [April 2, 2018, 12:29am UTC](https://discuss.elastic.co/t/convert-inline-ruby-to-ruby-script/126357 "2018-04-02T00:29:36Z")
I was given some inline ruby code to do some work for me. I've put it to good use but it makes my pipeline very messy and had a couple questions. First, here's what this part of the pipe looks like right now: if \[ex… --- ## [Logstash: No data in kibana](https://discuss.elastic.co/t/logstash-no-data-in-kibana/126339)
**Author:** [@syfy323](https://discuss.elastic.co/u/syfy323)\ **Replies:** 2\ **Last updated:** [April 1, 2018, 7:50am UTC](https://discuss.elastic.co/t/logstash-no-data-in-kibana/126339 "2018-04-01T07:50:50Z")
Hi! I switched syslog from Graylog to ELK (same port) but I am unable to push data to ES / Kibana. Filebeat and Packetbeat are working fine but syslog and netflow still have issues. \[2018-03-31T22:22:38,565\]\[INFO \]\[l… --- ## [Logstash parsing same contents again and again](https://discuss.elastic.co/t/logstash-parsing-same-contents-again-and-again/126306)
**Author:** [@xsallowed](https://discuss.elastic.co/u/xsallowed)\ **Replies:** 10\ **Last updated:** [April 1, 2018, 4:46am UTC](https://discuss.elastic.co/t/logstash-parsing-same-contents-again-and-again/126306 "2018-04-01T04:46:15Z")
Hi All, I am having a weird problem. Logstash is parsing and indexing same contents of a file over and again after a certain time period. Sincedb\_path is note created. and if i let logstash run for days it the number of… --- ## [How i to parse log](https://discuss.elastic.co/t/how-i-to-parse-log/126310)
**Author:** [@edwinhsieh](https://discuss.elastic.co/u/edwinhsieh)\ **Replies:** 4\ **Last updated:** [April 1, 2018, 4:02am UTC](https://discuss.elastic.co/t/how-i-to-parse-log/126310 "2018-04-01T04:02:10Z")
Hi everyone, I'm a novice to handle log with grok pattern on logstash. I have a log like that: \[32mINFO \\u001b\[0m\[03-31|09:09:27|print/print.go:20\] fire a hole in the room, give you a burst way \\tprint=testing tickat… --- ## [Filebeat unable to dial tcp on port logstash port 5044](https://discuss.elastic.co/t/filebeat-unable-to-dial-tcp-on-port-logstash-port-5044/126324)
**Author:** [@Chandan\_Mishra](https://discuss.elastic.co/u/Chandan_Mishra)\ **Replies:** 2\ **Last updated:** [April 1, 2018, 3:16am UTC](https://discuss.elastic.co/t/filebeat-unable-to-dial-tcp-on-port-logstash-port-5044/126324 "2018-04-01T03:16:10Z")
I am running logstash on remote Centos 6.8 machine. While sending logs via filebeat getting error ERROR pipeline/output.go:74 Failed to connect: dial tcp 192.168.180.72:5044: i/o timeout Tried telnet from filebeat … --- ## [Setting "xpack.monitoring.enabled" hasn't been registered](https://discuss.elastic.co/t/setting-xpack-monitoring-enabled-hasnt-been-registered/126345)
**Author:** [@paano](https://discuss.elastic.co/u/paano)\ **Replies:** 0\ **Last updated:** [April 1, 2018, 3:07am UTC](https://discuss.elastic.co/t/setting-xpack-monitoring-enabled-hasnt-been-registered/126345 "2018-04-01T03:07:51Z")
I have been struggling to get xpack working with logstash. When disabled it works. logstash version is 6.2.3. error i see is: \[2018-03-31T22:02:18,031\]\[INFO \]\[logstash.modules.scaffold\] Initializing module {:module\_na… --- ## [Logstash filter that supports several lines](https://discuss.elastic.co/t/logstash-filter-that-supports-several-lines/124711)
**Author:** [@solo](https://discuss.elastic.co/u/solo)\ **Replies:** 1\ **Last updated:** [March 31, 2018, 10:05pm UTC](https://discuss.elastic.co/t/logstash-filter-that-supports-several-lines/124711 "2018-03-31T22:05:05Z")
Hi. I've created a filter that takes in JBoss fuse logs, but it has problems when I get a stack trace or other information that takes more than one line. My filter: input { beats { port =\> "5… --- ## [Logstash 6.2.2 unable to start](https://discuss.elastic.co/t/logstash-6-2-2-unable-to-start/124510)
**Author:** [@Dvikas](https://discuss.elastic.co/u/Dvikas)\ **Replies:** 1\ **Last updated:** [March 31, 2018, 9:56pm UTC](https://discuss.elastic.co/t/logstash-6-2-2-unable-to-start/124510 "2018-03-31T21:56:25Z")
I got the following error when i started logstash. error no 1=\>\[ERROR\]\[logstash.pipeline \] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"#\ **Author:** [@mladen](https://discuss.elastic.co/u/mladen)\ **Replies:** 3\ **Last updated:** [March 31, 2018, 9:28pm UTC](https://discuss.elastic.co/t/parsing-ibm-integration-bus-logs-with-grok/126336 "2018-03-31T21:28:03Z") Hello, searching the internet I didn't find the right grok filter for IBM integration bus logs. At the end I decided to try by may self. Filter is working but I have opinion that it can be better. Below you can find my … --- ## [Date Filter Plugin Timezone documentation misleading](https://discuss.elastic.co/t/date-filter-plugin-timezone-documentation-misleading/126299)
**Author:** [@Steven\_Ensslen](https://discuss.elastic.co/u/Steven_Ensslen)\ **Replies:** 1\ **Last updated:** [March 31, 2018, 6:59pm UTC](https://discuss.elastic.co/t/date-filter-plugin-timezone-documentation-misleading/126299 "2018-03-31T18:59:48Z")
The timezone documentation says that the timezone is optional and has no default. Both of these statements are misleading. It is true that the filter runs without specifying the timezone option. What happens in that c… --- ## [That would be my first topic generally about elk stack](https://discuss.elastic.co/t/that-would-be-my-first-topic-generally-about-elk-stack/126326)
**Author:** [@otpisani](https://discuss.elastic.co/u/otpisani)\ **Replies:** 1\ **Last updated:** [March 31, 2018, 6:36pm UTC](https://discuss.elastic.co/t/that-would-be-my-first-topic-generally-about-elk-stack/126326 "2018-03-31T18:36:02Z")
Hello, Don't know if I choose right category but need some answers about elk stack: Can I find online elk stack emulator for config testing How much memory ELK stack will drain for small network monitoring suricata lo… --- ## [How do I encrypt multiple fields?](https://discuss.elastic.co/t/how-do-i-encrypt-multiple-fields/126153)
**Author:** [@wakame](https://discuss.elastic.co/u/wakame)\ **Replies:** 3\ **Last updated:** [March 31, 2018, 6:12pm UTC](https://discuss.elastic.co/t/how-do-i-encrypt-multiple-fields/126153 "2018-03-31T18:12:27Z")
how do I encrypt multiple fields? cipher{ source =\> "field1" target =\> "field1" } cipher{ source =\> "field2" target =\> "field2" } cipher{ source =\> "field3" target =\> "field3" } --- ## [Querying elastic for very recent log](https://discuss.elastic.co/t/querying-elastic-for-very-recent-log/126127)
**Author:** [@artonge](https://discuss.elastic.co/u/artonge)\ **Replies:** 5\ **Last updated:** [March 31, 2018, 6:55am UTC](https://discuss.elastic.co/t/querying-elastic-for-very-recent-log/126127 "2018-03-31T06:55:43Z")
Hi, I would like to do something similar to the first example of this doc. My problem is that the two logs are indexed almost one after the other. Could this lead to an error when querying for the first log ? I actuall… --- ## [Logstash Consuming high CPU](https://discuss.elastic.co/t/logstash-consuming-high-cpu/123245)
**Author:** [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Replies:** 11\ **Last updated:** [March 31, 2018, 4:49am UTC](https://discuss.elastic.co/t/logstash-consuming-high-cpu/123245 "2018-03-31T04:49:01Z")
Team, Below is my CPU configuration CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 8 NUMA node(s): 1 Vendor ID: Genuine… --- ## [How to Generate logs in json format using .yml file](https://discuss.elastic.co/t/how-to-generate-logs-in-json-format-using-yml-file/124575)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [March 31, 2018, 3:56am UTC](https://discuss.elastic.co/t/how-to-generate-logs-in-json-format-using-yml-file/124575 "2018-03-31T03:56:45Z")
Hi I am uisng LS 5.2.2 version, i am able to generate logs in json format using command line like this /bin\> logstash.bat --debug -f xyz.conf --log.format json i want to generate the json logs through control of logs… --- ## [Codec line not working as expected](https://discuss.elastic.co/t/codec-line-not-working-as-expected/126219)
**Author:** [@ddebast](https://discuss.elastic.co/u/ddebast)\ **Replies:** 1\ **Last updated:** [March 31, 2018, 1:15am UTC](https://discuss.elastic.co/t/codec-line-not-working-as-expected/126219 "2018-03-31T01:15:27Z")
logstash version logstash 5.4.1 I have a beat input that ships logs from a linux host. What I'm receiving: "message" =\> "Mar 30 13:01:34 host su: pam\_unix(su-l:session): session closed for user user", Since I want … --- ## [How to aggregate using name in Logstash Filter Elasticsearch Query](https://discuss.elastic.co/t/how-to-aggregate-using-name-in-logstash-filter-elasticsearch-query/124810)
**Author:** [@rajkamalkool6](https://discuss.elastic.co/u/rajkamalkool6)\ **Replies:** 1\ **Last updated:** [March 30, 2018, 9:28pm UTC](https://discuss.elastic.co/t/how-to-aggregate-using-name-in-logstash-filter-elasticsearch-query/124810 "2018-03-30T21:28:28Z")
I amusing ELK version 5.2.0 and I am trying to aggregate some fields from another index in logstash-filter-elasticsearch using the name or email or userid. Below is my query query =\> "type:user AND (email.keyword:%{use… --- ## [Explanation of string interpolation?](https://discuss.elastic.co/t/explanation-of-string-interpolation/126275)
**Author:** [@Shannon\_Carey](https://discuss.elastic.co/u/Shannon_Carey)\ **Replies:** 1\ **Last updated:** [March 30, 2018, 8:00pm UTC](https://discuss.elastic.co/t/explanation-of-string-interpolation/126275 "2018-03-30T20:00:01Z")
I cannot find where in the docs it explains how string interpolation with %{myvar} works... Is there an explanation? I have an "http" output which uses the "mapping" config to control the structure of the output like so… --- ## [\[Solved\]Mutate not working](https://discuss.elastic.co/t/solved-mutate-not-working/125948)
**Author:** [@wakame](https://discuss.elastic.co/u/wakame)\ **Replies:** 4\ **Last updated:** [March 30, 2018, 7:54pm UTC](https://discuss.elastic.co/t/solved-mutate-not-working/125948 "2018-03-30T19:54:29Z")
I upgraded logstash from 2.3.4 to 5.6.7, after the upgrade, mutate failed to remove any of the field. What do I need to check to make sure I have the right plugin? LS 5.6.7 ES 2.4.1 --- ## [Unrecognized service](https://discuss.elastic.co/t/unrecognized-service/126231)
**Author:** [@nadine](https://discuss.elastic.co/u/nadine)\ **Replies:** 4\ **Last updated:** [March 30, 2018, 6:13pm UTC](https://discuss.elastic.co/t/unrecognized-service/126231 "2018-03-30T18:13:35Z")
I'm a begginer in ELK. When i execute " service logstash configtest " I get this error message: "logstash: unrecognized service" Although the logstash in already running. This is my configuration : input { file { p… --- ## [Count message for user](https://discuss.elastic.co/t/count-message-for-user/126042)
**Author:** [@zeliko79](https://discuss.elastic.co/u/zeliko79)\ **Replies:** 1\ **Last updated:** [March 30, 2018, 5:15pm UTC](https://discuss.elastic.co/t/count-message-for-user/126042 "2018-03-30T17:15:30Z")
Hello everyone, I was able to configure ELK cluster and grok for postfix, now I have a problem in finding information. For every message sent I have 2 records similar to these: postfix\_queueid, postfix\_from postfix\_q… --- ## [How to add field to every element from an array with variable length](https://discuss.elastic.co/t/how-to-add-field-to-every-element-from-an-array-with-variable-length/126003)
**Author:** [@terrymu](https://discuss.elastic.co/u/terrymu)\ **Replies:** 1\ **Last updated:** [March 30, 2018, 5:12pm UTC](https://discuss.elastic.co/t/how-to-add-field-to-every-element-from-an-array-with-variable-length/126003 "2018-03-30T17:12:21Z")
Hi experts, I have an array such like this: "NODE" : \["node1", "node2", "node3"\] And my first requirement is add field to every element from this array. I have used a hard code to add three fields for these three elem… --- ## [Empty result when using TCP input to collect NLog JSON events](https://discuss.elastic.co/t/empty-result-when-using-tcp-input-to-collect-nlog-json-events/126107)
**Author:** [@tbugbee](https://discuss.elastic.co/u/tbugbee)\ **Replies:** 5\ **Last updated:** [March 30, 2018, 4:32pm UTC](https://discuss.elastic.co/t/empty-result-when-using-tcp-input-to-collect-nlog-json-events/126107 "2018-03-30T16:32:17Z")
Hello, I am attempting to collect logs using NLog targets and save them into a log file. The NLog target is sending events to Logstash using TCP and formats them in this way: \