# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=908 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 909 --- ## [Logstash grok filter unexpected duplication of values](https://discuss.elastic.co/t/logstash-grok-filter-unexpected-duplication-of-values/100457)
**Author:** [@mulgar](https://discuss.elastic.co/u/mulgar)\ **Replies:** 12\ **Last updated:** [September 15, 2017, 11:54am UTC](https://discuss.elastic.co/t/logstash-grok-filter-unexpected-duplication-of-values/100457 "2017-09-15T11:54:52Z")
I have a simple custom grok pattern to "normalise" version numbers from an input to capture just major and minor release version information from a full version field. Some examples to give you an idea: 1.2(3)U2 =\> 1.2 … --- ## [Json vs plaintext output](https://discuss.elastic.co/t/json-vs-plaintext-output/100669)
**Author:** [@drmason13](https://discuss.elastic.co/u/drmason13)\ **Replies:** 1\ **Last updated:** [September 15, 2017, 11:47am UTC](https://discuss.elastic.co/t/json-vs-plaintext-output/100669 "2017-09-15T11:47:24Z")
I am confused by the exact role of output codec plugins. In particular I have been struggling to understand why the syslog output plugin appears to output json. When running the following config: input { file { path =\>… --- ## [Parser for Modsecurity logs?](https://discuss.elastic.co/t/parser-for-modsecurity-logs/100672)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 0\ **Last updated:** [September 15, 2017, 10:43am UTC](https://discuss.elastic.co/t/parser-for-modsecurity-logs/100672 "2017-09-15T10:43:20Z")
Hi Guys, Does any one have a parser or logstash configuration file built for modsecurity logs which is a open source WAF These are the sample modsec logs 2017/09/15 06:07:57 \[info\] 43541#43541: 5 \[client 192.168.1.50\]… --- ## [Logstash subtracting 1 hour from time field](https://discuss.elastic.co/t/logstash-subtracting-1-hour-from-time-field/100647)
**Author:** [@patrickian](https://discuss.elastic.co/u/patrickian)\ **Replies:** 0\ **Last updated:** [September 15, 2017, 7:03am UTC](https://discuss.elastic.co/t/logstash-subtracting-1-hour-from-time-field/100647 "2017-09-15T07:03:16Z")
I have a source text in string like this: 09/15/2017 14:50:09.866800 Let's call this field action\_end\_raw. I use a date filter to parse it to a date time: date { match =\> \["action\_end\_raw", "MM/dd/yyyy HH:mm:ss.SSSS… --- ## [Match multiple ip segment Regex](https://discuss.elastic.co/t/match-multiple-ip-segment-regex/100484)
**Author:** [@sreejiths](https://discuss.elastic.co/u/sreejiths)\ **Replies:** 3\ **Last updated:** [September 15, 2017, 6:57am UTC](https://discuss.elastic.co/t/match-multiple-ip-segment-regex/100484 "2017-09-15T06:57:52Z")
Below is the if statement which is working fine if \[host\] =~ "^10.255.212.(\[1-9\]|\[1-9\]\[0-9\]|1(\[0-9\]\[0-9\])|2(\[0-4\]\[0-9\]|5\[0-5\]))$" { mutate { add\_field =\> \[ "host\_group", "XXX" \] } } else { mutate { add\_field =\> \[ "h… --- ## [Logstash not showing output anymore](https://discuss.elastic.co/t/logstash-not-showing-output-anymore/100274)
**Author:** [@hanifhan97](https://discuss.elastic.co/u/hanifhan97)\ **Replies:** 1\ **Last updated:** [September 15, 2017, 6:11am UTC](https://discuss.elastic.co/t/logstash-not-showing-output-anymore/100274 "2017-09-15T06:11:31Z")
Hello everyone, I've trying to parse xml file to logstash. when i try for a first time logstash show my output (even tho im still got the wrong output) but from now on i cant even read my output anymore. i already put s… --- ## [How to stop logstash automatically after some time](https://discuss.elastic.co/t/how-to-stop-logstash-automatically-after-some-time/100549)
**Author:** [@vishnudivakar31](https://discuss.elastic.co/u/vishnudivakar31)\ **Replies:** 4\ **Last updated:** [September 15, 2017, 6:06am UTC](https://discuss.elastic.co/t/how-to-stop-logstash-automatically-after-some-time/100549 "2017-09-15T06:06:53Z")
I am running logstash -f \ . I want this to be stopped automatically after 5 mins. --- ## [Plugin-input-snmptrap broken in 5.6?](https://discuss.elastic.co/t/plugin-input-snmptrap-broken-in-5-6/100633)
**Author:** [@MarcusCaepio](https://discuss.elastic.co/u/MarcusCaepio)\ **Replies:** 1\ **Last updated:** [September 15, 2017, 5:34am UTC](https://discuss.elastic.co/t/plugin-input-snmptrap-broken-in-5-6/100633 "2017-09-15T05:34:46Z")
Hi all, yesterday I wrote an snmptrap input for my test system, which was elk 5.2. Everything worked fine, I also found a script to translate missing MIBs into yml (the original import.rb script does not work at all...… --- ## [Help with identifying cause of Netflow IPFIX UDP service crashing randomly](https://discuss.elastic.co/t/help-with-identifying-cause-of-netflow-ipfix-udp-service-crashing-randomly/100609)
**Author:** [@gtyrer](https://discuss.elastic.co/u/gtyrer)\ **Replies:** 2\ **Last updated:** [September 15, 2017, 3:37am UTC](https://discuss.elastic.co/t/help-with-identifying-cause-of-netflow-ipfix-udp-service-crashing-randomly/100609 "2017-09-15T03:37:22Z")
Hi, complete newbie here. I have setup a Debian 9.1 server running logstash 5.6.0 along with the logstash-codec-netflow (3.5.2) plugin. Input is IPFIX and output to another server running Elasticsearch. (Test environme… --- ## [Google BigQuery Output Plugin Authentication Issue](https://discuss.elastic.co/t/google-bigquery-output-plugin-authentication-issue/100598)
**Author:** [@jeremytoce](https://discuss.elastic.co/u/jeremytoce)\ **Replies:** 0\ **Last updated:** [September 14, 2017, 7:38pm UTC](https://discuss.elastic.co/t/google-bigquery-output-plugin-authentication-issue/100598 "2017-09-14T19:38:07Z")
Hey all - I am struggling a bit with authenticating BigQuery for Logstash output use. I have an issue that I am sure is a matter of clarification, but haven't been able to solve for several days. I am receiving an inva… --- ## [Logstash sending to all output hosts](https://discuss.elastic.co/t/logstash-sending-to-all-output-hosts/100583)
**Author:** [@criddock](https://discuss.elastic.co/u/criddock)\ **Replies:** 4\ **Last updated:** [September 14, 2017, 7:21pm UTC](https://discuss.elastic.co/t/logstash-sending-to-all-output-hosts/100583 "2017-09-14T19:21:11Z")
So, here is a really odd situation. I have two pipeline files: Receive "Beats" fwd'ed logs over 5044 - send to ElasticSearch (x.x.x.164) input { beats { port =\> "5044" } } output { elasticsearch { … --- ## [How to handle the log partialy updated](https://discuss.elastic.co/t/how-to-handle-the-log-partialy-updated/99759)
**Author:** [@feijiangnan](https://discuss.elastic.co/u/feijiangnan)\ **Replies:** 2\ **Last updated:** [September 14, 2017, 6:27pm UTC](https://discuss.elastic.co/t/how-to-handle-the-log-partialy-updated/99759 "2017-09-14T18:27:41Z")
Hi there, WebSphere Application Server has ffdc exception log, and it looks like below, when the ffdc event happened, it search the log first, if it find same exception, it updates count only, otherwise append the log t… --- ## [How does Logstash keep track of Winlogbeat events?](https://discuss.elastic.co/t/how-does-logstash-keep-track-of-winlogbeat-events/100426)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 3\ **Last updated:** [September 14, 2017, 6:09pm UTC](https://discuss.elastic.co/t/how-does-logstash-keep-track-of-winlogbeat-events/100426 "2017-09-14T18:09:53Z")
Hi, I am trying to use winlogbeat and Logstash. How does Logstash keep track of winlogbeat events if Logstash is restarted? Thanks --- ## [CPU Management of Logstash](https://discuss.elastic.co/t/cpu-management-of-logstash/99487)
**Author:** [@CDR](https://discuss.elastic.co/u/CDR)\ **Replies:** 6\ **Last updated:** [September 14, 2017, 4:17pm UTC](https://discuss.elastic.co/t/cpu-management-of-logstash/99487 "2017-09-14T16:17:50Z")
I noticed that in the logstash.yml file there is a field called pipeline.workers and you can specify the amount of workers that will work in parallel. The number defaults to the host’s CPU cores. My question is whether … --- ## [Please help with pattern](https://discuss.elastic.co/t/please-help-with-pattern/97897)
**Author:** [@vitich](https://discuss.elastic.co/u/vitich)\ **Replies:** 4\ **Last updated:** [September 14, 2017, 2:42pm UTC](https://discuss.elastic.co/t/please-help-with-pattern/97897 "2017-09-14T14:42:26Z")
My nginx access log format: log\_format main '\[$time\_local\] $remote\_addr \[$geoip\_city\_country\_code:$geoip\_region\_name:$geoip\_city\] - "$status" "$http\_host" ' '"$request" $body\_bytes\_sent \[$request\_time s\] … --- ## [Logstash input exception stops pipeline](https://discuss.elastic.co/t/logstash-input-exception-stops-pipeline/100576)
**Author:** [@dunedain289](https://discuss.elastic.co/u/dunedain289)\ **Replies:** 0\ **Last updated:** [September 14, 2017, 4:30pm UTC](https://discuss.elastic.co/t/logstash-input-exception-stops-pipeline/100576 "2017-09-14T16:30:16Z")
I'm running 2.4.1, but in the process of upgrading the whole stack to 5.x. I see an java.lang.ArrayIndexOutOfBoundsException in my logs, and then one of the inputs shuts down without taking down the whole logstash proce… --- ## [Logstash output to multiple e-mail addresses](https://discuss.elastic.co/t/logstash-output-to-multiple-e-mail-addresses/100559)
**Author:** [@eds](https://discuss.elastic.co/u/eds)\ **Replies:** 1\ **Last updated:** [September 14, 2017, 4:06pm UTC](https://discuss.elastic.co/t/logstash-output-to-multiple-e-mail-addresses/100559 "2017-09-14T16:06:17Z")
Hi Magnus. My ELK version 5.5.2 For some events i use email output plugin, and send events to local exchange, works fine. But then i add new email output block with different email and server i get error: \[root@logst… --- ## [Date pattern in file input path](https://discuss.elastic.co/t/date-pattern-in-file-input-path/100538)
**Author:** [@shashwat](https://discuss.elastic.co/u/shashwat)\ **Replies:** 4\ **Last updated:** [September 14, 2017, 4:03pm UTC](https://discuss.elastic.co/t/date-pattern-in-file-input-path/100538 "2017-09-14T16:03:38Z")
Hi, I am in need to read files from different folders. These folders are maintained daily in the format of dates. e.g. 20170912, 20170911 etc. I am trying the file input path as "var/opt/data/%{+YYYY}%{+MM}%{+dd}/\*.txt… --- ## [Log rotation logstash](https://discuss.elastic.co/t/log-rotation-logstash/100324)
**Author:** [@zakabluk](https://discuss.elastic.co/u/zakabluk)\ **Replies:** 3\ **Last updated:** [September 14, 2017, 4:01pm UTC](https://discuss.elastic.co/t/log-rotation-logstash/100324 "2017-09-14T16:01:22Z")
Hi! I save logs if \[type\] == "tibco\_4" { file { path =\> "/opt/GET\_TRACE/ELK/tibco\_4/%{source}" codec =\> line { format =\> "custom format: %{message}"} gzip=\> true } } but in my directory /opt/GET\_TRACE/ELK/tibco\_… --- ## [Need help with grok parsing](https://discuss.elastic.co/t/need-help-with-grok-parsing/100437)
**Author:** [@Benoit\_Martin](https://discuss.elastic.co/u/Benoit_Martin)\ **Replies:** 2\ **Last updated:** [September 14, 2017, 3:27pm UTC](https://discuss.elastic.co/t/need-help-with-grok-parsing/100437 "2017-09-14T15:27:43Z")
Hi, I try to use grok and parse two type of log entry possibility. The field is a hostname receive by Cisco Prime. Possibility 1 : AP23132\_01,80:e0:1d:e0:c2:10 Possibility 2 : 80:e0:1d:e0:c2:10:AP23132\_01 AP23132\_01… --- ## [Does Logstash "Elasticsearch" output plugin support client certificates](https://discuss.elastic.co/t/does-logstash-elasticsearch-output-plugin-support-client-certificates/100167)
**Author:** [@Bozzerreaneaux](https://discuss.elastic.co/u/Bozzerreaneaux)\ **Replies:** 2\ **Last updated:** [September 12, 2017, 8:21am UTC](https://discuss.elastic.co/t/does-logstash-elasticsearch-output-plugin-support-client-certificates/100167 "2017-09-12T08:21:39Z")
Hi Team , Does Logstash "Elasticsearch" output plugin support client certificates? I can only see option there (cacert) to verify the ES node certificate but I would also like the Logstash pipeline to also identify it… --- ## [Read a file for token and use in it config file](https://discuss.elastic.co/t/read-a-file-for-token-and-use-in-it-config-file/100504)
**Author:** [@giridharan78](https://discuss.elastic.co/u/giridharan78)\ **Replies:** 4\ **Last updated:** [September 14, 2017, 2:29pm UTC](https://discuss.elastic.co/t/read-a-file-for-token-and-use-in-it-config-file/100504 "2017-09-14T14:29:22Z")
I've a token generated by a shell script and stored in a file(api\_calls.log)). I wanted the logstash config file to read the file and use the token in the request in http\_poller. The token has to be used in Authorizatio… --- ## [Input tcp error](https://discuss.elastic.co/t/input-tcp-error/100336)
**Author:** [@xiaopeng\_zhang](https://discuss.elastic.co/u/xiaopeng_zhang)\ **Replies:** 4\ **Last updated:** [September 14, 2017, 2:15pm UTC](https://discuss.elastic.co/t/input-tcp-error/100336 "2017-09-14T14:15:37Z")
i use the logback to send the log by tcp port 80, and i use the logstash-logback-encoder plugin. pom.xml: \4.0.0\ \edu.zxp\ \log4jlearn\ \ **Author:** [@NandhaKrishna](https://discuss.elastic.co/u/NandhaKrishna)\ **Replies:** 1\ **Last updated:** [September 14, 2017, 1:34pm UTC](https://discuss.elastic.co/t/need-help-to-grok-this-pattern/100541 "2017-09-14T13:34:49Z") This is the log message Shipper-CMN-CMN-WMQ-TOPOLOGY | 2017-09-13T12:23:03.512 | f20ad7a49557c605e729a2db16bb295d1d4922e32f2a2ffbb1d4b0b8ae34c28b I tried various grok patterns but it isnt working! --- ## [Grok isn't loading patterns](https://discuss.elastic.co/t/grok-isnt-loading-patterns/100327)
**Author:** [@nivo33](https://discuss.elastic.co/u/nivo33)\ **Replies:** 4\ **Last updated:** [September 14, 2017, 1:32pm UTC](https://discuss.elastic.co/t/grok-isnt-loading-patterns/100327 "2017-09-14T13:32:23Z")
Hi, I have tried to implement some custom grok patterns into a separate file and load it into logstash conf by using patterns\_dir. Each time running logstash with the following conf output a pattern not defined error me… --- ## [Logstash-input-beats rubygems,org](https://discuss.elastic.co/t/logstash-input-beats-rubygems-org/100214)
**Author:** [@Invictus](https://discuss.elastic.co/u/Invictus)\ **Replies:** 2\ **Last updated:** [September 14, 2017, 1:01pm UTC](https://discuss.elastic.co/t/logstash-input-beats-rubygems-org/100214 "2017-09-14T13:01:22Z")
I have a problem with the installation of the plugin logstash-input-beats. \[2017-09-12 15\_05\_37-Administrator\_ Eingabeaufforderung\] It always shows this advice. I already installed jruby and rubygem nothing changed. --- ## [Logstash - Nested fields](https://discuss.elastic.co/t/logstash-nested-fields/100201)
**Author:** [@NerdSec](https://discuss.elastic.co/u/NerdSec)\ **Replies:** 8\ **Last updated:** [September 14, 2017, 12:49pm UTC](https://discuss.elastic.co/t/logstash-nested-fields/100201 "2017-09-14T12:49:48Z")
Hi, I wish to parse proxy logs using logstash. I have multiple IP fields in each document (log). I wish to parse the IP field as an object/array, not quite sure which is best suited for the requirement. The final output… --- ## [Facing issue while starting multiple instance of logstash](https://discuss.elastic.co/t/facing-issue-while-starting-multiple-instance-of-logstash/100491)
**Author:** [@sukeerthi\_bj](https://discuss.elastic.co/u/sukeerthi_bj)\ **Replies:** 5\ **Last updated:** [September 14, 2017, 12:30pm UTC](https://discuss.elastic.co/t/facing-issue-while-starting-multiple-instance-of-logstash/100491 "2017-09-14T12:30:28Z")
Hi, I created a separate folder for new instance of logstash as mentioned in below link but getting logstash status as failed systemctl status logstash-suk.service ● logstash-suk.service - logstash-suk Loaded: loa… --- ## [Facing issues in starting Logstash using kafka input plugin and avro codec](https://discuss.elastic.co/t/facing-issues-in-starting-logstash-using-kafka-input-plugin-and-avro-codec/100525)
**Author:** [@Aarthy.Krishnamurthy](https://discuss.elastic.co/u/Aarthy.Krishnamurthy)\ **Replies:** 0\ **Last updated:** [September 14, 2017, 11:50am UTC](https://discuss.elastic.co/t/facing-issues-in-starting-logstash-using-kafka-input-plugin-and-avro-codec/100525 "2017-09-14T11:50:38Z")
Hi, I am entirely new to Logstash and elasticsearch Currently I am facing a problem creating LS pipeline using kafka input which receives an avro message. \[LS\_Error\] The following is the configuration that is being u… --- ## [Create array in logstash -\> elastic](https://discuss.elastic.co/t/create-array-in-logstash-elastic/100512)
**Author:** [@Frances\_Buontempo](https://discuss.elastic.co/u/Frances_Buontempo)\ **Replies:** 0\ **Last updated:** [September 14, 2017, 10:24am UTC](https://discuss.elastic.co/t/create-array-in-logstash-elastic/100512 "2017-09-14T10:24:39Z")
I want to update documents, and add some of the fields to an array, e.g. with documents from json like { "name": "bill, "hobbies": "a" , "id": 1} if I run logstash on more data like { "name": "ted, "hobbies": "a" , "… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=907) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=909)