# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1026 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1027 --- ## [Multiple Pipeline configuration file which belong to same type](https://discuss.elastic.co/t/multiple-pipeline-configuration-file-which-belong-to-same-type/77088)
**Author:** [@JasonT](https://discuss.elastic.co/u/JasonT)\ **Replies:** 2\ **Last updated:** [March 2, 2017, 4:19pm UTC](https://discuss.elastic.co/t/multiple-pipeline-configuration-file-which-belong-to-same-type/77088 "2017-03-02T16:19:34Z")
Hi, all, If I have multiple Pipeline configuration file which belong to same type (all input from syslog ), each file content is difference type device( e.g,. firewall, switch, server ). How to identify(like added any d… --- ## [Moving from JSON to non-JSON input filter](https://discuss.elastic.co/t/moving-from-json-to-non-json-input-filter/77159)
**Author:** [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Replies:** 0\ **Last updated:** [March 2, 2017, 3:09pm UTC](https://discuss.elastic.co/t/moving-from-json-to-non-json-input-filter/77159 "2017-03-02T15:09:57Z")
I've been sending our application logs via UDP wrapped in a JSON format, so the log looks something like this { "key1" : "value1", "key2" : "value2", "message" : "2017-03-02 INFO com.company.app Hello world!" } a… --- ## [Parsing a message after the pattern](https://discuss.elastic.co/t/parsing-a-message-after-the-pattern/76246)
**Author:** [@cisaksen](https://discuss.elastic.co/u/cisaksen)\ **Replies:** 3\ **Last updated:** [March 2, 2017, 3:02pm UTC](https://discuss.elastic.co/t/parsing-a-message-after-the-pattern/76246 "2017-03-02T15:02:07Z")
New to logstash - I'm trying to add a field that contains the http host value (http://host.domin.com/). I have the combinedlog pattern to match the "message" field and now I'm trying to use the addfield but i don't … --- ## [Logstash-filter-zeromq is timing out with ZeroMQ](https://discuss.elastic.co/t/logstash-filter-zeromq-is-timing-out-with-zeromq/77151)
**Author:** [@anmoljagetia](https://discuss.elastic.co/u/anmoljagetia)\ **Replies:** 0\ **Last updated:** [March 2, 2017, 2:29pm UTC](https://discuss.elastic.co/t/logstash-filter-zeromq-is-timing-out-with-zeromq/77151 "2017-03-02T14:29:11Z")
zeromq { address =\> \["tcp://10.6.17.39:5555"\] field =\> \["ZMQ\_Response"\] timeout =\> 1000 retries =\> 10 } When I send the data to ZMQ, I am getting zmqtimeouts. To investigate, I have… --- ## [Need to have true timestamp from the log](https://discuss.elastic.co/t/need-to-have-true-timestamp-from-the-log/77116)
**Author:** [@raamee](https://discuss.elastic.co/u/raamee)\ **Replies:** 4\ **Last updated:** [March 2, 2017, 12:12pm UTC](https://discuss.elastic.co/t/need-to-have-true-timestamp-from-the-log/77116 "2017-03-02T12:12:50Z")
Hi, I need help with sending the timestamp of the log properly to the Elasticsearch via logstash. My log looks like: Thu Feb 2 07:24:00 2017 - \[Client: WIFI-US001NW5 IPAddress: 10.29.24.6\] - \[User-Name: johndoe\] - \[A… --- ## [Parsed JSON object/hash requires a target configuration option](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354)
**Author:** [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Replies:** 11\ **Last updated:** [March 2, 2017, 10:46am UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354 "2017-03-02T10:46:00Z")
I'm seeing these messages in my logstash.log file: {:timestamp=\>"2016-08-30T11:36:51.811000-0700", :message=\>"Parsed JSON object/hash requires a target configuration option", :source=\>"message", :raw=\>"", :level=\>:warn… --- ## [Logstash is goving ssl error on installing plugin](https://discuss.elastic.co/t/logstash-is-goving-ssl-error-on-installing-plugin/77123)
**Author:** [@wolfghost](https://discuss.elastic.co/u/wolfghost)\ **Replies:** 0\ **Last updated:** [March 2, 2017, 10:18am UTC](https://discuss.elastic.co/t/logstash-is-goving-ssl-error-on-installing-plugin/77123 "2017-03-02T10:18:56Z")
c:\\ELK\_5.1.1\\logstash-5.1.1\\bin\>logstash-plugin install logstash-filter-split Validating logstash-filter-split Installing logstash-filter-split Error Bundler::InstallError, retrying 1/10 An error occurred while installin… --- ## [How to call python script from Logstash to process (partialy) logs](https://discuss.elastic.co/t/how-to-call-python-script-from-logstash-to-process-partialy-logs/40009)
**Author:** [@vitaly\_r](https://discuss.elastic.co/u/vitaly_r)\ **Replies:** 5\ **Last updated:** [March 2, 2017, 9:17am UTC](https://discuss.elastic.co/t/how-to-call-python-script-from-logstash-to-process-partialy-logs/40009 "2017-03-02T09:17:10Z")
Hello, I have to process parts of the logging information using python script (python beautifulsoup and other useful stuff which I do not want to reimnplement in Ruby). What is the simplest way to do this? As of no… --- ## [Split date(yyyymmdd:hhmmss) and convert into default timestamp logstash](https://discuss.elastic.co/t/split-date-yyyymmdd-hhmmss-and-convert-into-default-timestamp-logstash/76986)
**Author:** [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)\ **Replies:** 4\ **Last updated:** [March 2, 2017, 9:16am UTC](https://discuss.elastic.co/t/split-date-yyyymmdd-hhmmss-and-convert-into-default-timestamp-logstash/76986 "2017-03-02T09:16:10Z")
I have an input as 20170301:18544482:INFO 10.0.0.67 ABCLOG sending request 20170301:18321276:ERROR 10.0.0.67 ABCLOG got response Above are sample input for my logstash, where input is in foll format yyyymmdd:hhmmssSS… --- ## [Logstash to create the complete suggester](https://discuss.elastic.co/t/logstash-to-create-the-complete-suggester/77102)
**Author:** [@Greentea](https://discuss.elastic.co/u/Greentea)\ **Replies:** 3\ **Last updated:** [March 2, 2017, 8:18am UTC](https://discuss.elastic.co/t/logstash-to-create-the-complete-suggester/77102 "2017-03-02T08:18:02Z")
Hi! I am going to make completion suggester in my search page, I used the display\_name as the text, and I would like to insert id as the weight for sorting the suggester. Here is my logstash filter filter { if \[type\] =… --- ## [Writing plug-in](https://discuss.elastic.co/t/writing-plug-in/77103)
**Author:** [@bertvervaele](https://discuss.elastic.co/u/bertvervaele)\ **Replies:** 1\ **Last updated:** [March 2, 2017, 8:13am UTC](https://discuss.elastic.co/t/writing-plug-in/77103 "2017-03-02T08:13:19Z")
Hello I started using elastic stack and realy like it so already thanks a lot! I also would like to contribute a bit by writing a plug-in. I need it for my internschip and I would like to share it with the community if … --- ## [Can I add (arbitrary) new fields from inside a grok pattern?](https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497)
**Author:** [@nick-george](https://discuss.elastic.co/u/nick-george)\ **Replies:** 4\ **Last updated:** [March 2, 2017, 7:43am UTC](https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497 "2017-03-02T07:43:54Z")
Hi, I have been making use of freely available logstash configs and patterns, like this (https://github.com/nxhack/logstash/blob/master/patterns/sshd) awesome one for SSHD. I've copied a small section of the file for … --- ## [Logstash-output-elasticsearch error retry Individual action](https://discuss.elastic.co/t/logstash-output-elasticsearch-error-retry-individual-action/76718)
**Author:** [@jamal](https://discuss.elastic.co/u/jamal)\ **Replies:** 3\ **Last updated:** [March 2, 2017, 7:18am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-error-retry-individual-action/76718 "2017-03-02T07:18:47Z")
I am using twitter-input to fetch data from twitter and ouput-elasticsearch to store it in elasticsearch and I am using logstash 5.2.1 in Ubuntu OS when I run it through the following error \> \[2017-03-02T08:18:45,576… --- ## [All logs on 1 line](https://discuss.elastic.co/t/all-logs-on-1-line/76981)
**Author:** [@bertvervaele](https://discuss.elastic.co/u/bertvervaele)\ **Replies:** 2\ **Last updated:** [March 2, 2017, 7:15am UTC](https://discuss.elastic.co/t/all-logs-on-1-line/76981 "2017-03-02T07:15:56Z")
I want to get my SAP log's into logstash. The problem is SAP writes all his logs after each other without a new line. between 2 log lines it puts \*\*\*\*\*\*\*\* instead of a new line. I know I can use split to separate them bu… --- ## [Logstash aggregate filter - usage help needed](https://discuss.elastic.co/t/logstash-aggregate-filter-usage-help-needed/76176)
**Author:** [@Ruthu\_Ramya](https://discuss.elastic.co/u/Ruthu_Ramya)\ **Replies:** 4\ **Last updated:** [March 2, 2017, 7:12am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-usage-help-needed/76176 "2017-03-02T07:12:18Z")
Hi, I am using the aggregate filter to read repeating blocks of log data. I need block-wise data which at this moment I am not able to get. Log file: Nov 30,15:02:42.081 INFO CD playback started Nov 30,15:02:42.08… --- ## [Filter multiple different file beat logs in logstash](https://discuss.elastic.co/t/filter-multiple-different-file-beat-logs-in-logstash/76847)
**Author:** [@mohsin106](https://discuss.elastic.co/u/mohsin106)\ **Replies:** 4\ **Last updated:** [March 2, 2017, 7:04am UTC](https://discuss.elastic.co/t/filter-multiple-different-file-beat-logs-in-logstash/76847 "2017-03-02T07:04:39Z")
Hi, I am currently sending apache access logs from a remote server to my Logstash server by running FileBeat on the remote server. This is working like a charm. I would like to also send other logs with different log… --- ## [Logstash 5.2.0 - Cannot install any plugin on Windows 10](https://discuss.elastic.co/t/logstash-5-2-0-cannot-install-any-plugin-on-windows-10/73976)
**Author:** [@Martin\_Hatas](https://discuss.elastic.co/u/Martin_Hatas)\ **Replies:** 1\ **Last updated:** [March 2, 2017, 5:46am UTC](https://discuss.elastic.co/t/logstash-5-2-0-cannot-install-any-plugin-on-windows-10/73976 "2017-03-02T05:46:18Z")
C:\\Java\\temperature\\logstash-5.2.0 λ bin\\logstash-plugin install logstash-input-beats Validating logstash-input-beats Encoding::UndefinedConversionError: ""o\\x01"" from UTF-16LE to Windows-1252 encode at… --- ## [Logstash warning template](https://discuss.elastic.co/t/logstash-warning-template/77077)
**Author:** [@Greentea](https://discuss.elastic.co/u/Greentea)\ **Replies:** 0\ **Last updated:** [March 2, 2017, 2:23am UTC](https://discuss.elastic.co/t/logstash-warning-template/77077 "2017-03-02T02:23:15Z")
Hi! I got a question for a longtime. Is there any way to change to logstash warning out put? Every time it output the warning like this 10:16:52.512 \[\[main\]\>worker1\] WARN logstash.outputs.elasticsearch - Failed action… --- ## [How to remove fields in logstash/es](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-es/77039)
**Author:** [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Replies:** 5\ **Last updated:** [March 1, 2017, 9:37pm UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-es/77039 "2017-03-01T21:37:54Z")
This was my index looks like in kibana. I don't want so many fields below, so how can i delete/avoid those from my index. Ex.: id, \_score, version, beat info. { "\_index": "test", "\_type": "testing", "\_id": "AVqK… --- ## [Logstash 5.2 broke CSV output](https://discuss.elastic.co/t/logstash-5-2-broke-csv-output/77053)
**Author:** [@j2man](https://discuss.elastic.co/u/j2man)\ **Replies:** 0\ **Last updated:** [March 1, 2017, 9:17pm UTC](https://discuss.elastic.co/t/logstash-5-2-broke-csv-output/77053 "2017-03-01T21:17:25Z")
Seems that 5.2 broke the CSV output plugin. This generates a "csv" with all the message fields all the time NOT separated by comma's but spaces even though "blah" does not exist as a field. This works correctly 2.3 and … --- ## [Logstash instead of logstash forwarder/beats](https://discuss.elastic.co/t/logstash-instead-of-logstash-forwarder-beats/77045)
**Author:** [@kishorramanan](https://discuss.elastic.co/u/kishorramanan)\ **Replies:** 1\ **Last updated:** [March 1, 2017, 8:25pm UTC](https://discuss.elastic.co/t/logstash-instead-of-logstash-forwarder-beats/77045 "2017-03-01T20:25:37Z")
What if i use logstash instead of logstash forwarder/beats in my environment, and directly send the logs to elasticsearch. What will be the impact --- ## [Aggregate metrics for Graphite](https://discuss.elastic.co/t/aggregate-metrics-for-graphite/77042)
**Author:** [@Exocomp](https://discuss.elastic.co/u/Exocomp)\ **Replies:** 0\ **Last updated:** [March 1, 2017, 7:29pm UTC](https://discuss.elastic.co/t/aggregate-metrics-for-graphite/77042 "2017-03-01T19:29:12Z")
What are folks recommendation for aggregating data to send to Graphite. What I mean is for cases like http status codes, I found statsd and it works but it unfortunately does not support passing timestamps (it generates… --- ## [Showing multiline messages as "multi lines?"](https://discuss.elastic.co/t/showing-multiline-messages-as-multi-lines/76334)
**Author:** [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Replies:** 1\ **Last updated:** [March 1, 2017, 6:25pm UTC](https://discuss.elastic.co/t/showing-multiline-messages-as-multi-lines/76334 "2017-03-01T18:25:52Z")
I use the ELK stack as my central log server, and I have all the parts working so now I'm tweaking it. I see some of my logs (sent remotely via rsyslog) have "\\n" in them, that is I literally see the "\\n" characters. H… --- ## [Logstash inputs imx( imx plugin) cannot find the config file](https://discuss.elastic.co/t/logstash-inputs-imx-imx-plugin-cannot-find-the-config-file/77035)
**Author:** [@dhirenshumsher](https://discuss.elastic.co/u/dhirenshumsher)\ **Replies:** 0\ **Last updated:** [March 1, 2017, 5:24pm UTC](https://discuss.elastic.co/t/logstash-inputs-imx-imx-plugin-cannot-find-the-config-file/77035 "2017-03-01T17:24:04Z")
Hi Guys, Could someone help me with this please? Logstash JMX plugin cannot seem to find/load the config file, the file has the right ownership/permission. Thanks, \[2017-03-01T17:09:10,007\]\[ERROR\]\[logstash.inputs.jm… --- ## [Kafka input painfully slow](https://discuss.elastic.co/t/kafka-input-painfully-slow/77013)
**Author:** [@ArunANayagam](https://discuss.elastic.co/u/ArunANayagam)\ **Replies:** 0\ **Last updated:** [March 1, 2017, 3:39pm UTC](https://discuss.elastic.co/t/kafka-input-painfully-slow/77013 "2017-03-01T15:39:26Z")
Hi, I was using logstash-input-jms, to read messages transform them (xml parsing) and putting them into elasticsearch. It worked quite well although under marginally higher load (42k messages per minute) it would slow… --- ## [Logstash.conf.save permission denied](https://discuss.elastic.co/t/logstash-conf-save-permission-denied/77005)
**Author:** [@shookshank](https://discuss.elastic.co/u/shookshank)\ **Replies:** 1\ **Last updated:** [March 1, 2017, 3:07pm UTC](https://discuss.elastic.co/t/logstash-conf-save-permission-denied/77005 "2017-03-01T15:07:03Z")
Full Error Message: {:timestamp=\>"2017-03-01T09:05:28.458000-0500", :message=\>"The error reported is: \\n Permission denied - /etc/logstash/conf.d/logstash.conf.save"} Config Contents: logstash.conf: input { syslo… --- ## [Inserting mongodb collection in to elasticsearch give me error](https://discuss.elastic.co/t/inserting-mongodb-collection-in-to-elasticsearch-give-me-error/77001)
**Author:** [@111159](https://discuss.elastic.co/u/111159)\ **Replies:** 0\ **Last updated:** [March 1, 2017, 2:45pm UTC](https://discuss.elastic.co/t/inserting-mongodb-collection-in-to-elasticsearch-give-me-error/77001 "2017-03-01T14:45:38Z")
Hello I'm trying to insert mongodb collection to elasticsearch with this conf file but it give me following error. "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"Field \[\_id\] is a metadata field and cannot be a… --- ## [CIDR,subnet relating](https://discuss.elastic.co/t/cidr-subnet-relating/76757)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 8\ **Last updated:** [March 1, 2017, 2:11pm UTC](https://discuss.elastic.co/t/cidr-subnet-relating/76757 "2017-03-01T14:11:21Z")
Hi All, Will I be able to match my ip address to subnets ,i know we can use CIDR ,but I have around 800 subnets is there any way I can match the ip addresses with that using translate and CIDR together . I want t… --- ## [Parsing an xml file](https://discuss.elastic.co/t/parsing-an-xml-file/76971)
**Author:** [@Mohamed](https://discuss.elastic.co/u/Mohamed)\ **Replies:** 3\ **Last updated:** [March 1, 2017, 2:04pm UTC](https://discuss.elastic.co/t/parsing-an-xml-file/76971 "2017-03-01T14:04:09Z")
Hello, I am having a sample xml log file like below: \ \XX\ \YY\ \ I want to parse the above code and want to display the data in an output file like… --- ## [Bug in http\_poller: Environment variables don't always evaluate with http\_poller](https://discuss.elastic.co/t/bug-in-http-poller-environment-variables-dont-always-evaluate-with-http-poller/53070)
**Author:** [@iamthealex](https://discuss.elastic.co/u/iamthealex)\ **Replies:** 4\ **Last updated:** [March 1, 2017, 1:49pm UTC](https://discuss.elastic.co/t/bug-in-http-poller-environment-variables-dont-always-evaluate-with-http-poller/53070 "2017-03-01T13:49:42Z")
You can see the full details here:http://pastebin.com/3sLsemjU but the basic problem is that env substitution is not happening on all the fields. Specifically, this env substitution doesn't work: I need this form so th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1025) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1027)