# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=668 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 669 --- ## [Help with RUBY. Recognising field with prefix and changing its type](https://discuss.elastic.co/t/help-with-ruby-recognising-field-with-prefix-and-changing-its-type/152157)
**Author:** [@Harshet\_Jain](https://discuss.elastic.co/u/Harshet_Jain)\ **Replies:** 10\ **Last updated:** [October 15, 2018, 8:28am UTC](https://discuss.elastic.co/t/help-with-ruby-recognising-field-with-prefix-and-changing-its-type/152157 "2018-10-15T08:28:38Z")
Hi, I am trying to convert fields with a certain prefix into an integer event.to\_hash.keys.each { |k| if event.get\[k\].is\_a?(String) and k.start\_with?('behaviour\_' || 'top\_disease\_') event.get\[k\] = event.… --- ## [Wildcard matches in logstash filters](https://discuss.elastic.co/t/wildcard-matches-in-logstash-filters/152445)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 1\ **Last updated:** [October 15, 2018, 8:23am UTC](https://discuss.elastic.co/t/wildcard-matches-in-logstash-filters/152445 "2018-10-15T08:23:47Z")
Hi, I want to match every ip adress that starts with 10.99 in our field "dst\_ip". This is our logstash conf: filter { if "syslog" in \[tags\] and "pre-processed" not in \[tags\] { if "test" in \[field1\] { muta… --- ## [Split xml like string](https://discuss.elastic.co/t/split-xml-like-string/152217)
**Author:** [@mortenb123](https://discuss.elastic.co/u/mortenb123)\ **Replies:** 1\ **Last updated:** [October 15, 2018, 7:10am UTC](https://discuss.elastic.co/t/split-xml-like-string/152217 "2018-10-15T07:10:19Z")
Hi How can I easily parse the following xml like string into kv strings: "controllerstring" =\> "\SC-M\\SC-1 6.1.0-1 Jun 19 2018\\48DA97EA-000229\\\\1222285290\\00:… --- ## [Grok filter match with joda datetime pattern?](https://discuss.elastic.co/t/grok-filter-match-with-joda-datetime-pattern/151749)
**Author:** [@Wurzelseppi](https://discuss.elastic.co/u/Wurzelseppi)\ **Replies:** 1\ **Last updated:** [October 15, 2018, 5:22am UTC](https://discuss.elastic.co/t/grok-filter-match-with-joda-datetime-pattern/151749 "2018-10-15T05:22:31Z")
Hi guys, I have this configuration (part of) filter { if \[type\] == "gflogs" { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:TimestampTemp}" } } date { match =\> \[ "TimestampTemp", "ISO8601" \] remove\_field… --- ## [Logstash filter date](https://discuss.elastic.co/t/logstash-filter-date/152382)
**Author:** [@Pirate\_Trkmn](https://discuss.elastic.co/u/Pirate_Trkmn)\ **Replies:** 2\ **Last updated:** [October 14, 2018, 4:48pm UTC](https://discuss.elastic.co/t/logstash-filter-date/152382 "2018-10-14T16:48:56Z")
hi i used date filter as: date { match =\> \["log\_timestamp", "yyyy-MM-dd HH:mm:ss" \] } filter is working but in log\_timestamp it shows 2016-12-31 22:31:24, in @timestamp it is January 1st 2017, 02:31:24. how can i so… --- ## [Initial setup of Logstash](https://discuss.elastic.co/t/initial-setup-of-logstash/152369)
**Author:** [@Thor-Egil\_Ekeli](https://discuss.elastic.co/u/Thor-Egil_Ekeli)\ **Replies:** 2\ **Last updated:** [October 14, 2018, 2:52pm UTC](https://discuss.elastic.co/t/initial-setup-of-logstash/152369 "2018-10-14T14:52:01Z")
Hi, I am new to ELK and have a basic question about how to start. I have installed the ELK-stack on Ubuntu 18.04, and as a start I want to receive syslog from my Cisco ASA-devices into Logstash so that I can view the lo… --- ## ["LogStash::ConfigurationError", :message=\>"Expected one of #](https://discuss.elastic.co/t/logstash-configurationerror-message-expected-one-of/152367)
**Author:** [@idrees](https://discuss.elastic.co/u/idrees)\ **Replies:** 1\ **Last updated:** [October 14, 2018, 10:00am UTC](https://discuss.elastic.co/t/logstash-configurationerror-message-expected-one-of/152367 "2018-10-14T10:00:53Z")
Dear all, i am using elasticsearch 6.4 with logstash and searchguard with ssl enabled on centos 7. my logstash configuration file is below input { file { path =\>"/root/logstashProjects/studentwithdraw/student\_withd… --- ## [Cannot Run Logstsh as a service](https://discuss.elastic.co/t/cannot-run-logstsh-as-a-service/152377)
**Author:** [@Billz1026](https://discuss.elastic.co/u/Billz1026)\ **Replies:** 2\ **Last updated:** [October 14, 2018, 9:15am UTC](https://discuss.elastic.co/t/cannot-run-logstsh-as-a-service/152377 "2018-10-14T09:15:04Z")
Hi, I had installed logstash using the rpm file. following is the way I used to install logstash. rpm -ivh logstash-6.4.2.rpm And my config file is located in below directory. /etc/logstash/conf.d/ When I am using c… --- ## [YML not fetching](https://discuss.elastic.co/t/yml-not-fetching/152370)
**Author:** [@DavisDxb](https://discuss.elastic.co/u/DavisDxb)\ **Replies:** 0\ **Last updated:** [October 14, 2018, 6:05am UTC](https://discuss.elastic.co/t/yml-not-fetching/152370 "2018-10-14T06:05:15Z")
When I try to Install BlueLiv on ELK ,one message shows, logstash.yml not identified. But in my system logstash installed properly and yml file ready in /etc/logstash. But it shows this error. Logstash service working … --- ## [Logstash file input not working 6.4.0](https://discuss.elastic.co/t/logstash-file-input-not-working-6-4-0/152152)
**Author:** [@envy7](https://discuss.elastic.co/u/envy7)\ **Replies:** 2\ **Last updated:** [October 14, 2018, 3:30am UTC](https://discuss.elastic.co/t/logstash-file-input-not-working-6-4-0/152152 "2018-10-14T03:30:01Z")
Hi, I have been trying to use the logstash file input but it is not reading the file, the sincedb file is empty, I have tried all options available, mentioned in all similar threads. Setting sincedb\_path to null, tried c… --- ## [How to eliminate swapping of logstash container](https://discuss.elastic.co/t/how-to-eliminate-swapping-of-logstash-container/151873)
**Author:** [@netvmdb](https://discuss.elastic.co/u/netvmdb)\ **Replies:** 1\ **Last updated:** [October 12, 2018, 6:22pm UTC](https://discuss.elastic.co/t/how-to-eliminate-swapping-of-logstash-container/151873 "2018-10-12T18:22:36Z")
elasticsearch / logstash / kibana oss 6.2.4 running as docker containers in one physical host. Already set vm.swappiness=1 on the host. But swappiness still happening. I tried following setting for elasticsearch nodes,… --- ## [\ is missing in tag \\>](https://discuss.elastic.co/t/rss-tag-author-is-missing-in-tag-entry/152322)
**Author:** [@JohnOllhorn](https://discuss.elastic.co/u/JohnOllhorn)\ **Replies:** 0\ **Last updated:** [October 13, 2018, 7:37am UTC](https://discuss.elastic.co/t/rss-tag-author-is-missing-in-tag-entry/152322 "2018-10-13T07:37:55Z")
\[ERROR\] 2018-10-13 07:30:11.771 \[\[main\]\#\} Hello, folks, I'm just using ELK, but I'm already totally excited. However, I already … --- ## [\_grokparsefailure: All fields are here but still get the error](https://discuss.elastic.co/t/grokparsefailure-all-fields-are-here-but-still-get-the-error/152213)
**Author:** [@faulander](https://discuss.elastic.co/u/faulander)\ **Replies:** 3\ **Last updated:** [October 13, 2018, 5:00am UTC](https://discuss.elastic.co/t/grokparsefailure-all-fields-are-here-but-still-get-the-error/152213 "2018-10-13T05:00:15Z")
Here is the GROK: %{IPORHOST:remote\_ip} %{USER:ident} %{USER:auth} \\\[%{HTTPDATE:access\_time}\\\] TLSv%{DATA:tls} %{NOTSPACE:RSA} \\"%{WORD:method} %{NOTSPACE:url} %{DATA:protocol}\\/%{NUMBER:version}\\" %{NUMBER:Statuscode} … --- ## [Elasticsearch connection refused error](https://discuss.elastic.co/t/elasticsearch-connection-refused-error/152292)
**Author:** [@cmahan](https://discuss.elastic.co/u/cmahan)\ **Replies:** 1\ **Last updated:** [October 13, 2018, 3:09am UTC](https://discuss.elastic.co/t/elasticsearch-connection-refused-error/152292 "2018-10-13T03:09:15Z")
I start logstash my input uses the s3 input plugin. I'm try to output to elasticsearch. However, I get an error and warning. \[ERROR\] 2018-10-12 19:12:03.039 \[\[main\]\ **Author:** [@Aayush\_Nagpal](https://discuss.elastic.co/u/Aayush_Nagpal)\ **Replies:** 0\ **Last updated:** [October 13, 2018, 12:20am UTC](https://discuss.elastic.co/t/need-help-with-grok-pattern/152311 "2018-10-13T00:20:20Z") My application generates log in the following pattern: 2018-10-12 14:45:20.583 UTC DEBUG 123456 pool-default-dummytext-dummytext-thread-1 text text text received from server: \[ \[ multiplelinetext multiplelinetext m… --- ## [Attempted to send a bulk request to Elasticsearch](https://discuss.elastic.co/t/attempted-to-send-a-bulk-request-to-elasticsearch/152052)
**Author:** [@Dani\_Pinol](https://discuss.elastic.co/u/Dani_Pinol)\ **Replies:** 7\ **Last updated:** [October 12, 2018, 7:05pm UTC](https://discuss.elastic.co/t/attempted-to-send-a-bulk-request-to-elasticsearch/152052 "2018-10-12T19:05:44Z")
Hi, My ELK stack has been running great for months, until recently. I've been receiving the following error: timestamp=\>"2018-10-10T22:00:08.928000+0200", :message=\>"Attempted to send a bulk request to Elasticsearch co… --- ## [COMBINEDAPACHELOG Outliers](https://discuss.elastic.co/t/combinedapachelog-outliers/151928)
**Author:** [@pooch](https://discuss.elastic.co/u/pooch)\ **Replies:** 3\ **Last updated:** [October 12, 2018, 6:36pm UTC](https://discuss.elastic.co/t/combinedapachelog-outliers/151928 "2018-10-12T18:36:39Z")
Hi All, So I have the following GROK filter going for my Apache logs. else if "apache" in \[tags\] { grok { match =\> { "message" =\> '%{COMBINEDAPACHELOG} } This is working for ~75% of ingested logs but I s… --- ## [Logstash file input works on one path but not others](https://discuss.elastic.co/t/logstash-file-input-works-on-one-path-but-not-others/152126)
**Author:** [@Scott\_Newby](https://discuss.elastic.co/u/Scott_Newby)\ **Replies:** 3\ **Last updated:** [October 12, 2018, 6:08pm UTC](https://discuss.elastic.co/t/logstash-file-input-works-on-one-path-but-not-others/152126 "2018-10-12T18:08:00Z")
I was able to load the json logs we're generating, flattening with a ruby script, and then loading to elastic. We're on a windows server 2012 environment. Logstash 6.3.0. This works on one path, but not another inpu… --- ## [How to monitor logstash process](https://discuss.elastic.co/t/how-to-monitor-logstash-process/152241)
**Author:** [@cloudg0102](https://discuss.elastic.co/u/cloudg0102)\ **Replies:** 3\ **Last updated:** [October 12, 2018, 3:33pm UTC](https://discuss.elastic.co/t/how-to-monitor-logstash-process/152241 "2018-10-12T15:33:02Z")
currently we have couple of logstash process where we go manually run them so sometimes they go down and its very difficult to track and how do we get notified if any individual process goes down we are working on runni… --- ## [Represent logstash output in proper json format](https://discuss.elastic.co/t/represent-logstash-output-in-proper-json-format/152185)
**Author:** [@balumurari1](https://discuss.elastic.co/u/balumurari1)\ **Replies:** 1\ **Last updated:** [October 12, 2018, 3:23pm UTC](https://discuss.elastic.co/t/represent-logstash-output-in-proper-json-format/152185 "2018-10-12T15:23:36Z")
How to represent logstash output in proper json format event name represents station name, event id represent station id. How can i combine them to get proper json data My conf file: input { file { path =\> "D:/EL… --- ## [Configure Grok Filters](https://discuss.elastic.co/t/configure-grok-filters/152169)
**Author:** [@shirm](https://discuss.elastic.co/u/shirm)\ **Replies:** 1\ **Last updated:** [October 12, 2018, 3:11pm UTC](https://discuss.elastic.co/t/configure-grok-filters/152169 "2018-10-12T15:11:16Z")
Hi All, I am just starting out with Logstash. I am trying to parse a logfile with error contents as 2018-09-03 15:31:51.9682 Error ActiveDirectory Application MyApp is referenced by groups uADGROUP1 and uADGROUP2 - onl… --- ## [Grokparse failures issues](https://discuss.elastic.co/t/grokparse-failures-issues/152158)
**Author:** [@vishnuduttpv](https://discuss.elastic.co/u/vishnuduttpv)\ **Replies:** 1\ **Last updated:** [October 12, 2018, 3:07pm UTC](https://discuss.elastic.co/t/grokparse-failures-issues/152158 "2018-10-12T15:07:30Z")
We have created a logstash configuration with tomcat logs as the input and provide as the output in Json format to a file. This is the configuration : =================================================================== … --- ## [Is this possible with Aggregate plugin or Ruby? Any ideas?](https://discuss.elastic.co/t/is-this-possible-with-aggregate-plugin-or-ruby-any-ideas/152182)
**Author:** [@Harshet\_Jain](https://discuss.elastic.co/u/Harshet_Jain)\ **Replies:** 0\ **Last updated:** [October 12, 2018, 7:42am UTC](https://discuss.elastic.co/t/is-this-possible-with-aggregate-plugin-or-ruby-any-ideas/152182 "2018-10-12T07:42:19Z")
I am trying to combine records from two (or more) sources. Here is the example: Source A contains: doc\_id, RecA Source B contains: doc\_id, RecB I want to aggregate them to create "doc\_id": 44342 "RecA": "hello" "R… --- ## [Pipeline-to-pipeline, distributor pattern best practice / known limitation](https://discuss.elastic.co/t/pipeline-to-pipeline-distributor-pattern-best-practice-known-limitation/151365)
**Author:** [@yarilc](https://discuss.elastic.co/u/yarilc)\ **Replies:** 1\ **Last updated:** [October 12, 2018, 12:01pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-distributor-pattern-best-practice-known-limitation/151365 "2018-10-12T12:01:23Z")
Since I have moved my logging infrastructure to elastic 6.3 I'm evaluating to split and simplifying our existing long chain of 'if' into multiple pipeline-to-pipeline files that end to something like 20 different pipelin… --- ## [Beats plugin crashes on Logstash: A plugin had an unrecoverable error. Will restart this plugin](https://discuss.elastic.co/t/beats-plugin-crashes-on-logstash-a-plugin-had-an-unrecoverable-error-will-restart-this-plugin/152231)
**Author:** [@faulander](https://discuss.elastic.co/u/faulander)\ **Replies:** 1\ **Last updated:** [October 12, 2018, 11:57am UTC](https://discuss.elastic.co/t/beats-plugin-crashes-on-logstash-a-plugin-had-an-unrecoverable-error-will-restart-this-plugin/152231 "2018-10-12T11:57:40Z")
This only happens, if i have 2 different conf files in conf.d directory: Pipeline\_id:main Plugin: \5044, id=\>"4ba10da00dcd2f166d6b349db9c4681f6ec354c312b7bc450dc25aca76c0511b", enable\_metr… --- ## [New to Logstash and am trying to set up Sonicwall syslog](https://discuss.elastic.co/t/new-to-logstash-and-am-trying-to-set-up-sonicwall-syslog/152226)
**Author:** [@trader142](https://discuss.elastic.co/u/trader142)\ **Replies:** 0\ **Last updated:** [October 12, 2018, 10:58am UTC](https://discuss.elastic.co/t/new-to-logstash-and-am-trying-to-set-up-sonicwall-syslog/152226 "2018-10-12T10:58:33Z")
I have set up ElasticSearch and am sending Winlogbeat data from my servers and am now trying to send Syslog data from the Sonicwall. I have started Logstash and have configured the Syslog to point to the server. Do I nee… --- ## [Loop over array in output](https://discuss.elastic.co/t/loop-over-array-in-output/151997)
**Author:** [@drnjst](https://discuss.elastic.co/u/drnjst)\ **Replies:** 1\ **Last updated:** [October 12, 2018, 9:43am UTC](https://discuss.elastic.co/t/loop-over-array-in-output/151997 "2018-10-12T09:43:36Z")
Hello currently I have code that works, but could be improved if I would be able to iterate. Now I say if the 3rd element of the array exists, output the 3rd element (into a file). See code below: output { if \[value… --- ## [Remove dynamic nested field in JSON](https://discuss.elastic.co/t/remove-dynamic-nested-field-in-json/152062)
**Author:** [@mahipal\_baid](https://discuss.elastic.co/u/mahipal_baid)\ **Replies:** 1\ **Last updated:** [October 12, 2018, 9:23am UTC](https://discuss.elastic.co/t/remove-dynamic-nested-field-in-json/152062 "2018-10-12T09:23:21Z")
Hi , I have an input (JSON) like below: // Sample input JSON doc "TestCases": { "0": { "Verdict": "PASS", "tcId": 0, "NoOfFail": 0 }, "1": { } Where number of test cases is not static and I want t… --- ## [Unable to output to elastic cloud](https://discuss.elastic.co/t/unable-to-output-to-elastic-cloud/152190)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [October 12, 2018, 8:51am UTC](https://discuss.elastic.co/t/unable-to-output-to-elastic-cloud/152190 "2018-10-12T08:51:31Z")
Hi, Logstash is trowing the following error when trying to output to elastic cloud. I've tried using the elastic user account and password as well to avoid any permission issues. var hash = window.location.hash; if (h… --- ## [Trying to separate key-value pairs in nested array (help with RUBY)](https://discuss.elastic.co/t/trying-to-separate-key-value-pairs-in-nested-array-help-with-ruby/152035)
**Author:** [@Harshet\_Jain](https://discuss.elastic.co/u/Harshet_Jain)\ **Replies:** 6\ **Last updated:** [October 12, 2018, 8:49am UTC](https://discuss.elastic.co/t/trying-to-separate-key-value-pairs-in-nested-array-help-with-ruby/152035 "2018-10-12T08:49:21Z")
I have been trying to import my mySQL database into Elasticsearch through Logstash but I am stuck (for two days now) to separate three columns into separate fields. I want to split these into different fields so it disp… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=667) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=669)