# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=747 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 748 --- ## [Logstash Beat pipeline blocked](https://discuss.elastic.co/t/logstash-beat-pipeline-blocked/133980)
**Author:** [@tharu85](https://discuss.elastic.co/u/tharu85)\ **Replies:** 1\ **Last updated:** [June 5, 2018, 8:54am UTC](https://discuss.elastic.co/t/logstash-beat-pipeline-blocked/133980 "2018-06-05T08:54:10Z")
When I run logstash.conf, I am getting following error log (refer below logs). NOTE: I used same logstash.conf script for more than 6 months without any issue. Today I have add only following lines in one of filter used… --- ## [Split function in ruby filter](https://discuss.elastic.co/t/split-function-in-ruby-filter/134538)
**Author:** [@Shivom\_Kumar](https://discuss.elastic.co/u/Shivom_Kumar)\ **Replies:** 0\ **Last updated:** [June 5, 2018, 6:52am UTC](https://discuss.elastic.co/t/split-function-in-ruby-filter/134538 "2018-06-05T06:52:51Z")
i want to use spilt function in ruby to split one list into two arrays. however there are some type of values that i want to ignore based on some regex. i have defined some regex in patterns dir of logstash. can someon… --- ## [Logstash Event Processing is decreasing as times go by](https://discuss.elastic.co/t/logstash-event-processing-is-decreasing-as-times-go-by/133963)
**Author:** [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Replies:** 1\ **Last updated:** [June 5, 2018, 5:20am UTC](https://discuss.elastic.co/t/logstash-event-processing-is-decreasing-as-times-go-by/133963 "2018-06-05T05:20:59Z")
Here's what we did: Started the logstash. After few hours, logstash performance is going down and little by little, few events are being processed. We checked the metricbeat report before restarting the logstash and fo… --- ## [Relative value for\`patterns\_dir\`](https://discuss.elastic.co/t/relative-value-for-patterns-dir/134505)
**Author:** [@amomchilov](https://discuss.elastic.co/u/amomchilov)\ **Replies:** 0\ **Last updated:** [June 5, 2018, 1:32am UTC](https://discuss.elastic.co/t/relative-value-for-patterns-dir/134505 "2018-06-05T01:32:07Z")
Can the value of patterns\_dir be relative? If so, where is it relative to? I'm deploying Logstash using a dependency management system that could effect its install location. I don't want my configs to be sensitive to t… --- ## [SEMANTIC Name Issue](https://discuss.elastic.co/t/semantic-name-issue/134494)
**Author:** [@Sivachandra](https://discuss.elastic.co/u/Sivachandra)\ **Replies:** 0\ **Last updated:** [June 4, 2018, 9:42pm UTC](https://discuss.elastic.co/t/semantic-name-issue/134494 "2018-06-04T21:42:50Z")
Hi, I am using grok filter in my logstash.conf file and I would like to create SEMANTIC name like processing\_time(sec). I have tried like %{NUMBER:processing\_time(sec)}. However, I could still see 'processing\_time' as a… --- ## [Kibana Ingest Pipeline Parsing exception](https://discuss.elastic.co/t/kibana-ingest-pipeline-parsing-exception/134115)
**Author:** [@murali\_nutakkivikjuv](https://discuss.elastic.co/u/murali_nutakkivikjuv)\ **Replies:** 5\ **Last updated:** [June 4, 2018, 8:30pm UTC](https://discuss.elastic.co/t/kibana-ingest-pipeline-parsing-exception/134115 "2018-06-04T20:30:45Z")
I'm trying to ingest an HTTP log through file beat into kibana Pipeline and getting the parse error. The Grok Pattern worked well with grok debugger but the same pattern failed when I attempt to ingest through kibana pip… --- ## [LogStash JDBC input Error - sqlanywhere - Sybase - undefined method \`new](https://discuss.elastic.co/t/logstash-jdbc-input-error-sqlanywhere-sybase-undefined-method-new/132907)
**Author:** [@pquery](https://discuss.elastic.co/u/pquery)\ **Replies:** 1\ **Last updated:** [June 4, 2018, 6:08pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-error-sqlanywhere-sybase-undefined-method-new/132907 "2018-06-04T18:08:18Z")
I have been trying to use the JDBC input plugin to connect to my Sybase database (running on a SUSE server) and I think that I have it close but it is throwing a strange error at the moment: \[2018-05-23T00:47:43,415\]\[ER… --- ## [Send data to multiple destinations - store data in multiple kafka topics?](https://discuss.elastic.co/t/send-data-to-multiple-destinations-store-data-in-multiple-kafka-topics/134300)
**Author:** [@vb4t](https://discuss.elastic.co/u/vb4t)\ **Replies:** 3\ **Last updated:** [June 4, 2018, 5:48pm UTC](https://discuss.elastic.co/t/send-data-to-multiple-destinations-store-data-in-multiple-kafka-topics/134300 "2018-06-04T17:48:15Z")
Hello, I would like to send some events to more than one ES cluster. I have following architecture: LOG SOURCE -\> LOGSTASH -\> KAFKA -\> LOGSTASH -\> ES The first logstash process logs and sends it into kafka. The second… --- ## [Export From elasticsearch to S3](https://discuss.elastic.co/t/export-from-elasticsearch-to-s3/134112)
**Author:** [@praveen.vemuri](https://discuss.elastic.co/u/praveen.vemuri)\ **Replies:** 5\ **Last updated:** [June 4, 2018, 5:36pm UTC](https://discuss.elastic.co/t/export-from-elasticsearch-to-s3/134112 "2018-06-04T17:36:23Z")
Hi, What is the best way to export raw data from elasticsearch to s3? i read about elasticdump and there we have to download to local filesystem and then copy to s3? --- ## [Extracting a substring](https://discuss.elastic.co/t/extracting-a-substring/134268)
**Author:** [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\ **Replies:** 2\ **Last updated:** [June 4, 2018, 3:59pm UTC](https://discuss.elastic.co/t/extracting-a-substring/134268 "2018-06-04T15:59:05Z")
I was wondering what I can use to extract the first three letters of a given string "/opt/LogStashOutputFormatted/TMOES/TNTMO072.2018-05-02\_08:40.log" I want to extract " TNT" from the string was able to extract TNTMO0… --- ## [Prune plugin cannot be found by systems after being installed](https://discuss.elastic.co/t/prune-plugin-cannot-be-found-by-systems-after-being-installed/134453)
**Author:** [@Mr\_BAU](https://discuss.elastic.co/u/Mr_BAU)\ **Replies:** 0\ **Last updated:** [June 4, 2018, 3:37pm UTC](https://discuss.elastic.co/t/prune-plugin-cannot-be-found-by-systems-after-being-installed/134453 "2018-06-04T15:37:02Z")
Hi, maybe I'm looking at this all wrong, but I can't find what I"m doing wrong here. I installed the prune filter, with this command: bin/logstash-plugin install logstash-filter-prune which was successful and I can l… --- ## [Logstash Arcsight Module issue](https://discuss.elastic.co/t/logstash-arcsight-module-issue/133457)
**Author:** [@Madhukar](https://discuss.elastic.co/u/Madhukar)\ **Replies:** 1\ **Last updated:** [June 4, 2018, 3:29pm UTC](https://discuss.elastic.co/t/logstash-arcsight-module-issue/133457 "2018-06-04T15:29:45Z")
I've to ingest logs via smart connector to logstash. I installed x-pack and the arcsight module (Followed this link https://www.elastic.co/guide/en/logstash/current/arcsight-module.html) In logstash.yml, I have mentione… --- ## [Same pattern file gives pattern not define for only one pattern](https://discuss.elastic.co/t/same-pattern-file-gives-pattern-not-define-for-only-one-pattern/134368)
**Author:** [@SashaAlex](https://discuss.elastic.co/u/SashaAlex)\ **Replies:** 1\ **Last updated:** [June 4, 2018, 2:59pm UTC](https://discuss.elastic.co/t/same-pattern-file-gives-pattern-not-define-for-only-one-pattern/134368 "2018-06-04T14:59:16Z")
I have following pattern file. CPAT (\[\\w-\]+) FRDOM (\[\\w-\]+) DEVC (\[\\w\]+) I had given permission for pattern file. but it gives the following error. Pipeline aborted due to error {:exception=\>#\"main"}](https://discuss.elastic.co/t/stopping-pipeline-id-main/134260)
**Author:** [@Pavan\_Kumar1](https://discuss.elastic.co/u/Pavan_Kumar1)\ **Replies:** 4\ **Last updated:** [June 4, 2018, 1:41pm UTC](https://discuss.elastic.co/t/stopping-pipeline-id-main/134260 "2018-06-04T13:41:38Z")
Hi, I am having an issue when starting logstash. I have checked the other solutions in the forum but they dont seem to be related to the issue I am having. Can someone please help? Thank you. \[2018-06-01T12:32:23,733\]\[… --- ## [Logstash common output for all configured pipelines](https://discuss.elastic.co/t/logstash-common-output-for-all-configured-pipelines/134412)
**Author:** [@laurentiusoica](https://discuss.elastic.co/u/laurentiusoica)\ **Replies:** 1\ **Last updated:** [June 4, 2018, 1:33pm UTC](https://discuss.elastic.co/t/logstash-common-output-for-all-configured-pipelines/134412 "2018-06-04T13:33:36Z")
Hello, In a setup with multiple Logstash pipelines, is it possible to have an output that applies to all pipelines ? --- ## [If condition in Logstash](https://discuss.elastic.co/t/if-condition-in-logstash/134400)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 2\ **Last updated:** [June 4, 2018, 1:24pm UTC](https://discuss.elastic.co/t/if-condition-in-logstash/134400 "2018-06-04T13:24:07Z")
Hi, I've the logs coming from different sources and I have running thru the ruby code to parse them. I need to create an IF condition, so that based on the Job name it has to move to the other ruby loops. The Job name… --- ## [Docker Logstash Container](https://discuss.elastic.co/t/docker-logstash-container/133833)
**Author:** [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)\ **Replies:** 2\ **Last updated:** [June 4, 2018, 10:54am UTC](https://discuss.elastic.co/t/docker-logstash-container/133833 "2018-06-04T10:54:49Z")
Hi all, Is it possible to run a Logstash Docker Container on a Windows Server 2016 or later. Using: https://www.elastic.co/guide/en/logstash/current/docker-config.html Jason --- ## [Ruby filter in logstash](https://discuss.elastic.co/t/ruby-filter-in-logstash/134404)
**Author:** [@Shivom\_Kumar](https://discuss.elastic.co/u/Shivom_Kumar)\ **Replies:** 0\ **Last updated:** [June 4, 2018, 10:50am UTC](https://discuss.elastic.co/t/ruby-filter-in-logstash/134404 "2018-06-04T10:50:58Z")
i have my log something like this : May 23 2018 06:32:47 GMT: INFO (xdr): (xdr.c:607) \[030\]: dc-state CLUSTER\_UP timelag-sec 0 lst 1527057166864 mlst 1527057166864 (2018-05-23 06:32:46 GMT) fnlst 0 (-) wslst 0 (-) shlat… --- ## [Filter grok email](https://discuss.elastic.co/t/filter-grok-email/134398)
**Author:** [@rouchad\_rouchad](https://discuss.elastic.co/u/rouchad_rouchad)\ **Replies:** 0\ **Last updated:** [June 4, 2018, 10:04am UTC](https://discuss.elastic.co/t/filter-grok-email/134398 "2018-06-04T10:04:17Z")
please i need your help, i have a problem i do not know how to use the grok filter and the conditions properly .. i have a little bit 6 types of emails and i have done some filters but it does not work if you if any of … --- ## [Jdbc driver not loading](https://discuss.elastic.co/t/jdbc-driver-not-loading/134394)
**Author:** [@hitman](https://discuss.elastic.co/u/hitman)\ **Replies:** 1\ **Last updated:** [June 4, 2018, 10:00am UTC](https://discuss.elastic.co/t/jdbc-driver-not-loading/134394 "2018-06-04T10:00:04Z")
Error: com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc\_driver\_library? that is the error i get?can anyone comeup with a solution --- ## [Generic "Something is wrong with your configuration"](https://discuss.elastic.co/t/generic-something-is-wrong-with-your-configuration/134342)
**Author:** [@Michael\_Hatch](https://discuss.elastic.co/u/Michael_Hatch)\ **Replies:** 3\ **Last updated:** [June 4, 2018, 9:16am UTC](https://discuss.elastic.co/t/generic-something-is-wrong-with-your-configuration/134342 "2018-06-04T09:16:47Z")
I would think that if you could detect an issue with plugin configuration, you would be able to supply a useful error message, but alas...Any thoughts? Elastic 6.2 jdbc plugin conf input { jdbc { id =\> "new\_user" j… --- ## [Unrecognized @timestamp value type=class org.jruby.RubyFloat](https://discuss.elastic.co/t/unrecognized-timestamp-value-type-class-org-jruby-rubyfloat/134371)
**Author:** [@Akash\_Kumar\_Dutta](https://discuss.elastic.co/u/Akash_Kumar_Dutta)\ **Replies:** 1\ **Last updated:** [June 4, 2018, 8:58am UTC](https://discuss.elastic.co/t/unrecognized-timestamp-value-type-class-org-jruby-rubyfloat/134371 "2018-06-04T08:58:32Z")
The configuration I am using for logstash is:- kafka { type =\> "docker-logs" codec =\> "json" bootstrap\_servers =\> "kafka-0.kafka-svc.ns-adutta.svc.cluster.local:9093,kafka-1.kafka-svc.ns-adutta.svc.cluster.l… --- ## [Grok tag\_on\_failure](https://discuss.elastic.co/t/grok-tag-on-failure/134310)
**Author:** [@nin77](https://discuss.elastic.co/u/nin77)\ **Replies:** 4\ **Last updated:** [June 4, 2018, 8:28am UTC](https://discuss.elastic.co/t/grok-tag-on-failure/134310 "2018-06-04T08:28:58Z")
Hey, I am using: grok { match =\> \[ "ip\_filter" , " %{IPV4:clientip}" \] tag\_on\_failure =\> \[ "\_todelete" \] } So every line with a non valid IP should be tagged \_todelete, but on Kibana I see every line tagged with it.… --- ## [Logstash agent: Cannot create pipeline error](https://discuss.elastic.co/t/logstash-agent-cannot-create-pipeline-error/134370)
**Author:** [@nikhil.k](https://discuss.elastic.co/u/nikhil.k)\ **Replies:** 1\ **Last updated:** [June 4, 2018, 7:35am UTC](https://discuss.elastic.co/t/logstash-agent-cannot-create-pipeline-error/134370 "2018-06-04T07:35:17Z")
Hi Everyone, I want to filter apache log using grok filter. eg:- 64.242.88.10 - - \[07/Mar/2004:16:05:49 -0800\] "GET /twiki/bin/edit/Main/Double\_bounce\_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846 … --- ## [ELK - Aggregating Fields from 2 different messages into a single log](https://discuss.elastic.co/t/elk-aggregating-fields-from-2-different-messages-into-a-single-log/134138)
**Author:** [@steelerz\_06](https://discuss.elastic.co/u/steelerz_06)\ **Replies:** 1\ **Last updated:** [June 4, 2018, 6:29am UTC](https://discuss.elastic.co/t/elk-aggregating-fields-from-2-different-messages-into-a-single-log/134138 "2018-06-04T06:29:48Z")
Hi All, We are a newbie to Logstash and Elastic search and trying to bring more analytics view for Firewalls. We have the following two messages (Access-Log & Flow-log) coming at different intervals: 2018-06-01T04:43:… --- ## [Heap dump file created](https://discuss.elastic.co/t/heap-dump-file-created/134169)
**Author:** [@Nikhil\_Jaiswal](https://discuss.elastic.co/u/Nikhil_Jaiswal)\ **Replies:** 2\ **Last updated:** [June 4, 2018, 3:43am UTC](https://discuss.elastic.co/t/heap-dump-file-created/134169 "2018-06-04T03:43:44Z")
Hi All, i am getting below error mesgae while running logstash 6.2.4 \[image\] and this process is creating 8Gb dump file inside logstash and logstash is crashing after this. Any help would be appreciated. --- ## [Filtering IPs with Grok](https://discuss.elastic.co/t/filtering-ips-with-grok/134296)
**Author:** [@nin77](https://discuss.elastic.co/u/nin77)\ **Replies:** 2\ **Last updated:** [June 3, 2018, 10:13pm UTC](https://discuss.elastic.co/t/filtering-ips-with-grok/134296 "2018-06-03T22:13:26Z")
Hello, I want to filter apache access logs before giving them to elastic search. I want to filter specific IPs and remove the lines where the IPs dont occur. But at the moment I am struggeling with the grok filter. I a… --- ## [Issue with logstash - rabbitmq input plugin](https://discuss.elastic.co/t/issue-with-logstash-rabbitmq-input-plugin/134287)
**Author:** [@azhar](https://discuss.elastic.co/u/azhar)\ **Replies:** 2\ **Last updated:** [June 3, 2018, 10:10pm UTC](https://discuss.elastic.co/t/issue-with-logstash-rabbitmq-input-plugin/134287 "2018-06-03T22:10:59Z")
The aim is to have a logstash server consuming the msgs from a rabbitmq queue and flushing the messages to an elasticsearch DB. My logstash.conf file content is as follows: input { rabbitmq { codec =\> "line" … --- ## [Read one file, make two filters and send for two different indexes](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291)
**Author:** [@paulobezerr](https://discuss.elastic.co/u/paulobezerr)\ **Replies:** 8\ **Last updated:** [June 2, 2018, 6:30pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291 "2018-06-02T18:30:21Z")
I have a big log CSV file, receiving logs from different equipments. My logstash read the lines and send to a daily index. I need to keep the last equipment report in a different index. Too many slow visualizations ag… --- ## [Logstash conditional each line (messages) not sending data to elasticsearch?](https://discuss.elastic.co/t/logstash-conditional-each-line-messages-not-sending-data-to-elasticsearch/133990)
**Author:** [@beckblurry](https://discuss.elastic.co/u/beckblurry)\ **Replies:** 5\ **Last updated:** [June 2, 2018, 12:55pm UTC](https://discuss.elastic.co/t/logstash-conditional-each-line-messages-not-sending-data-to-elasticsearch/133990 "2018-06-02T12:55:14Z")
Hello, I run filebeat and filebeat run each line from log to logstash, and I have each grok for each line message. But why grok not sending the second data (different field) to Elasticsearch.. This bellow my configura… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=746) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=748)