# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1075 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1076 --- ## [Logstash fails to start the simplest pipeline](https://discuss.elastic.co/t/logstash-fails-to-start-the-simplest-pipeline/66307)
**Author:** [@letharion](https://discuss.elastic.co/u/letharion)\ **Replies:** 2\ **Last updated:** [November 21, 2016, 7:05am UTC](https://discuss.elastic.co/t/logstash-fails-to-start-the-simplest-pipeline/66307 "2016-11-21T07:05:08Z")
\# java -version openjdk version "1.8.0\_111" OpenJDK Runtime Environment (build 1.8.0\_111-8u111-b14-2ubuntu0.16.04.2-b14) OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode) # bin/logstash --version logstash 5.0.1 #… --- ## [Importing lots from journald](https://discuss.elastic.co/t/importing-lots-from-journald/66286)
**Author:** [@levian](https://discuss.elastic.co/u/levian)\ **Replies:** 1\ **Last updated:** [November 21, 2016, 6:42am UTC](https://discuss.elastic.co/t/importing-lots-from-journald/66286 "2016-11-21T06:42:23Z")
I've written an HTTP server that outputs access logs to STDOUT. The server is setup as a service using systemd, so all the access logs end up in journald. I'm running the ELK stack on a second server that I would like to… --- ## [Timelion simple query help please](https://discuss.elastic.co/t/timelion-simple-query-help-please/66264)
**Author:** [@DustinVest](https://discuss.elastic.co/u/DustinVest)\ **Replies:** 1\ **Last updated:** [November 21, 2016, 6:40am UTC](https://discuss.elastic.co/t/timelion-simple-query-help-please/66264 "2016-11-21T06:40:14Z")
I am trying to figure out a very simple query and im not having much luck. Most of the information I am getting from google relates to numbers, but how would I get a time series on bool values. For example active directo… --- ## [Logstash query to text output TOO SLOW](https://discuss.elastic.co/t/logstash-query-to-text-output-too-slow/66246)
**Author:** [@billy6](https://discuss.elastic.co/u/billy6)\ **Replies:** 1\ **Last updated:** [November 21, 2016, 6:39am UTC](https://discuss.elastic.co/t/logstash-query-to-text-output-too-slow/66246 "2016-11-21T06:39:11Z")
Hello, I have this logstash configuration \> input{ elasticsearch { hosts=\> \[ "HOST\_A"\] index =\> "logstash-syslog-2016.11.17" query =\> '{"filter" : { "or" : \[ { … --- ## [Logstash vs spark streaming and storm](https://discuss.elastic.co/t/logstash-vs-spark-streaming-and-storm/55133)
**Author:** [@sen](https://discuss.elastic.co/u/sen)\ **Replies:** 2\ **Last updated:** [November 21, 2016, 6:29am UTC](https://discuss.elastic.co/t/logstash-vs-spark-streaming-and-storm/55133 "2016-11-21T06:29:34Z")
Hi folks, I am working on building a distributed real time cluster system to supervise and analyze a network. I did several researches on internet and I came out with few technologies: -for real time processing : logst… --- ## [Twitter Client Error "Connection Refused"](https://discuss.elastic.co/t/twitter-client-error-connection-refused/66205)
**Author:** [@Vikas\_Reddy\_Aravabhu](https://discuss.elastic.co/u/Vikas_Reddy_Aravabhu)\ **Replies:** 2\ **Last updated:** [November 21, 2016, 2:03am UTC](https://discuss.elastic.co/t/twitter-client-error-connection-refused/66205 "2016-11-21T02:03:54Z")
Twitter client error {:message=\>"Connection refused - Connection refused", :exception=\>"Errno::ECONNREFUSED", :backtrace=\>\["org/jruby/ext/socket/RubyTCPSocket.java:126:in initialize'", "org/jruby/RubyIO.java:871:innew'",… --- ## [JDBC plugin doesn't work](https://discuss.elastic.co/t/jdbc-plugin-doesnt-work/66244)
**Author:** [@Idan](https://discuss.elastic.co/u/Idan)\ **Replies:** 3\ **Last updated:** [November 20, 2016, 4:43pm UTC](https://discuss.elastic.co/t/jdbc-plugin-doesnt-work/66244 "2016-11-20T16:43:01Z")
I'm trying to use the JDBC plugin has you recommended to connect to MySQL retrive query data and send to Elasticsearch for graph. I get this error: \[2016-11-15T14:16:28,314\]\[ERROR\]\[logstash.agent \] fetched an invalid con… --- ## [Logstash in windows](https://discuss.elastic.co/t/logstash-in-windows/66635)
**Author:** [@xsallowed](https://discuss.elastic.co/u/xsallowed)\ **Replies:** 1\ **Last updated:** [November 20, 2016, 4:36pm UTC](https://discuss.elastic.co/t/logstash-in-windows/66635 "2016-11-20T16:36:51Z")
Hi Folks I am a newbie here and all I am trying to do is find a way to see if logstash is able to read my files and is it able to parse it correctly. I took a basic config file from internet and trying to run logstash wi… --- ## [How filter twitter input data based on @timestamps field in logstash?](https://discuss.elastic.co/t/how-filter-twitter-input-data-based-on-timestamps-field-in-logstash/66198)
**Author:** [@jamal](https://discuss.elastic.co/u/jamal)\ **Replies:** 1\ **Last updated:** [November 20, 2016, 4:32pm UTC](https://discuss.elastic.co/t/how-filter-twitter-input-data-based-on-timestamps-field-in-logstash/66198 "2016-11-20T16:32:15Z")
I want to filter twitter input data based on @timestamp field my mean just get new twitt of twiiter not old twitt, any one help to how filter it with logstash-filter-date plugin --- ## [Translate return multiple fields](https://discuss.elastic.co/t/translate-return-multiple-fields/64590)
**Author:** [@Hans](https://discuss.elastic.co/u/Hans)\ **Replies:** 11\ **Last updated:** [November 20, 2016, 4:30pm UTC](https://discuss.elastic.co/t/translate-return-multiple-fields/64590 "2016-11-20T16:30:35Z")
Hi, I have been using the translate plugin that works great however I have multiple fields that all reference to the one field used for the translate. So is it possible to use the translate plugin to lookup a value again… --- ## [How to configure logstash to identify column headers?](https://discuss.elastic.co/t/how-to-configure-logstash-to-identify-column-headers/66181)
**Author:** [@deepu.sundar](https://discuss.elastic.co/u/deepu.sundar)\ **Replies:** 1\ **Last updated:** [November 20, 2016, 4:25pm UTC](https://discuss.elastic.co/t/how-to-configure-logstash-to-identify-column-headers/66181 "2016-11-20T16:25:53Z")
Hi, Is there way to configure Logstash (in ELK) such that the first line wil be identified as the column header and then it will create and index the remaining records based on that. Appreciate any help on this . --- ## [All values become strings](https://discuss.elastic.co/t/all-values-become-strings/66176)
**Author:** [@rolfberkenbosch](https://discuss.elastic.co/u/rolfberkenbosch)\ **Replies:** 1\ **Last updated:** [November 20, 2016, 4:15pm UTC](https://discuss.elastic.co/t/all-values-become-strings/66176 "2016-11-20T16:15:27Z")
Hi, I have the following code, but the ruby code is not working for detecting numeric values. So all fields are strings: input { tcp { host =\> "172.30.19.177" port =\> 5000 type =\> syslog } udp { … --- ## [How to define index name for kibana index pattern](https://discuss.elastic.co/t/how-to-define-index-name-for-kibana-index-pattern/66614)
**Author:** [@student975](https://discuss.elastic.co/u/student975)\ **Replies:** 9\ **Last updated:** [November 20, 2016, 10:07am UTC](https://discuss.elastic.co/t/how-to-define-index-name-for-kibana-index-pattern/66614 "2016-11-20T10:07:58Z")
Hi! How to define index name in multi-ls-on-hosts -\> rabbit -\> ls -\> kibana chain to be further used in kibana index pattern? --- ## [5.x: input/filter/output in different conf files](https://discuss.elastic.co/t/5-x-input-filter-output-in-different-conf-files/66627)
**Author:** [@strarsis](https://discuss.elastic.co/u/strarsis)\ **Replies:** 1\ **Last updated:** [November 20, 2016, 7:18am UTC](https://discuss.elastic.co/t/5-x-input-filter-output-in-different-conf-files/66627 "2016-11-20T07:18:13Z")
In logstash previous 5.x, it was possible to specify the input/filter/output blocks in different configuration files (in conf.d/ directory). With 5.x, apparently this isn't possible anymore, it will terminate when ther… --- ## [Can I create different index patterns for different logs types in logstash?](https://discuss.elastic.co/t/can-i-create-different-index-patterns-for-different-logs-types-in-logstash/66588)
**Author:** [@robertchen117](https://discuss.elastic.co/u/robertchen117)\ **Replies:** 1\ **Last updated:** [November 19, 2016, 4:37am UTC](https://discuss.elastic.co/t/can-i-create-different-index-patterns-for-different-logs-types-in-logstash/66588 "2016-11-19T04:37:42Z")
in logstash 2.4.1, can I create different index patterns for different logs types? I see if just one logstash-\* index pattern, it mess up all fields together. thanks. --- ## [Translate filter not working for numbers extracted through Regular Expression](https://discuss.elastic.co/t/translate-filter-not-working-for-numbers-extracted-through-regular-expression/66623)
**Author:** [@Hans](https://discuss.elastic.co/u/Hans)\ **Replies:** 2\ **Last updated:** [November 19, 2016, 3:22pm UTC](https://discuss.elastic.co/t/translate-filter-not-working-for-numbers-extracted-through-regular-expression/66623 "2016-11-19T15:22:57Z")
Hi, I have difficulties with a filed that I extracted from a long number to lookup with the translate filter. Here is the initial number: 649032011715511 However I only require the last 5 digits and extract these wi… --- ## [Ruby Filter change in 5](https://discuss.elastic.co/t/ruby-filter-change-in-5/66437)
**Author:** [@Hans](https://discuss.elastic.co/u/Hans)\ **Replies:** 3\ **Last updated:** [November 19, 2016, 2:29pm UTC](https://discuss.elastic.co/t/ruby-filter-change-in-5/66437 "2016-11-19T14:29:24Z")
Hi, I have been using a Ruby filter for some of by information, however since I have upgraded to 5 these no longer work. I have tried to understand exactly what changes under https://www.elastic.co/guide/en/logstash/5.x/… --- ## [Logstash Authentication doesn't work after x-tools install](https://discuss.elastic.co/t/logstash-authentication-doesnt-work-after-x-tools-install/66595)
**Author:** [@Frank\_Trezza](https://discuss.elastic.co/u/Frank_Trezza)\ **Replies:** 1\ **Last updated:** [November 19, 2016, 8:58am UTC](https://discuss.elastic.co/t/logstash-authentication-doesnt-work-after-x-tools-install/66595 "2016-11-19T08:58:58Z")
I installed X-tools and my logstash.conf file no longer was able to connect, getting 401 errors. I followed the steps outlined in https://www.elastic.co/guide/en/x-pack/current/logstash.html and created the users and con… --- ## [Logstash parses log fine, elasticsearch throws date error?](https://discuss.elastic.co/t/logstash-parses-log-fine-elasticsearch-throws-date-error/66611)
**Author:** [@encee](https://discuss.elastic.co/u/encee)\ **Replies:** 1\ **Last updated:** [November 19, 2016, 6:12am UTC](https://discuss.elastic.co/t/logstash-parses-log-fine-elasticsearch-throws-date-error/66611 "2016-11-19T06:12:12Z")
Log: 2016-11-19 05:15:34.537 \[INFO \] \[15722:119885\] Blah blah blah Date Config: date { match =\> \[ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" \] timezone =\> 'UTC' locale =\> 'en' } Error: {:timestamp=\>"2… --- ## [Couchdb\_changes plugin and logstash 5.0.1](https://discuss.elastic.co/t/couchdb-changes-plugin-and-logstash-5-0-1/66600)
**Author:** [@DrRob](https://discuss.elastic.co/u/DrRob)\ **Replies:** 0\ **Last updated:** [November 18, 2016, 11:21pm UTC](https://discuss.elastic.co/t/couchdb-changes-plugin-and-logstash-5-0-1/66600 "2016-11-18T23:21:20Z")
Hi, The couchdb\_changes plugin doesn't work in logstash 5.0.1. I installed it to CentOS 7 from the RPM repository. The error message says: Error: Direct event field references (i.e. event\['field'\]) have been disabl… --- ## [How to keep track of failed records or events from logstash](https://discuss.elastic.co/t/how-to-keep-track-of-failed-records-or-events-from-logstash/66585)
**Author:** [@deepu.sundar](https://discuss.elastic.co/u/deepu.sundar)\ **Replies:** 0\ **Last updated:** [November 18, 2016, 8:11pm UTC](https://discuss.elastic.co/t/how-to-keep-track-of-failed-records-or-events-from-logstash/66585 "2016-11-18T20:11:31Z")
Hi Team, What is the best way to know that how many records got failed while indexing into ES from logstash. We are trying to index AWS monthly billing files (of size 18G+, having 35+ million records) the record count… --- ## [\[NEWUSER\] JRuby ext built for wrong Java version](https://discuss.elastic.co/t/newuser-jruby-ext-built-for-wrong-java-version/66571)
**Author:** [@utdrmac](https://discuss.elastic.co/u/utdrmac)\ **Replies:** 1\ **Last updated:** [November 18, 2016, 8:06pm UTC](https://discuss.elastic.co/t/newuser-jruby-ext-built-for-wrong-java-version/66571 "2016-11-18T20:06:35Z")
Hi there. First time user of Logstash. I just installed OpenJDK 1.8 onto this machine \[root@hotspare ~\]# rpm -qa | grep openjdk java-1.8.0-openjdk-1.8.0.111-0.b15.el6\_8.x86\_64 java-1.8.0-openjdk-headless-1.8.0.111-0.b1… --- ## [Email plugin](https://discuss.elastic.co/t/email-plugin/66583)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 0\ **Last updated:** [November 18, 2016, 7:59pm UTC](https://discuss.elastic.co/t/email-plugin/66583 "2016-11-18T19:59:21Z")
I've been trouble shooting the email plugin for a while now and there has been very little information generally about it. It is very difficult to know which fields are required for different situations. Specifically exa… --- ## [Logstash http\_poller POST Syntax](https://discuss.elastic.co/t/logstash-http-poller-post-syntax/51674)
**Author:** [@pramodxyle](https://discuss.elastic.co/u/pramodxyle)\ **Replies:** 1\ **Last updated:** [November 18, 2016, 7:39pm UTC](https://discuss.elastic.co/t/logstash-http-poller-post-syntax/51674 "2016-11-18T19:39:45Z")
Hi, I am trying to POST data to a REST API and get the response back in logstash and output it to elastic. What is the syntax to POST data ? Is this correct ? http\_poller { urls =\> { url1 =\> { … --- ## [LoadError: no such file to load -- sequel](https://discuss.elastic.co/t/loaderror-no-such-file-to-load-sequel/66568)
**Author:** [@cowarish](https://discuss.elastic.co/u/cowarish)\ **Replies:** 0\ **Last updated:** [November 18, 2016, 5:48pm UTC](https://discuss.elastic.co/t/loaderror-no-such-file-to-load-sequel/66568 "2016-11-18T17:48:34Z")
I am trying to connect logstash to SQL via Jdbc. My input is as such: input { jdbc { jdbc\_connection\_string =\> "jdbc:mysql://sqlservername\\INST1;databaseName=dbname;integratedSecurity=true" jdbc\_user =\> "" jdbc\_passw… --- ## [Storage Nodes Also Using Logstash?](https://discuss.elastic.co/t/storage-nodes-also-using-logstash/66015)
**Author:** [@seth.yes](https://discuss.elastic.co/u/seth.yes)\ **Replies:** 14\ **Last updated:** [November 18, 2016, 4:35pm UTC](https://discuss.elastic.co/t/storage-nodes-also-using-logstash/66015 "2016-11-18T16:35:34Z")
I was recently reviewing some of my logs being ingested -- some of them have a timestamp two weeks older than when they were finally ingested. To me, this indicates a two-week latency, likely in Logstash.. In order to… --- ## [Local GEM installation fails with logstash 5.0.1 on RHEL7](https://discuss.elastic.co/t/local-gem-installation-fails-with-logstash-5-0-1-on-rhel7/66517)
**Author:** [@arnold79](https://discuss.elastic.co/u/arnold79)\ **Replies:** 1\ **Last updated:** [November 18, 2016, 3:44pm UTC](https://discuss.elastic.co/t/local-gem-installation-fails-with-logstash-5-0-1-on-rhel7/66517 "2016-11-18T15:44:55Z")
Hi All, Currently have an issue with installing a local GEM file, which worked in the previous editions like a charm. We want to install a local GEM file, which delivers us a 'logstash-codec-leef' ( a fork with minor… --- ## [Matched field truncated by grok](https://discuss.elastic.co/t/matched-field-truncated-by-grok/66526)
**Author:** [@arussel](https://discuss.elastic.co/u/arussel)\ **Replies:** 1\ **Last updated:** [November 18, 2016, 1:04pm UTC](https://discuss.elastic.co/t/matched-field-truncated-by-grok/66526 "2016-11-18T13:04:01Z")
I do: grok { match =\> { "message" =\> "%{GREEDYDATA:foo}" } } and I would expect that message and foo are the same, but foo is only the first ~4000 chars of message. I don't believe this is an event size issue, as I'… --- ## ["Do something" when the file is done being read](https://discuss.elastic.co/t/do-something-when-the-file-is-done-being-read/66536)
**Author:** [@asac2](https://discuss.elastic.co/u/asac2)\ **Replies:** 1\ **Last updated:** [November 18, 2016, 12:39pm UTC](https://discuss.elastic.co/t/do-something-when-the-file-is-done-being-read/66536 "2016-11-18T12:39:46Z")
I want to call a shellscript, when the logstash is done with sending file to ElasticSearch. I have tried "done\_do: '/my\_script.sh'" but it dint worked for me me. Any guidance will be appreciated. Regards Aman --- ## [Beats Pipeline blocked on the hour](https://discuss.elastic.co/t/beats-pipeline-blocked-on-the-hour/65765)
**Author:** [@Omar\_Al\_Zabir](https://discuss.elastic.co/u/Omar_Al_Zabir)\ **Replies:** 9\ **Last updated:** [November 18, 2016, 11:25am UTC](https://discuss.elastic.co/t/beats-pipeline-blocked-on-the-hour/65765 "2016-11-18T11:25:35Z")
Every hour, pipeline gets blocked and I get this error from logstash. I have a simple logstash config file which has in input { beats { ... } } and no filter, direct output { ... } do elasticsearch. Elasticsearch has no … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1074) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1076)