# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1006 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1007 --- ## [Errors running logstash, bit of a newb](https://discuss.elastic.co/t/errors-running-logstash-bit-of-a-newb/80269)
**Author:** [@Keith\_Sanks](https://discuss.elastic.co/u/Keith_Sanks)\ **Replies:** 5\ **Last updated:** [April 3, 2017, 11:25pm UTC](https://discuss.elastic.co/t/errors-running-logstash-bit-of-a-newb/80269 "2017-04-03T23:25:58Z")
Trying to set logstash up properly but I'm getting the following message after running it with my config file: Attempting to install template {:manage\_template=\>{"template"=\>"logstash-", "settings"=\>{"index.refresh\_inte… --- ## [\[Newbie Alert\] How would you process this data?](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111)
**Author:** [@Toontje](https://discuss.elastic.co/u/Toontje)\ **Replies:** 7\ **Last updated:** [April 3, 2017, 8:09pm UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111 "2017-04-03T20:09:20Z")
I would like to import this data into ELK. Data structure see below. --- ## [Logstash priority order when using multiple config files](https://discuss.elastic.co/t/logstash-priority-order-when-using-multiple-config-files/80507)
**Author:** [@Jayanna\_Hallur](https://discuss.elastic.co/u/Jayanna_Hallur)\ **Replies:** 3\ **Last updated:** [April 3, 2017, 7:55pm UTC](https://discuss.elastic.co/t/logstash-priority-order-when-using-multiple-config-files/80507 "2017-04-03T19:55:48Z")
Hi All I'm planning to use multiple config files in our deployment. One config file with input for beat and output to elastic search only I would like to maintain separate config file for each applications teams for… --- ## [Anyone know if all these error message will keep same in 5.x version?](https://discuss.elastic.co/t/anyone-know-if-all-these-error-message-will-keep-same-in-5-x-version/80738)
**Author:** [@DravenJohnson](https://discuss.elastic.co/u/DravenJohnson)\ **Replies:** 2\ **Last updated:** [April 3, 2017, 7:41pm UTC](https://discuss.elastic.co/t/anyone-know-if-all-these-error-message-will-keep-same-in-5-x-version/80738 "2017-04-03T19:41:27Z")
We using some key words to monitoring Logstash error: "Beats input: The circuit breaker has detected a slowdown or stall in the pipeline" "LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker" "Beats input:… --- ## [Few messages go unparsed from Logstash into Elastic without any failure](https://discuss.elastic.co/t/few-messages-go-unparsed-from-logstash-into-elastic-without-any-failure/80921)
**Author:** [@aroraj08](https://discuss.elastic.co/u/aroraj08)\ **Replies:** 3\ **Last updated:** [April 3, 2017, 3:45pm UTC](https://discuss.elastic.co/t/few-messages-go-unparsed-from-logstash-into-elastic-without-any-failure/80921 "2017-04-03T15:45:24Z")
I have deployed Filebeat -\> Logstash -\> ElasticSearch -\> Kibana stack for log monitoring in my testing environment. I have two GROK patterns specified in the Logstash configuration. The problem is that few of the message… --- ## [Custom Plugin not found](https://discuss.elastic.co/t/custom-plugin-not-found/79565)
**Author:** [@ohk](https://discuss.elastic.co/u/ohk)\ **Replies:** 5\ **Last updated:** [April 3, 2017, 3:13pm UTC](https://discuss.elastic.co/t/custom-plugin-not-found/79565 "2017-04-03T15:13:03Z")
Im working this plugin to fetch logs from hdfs, this is the gemspec file: s.name = 'logstash-input-webhdfs' s.version = '0.1.0' s.licenses = \['Apache License (2.0)'\] s.summary = 'Write… --- ## [Logstash Plugins: input-webhdfs](https://discuss.elastic.co/t/logstash-plugins-input-webhdfs/80656)
**Author:** [@ohk](https://discuss.elastic.co/u/ohk)\ **Replies:** 2\ **Last updated:** [April 3, 2017, 3:04pm UTC](https://discuss.elastic.co/t/logstash-plugins-input-webhdfs/80656 "2017-04-03T15:04:33Z")
Hi, I need a plugin able to read logs files from HDFS. I took the output-webhdfs plugin and changed to download files from HDFS using its REST API then parsing this temporary files with adequate codec. For now I can co… --- ## [Appending app field to coreos logs](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312)
**Author:** [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\ **Replies:** 9\ **Last updated:** [April 3, 2017, 2:48pm UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312 "2017-04-03T14:48:02Z")
I have logs coming in from coreos nodes via filebeat, the "source" field is of the general form: /var/log/containers/\-\ I'd like to parse the part after /v… --- ## [Using Logstash JDBC connector for geo\_shape datatype](https://discuss.elastic.co/t/using-logstash-jdbc-connector-for-geo-shape-datatype/80767)
**Author:** [@Rory](https://discuss.elastic.co/u/Rory)\ **Replies:** 5\ **Last updated:** [April 3, 2017, 2:16pm UTC](https://discuss.elastic.co/t/using-logstash-jdbc-connector-for-geo-shape-datatype/80767 "2017-04-03T14:16:27Z")
Can we use the Logstash JDBC connector to store data into geo\_shape datatype field? The sql statement outputs 2 columns \[Latitude\] and \[Longitude\], how do we use these columns to create a geo\_shape field and Index into E… --- ## [How to write the pattern for below log](https://discuss.elastic.co/t/how-to-write-the-pattern-for-below-log/81004)
**Author:** [@jagadees](https://discuss.elastic.co/u/jagadees)\ **Replies:** 1\ **Last updated:** [April 3, 2017, 1:57pm UTC](https://discuss.elastic.co/t/how-to-write-the-pattern-for-below-log/81004 "2017-04-03T13:57:34Z")
"165.214.4.20 - - \[27/Mar/2017:23:15:38 -0400\] 'POST /servlet/rcopia.servlet.EngineServlet HTTP/1.1' 200 1123 23 - 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)'" --- ## [Date filter for logstash seems to change date to older value](https://discuss.elastic.co/t/date-filter-for-logstash-seems-to-change-date-to-older-value/80983)
**Author:** [@raazeev64](https://discuss.elastic.co/u/raazeev64)\ **Replies:** 2\ **Last updated:** [April 3, 2017, 1:06pm UTC](https://discuss.elastic.co/t/date-filter-for-logstash-seems-to-change-date-to-older-value/80983 "2017-04-03T13:06:18Z")
I am parsing logs in the format 2017-04-03 05:48:16,129.... I am parsing this using grok as match =\> {"message" =\> "%{TIMESTAMP\_ISO8601:timestamp},.\*\\|%{DATA:logLevel}\\|.\*\\|.\*\\|.\*\\|.\*\\|.\*\\|%{DATA:service}\\|.\*\\… --- ## [Video log processing logstash](https://discuss.elastic.co/t/video-log-processing-logstash/80985)
**Author:** [@rach](https://discuss.elastic.co/u/rach)\ **Replies:** 2\ **Last updated:** [April 3, 2017, 11:52am UTC](https://discuss.elastic.co/t/video-log-processing-logstash/80985 "2017-04-03T11:52:08Z")
hi :slight\_smile: can you please help me process the following video log? I will have multiple dynamic fields (which not appear all the time), and this is my log: 02/04 10:07:59.216,DEBUG,10024,JDM,"Received: FR\_JOB\_RE… --- ## [Cisco Log Processing](https://discuss.elastic.co/t/cisco-log-processing/79870)
**Author:** [@mhalatuituia](https://discuss.elastic.co/u/mhalatuituia)\ **Replies:** 22\ **Last updated:** [April 3, 2017, 10:17am UTC](https://discuss.elastic.co/t/cisco-log-processing/79870 "2017-04-03T10:17:56Z")
Anyone with working Logstash Working config for processinf Cisco Logs --- ## [Cisco Log](https://discuss.elastic.co/t/cisco-log/80280)
**Author:** [@mhalatuituia](https://discuss.elastic.co/u/mhalatuituia)\ **Replies:** 5\ **Last updated:** [April 3, 2017, 10:17am UTC](https://discuss.elastic.co/t/cisco-log/80280 "2017-04-03T10:17:16Z")
hi i have been trying for quite some time but i cannot get it to work .... here is my config .. file \>\>\>\>\>\>\>\> 02-beat-input.conf input { beats { port =\> 5044 ssl =\> true ssl\_certificate =\> "/etc/pki/tls/… --- ## [Logs are not displaying in order](https://discuss.elastic.co/t/logs-are-not-displaying-in-order/80409)
**Author:** [@Raja1](https://discuss.elastic.co/u/Raja1)\ **Replies:** 49\ **Last updated:** [April 3, 2017, 9:54am UTC](https://discuss.elastic.co/t/logs-are-not-displaying-in-order/80409 "2017-04-03T09:54:50Z")
Hi All, I have configured ELK with Layer 7 logs locally by taking the logs from gateway. I am able to get the logs in Kibana dashboard but the logs are not displayed in order. Below log is complete one entry but in … --- ## [Logstash stop parsing IIS logs on Windows](https://discuss.elastic.co/t/logstash-stop-parsing-iis-logs-on-windows/80831)
**Author:** [@pad](https://discuss.elastic.co/u/pad)\ **Replies:** 3\ **Last updated:** [April 3, 2017, 8:49am UTC](https://discuss.elastic.co/t/logstash-stop-parsing-iis-logs-on-windows/80831 "2017-04-03T08:49:45Z")
Hi, I'm using Logstash (2.3.4) on Windows to parse IIS logs (IIS 8.5) on two different server, one on Azure and one on our datacenter. With the same configuration and the same version of Logstash, the one on our data… --- ## [Unable to reroute logs emitted from logstash](https://discuss.elastic.co/t/unable-to-reroute-logs-emitted-from-logstash/80954)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 2\ **Last updated:** [April 3, 2017, 7:30am UTC](https://discuss.elastic.co/t/unable-to-reroute-logs-emitted-from-logstash/80954 "2017-04-03T07:30:13Z")
Hi, I am stuck trying to reroute the path of logs coming from logstash. In logstash.yml that is in etc/logstash I configured to send logs to ~/logstash as a test: # ------------ Debugging Settings -------------- # # … --- ## [Dealing with improperly formatted XML](https://discuss.elastic.co/t/dealing-with-improperly-formatted-xml/78638)
**Author:** [@Kvetch](https://discuss.elastic.co/u/Kvetch)\ **Replies:** 12\ **Last updated:** [April 3, 2017, 5:08am UTC](https://discuss.elastic.co/t/dealing-with-improperly-formatted-xml/78638 "2017-04-03T05:08:08Z")
I have an XML that isn't parsing correctly unless I use something like xmllint to prettify it. The XML file is written line by line but doesn't contain indentation and such and for whatever reason my logstash config doe… --- ## [Unable to install logstash filter plugin](https://discuss.elastic.co/t/unable-to-install-logstash-filter-plugin/80775)
**Author:** [@vits23](https://discuss.elastic.co/u/vits23)\ **Replies:** 2\ **Last updated:** [April 3, 2017, 4:54am UTC](https://discuss.elastic.co/t/unable-to-install-logstash-filter-plugin/80775 "2017-04-03T04:54:14Z")
I have cloned one of the filter plugin available and trying to install the plugin but it is getting failed due to below: Could not find gem 'logstash-devutils x64-mingw32' in any of the gem sources listed in your Gemfile… --- ## [String field to date, parsing from xpath](https://discuss.elastic.co/t/string-field-to-date-parsing-from-xpath/80872)
**Author:** [@daian.martinho](https://discuss.elastic.co/u/daian.martinho)\ **Replies:** 3\ **Last updated:** [April 3, 2017, 2:49am UTC](https://discuss.elastic.co/t/string-field-to-date-parsing-from-xpath/80872 "2017-04-03T02:49:26Z")
Hi, I'm trying to parse string field as date. i got the value from an xml using xpath.. xpath =\>\[ "/ReportHost/HostProperties/tag\[@name='HOST\_START'\]/text()","report\_host\_start"… --- ## [Apache Logs - Logstash Book Example](https://discuss.elastic.co/t/apache-logs-logstash-book-example/80884)
**Author:** [@seag33k](https://discuss.elastic.co/u/seag33k)\ **Replies:** 1\ **Last updated:** [April 3, 2017, 2:31am UTC](https://discuss.elastic.co/t/apache-logs-logstash-book-example/80884 "2017-04-03T02:31:22Z")
I am reading The Logstash Book and there is an example of custom Apache logging. I am sorry if this is more of an apache questions, but it is directly related to logstash. I've created the following conf file for my Ub… --- ## [Date filter not working](https://discuss.elastic.co/t/date-filter-not-working/79679)
**Author:** [@techols](https://discuss.elastic.co/u/techols)\ **Replies:** 5\ **Last updated:** [April 1, 2017, 11:29pm UTC](https://discuss.elastic.co/t/date-filter-not-working/79679 "2017-04-01T23:29:31Z")
I have an xml file being ingested by logstash. The xml file has a timestamp element with a unix\_ms format (e.g., 1458827492928). My date filter is not properly parsing this and setting the @timestamp variable. I creat… --- ## [Trying to use logstash-input-google\_pubsub](https://discuss.elastic.co/t/trying-to-use-logstash-input-google-pubsub/80469)
**Author:** [@dinaWork](https://discuss.elastic.co/u/dinaWork)\ **Replies:** 2\ **Last updated:** [April 1, 2017, 5:18pm UTC](https://discuss.elastic.co/t/trying-to-use-logstash-input-google-pubsub/80469 "2017-04-01T17:18:00Z")
here is conf/logstash.conf input { google\_pubsub { project\_id =\> "myProjectId" topic =\> "logstash-input-dev" subscription =\> "logstash-sub" json\_key\_file =\> "/usr/share/logstash/ke… --- ## [Questions about the timezone](https://discuss.elastic.co/t/questions-about-the-timezone/80894)
**Author:** [@qin](https://discuss.elastic.co/u/qin)\ **Replies:** 4\ **Last updated:** [April 1, 2017, 2:27pm UTC](https://discuss.elastic.co/t/questions-about-the-timezone/80894 "2017-04-01T14:27:06Z")
I want to let logstash output to elasticsearch, My configuration is input { tcp { host =\> "0.0.0.0" port =\> 4560 mode =\> "server" codec =\> json\_lines } } output { s… --- ## [Processing bro output logs (from pcaps) with logstash](https://discuss.elastic.co/t/processing-bro-output-logs-from-pcaps-with-logstash/80853)
**Author:** [@10past10](https://discuss.elastic.co/u/10past10)\ **Replies:** 2\ **Last updated:** [April 1, 2017, 1:27pm UTC](https://discuss.elastic.co/t/processing-bro-output-logs-from-pcaps-with-logstash/80853 "2017-04-01T13:27:09Z")
I'm trying to get bro output processed with logstash to elasticsearch, but for some reason I can't get it to work. Am I missing something? Below is some sample output that I'm trying to process. I got this from a pcap … --- ## [Offline install x-pack on windows fails v5.3.0](https://discuss.elastic.co/t/offline-install-x-pack-on-windows-fails-v5-3-0/80722)
**Author:** [@Kim-Kruse-Hansen](https://discuss.elastic.co/u/Kim-Kruse-Hansen)\ **Replies:** 3\ **Last updated:** [April 1, 2017, 10:12am UTC](https://discuss.elastic.co/t/offline-install-x-pack-on-windows-fails-v5-3-0/80722 "2017-04-01T10:12:52Z")
Hi I am trying to incorporate x-pack in my logstash setup scripts. However offline install of x-pack seems to be broken on windows. If I install it online , it works fine. c:\\elastic\\logstash\\bin\>call logstash-pl… --- ## [Can Logstash output Mongo 3.4's new type Decimal128?](https://discuss.elastic.co/t/can-logstash-output-mongo-3-4s-new-type-decimal128/80889)
**Author:** [@ddxv](https://discuss.elastic.co/u/ddxv)\ **Replies:** 0\ **Last updated:** [April 1, 2017, 6:05am UTC](https://discuss.elastic.co/t/can-logstash-output-mongo-3-4s-new-type-decimal128/80889 "2017-04-01T06:05:30Z")
Mongo 3.4 has a new BSON Type: Decimal128 ("decimal") along with a new constructor NumberDecimal(). An example field looks like: NumberDecimal("123.456") I can not find any documentation on filters that would support… --- ## ["Failed to flush outgoing items" Error with logstash-output-webhdfs plugin](https://discuss.elastic.co/t/failed-to-flush-outgoing-items-error-with-logstash-output-webhdfs-plugin/80601)
**Author:** [@jackhe](https://discuss.elastic.co/u/jackhe)\ **Replies:** 2\ **Last updated:** [April 1, 2017, 3:34am UTC](https://discuss.elastic.co/t/failed-to-flush-outgoing-items-error-with-logstash-output-webhdfs-plugin/80601 "2017-04-01T03:34:19Z")
Dear all, When I tried use logstash-output-webhdfs plugin to archive logs into hdfs, I met below error occasionly(not always). Does anyone know what happened and how to fix this ? Failed to flush outgoing items {:out… --- ## [Logstash filter plugin - Missing class name](https://discuss.elastic.co/t/logstash-filter-plugin-missing-class-name/80631)
**Author:** [@Conall\_Prendergast](https://discuss.elastic.co/u/Conall_Prendergast)\ **Replies:** 2\ **Last updated:** [March 31, 2017, 10:33pm UTC](https://discuss.elastic.co/t/logstash-filter-plugin-missing-class-name/80631 "2017-03-31T22:33:56Z")
Hi, I am trying to convert a logstash 2.4 plugin to be compatible with 5.x. I made the necessary updates and when I run "rspec", I get the below error. NameError: missing class name (\`org.apache.logging.log4j.Level')… --- ## [Convert String to timestamp example](https://discuss.elastic.co/t/convert-string-to-timestamp-example/80856)
**Author:** [@cabishek188](https://discuss.elastic.co/u/cabishek188)\ **Replies:** 6\ **Last updated:** [March 31, 2017, 8:36pm UTC](https://discuss.elastic.co/t/convert-string-to-timestamp-example/80856 "2017-03-31T20:36:23Z")
Hi all, I want to convert a String to be used in addition to @timestamp. Please help me as to how I do it. I used the "date" filter as mentioned in lot of posts but was unable to get it working. My message: message… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1005) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1007)