# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=866 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 867 --- ## [Something want to know about logstash persisted queue](https://discuss.elastic.co/t/something-want-to-know-about-logstash-persisted-queue/109005)
**Author:** [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\ **Replies:** 2\ **Last updated:** [November 29, 2017, 9:28am UTC](https://discuss.elastic.co/t/something-want-to-know-about-logstash-persisted-queue/109005 "2017-11-29T09:28:30Z")
Hi folks, Recently I checked the new version of Logstash, which is logstash 6.x, There are a new feature of queue based disk can instead of message broker like radis/rabbitMQ/kafaka etc.. My question is here: 1.If we… --- ## [General question on filter order (with grok filters)](https://discuss.elastic.co/t/general-question-on-filter-order-with-grok-filters/109541)
**Author:** [@pir1981](https://discuss.elastic.co/u/pir1981)\ **Replies:** 0\ **Last updated:** [November 29, 2017, 9:04am UTC](https://discuss.elastic.co/t/general-question-on-filter-order-with-grok-filters/109541 "2017-11-29T09:04:02Z")
Hi, I am new to the ELK stack and loving and learning it. I have a likely basic question, so if you know of a good thread that I missed I would love if you could let me know of it. The question is how are filter proces… --- ## [Multiple-pipelines](https://discuss.elastic.co/t/multiple-pipelines/109274)
**Author:** [@hispeed](https://discuss.elastic.co/u/hispeed)\ **Replies:** 3\ **Last updated:** [November 29, 2017, 8:46am UTC](https://discuss.elastic.co/t/multiple-pipelines/109274 "2017-11-29T08:46:53Z")
Hi, Everything worked on 5.6 so I made a snapshot and updated to V.6.0. All services are running fine. I have winlogbeat sending logs to logstash and I have a Pfsense and other stuff comming in directly to logstash. D… --- ## [\[logstash-input-jdbc plugin\] Additional tracking column types](https://discuss.elastic.co/t/logstash-input-jdbc-plugin-additional-tracking-column-types/109523)
**Author:** [@ashabunevich](https://discuss.elastic.co/u/ashabunevich)\ **Replies:** 0\ **Last updated:** [November 29, 2017, 7:20am UTC](https://discuss.elastic.co/t/logstash-input-jdbc-plugin-additional-tracking-column-types/109523 "2017-11-29T07:20:57Z")
Hello, In current version of Logstash (6.0) only two data types are supported for "tracking\_column\_type". It is numeric and timestamp. Is it possible somehow to use RowVersion data type which is semantically equivalent… --- ## [Is there any option to add a temporary tag / similar variable to logstash entry, not to insert into Elasticsearch?](https://discuss.elastic.co/t/is-there-any-option-to-add-a-temporary-tag-similar-variable-to-logstash-entry-not-to-insert-into-elasticsearch/109511)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 1\ **Last updated:** [November 29, 2017, 6:36am UTC](https://discuss.elastic.co/t/is-there-any-option-to-add-a-temporary-tag-similar-variable-to-logstash-entry-not-to-insert-into-elasticsearch/109511 "2017-11-29T06:36:56Z")
Hi, I am using ELK GA 6. In my logstash, I am taking inputs from 3 kafka topics and I perform different filters and outputs based on the kafka topic. Below is a small snippet of how I am doing it now; input { kafka{ … --- ## [Using my own document\_id - is there a faster way?](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667)
**Author:** [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\ **Replies:** 25\ **Last updated:** [November 29, 2017, 6:24am UTC](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667 "2017-11-29T06:24:54Z")
Hi Everyone, For years I have been using my own document \_id to de-duplicate my data. The principle is that I use a ruby filter to create a unique hash of the IP address and the SHA fingerprint, and use this as the docu… --- ## [Convert time duration to float](https://discuss.elastic.co/t/convert-time-duration-to-float/109449)
**Author:** [@yahoo](https://discuss.elastic.co/u/yahoo)\ **Replies:** 3\ **Last updated:** [November 29, 2017, 3:35am UTC](https://discuss.elastic.co/t/convert-time-duration-to-float/109449 "2017-11-29T03:35:05Z")
I am trying to convert a time duration into float. The error message: Ruby exception occurred: undefined method \`/' for 2017-01-01T05:00:00.000Z:LogStash::Timestamp { "voice\_duration" =\> "00:00:00", "voi… --- ## [Logstash ruby filter something is wrong](https://discuss.elastic.co/t/logstash-ruby-filter-something-is-wrong/109443)
**Author:** [@wentao](https://discuss.elastic.co/u/wentao)\ **Replies:** 2\ **Last updated:** [November 29, 2017, 1:45am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-something-is-wrong/109443 "2017-11-29T01:45:00Z")
my filter is filter { ruby { init =\> "@kname = \['time\_local','domain'\]" code =\> " new\_event = Logstash::Event.new(Hash\[@kname.zip(event.get('message').split('|'))\]) new\_event.remove('@timestamp') event.append(new\_… --- ## [The unknown output information in logstash](https://discuss.elastic.co/t/the-unknown-output-information-in-logstash/109334)
**Author:** [@antony.y](https://discuss.elastic.co/u/antony.y)\ **Replies:** 4\ **Last updated:** [November 29, 2017, 1:01am UTC](https://discuss.elastic.co/t/the-unknown-output-information-in-logstash/109334 "2017-11-29T01:01:50Z")
I have used some output plugin in my logstash and conduct the different actions through the condition i've defined. if "cisco" in \[tags\] { elasticsearch { hosts =\> \[ "ph71v-esn01.ae007.com:9200", "ph71v-… --- ## [Incomplete parsing by CEF codec?](https://discuss.elastic.co/t/incomplete-parsing-by-cef-codec/109483)
**Author:** [@errantpixel](https://discuss.elastic.co/u/errantpixel)\ **Replies:** 0\ **Last updated:** [November 28, 2017, 10:12pm UTC](https://discuss.elastic.co/t/incomplete-parsing-by-cef-codec/109483 "2017-11-28T22:12:36Z")
Taking input from some network elements that are forwarding CEF, and using the CEF codec, there's an issue where if one the keys in the body has a two character sub-name (e.g ad.vd=) , it doesn't get parsed properly and … --- ## [Inject data from a log file](https://discuss.elastic.co/t/inject-data-from-a-log-file/109241)
**Author:** [@Diego\_Gomez](https://discuss.elastic.co/u/Diego_Gomez)\ **Replies:** 7\ **Last updated:** [November 28, 2017, 8:22pm UTC](https://discuss.elastic.co/t/inject-data-from-a-log-file/109241 "2017-11-28T20:22:57Z")
Hello everyone, I have a .log file with a format that I show below, I needed to know what code I use to put it in logstash and index it to elastisearch? File.log MODSEN=000&I=216F6F057C105410&TS=Sun, 17 /9/10, 17:30:0&B… --- ## [Logstash condition statement](https://discuss.elastic.co/t/logstash-condition-statement/109448)
**Author:** [@saikrishnagaddipati](https://discuss.elastic.co/u/saikrishnagaddipati)\ **Replies:** 1\ **Last updated:** [November 28, 2017, 7:13pm UTC](https://discuss.elastic.co/t/logstash-condition-statement/109448 "2017-11-28T19:13:32Z")
Hi, Below is my logstash output config file, I am using too many if, else if conditional statements in the code. Is it possible to do a case statement like in ruby or something which will reduce the repetition. output… --- ## [java::mongodb.jdbc.MongoDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc\_driver\_library](https://discuss.elastic.co/t/java-mongodb-jdbc-mongodriver-not-loaded-are-you-sure-youve-included-the-correct-jdbc-driver-in-jdbc-driver-library/107025)
**Author:** [@pavi](https://discuss.elastic.co/u/pavi)\ **Replies:** 3\ **Last updated:** [November 28, 2017, 7:10pm UTC](https://discuss.elastic.co/t/java-mongodb-jdbc-mongodriver-not-loaded-are-you-sure-youve-included-the-correct-jdbc-driver-in-jdbc-driver-library/107025 "2017-11-28T19:10:23Z")
Hi, I am getting error while using Mongodb Driver. logstash conf file input { jdbc { jdbc\_driver\_library =\> "C://elk//logstash//bin//unityjdbc.jar" jdbc\_driver\_class =\> "java::mongodb.jdbc.MongoDriver" jdbc\_connecti… --- ## [Overriding field values in logstash-filter-metrics events](https://discuss.elastic.co/t/overriding-field-values-in-logstash-filter-metrics-events/109446)
**Author:** [@titiu7](https://discuss.elastic.co/u/titiu7)\ **Replies:** 0\ **Last updated:** [November 28, 2017, 5:24pm UTC](https://discuss.elastic.co/t/overriding-field-values-in-logstash-filter-metrics-events/109446 "2017-11-28T17:24:26Z")
Hey all, I'm using the filter metrics plugin to generate metrics from tomcat access logs. I'm trying to the get the @timestamp field value to match the timestamp value from the log. To do this, i can use the date filter… --- ## [Problem with logstash passing files to ES](https://discuss.elastic.co/t/problem-with-logstash-passing-files-to-es/109399)
**Author:** [@Rodolphe\_Redoute](https://discuss.elastic.co/u/Rodolphe_Redoute)\ **Replies:** 11\ **Last updated:** [November 28, 2017, 4:18pm UTC](https://discuss.elastic.co/t/problem-with-logstash-passing-files-to-es/109399 "2017-11-28T16:18:06Z")
Hi, i'm having an issue with logstash. I'm getting logs files from a Filebeat and doing some grok pattern for every line of the log (tested on http://grokconstructor.appspot.com and working well for the grok part). But … --- ## [Remove special characters in a filed value](https://discuss.elastic.co/t/remove-special-characters-in-a-filed-value/107016)
**Author:** [@tharu85](https://discuss.elastic.co/u/tharu85)\ **Replies:** 2\ **Last updated:** [November 28, 2017, 3:43pm UTC](https://discuss.elastic.co/t/remove-special-characters-in-a-filed-value/107016 "2017-11-28T15:43:55Z")
In my logstash filter, I received following filed and its value.. In order to remove duplicate filed value, first I need to remove special characters in that filed value. For given sample the field name "log\_name" liste… --- ## [Parsing array elements in logs into array](https://discuss.elastic.co/t/parsing-array-elements-in-logs-into-array/109407)
**Author:** [@tanji](https://discuss.elastic.co/u/tanji)\ **Replies:** 4\ **Last updated:** [November 28, 2017, 3:21pm UTC](https://discuss.elastic.co/t/parsing-array-elements-in-logs-into-array/109407 "2017-11-28T15:21:18Z")
My logs contain arbitrary array elements e.g. item\[0\]=value0, item\[1\]=value1. The number of elements can vary from logline to logline. I'm looking for a way to parse those elements and combine them into an array, e.g. i… --- ## [%{\[@metadata\]\[version\]} not being rendered](https://discuss.elastic.co/t/metadata-version-not-being-rendered/109409)
**Author:** [@rj-reilly](https://discuss.elastic.co/u/rj-reilly)\ **Replies:** 0\ **Last updated:** [November 28, 2017, 2:10pm UTC](https://discuss.elastic.co/t/metadata-version-not-being-rendered/109409 "2017-11-28T14:10:26Z")
Hi, Using logstash 6.0 and filebeat 6.0, I added the new filebeat version to the logstash output but i am getting an index named: filebeat-%{\[@metadata\]\[version\]}-2017.11.28, I am using elasticsearch 5.6.4 my output is… --- ## [Log date want to index as actual date for Old Log file](https://discuss.elastic.co/t/log-date-want-to-index-as-actual-date-for-old-log-file/109388)
**Author:** [@rijinmp](https://discuss.elastic.co/u/rijinmp)\ **Replies:** 1\ **Last updated:** [November 28, 2017, 1:53pm UTC](https://discuss.elastic.co/t/log-date-want-to-index-as-actual-date-for-old-log-file/109388 "2017-11-28T13:53:29Z")
Hi All , When I am indexing data from Apache access.log , current system date ( Actual date ) is using for indexing. Actually I am indexing 2014 year logs . But these logs are indexing as in today's log . I want to inde… --- ## [Csv parse error](https://discuss.elastic.co/t/csv-parse-error/109187)
**Author:** [@cilzzz](https://discuss.elastic.co/u/cilzzz)\ **Replies:** 13\ **Last updated:** [November 28, 2017, 1:24pm UTC](https://discuss.elastic.co/t/csv-parse-error/109187 "2017-11-28T13:24:03Z")
Hello, i am trying to parse a csv file but when logstash piupeline started this error is showing my filter in logstash is : filter { if \[type\] == "soe\_phys\_tsm" { csv { columns =\> \["Pa… --- ## [MYSQL DB to fetch data from table and push the same to elastic](https://discuss.elastic.co/t/mysql-db-to-fetch-data-from-table-and-push-the-same-to-elastic/109264)
**Author:** [@guruprasadh4](https://discuss.elastic.co/u/guruprasadh4)\ **Replies:** 0\ **Last updated:** [November 27, 2017, 5:57pm UTC](https://discuss.elastic.co/t/mysql-db-to-fetch-data-from-table-and-push-the-same-to-elastic/109264 "2017-11-27T17:57:45Z")
Hello All, I am trying to fetch the data from mysql db table and trying to push the same to elastic. Below is my config file and the error message that I am getting. Kindly review and let me know if there is something w… --- ## [How much is the syslog events storage capacity of logstash when output is offline?](https://discuss.elastic.co/t/how-much-is-the-syslog-events-storage-capacity-of-logstash-when-output-is-offline/109374)
**Author:** [@vishal0708gautam](https://discuss.elastic.co/u/vishal0708gautam)\ **Replies:** 1\ **Last updated:** [November 28, 2017, 12:55pm UTC](https://discuss.elastic.co/t/how-much-is-the-syslog-events-storage-capacity-of-logstash-when-output-is-offline/109374 "2017-11-28T12:55:20Z")
Hello Experts, We are using Logstash for syslog event receiving & forwarding and its working as expected, but in between sometimes the output server (which receives syslog events forwarded by logstash) goes offline and … --- ## [Date does not convert date/time string properly](https://discuss.elastic.co/t/date-does-not-convert-date-time-string-properly/109243)
**Author:** [@pgervais](https://discuss.elastic.co/u/pgervais)\ **Replies:** 2\ **Last updated:** [November 28, 2017, 12:38pm UTC](https://discuss.elastic.co/t/date-does-not-convert-date-time-string-properly/109243 "2017-11-28T12:38:52Z")
I have the following input date , time string as input: Date == 03OCT17 Hour == 20 Minute == 45 These 3 fields are combined into a single string. "03OCT17:20:45". where the timezone is mutate { add\_field =\> { "t… --- ## [Webhdfs and clone plugin](https://discuss.elastic.co/t/webhdfs-and-clone-plugin/109355)
**Author:** [@gasparuben](https://discuss.elastic.co/u/gasparuben)\ **Replies:** 0\ **Last updated:** [November 28, 2017, 9:58am UTC](https://discuss.elastic.co/t/webhdfs-and-clone-plugin/109355 "2017-11-28T09:58:26Z")
Hi, As part of a data protection project, I need to given a certain document, hash some of its values like IP, username etc. In order to do so I use following logstash file: input { stdin { add\_field =\> … --- ## [Logstash with GlusterFS, sincedb issue](https://discuss.elastic.co/t/logstash-with-glusterfs-sincedb-issue/109234)
**Author:** [@Bostjan](https://discuss.elastic.co/u/Bostjan)\ **Replies:** 0\ **Last updated:** [November 27, 2017, 2:32pm UTC](https://discuss.elastic.co/t/logstash-with-glusterfs-sincedb-issue/109234 "2017-11-27T14:32:39Z")
Hello! I am running into issue with (unscheduled) reimport of data over logstash. I am currently deploying Logstash (version 5.1.1) via Docker. It is taking files from filesystem (GlusterFS is used, 4 replicas) and imp… --- ## [2 grok filter NOT WOERKED ! :|](https://discuss.elastic.co/t/2-grok-filter-not-woerked/109125)
**Author:** [@omid](https://discuss.elastic.co/u/omid)\ **Replies:** 5\ **Last updated:** [November 28, 2017, 10:26am UTC](https://discuss.elastic.co/t/2-grok-filter-not-woerked/109125 "2017-11-28T10:26:54Z")
im use 2 grok filter to one config file but not worked !!! input { beats { port =\> 5443 type =\> syslog ssl =\> true ssl\_certificate =\> "/etc/logstash/logstash.crt" ssl\_key =\> "/etc/logstash/logstash… --- ## [Send logs to Arcsight from Logstash](https://discuss.elastic.co/t/send-logs-to-arcsight-from-logstash/109284)
**Author:** [@saad](https://discuss.elastic.co/u/saad)\ **Replies:** 1\ **Last updated:** [November 28, 2017, 9:49am UTC](https://discuss.elastic.co/t/send-logs-to-arcsight-from-logstash/109284 "2017-11-28T09:49:47Z")
Can we send CEF logs from logstash to arcsight? If yes, How we can do this. --- ## [Not able to make a fileld conditional in grok](https://discuss.elastic.co/t/not-able-to-make-a-fileld-conditional-in-grok/109242)
**Author:** [@George\_Cherian](https://discuss.elastic.co/u/George_Cherian)\ **Replies:** 6\ **Last updated:** [November 28, 2017, 9:08am UTC](https://discuss.elastic.co/t/not-able-to-make-a-fileld-conditional-in-grok/109242 "2017-11-28T09:08:00Z")
The grok filter what is written is as follows: grok { match =\> { "correlationid:" =\> "(?\\w{8}-\\w{4}-\\w{4}-\\w{4}-\\w{12})?" } } For the below text it is not showing the output: "correlationid:" =\> "1000 | callhalf-138… --- ## [Logstash event ruby code](https://discuss.elastic.co/t/logstash-event-ruby-code/108683)
**Author:** [@hsiuming](https://discuss.elastic.co/u/hsiuming)\ **Replies:** 12\ **Last updated:** [November 28, 2017, 8:44am UTC](https://discuss.elastic.co/t/logstash-event-ruby-code/108683 "2017-11-28T08:44:08Z")
Hello We now set our own field by ruby code in logstash. There are three Binary fields. We have two question about: (1) The logstash setting code is: event.set('\[Binary1\]', event.get('\[event\_data\]\[Binary\]')\[0..7\]) e… --- ## [Logstash pipeline continue on error?](https://discuss.elastic.co/t/logstash-pipeline-continue-on-error/108787)
**Author:** [@troy](https://discuss.elastic.co/u/troy)\ **Replies:** 3\ **Last updated:** [November 28, 2017, 7:43am UTC](https://discuss.elastic.co/t/logstash-pipeline-continue-on-error/108787 "2017-11-28T07:43:52Z")
We're running a logstash 6.0.0 instance w/ an http\_input plugin. The plugin expects to receive an array of objects. Some bad data made it to the instance last night that was an array w/ an integer in it instead of an obj… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=865) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=867)