# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=595 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 596 --- ## [Translate filter errors on 's values](https://discuss.elastic.co/t/translate-filter-errors-on-s-values/168621)
**Author:** [@Ant](https://discuss.elastic.co/u/Ant)\ **Replies:** 2\ **Last updated:** [February 23, 2019, 8:05pm UTC](https://discuss.elastic.co/t/translate-filter-errors-on-s-values/168621 "2019-02-23T20:05:30Z")
I've got an id that I want to translate to 4 other variables so having read over the forum I found a suggestion to use JSON to make life easier on myself but some records are giving me issues '1148' : '{"description": "… --- ## [Parse json message events using split filter](https://discuss.elastic.co/t/parse-json-message-events-using-split-filter/169673)
**Author:** [@saroja](https://discuss.elastic.co/u/saroja)\ **Replies:** 3\ **Last updated:** [February 23, 2019, 6:33pm UTC](https://discuss.elastic.co/t/parse-json-message-events-using-split-filter/169673 "2019-02-23T18:33:07Z")
Continuing the discussion from Split nested json array: How to parse a json message events :- If any json key value, passed as null, It would not be parsed by the split filter and wouldn't be render.See the below json… --- ## [Range and size of tracking\_column](https://discuss.elastic.co/t/range-and-size-of-tracking-column/169674)
**Author:** [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\ **Replies:** 0\ **Last updated:** [February 23, 2019, 4:48pm UTC](https://discuss.elastic.co/t/range-and-size-of-tracking-column/169674 "2019-02-23T16:48:27Z")
Hello Team, I am using JDBC input plugin and setting tracking value as numeric value (date converted to number), however it seems tracking column is not working, so just wanted to check if it is because of the number ge… --- ## [No logs are displayed on Kibana with Filebeat](https://discuss.elastic.co/t/no-logs-are-displayed-on-kibana-with-filebeat/169659)
**Author:** [@Eva](https://discuss.elastic.co/u/Eva)\ **Replies:** 0\ **Last updated:** [February 23, 2019, 12:16pm UTC](https://discuss.elastic.co/t/no-logs-are-displayed-on-kibana-with-filebeat/169659 "2019-02-23T12:16:54Z")
Hello there, I have been playing around with Grok filters and I am using a static file that has logs and trying to use grok on the same. I have specified this log file's path in Filebeat config. Though i do not see any … --- ## [Logstash pipeline feature replacement for rather using message queue?](https://discuss.elastic.co/t/logstash-pipeline-feature-replacement-for-rather-using-message-queue/169605)
**Author:** [@chitraj8](https://discuss.elastic.co/u/chitraj8)\ **Replies:** 1\ **Last updated:** [February 23, 2019, 8:31am UTC](https://discuss.elastic.co/t/logstash-pipeline-feature-replacement-for-rather-using-message-queue/169605 "2019-02-23T08:31:49Z")
I evaluating logstash 6.6 and upgrading from 2.4 version. With older logstash 2.4 version we used kafka message queue to ingest data and then push to logstash to handle high streams. With logstash 6.6 and pipeline enha… --- ## [Encryption in Logstash 5.6](https://discuss.elastic.co/t/encryption-in-logstash-5-6/169456)
**Author:** [@kb2295](https://discuss.elastic.co/u/kb2295)\ **Replies:** 1\ **Last updated:** [February 23, 2019, 8:11am UTC](https://discuss.elastic.co/t/encryption-in-logstash-5-6/169456 "2019-02-23T08:11:08Z")
Hello, I have a logstash configuration file with JDBC and Kibana passwords in plaintext. Wondering how I can go about encrypting both of those passwords. Logstash-Keystore seems like it is only available in 6.x version… --- ## [Congestion issues in Redis coming from Logstash](https://discuss.elastic.co/t/congestion-issues-in-redis-coming-from-logstash/167921)
**Author:** [@drivera](https://discuss.elastic.co/u/drivera)\ **Replies:** 6\ **Last updated:** [February 23, 2019, 12:35am UTC](https://discuss.elastic.co/t/congestion-issues-in-redis-coming-from-logstash/167921 "2019-02-23T00:35:14Z")
Hello, I am running LS 5.3 with 5 nodes load balanced and ES 5.3, using Redis to throttle events. I am receiving the error message below within Redis, I believe this is coming from Logstash but I'm not sure how to resolv… --- ## [Timeout executing grok error on LS docker container](https://discuss.elastic.co/t/timeout-executing-grok-error-on-ls-docker-container/169630)
**Author:** [@jwalls](https://discuss.elastic.co/u/jwalls)\ **Replies:** 1\ **Last updated:** [February 22, 2019, 9:37pm UTC](https://discuss.elastic.co/t/timeout-executing-grok-error-on-ls-docker-container/169630 "2019-02-22T21:37:05Z")
I'm running a docker container using LS 6.5.4, but in the docker logs I am getting these errors Timeout executing grok '%{USERNAME:level}: ?%{PATH:file} %{NUMBER:line\_num} %{GREEDYDATA:kvpairs}' against field 'message' … --- ## [Logstash with elasticsearch input and output keep looping results forever](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results-forever/169632)
**Author:** [@ksemaev](https://discuss.elastic.co/u/ksemaev)\ **Replies:** 2\ **Last updated:** [February 22, 2019, 9:17pm UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results-forever/169632 "2019-02-22T21:17:49Z")
The issue is absolutely equal to https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031 I have one elastic index as input, and another as output. I've played with scroll an… --- ## [Aggregate filter does not create a value in the output](https://discuss.elastic.co/t/aggregate-filter-does-not-create-a-value-in-the-output/169568)
**Author:** [@ozmhsh](https://discuss.elastic.co/u/ozmhsh)\ **Replies:** 2\ **Last updated:** [February 22, 2019, 8:22pm UTC](https://discuss.elastic.co/t/aggregate-filter-does-not-create-a-value-in-the-output/169568 "2019-02-22T20:22:44Z")
In my aggregate filter, I would expect to create new output values 'StartTime' and 'EndTime', by using the timestamp 'tmstp' value of the Start Event and End Event. As following: if \[Status\] == "Start" { ag… --- ## [Logstash using default template instead of the one I added](https://discuss.elastic.co/t/logstash-using-default-template-instead-of-the-one-i-added/168904)
**Author:** [@jwalls](https://discuss.elastic.co/u/jwalls)\ **Replies:** 4\ **Last updated:** [February 22, 2019, 8:10pm UTC](https://discuss.elastic.co/t/logstash-using-default-template-instead-of-the-one-i-added/168904 "2019-02-22T20:10:39Z")
when booting up logstash, i can't find the template for ES I added and I get: \[2019-02-18T21:24:30,994\]\[INFO \]\[logstash.outputs.elasticsearch\] Using mapping template from {:path=\>nil} \[2019-02-18T21:24:31,009\]\[INFO \]\[… --- ## [Date to CST](https://discuss.elastic.co/t/date-to-cst/169615)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 5\ **Last updated:** [February 22, 2019, 7:41pm UTC](https://discuss.elastic.co/t/date-to-cst/169615 "2019-02-22T19:41:17Z")
I have date filed coming from db. "completed" =\> "04-12-2018 11:45:17" would like to convert to CST date field. I used following date { match =\> \["completed", "dd-MM-YYY HH:mm:ss"\] target =\> "completed" } It become… --- ## [Error parsing json {:source=\>... Json::ParserError: Invalid numeric value: Leading zeroes not allowed at \[Source: (byte\[\])](https://discuss.elastic.co/t/error-parsing-json-source-json-invalid-numeric-value-leading-zeroes-not-allowed-at-source-byte/169619)
**Author:** [@lowdmt](https://discuss.elastic.co/u/lowdmt)\ **Replies:** 1\ **Last updated:** [February 22, 2019, 6:59pm UTC](https://discuss.elastic.co/t/error-parsing-json-source-json-invalid-numeric-value-leading-zeroes-not-allowed-at-source-byte/169619 "2019-02-22T18:59:20Z")
Hi, I have an issue with my logstash configuration and I can't figure out why. I have a custom logfile on a server which has filebeat installed. An example of the custom log 02/06/2019 11:51:14 Database Copy: DAG003-M… --- ## [Getting \_dateparsefailure on date filter](https://discuss.elastic.co/t/getting-dateparsefailure-on-date-filter/169575)
**Author:** [@bigster](https://discuss.elastic.co/u/bigster)\ **Replies:** 6\ **Last updated:** [February 22, 2019, 4:38pm UTC](https://discuss.elastic.co/t/getting-dateparsefailure-on-date-filter/169575 "2019-02-22T16:38:34Z")
Hi, I'm trying to do this stuff: date { match =\> \[ "date\_created", "ISO8601" \] target =\> "@timestamp" } and date { match =\> \[ "date\_created", "yyyy-MM-dd HH:mm:ss,SSS" \] target =\> "@timestamp" } But everything… --- ## [Why all csv columns are saved inside message field](https://discuss.elastic.co/t/why-all-csv-columns-are-saved-inside-message-field/169598)
**Author:** [@user\_csv](https://discuss.elastic.co/u/user_csv)\ **Replies:** 4\ **Last updated:** [February 22, 2019, 3:16pm UTC](https://discuss.elastic.co/t/why-all-csv-columns-are-saved-inside-message-field/169598 "2019-02-22T15:16:43Z")
Hi, I'm trying to load a CSV file into Elasticsearch using Logstash, but the problem is all the csv columns are saved inside message fiels heres the config file: input { file { path =\> "C:/Users/zya/Documents/ELK\_Sta… --- ## [How to aggregate two events from log lines with no common pattern](https://discuss.elastic.co/t/how-to-aggregate-two-events-from-log-lines-with-no-common-pattern/167939)
**Author:** [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Replies:** 5\ **Last updated:** [February 22, 2019, 2:56pm UTC](https://discuss.elastic.co/t/how-to-aggregate-two-events-from-log-lines-with-no-common-pattern/167939 "2019-02-22T14:56:31Z")
Hi I am reading the documentation on the Aggregate Filter, but all examples assume there is something common between the lines (events) to be aggregated. Usually, that common entity is also used as task\_id. But my case… --- ## [How to separate Date and time in HTTPDATE format in grok and assign in different fields](https://discuss.elastic.co/t/how-to-separate-date-and-time-in-httpdate-format-in-grok-and-assign-in-different-fields/169522)
**Author:** [@nitin.jangid](https://discuss.elastic.co/u/nitin.jangid)\ **Replies:** 1\ **Last updated:** [February 22, 2019, 1:46pm UTC](https://discuss.elastic.co/t/how-to-separate-date-and-time-in-httpdate-format-in-grok-and-assign-in-different-fields/169522 "2019-02-22T13:46:28Z")
Hi, This is my weblogic log 10.10.110.180 - - \[02/Jan/2018:12:36:37 +0530\] "GET /imatch/cavsar/css/cavsar.css HTTP/1.1" 404 14681 with grok filter. I am having field name DateAndTime that having value 02/Jan/2018:12… --- ## [Converting century date format to yyyymmdd format](https://discuss.elastic.co/t/converting-century-date-format-to-yyyymmdd-format/169048)
**Author:** [@shruti51](https://discuss.elastic.co/u/shruti51)\ **Replies:** 5\ **Last updated:** [February 22, 2019, 1:29pm UTC](https://discuss.elastic.co/t/converting-century-date-format-to-yyyymmdd-format/169048 "2019-02-22T13:29:40Z")
Hi, I am trying to convert century format CYYMMDD to YYYYMMDD format in logstash. Please provide some inputs of how to do it. --- ## [Duration calculation using date fields](https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407)
**Author:** [@vasudevan](https://discuss.elastic.co/u/vasudevan)\ **Replies:** 5\ **Last updated:** [February 22, 2019, 10:06am UTC](https://discuss.elastic.co/t/duration-calculation-using-date-fields/169407 "2019-02-22T10:06:39Z")
We have log file in which we have to capture the first line matching "TIMESTAMP\_ISO8601" against build\_StartTime filed and last line matching "TIMESTAMP\_ISO8601" against build\_EndTime filed. After this we have to calcula… --- ## [\_grokparsefailure in Logstash](https://discuss.elastic.co/t/grokparsefailure-in-logstash/169436)
**Author:** [@Astarandel](https://discuss.elastic.co/u/Astarandel)\ **Replies:** 2\ **Last updated:** [February 22, 2019, 8:11am UTC](https://discuss.elastic.co/t/grokparsefailure-in-logstash/169436 "2019-02-22T08:11:48Z")
Greetings everyone, I m pretty new to the whole ELK stack, please excuse me if this seems as a noob question. I am trying to match the following log message: frost: (Total of 4 licenses issued; Total of 0 licenses in … --- ## [Index is not detting generated](https://discuss.elastic.co/t/index-is-not-detting-generated/169417)
**Author:** [@cppatel](https://discuss.elastic.co/u/cppatel)\ **Replies:** 2\ **Last updated:** [February 22, 2019, 4:26am UTC](https://discuss.elastic.co/t/index-is-not-detting-generated/169417 "2019-02-22T04:26:54Z")
Index is not getting generated . Please find the following configurations related to the issue. I am not getting any error in logstash logs. Elasticsearch - 6.2.2 Logstash - 6.2.2 logstash.conf input { file { pat… --- ## [HTTP output - conditional mapping](https://discuss.elastic.co/t/http-output-conditional-mapping/169502)
**Author:** [@somni](https://discuss.elastic.co/u/somni)\ **Replies:** 1\ **Last updated:** [February 21, 2019, 11:50pm UTC](https://discuss.elastic.co/t/http-output-conditional-mapping/169502 "2019-02-21T23:50:43Z")
I'm doing mapping on my http output to slim down which fields for event logs are included. Currently I'm doing this in the http output: mapping =\> { "event" =\> { "event\_id" =\> "%{event\_id}" "message" =\> "%{message}" … --- ## [Rereading a refreshed file](https://discuss.elastic.co/t/rereading-a-refreshed-file/169480)
**Author:** [@lovecraft](https://discuss.elastic.co/u/lovecraft)\ **Replies:** 2\ **Last updated:** [February 21, 2019, 9:53pm UTC](https://discuss.elastic.co/t/rereading-a-refreshed-file/169480 "2019-02-21T21:53:29Z")
Hi All, I am new to Logstash and trying to work out a config. My scenario is that I have a utility which runs periodically and outputs a report as a multiline JSON object in a file. The filename remains the same across… --- ## [Deleting @Version and @Timestamp fields in Logstash 2.2.2](https://discuss.elastic.co/t/deleting-version-and-timestamp-fields-in-logstash-2-2-2/169475)
**Author:** [@stecino](https://discuss.elastic.co/u/stecino)\ **Replies:** 1\ **Last updated:** [February 21, 2019, 9:39pm UTC](https://discuss.elastic.co/t/deleting-version-and-timestamp-fields-in-logstash-2-2-2/169475 "2019-02-21T21:39:42Z")
Hello, I am using Logstash pipeline to re-index ES 2.1.2 index from cluster A to cluster B. New index is created, but logstash adds it's own @version and @timestamp to records. Is there a way to delete these 2 fields vi… --- ## [How to extract first line of message from Winlogbeat](https://discuss.elastic.co/t/how-to-extract-first-line-of-message-from-winlogbeat/169466)
**Author:** [@Mark\_Tellier](https://discuss.elastic.co/u/Mark_Tellier)\ **Replies:** 4\ **Last updated:** [February 21, 2019, 9:28pm UTC](https://discuss.elastic.co/t/how-to-extract-first-line-of-message-from-winlogbeat/169466 "2019-02-21T21:28:49Z")
Winlogbeat is sending Windows Event logs to logstash and I would like to create a new field by extracting the first line of the message field. "message": "Special privileges assigned to new logon.\\n\\nSubject:\\n\\tSecurit… --- ## [Jdbc\_static local\_lookups sql query with in operator](https://discuss.elastic.co/t/jdbc-static-local-lookups-sql-query-with-in-operator/169472)
**Author:** [@pablocoacci](https://discuss.elastic.co/u/pablocoacci)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 5:55pm UTC](https://discuss.elastic.co/t/jdbc-static-local-lookups-sql-query-with-in-operator/169472 "2019-02-21T17:55:45Z")
Hi. How are you? We are trying to execute the filter "Jdbc\_static" to enrich the events with static information present in a database. The field that we want to use is an arrays strings and we need to perform a query w… --- ## [Aggregation error - nil can't be coerced into Fixnum](https://discuss.elastic.co/t/aggregation-error-nil-cant-be-coerced-into-fixnum/168678)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 36\ **Last updated:** [February 21, 2019, 5:13pm UTC](https://discuss.elastic.co/t/aggregation-error-nil-cant-be-coerced-into-fixnum/168678 "2019-02-21T17:13:34Z")
Hi, I am trying to understand what is wrong with my aggregation: My code: aggregate { task\_id =\> "%{ResolutionDate}" code =\> " map\['Date'\] = event.get('ResolutionDate') … --- ## [Convert Microseconds to Milliseconds](https://discuss.elastic.co/t/convert-microseconds-to-milliseconds/169453)
**Author:** [@jwilson-chess](https://discuss.elastic.co/u/jwilson-chess)\ **Replies:** 3\ **Last updated:** [February 21, 2019, 5:05pm UTC](https://discuss.elastic.co/t/convert-microseconds-to-milliseconds/169453 "2019-02-21T17:05:57Z")
I have the following ruby code to do a conversion: ruby { code =\> "event.set('redis.slowlog.duration.ms', event.get('redis.slowlog.duration.us').to\_f / 1000)" } But it doesn't seem to produce any output: I've tr… --- ## [Filebeat agent configuration for two different endpoint(logstash ,kafka clusters)](https://discuss.elastic.co/t/filebeat-agent-configuration-for-two-different-endpoint-logstash-kafka-clusters/168863)
**Author:** [@msina](https://discuss.elastic.co/u/msina)\ **Replies:** 2\ **Last updated:** [February 21, 2019, 4:09pm UTC](https://discuss.elastic.co/t/filebeat-agent-configuration-for-two-different-endpoint-logstash-kafka-clusters/168863 "2019-02-21T16:09:53Z")
Hi, I have one apache HTTPD server and filebeat is already installed there and sending few specific logs to our internal logstatsh server. currently it is sending logs to our internal logstash server are /var/log/htt… --- ## [Issue with logstash and NNT syslog parsing](https://discuss.elastic.co/t/issue-with-logstash-and-nnt-syslog-parsing/169462)
**Author:** [@blenski](https://discuss.elastic.co/u/blenski)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 4:07pm UTC](https://discuss.elastic.co/t/issue-with-logstash-and-nnt-syslog-parsing/169462 "2019-02-21T16:07:47Z")
Hello I am using logstash to collect logs sent by the NNT Change Tracker Generation 7. It all configures so I can see message incoming when performing a test but messages are unreadable. I get for instance : "message"… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=594) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=596)