# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=651 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 652 --- ## [An index per host](https://discuss.elastic.co/t/an-index-per-host/156124)
**Author:** [@vman](https://discuss.elastic.co/u/vman)\ **Replies:** 2\ **Last updated:** [November 10, 2018, 9:08am UTC](https://discuss.elastic.co/t/an-index-per-host/156124 "2018-11-10T09:08:15Z")
if \[host\] in \["host1", "host2", "host3" , "host4" , "host5"\] { elasticsearch { hosts =\> \["localhost:9200"\] ... index =\> "host1-%{+YYYY.MM.dd}" document\_type =\> "%{\[@metadata\]\[type\]}" } } else { elasticsearch { h… --- ## [Grok expression works, but causes logstash config check to fail. Special characters in log an issue?](https://discuss.elastic.co/t/grok-expression-works-but-causes-logstash-config-check-to-fail-special-characters-in-log-an-issue/156107)
**Author:** [@hueyg](https://discuss.elastic.co/u/hueyg)\ **Replies:** 1\ **Last updated:** [November 10, 2018, 3:51am UTC](https://discuss.elastic.co/t/grok-expression-works-but-causes-logstash-config-check-to-fail-special-characters-in-log-an-issue/156107 "2018-11-10T03:51:01Z")
I have a grok expression for an application where the log contains square brackets, curly brackets, and pipe characters. The expression works in the grok debugger but fails when I use it in the logstash filter. Here is… --- ## [Syslog filters for Versa Networks](https://discuss.elastic.co/t/syslog-filters-for-versa-networks/156113)
**Author:** [@Zorkmid](https://discuss.elastic.co/u/Zorkmid)\ **Replies:** 0\ **Last updated:** [November 9, 2018, 10:28pm UTC](https://discuss.elastic.co/t/syslog-filters-for-versa-networks/156113 "2018-11-09T22:28:09Z")
Hello, Has any one work with Syslog output from Versa Network's SD-WAND products? I'm working on a POC with Versa and would like to see the output from the SD-WAN devices be ingested into ES via logstash. The following… --- ## [Timestamp off(delay) 5 hours](https://discuss.elastic.co/t/timestamp-off-delay-5-hours/156081)
**Author:** [@netvmdb](https://discuss.elastic.co/u/netvmdb)\ **Replies:** 7\ **Last updated:** [November 9, 2018, 11:03pm UTC](https://discuss.elastic.co/t/timestamp-off-delay-5-hours/156081 "2018-11-09T23:03:24Z")
Running version 6.2.4. Multiple logstash and each for different log type, like messages, secure... Index for messages log is off 5 hours. Everything else is correct. All logstash.conf files are similar. Here is the filte… --- ## [Kinesis input plugin - Configure Kinesis and DynamoDB using different AWS profiles](https://discuss.elastic.co/t/kinesis-input-plugin-configure-kinesis-and-dynamodb-using-different-aws-profiles/156111)
**Author:** [@Vijay\_Reddiar](https://discuss.elastic.co/u/Vijay_Reddiar)\ **Replies:** 0\ **Last updated:** [November 9, 2018, 9:51pm UTC](https://discuss.elastic.co/t/kinesis-input-plugin-configure-kinesis-and-dynamodb-using-different-aws-profiles/156111 "2018-11-09T21:51:07Z")
Kinesis input plugin is working great but now I am running into an security issue. In my integration scenario for monitoring the flow logs, I have to consume the Kinesis stream hosted by the client. Client security polic… --- ## [Logstash 6.4.2 template mapping does not working](https://discuss.elastic.co/t/logstash-6-4-2-template-mapping-does-not-working/155706)
**Author:** [@gogua](https://discuss.elastic.co/u/gogua)\ **Replies:** 1\ **Last updated:** [November 9, 2018, 7:45pm UTC](https://discuss.elastic.co/t/logstash-6-4-2-template-mapping-does-not-working/155706 "2018-11-09T19:45:57Z")
i have custom template mapping, but when the new index is created dynamically by logstash, index mapping is wrong, it differs from my template mappings and why? Logstash Config output { elasticsearch { hosts … --- ## [Logstash timetaken not recognized as an integer](https://discuss.elastic.co/t/logstash-timetaken-not-recognized-as-an-integer/155765)
**Author:** [@Daniel\_Carucha\_Altuz](https://discuss.elastic.co/u/Daniel_Carucha_Altuz)\ **Replies:** 1\ **Last updated:** [November 9, 2018, 7:41pm UTC](https://discuss.elastic.co/t/logstash-timetaken-not-recognized-as-an-integer/155765 "2018-11-09T19:41:11Z")
Hi, i have the next logline: 2018-11-07T13:48:26-03:00 localhost.localdomain haproxy\[40370\]: 10.5.38.210:55125 \[07/Nov/2018:13:48:26.791\] https-in~ default\_443/anseswww71.anses.gov.ar 101/0/1/83/185 200 604 - - ---- 335… --- ## [Netflow Sampling Rate](https://discuss.elastic.co/t/netflow-sampling-rate/156091)
**Author:** [@rfg](https://discuss.elastic.co/u/rfg)\ **Replies:** 0\ **Last updated:** [November 9, 2018, 5:36pm UTC](https://discuss.elastic.co/t/netflow-sampling-rate/156091 "2018-11-09T17:36:48Z")
Is there any way to set this value? I am exporting from a Huawei configured as sampler. Ex. ip netstream sampler fix-packets 1000 inbound ip netstream sampler fix-packets 1000 outbound --- ## [Avoid duplication](https://discuss.elastic.co/t/avoid-duplication/155731)
**Author:** [@totuyim](https://discuss.elastic.co/u/totuyim)\ **Replies:** 12\ **Last updated:** [November 9, 2018, 4:01pm UTC](https://discuss.elastic.co/t/avoid-duplication/155731 "2018-11-09T16:01:45Z")
Hi there, I've a lot of traffic logs (around 20 millions per day) to index and I would like to know suggestion of how to avoid duplication with these amount of data. I was reading this article but I haven't such experi… --- ## [Geoip.location is a number not a geopoint](https://discuss.elastic.co/t/geoip-location-is-a-number-not-a-geopoint/155965)
**Author:** [@dbutler](https://discuss.elastic.co/u/dbutler)\ **Replies:** 3\ **Last updated:** [November 9, 2018, 2:32pm UTC](https://discuss.elastic.co/t/geoip-location-is-a-number-not-a-geopoint/155965 "2018-11-09T14:32:25Z")
template mapping snippit "geoip": { "properties": { "city\_name": { "type": "text", "fields": { "keyword": { "type… --- ## [Config.reload.automatic for logstash docker container](https://discuss.elastic.co/t/config-reload-automatic-for-logstash-docker-container/153727)
**Author:** [@mortenb123](https://discuss.elastic.co/u/mortenb123)\ **Replies:** 2\ **Last updated:** [November 9, 2018, 1:38pm UTC](https://discuss.elastic.co/t/config-reload-automatic-for-logstash-docker-container/153727 "2018-11-09T13:38:53Z")
Hi I've added: config.reload.automatic: true config.reload.interval: 300 to my config/logstash/logstash.yml which points to /usr/share/logstash/config/logstash.yml in the running logstash docker container. I also tr… --- ## [Split filter - get index of splitted logs](https://discuss.elastic.co/t/split-filter-get-index-of-splitted-logs/156054)
**Author:** [@benoit\_elk](https://discuss.elastic.co/u/benoit_elk)\ **Replies:** 0\ **Last updated:** [November 9, 2018, 1:06pm UTC](https://discuss.elastic.co/t/split-filter-get-index-of-splitted-logs/156054 "2018-11-09T13:06:52Z")
I am using Logstash split filter plugin as described in this page : https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html Let say I have data looking like that : {"log\_id": "abcd", "logs": \[{"val"… --- ## [Retrieve json fields with regex](https://discuss.elastic.co/t/retrieve-json-fields-with-regex/156039)
**Author:** [@matpil](https://discuss.elastic.co/u/matpil)\ **Replies:** 0\ **Last updated:** [November 9, 2018, 11:58am UTC](https://discuss.elastic.co/t/retrieve-json-fields-with-regex/156039 "2018-11-09T11:58:43Z")
Hi all, I'm trying to estract a value from my json log using a regex/wildcard into path field (so, it's generic to all my json log). es. json log: { "appl": { "hostname": "host14-26102017", "containerid": "8410d66… --- ## [Error while inserting data from oracle db to Elasticsearch using logstash](https://discuss.elastic.co/t/error-while-inserting-data-from-oracle-db-to-elasticsearch-using-logstash/156026)
**Author:** [@khurana.savita](https://discuss.elastic.co/u/khurana.savita)\ **Replies:** 1\ **Last updated:** [November 9, 2018, 11:57am UTC](https://discuss.elastic.co/t/error-while-inserting-data-from-oracle-db-to-elasticsearch-using-logstash/156026 "2018-11-09T11:57:17Z")
Error Log Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, input, filter, output at line 1, column 1 (byte 1… --- ## [Time format parse. exmp - 2018-09-13T00:00:01.369623+00:00](https://discuss.elastic.co/t/time-format-parse-exmp-2018-09-13t0001-369623-00-00/151357)
**Author:** [@orsa](https://discuss.elastic.co/u/orsa)\ **Replies:** 2\ **Last updated:** [November 9, 2018, 11:22am UTC](https://discuss.elastic.co/t/time-format-parse-exmp-2018-09-13t0001-369623-00-00/151357 "2018-11-09T11:22:20Z")
Hi, my system writes a log with this date format - 2018-09-13T00:00:01.369623+00:00 what is the correct format I should use for the time codec? I tried YYYY-MM-dd HH:mm:ss.SSSZZ, but it not work ( --- ## [Trying to add a 2nd source of logs](https://discuss.elastic.co/t/trying-to-add-a-2nd-source-of-logs/155914)
**Author:** [@lokylax](https://discuss.elastic.co/u/lokylax)\ **Replies:** 0\ **Last updated:** [November 8, 2018, 3:03pm UTC](https://discuss.elastic.co/t/trying-to-add-a-2nd-source-of-logs/155914 "2018-11-08T15:03:29Z")
Hey everyone, I'm very new to the ELK stack and I've been learning it thanks to online resources. My next goal is to visualize logs from 2 different logs files. First, I have tried 2 different config files with 2 pipeli… --- ## [Escaped JSON in XML](https://discuss.elastic.co/t/escaped-json-in-xml/155854)
**Author:** [@ChristianCH](https://discuss.elastic.co/u/ChristianCH)\ **Replies:** 2\ **Last updated:** [November 9, 2018, 9:30am UTC](https://discuss.elastic.co/t/escaped-json-in-xml/155854 "2018-11-09T09:30:25Z")
Hi all, I have an escaped JSON in XML structure, so the XML looks like this: \ \ \test\ \test\ \ \ \ \ \ **Author:** [@roly](https://discuss.elastic.co/u/roly)\ **Replies:** 0\ **Last updated:** [November 9, 2018, 9:24am UTC](https://discuss.elastic.co/t/extracting-filename-from-existing-source-field/156015 "2018-11-09T09:24:08Z") How do i extract source filename from the existing field which standard field from logstash which is (Source)? input { beats { port =\> 5044 host =\> "0.0.0.0" } } filter { grok { patterns\_dir =\> \["/etc… --- ## [Logstash keystore, keys entered in UPPERCASE becomes lowercase](https://discuss.elastic.co/t/logstash-keystore-keys-entered-in-uppercase-becomes-lowercase/155919)
**Author:** [@Mattias\_Arbin](https://discuss.elastic.co/u/Mattias_Arbin)\ **Replies:** 1\ **Last updated:** [November 9, 2018, 9:13am UTC](https://discuss.elastic.co/t/logstash-keystore-keys-entered-in-uppercase-becomes-lowercase/155919 "2018-11-09T09:13:42Z")
I have this strange issue with the logstash keystore. I started adding one password in my keystore named ES\_PWD I am pretty sure I got my logstash config working by accessing the key as ${ES\_PWD} Now, after adding a f… --- ## [Elasticsearch didn't show index from logstash-filebeat](https://discuss.elastic.co/t/elasticsearch-didnt-show-index-from-logstash-filebeat/156009)
**Author:** [@lala](https://discuss.elastic.co/u/lala)\ **Replies:** 0\ **Last updated:** [November 9, 2018, 9:03am UTC](https://discuss.elastic.co/t/elasticsearch-didnt-show-index-from-logstash-filebeat/156009 "2018-11-09T09:03:24Z")
Hi I'm student, I'm new to ELK, I would like to ask about my configurations, because the elasticsearch didn't show the indices log from server SOA -\> server ELK -\> logstash -\> elasticsearch I'm using filebeat 6.4.2 th… --- ## [How to filter json log file in logstash and send it to elastic search](https://discuss.elastic.co/t/how-to-filter-json-log-file-in-logstash-and-send-it-to-elastic-search/155981)
**Author:** [@Jin1129](https://discuss.elastic.co/u/Jin1129)\ **Replies:** 4\ **Last updated:** [November 9, 2018, 7:44am UTC](https://discuss.elastic.co/t/how-to-filter-json-log-file-in-logstash-and-send-it-to-elastic-search/155981 "2018-11-09T07:44:50Z")
This is my json log file. I'm trying to store the file to my elastic-Search through my logstash. { "id": "135569", "title" : "Star Trek Beyond", "year":2016 , "genre": \["Action", "Adventure", "Sci-Fi"\] } after… --- ## [Logstash file descriptors increasing rapidly](https://discuss.elastic.co/t/logstash-file-descriptors-increasing-rapidly/155947)
**Author:** [@meetdave2611997](https://discuss.elastic.co/u/meetdave2611997)\ **Replies:** 1\ **Last updated:** [November 9, 2018, 7:22am UTC](https://discuss.elastic.co/t/logstash-file-descriptors-increasing-rapidly/155947 "2018-11-09T07:22:20Z")
Hi, We are using Elastic stack 6.2.4. We have 12 logstash nodes in prod. Each node was working perfectly. Somehow each node started consuming too much file descriptors. 65000 is the max limit of file descriptors set up … --- ## [I want to filter apache access log based on String matching in its request URL](https://discuss.elastic.co/t/i-want-to-filter-apache-access-log-based-on-string-matching-in-its-request-url/155994)
**Author:** [@abhisekparida](https://discuss.elastic.co/u/abhisekparida)\ **Replies:** 0\ **Last updated:** [November 9, 2018, 7:10am UTC](https://discuss.elastic.co/t/i-want-to-filter-apache-access-log-based-on-string-matching-in-its-request-url/155994 "2018-11-09T07:10:30Z")
I want to filter apache access log based on String matching in its request URL. Example :10.109.146.4 - - \[25/Oct/2018:08:55:10 +0100\] "GET /ngwfmt/resources/frameworkdata/getDataForProcessAndEventName?processName=VIS… --- ## [Is there any other way to store rejected records by elasticsearch?](https://discuss.elastic.co/t/is-there-any-other-way-to-store-rejected-records-by-elasticsearch/155985)
**Author:** [@Bhagi007](https://discuss.elastic.co/u/Bhagi007)\ **Replies:** 0\ **Last updated:** [November 9, 2018, 6:22am UTC](https://discuss.elastic.co/t/is-there-any-other-way-to-store-rejected-records-by-elasticsearch/155985 "2018-11-09T06:22:59Z")
I tried dead letter queue plugin but it's not giving consistent output. and also i am facing the below issues. Error: \[2018-10-16T06:56:42,248\]\[ERROR\]\[logstash.pipeline \] A plugin had an unrecoverable error. Will res… --- ## [How to use http\_poller to poll REST API with multiple inputs](https://discuss.elastic.co/t/how-to-use-http-poller-to-poll-rest-api-with-multiple-inputs/155980)
**Author:** [@Chuan\_Ydnk](https://discuss.elastic.co/u/Chuan_Ydnk)\ **Replies:** 0\ **Last updated:** [November 9, 2018, 5:59am UTC](https://discuss.elastic.co/t/how-to-use-http-poller-to-poll-rest-api-with-multiple-inputs/155980 "2018-11-09T05:59:50Z")
Hi Guys, I am newbie to logstashs, kibana and elk. Hence, I wanted to use http\_poller to periodically get data from the this REST API https://1.2.3.4/rest/dmiquery/getDMIData3 using POST method with JSON inputs as below … --- ## [Dead letter queue Logs with number 1](https://discuss.elastic.co/t/dead-letter-queue-logs-with-number-1/155942)
**Author:** [@daniloJava](https://discuss.elastic.co/u/daniloJava)\ **Replies:** 0\ **Last updated:** [November 8, 2018, 5:32pm UTC](https://discuss.elastic.co/t/dead-letter-queue-logs-with-number-1/155942 "2018-11-08T17:32:59Z")
Why the configuration dead\_letter\_queue generate files 1.log, 2.log with number 1 when not happen anything problem? There is anything configuration to delete this logs? --- ## [Logstash type vs Elasticsearch type](https://discuss.elastic.co/t/logstash-type-vs-elasticsearch-type/155934)
**Author:** [@rishabhguptaaec](https://discuss.elastic.co/u/rishabhguptaaec)\ **Replies:** 0\ **Last updated:** [November 8, 2018, 4:42pm UTC](https://discuss.elastic.co/t/logstash-type-vs-elasticsearch-type/155934 "2018-11-08T16:42:35Z")
Hi We are currently using version 5.5 for both Logstash and Elasticsearch. In our Logstash input configuration we specify "type" field value. We do not specify document\_type in elasticsearch output configuration. Howeve… --- ## [Filter to consolidate values in multiple logs](https://discuss.elastic.co/t/filter-to-consolidate-values-in-multiple-logs/155928)
**Author:** [@logger](https://discuss.elastic.co/u/logger)\ **Replies:** 0\ **Last updated:** [November 8, 2018, 4:32pm UTC](https://discuss.elastic.co/t/filter-to-consolidate-values-in-multiple-logs/155928 "2018-11-08T16:32:43Z")
Hi there, I need help to optimize my logstash. Szenario: We use IBM datapower and ship the logs to logstash. Logstash sends it to Elasticsearch of course. In these logs is field named gtid ("globalTransactionId"). … --- ## [Permission Denied Issue using Logstash](https://discuss.elastic.co/t/permission-denied-issue-using-logstash/155824)
**Author:** [@magician.kim](https://discuss.elastic.co/u/magician.kim)\ **Replies:** 2\ **Last updated:** [November 8, 2018, 3:56pm UTC](https://discuss.elastic.co/t/permission-denied-issue-using-logstash/155824 "2018-11-08T15:56:07Z")
Hi, I've been started ELK Stack (+Filebeat) for one week. I'm trying to transfer log files data using Filebeat and Logstash to ElasticSearch. Here, my Logstash configuration file. And I started Logstash and received… --- ## [Logstash error „mapper\_parsing\_exception“](https://discuss.elastic.co/t/logstash-error-mapper-parsing-exception/155922)
**Author:** [@niagato](https://discuss.elastic.co/u/niagato)\ **Replies:** 0\ **Last updated:** [November 8, 2018, 3:47pm UTC](https://discuss.elastic.co/t/logstash-error-mapper-parsing-exception/155922 "2018-11-08T15:47:08Z")
We use an instance of Elastic as part of Security Onion, because of that preconfigured config files exist on the system. Because of that all document fields with the name ‚host‘ will be renamed to ‚beat\_host‘ by Logstas… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=650) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=652)