# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=806 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 807 --- ## [Slowing Down File Ingest](https://discuss.elastic.co/t/slowing-down-file-ingest/122978)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 7\ **Last updated:** [March 9, 2018, 12:50am UTC](https://discuss.elastic.co/t/slowing-down-file-ingest/122978 "2018-03-09T00:50:31Z")
Is there a way to slow down the rate at which the file input ingests data from a file before delivering it to the filter. The problem I have is the file input using multiline codec is feeding data to my XML filter faste… --- ## [Logstash 6.2.2 high CPU usage](https://discuss.elastic.co/t/logstash-6-2-2-high-cpu-usage/122672)
**Author:** [@JulianYuan](https://discuss.elastic.co/u/JulianYuan)\ **Replies:** 2\ **Last updated:** [March 9, 2018, 12:42am UTC](https://discuss.elastic.co/t/logstash-6-2-2-high-cpu-usage/122672 "2018-03-09T00:42:40Z")
I upgraded Elastic Stack from 5.6.3 to 6.2.2, but the Logstash CPU usage go high and no error logs. Start command: ./logstash -f '/data/conf/main.conf' -l /data/logs/logstash/ Logstash config: input { beats { p… --- ## [Logstash keeps crashing with the same Fatal Error](https://discuss.elastic.co/t/logstash-keeps-crashing-with-the-same-fatal-error/122838)
**Author:** [@whicter](https://discuss.elastic.co/u/whicter)\ **Replies:** 3\ **Last updated:** [March 7, 2018, 7:47pm UTC](https://discuss.elastic.co/t/logstash-keeps-crashing-with-the-same-fatal-error/122838 "2018-03-07T19:47:34Z")
Hello, My logstash keeps crashing with the same reason and I couldn't find the reason. Could someone help? The log is as below: \[2018-03-06T17:44:09,393\]\[FATAL\]\[logstash.runner \] An unexpected error occurred! … --- ## [Extract timestamp from my log file along with loglevel?](https://discuss.elastic.co/t/extract-timestamp-from-my-log-file-along-with-loglevel/123069)
**Author:** [@zakkhan](https://discuss.elastic.co/u/zakkhan)\ **Replies:** 5\ **Last updated:** [March 8, 2018, 10:53pm UTC](https://discuss.elastic.co/t/extract-timestamp-from-my-log-file-along-with-loglevel/123069 "2018-03-08T22:53:30Z")
I am using grok to extract the logtime alongwith loglevel but everytime i got grokparsefailure error. Although on individual selection they work fine. Here are the details: My log file: 2018-01-31 07:35:49.899 \[Inform… --- ## [Match name in Logstash Filter Elasticsearch plugin](https://discuss.elastic.co/t/match-name-in-logstash-filter-elasticsearch-plugin/122788)
**Author:** [@rajkamalkool6](https://discuss.elastic.co/u/rajkamalkool6)\ **Replies:** 1\ **Last updated:** [March 8, 2018, 9:55pm UTC](https://discuss.elastic.co/t/match-name-in-logstash-filter-elasticsearch-plugin/122788 "2018-03-08T21:55:33Z")
I am using Logstash Elasticsearch filter to query matching data from another index. I am using below query indexA : logstash-name fields: user indexB: logstash-user fields: first\_name fields: type elasticsearch{ … --- ## [Logstash TLS with x-pack error registering plugin (Cipher is not available) 6.x](https://discuss.elastic.co/t/logstash-tls-with-x-pack-error-registering-plugin-cipher-is-not-available-6-x/117650)
**Author:** [@stephenpatten](https://discuss.elastic.co/u/stephenpatten)\ **Replies:** 7\ **Last updated:** [March 8, 2018, 6:06pm UTC](https://discuss.elastic.co/t/logstash-tls-with-x-pack-error-registering-plugin-cipher-is-not-available-6-x/117650 "2018-03-08T18:06:08Z")
Hello, Set up LS w/ x-pack on a 2 node TLS ES/Kibana windows cluster. ES and Kibana are running fine at this point. logstash.yml file: node.name: logstash.local xpack.monitoring.elasticsearch.username: logstash\_syste… --- ## [Logstash and open files question](https://discuss.elastic.co/t/logstash-and-open-files-question/123115)
**Author:** [@Anabella\_Cristaldi](https://discuss.elastic.co/u/Anabella_Cristaldi)\ **Replies:** 2\ **Last updated:** [March 8, 2018, 4:42pm UTC](https://discuss.elastic.co/t/logstash-and-open-files-question/123115 "2018-03-08T16:42:14Z")
Hi all, I have a question regarding how Logstash, when reading directly from files. Here is my input pluging conf: input { file { path =\> "/data/sbc/elkdev/\*" start\_position =\> "beginning" … --- ## [dd-MMM-yyyy HH:mm:ss.SSS not replacing with @timestamp](https://discuss.elastic.co/t/dd-mmm-yyyy-hh-mm-ss-sss-not-replacing-with-timestamp/123112)
**Author:** [@Uwais\_Ibrahim](https://discuss.elastic.co/u/Uwais_Ibrahim)\ **Replies:** 0\ **Last updated:** [March 8, 2018, 3:44pm UTC](https://discuss.elastic.co/t/dd-mmm-yyyy-hh-mm-ss-sss-not-replacing-with-timestamp/123112 "2018-03-08T15:44:31Z")
Hi @magnusbaeck , I have one log file in which the date format is as follows. 14-Jun-2017 04:40:28.120 INFO \[localhost-startStop-1\] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application direc… --- ## [Pipeline.workers \> 1 does not result in parallel tcp output connections](https://discuss.elastic.co/t/pipeline-workers-1-does-not-result-in-parallel-tcp-output-connections/123110)
**Author:** [@wyldfury](https://discuss.elastic.co/u/wyldfury)\ **Replies:** 0\ **Last updated:** [March 8, 2018, 3:29pm UTC](https://discuss.elastic.co/t/pipeline-workers-1-does-not-result-in-parallel-tcp-output-connections/123110 "2018-03-08T15:29:14Z")
Hi, I have multiple pipelines configured with kafka input -\> tcp output (client). With pipeline.workers set to 1 for a pipeline, I see one long lived tcp connection to the configured server for the tcp output, which is … --- ## [Conditions in logstash](https://discuss.elastic.co/t/conditions-in-logstash/123063)
**Author:** [@cilzzz](https://discuss.elastic.co/u/cilzzz)\ **Replies:** 2\ **Last updated:** [March 8, 2018, 3:27pm UTC](https://discuss.elastic.co/t/conditions-in-logstash/123063 "2018-03-08T15:27:12Z")
Hey everyone, i have two fileds in a csv file dump\_status and upload\_status then i added a third field called return\_code. the values of upload\_status and dump\_status are copy complete and copy failed. what i need to i… --- ## [6.1.2 performs existing ruby filter extremly SLOWER than 5.2.0. Need help!](https://discuss.elastic.co/t/6-1-2-performs-existing-ruby-filter-extremly-slower-than-5-2-0-need-help/119121)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 8\ **Last updated:** [March 8, 2018, 3:23pm UTC](https://discuss.elastic.co/t/6-1-2-performs-existing-ruby-filter-extremly-slower-than-5-2-0-need-help/119121 "2018-03-08T15:23:23Z")
Hi all, Last time I wanted to migrate from 5.1.2 to 6.1.0 I had an extremeperformance drop on production - so I needed to rollback. Currently I am analyzing the issues, trying to build some rally tracks to check where … --- ## [Logstash grok regex threw exception](https://discuss.elastic.co/t/logstash-grok-regex-threw-exception/123106)
**Author:** [@Dee](https://discuss.elastic.co/u/Dee)\ **Replies:** 0\ **Last updated:** [March 8, 2018, 3:16pm UTC](https://discuss.elastic.co/t/logstash-grok-regex-threw-exception/123106 "2018-03-08T15:16:44Z")
Hi, I'm migrating grok patterns from logstash 2.3 to logstash 6.2, so far its going well, except for my syslog logs. This is the grok pattern I have: if \[type\] == "syslog" { grok { break\_on\_match =\> false … --- ## [Batch processing](https://discuss.elastic.co/t/batch-processing/122939)
**Author:** [@GregoryB-T](https://discuss.elastic.co/u/GregoryB-T)\ **Replies:** 2\ **Last updated:** [March 8, 2018, 2:33pm UTC](https://discuss.elastic.co/t/batch-processing/122939 "2018-03-08T14:33:14Z")
Hi, is it possible to extract all the data from elastic and ingest them into logstash into elastic elasticsearch -\> Logstash -\> elasticsearch Thanks and Regards Grégory Boero-teyssier --- ## [Grok pattern to extract username on %{GREEDYDATA:log}](https://discuss.elastic.co/t/grok-pattern-to-extract-username-on-greedydata-log/123076)
**Author:** [@Tshepo\_Moganedi](https://discuss.elastic.co/u/Tshepo_Moganedi)\ **Replies:** 2\ **Last updated:** [March 8, 2018, 2:06pm UTC](https://discuss.elastic.co/t/grok-pattern-to-extract-username-on-greedydata-log/123076 "2018-03-08T14:06:51Z")
Hi, I am trying to strip out Username from the following line and put it in the variable: 2017-06-01 17:05:59,429 INFO - Obtaining UserName: \[admin \] roles and privileges from DataBase. What will be grok pattern for … --- ## [Can't Send @metadata to elasticsearch](https://discuss.elastic.co/t/cant-send-metadata-to-elasticsearch/123066)
**Author:** [@manya12](https://discuss.elastic.co/u/manya12)\ **Replies:** 0\ **Last updated:** [March 8, 2018, 12:46pm UTC](https://discuss.elastic.co/t/cant-send-metadata-to-elasticsearch/123066 "2018-03-08T12:46:33Z")
Hi Team, I want to include @metadata field contents in my elasticsearch output. This is the output when i am using stdout in my output filter- { "@timestamp" =\> 2018-03-08T08:17:42.059Z, "thread\_name" =\> "SimpleAsyn… --- ## [APM Logstash GROK](https://discuss.elastic.co/t/apm-logstash-grok/123037)
**Author:** [@mlala](https://discuss.elastic.co/u/mlala)\ **Replies:** 2\ **Last updated:** [March 8, 2018, 10:21am UTC](https://discuss.elastic.co/t/apm-logstash-grok/123037 "2018-03-08T10:21:30Z")
Hi, I am trying to prepare filter for F5 ASM but using GROK Debugger i am facing issue related to same can anyone help with with proper logstash filter. i will paste the log file for your convenience. \<134\>Mar 8 12:40… --- ## [Tags in input](https://discuss.elastic.co/t/tags-in-input/122899)
**Author:** [@Bakkali\_Amine](https://discuss.elastic.co/u/Bakkali_Amine)\ **Replies:** 5\ **Last updated:** [March 8, 2018, 10:00am UTC](https://discuss.elastic.co/t/tags-in-input/122899 "2018-03-08T10:00:52Z")
When I add a Tag in the input and I chek it isnt added Thank you --- ## [How to check if logstash is recieving data?](https://discuss.elastic.co/t/how-to-check-if-logstash-is-recieving-data/122916)
**Author:** [@olorien](https://discuss.elastic.co/u/olorien)\ **Replies:** 2\ **Last updated:** [March 8, 2018, 8:58am UTC](https://discuss.elastic.co/t/how-to-check-if-logstash-is-recieving-data/122916 "2018-03-08T08:58:37Z")
Hi everyone I am doing some troubleshooting for our logstash-elasticsearch setup, and I was wondering how to check if logstash is receiving any input at all. It simply stopped creating indexes at one point, and no log… --- ## [Logstash ingest & Logstash Indexer](https://discuss.elastic.co/t/logstash-ingest-logstash-indexer/122878)
**Author:** [@Eswar\_Kumar\_Musiboin](https://discuss.elastic.co/u/Eswar_Kumar_Musiboin)\ **Replies:** 4\ **Last updated:** [March 7, 2018, 1:56pm UTC](https://discuss.elastic.co/t/logstash-ingest-logstash-indexer/122878 "2018-03-07T13:56:03Z")
How to configure Logstash ingest nodes and logstash index nodes separately ? Heard logstash index and ingesters some where but no traces in elastic.co documentations. Correct me if invalid. --- ## [Weekday parsing using locale not working](https://discuss.elastic.co/t/weekday-parsing-using-locale-not-working/122479)
**Author:** [@friesoft](https://discuss.elastic.co/u/friesoft)\ **Replies:** 1\ **Last updated:** [March 8, 2018, 8:08am UTC](https://discuss.elastic.co/t/weekday-parsing-using-locale-not-working/122479 "2018-03-08T08:08:18Z")
Hi, I'm trying to parse a timestamp containing a weekday using logstash 6.2.2. German weekday: not working (dateparsefailure) echo "Die" | /opt/logstash-6.2.2/bin/logstash -e 'filter { date { locale =\> "de" match =\> … --- ## [Grok filter is not working using Filebeat](https://discuss.elastic.co/t/grok-filter-is-not-working-using-filebeat/122929)
**Author:** [@SarojELK](https://discuss.elastic.co/u/SarojELK)\ **Replies:** 6\ **Last updated:** [March 8, 2018, 7:24am UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-using-filebeat/122929 "2018-03-08T07:24:36Z")
Hi experts. I already setup ELK stack and it working normally. But I am having a tough time in setting up log shipping using Filebeat and adding custom fields (grok filter) via logstash. my sample log lines \[3/2/16 2:… --- ## [No Matches for the grok filter](https://discuss.elastic.co/t/no-matches-for-the-grok-filter/122820)
**Author:** [@kavinda](https://discuss.elastic.co/u/kavinda)\ **Replies:** 5\ **Last updated:** [March 8, 2018, 7:22am UTC](https://discuss.elastic.co/t/no-matches-for-the-grok-filter/122820 "2018-03-08T07:22:43Z")
HI all Im new to the ELK stack and i have face fallowing problem when i try to create a pattern for a custom log file This is the custom log line INFO \[01/Jul/2017:03:38:38 +0530\] REQUEST CarrierName Balance Check … --- ## [How to authenticate Logstash output to a secure Elasticsearch URL (version 5.6.5)](https://discuss.elastic.co/t/how-to-authenticate-logstash-output-to-a-secure-elasticsearch-url-version-5-6-5/122500)
**Author:** [@loganathan001](https://discuss.elastic.co/u/loganathan001)\ **Replies:** 2\ **Last updated:** [March 6, 2018, 8:39am UTC](https://discuss.elastic.co/t/how-to-authenticate-logstash-output-to-a-secure-elasticsearch-url-version-5-6-5/122500 "2018-03-06T08:39:28Z")
I am using Logstash 5.6.5. So far used elasticsearch (version 5.6.5) output with HTTP protocol and no authentication. Now Elasticsearch is being secured using basic authentication (user/password) and CA certified HTTPS U… --- ## [Hardware monitoring with ELK](https://discuss.elastic.co/t/hardware-monitoring-with-elk/122953)
**Author:** [@iamlearner123](https://discuss.elastic.co/u/iamlearner123)\ **Replies:** 3\ **Last updated:** [March 8, 2018, 6:33am UTC](https://discuss.elastic.co/t/hardware-monitoring-with-elk/122953 "2018-03-08T06:33:53Z")
Hi there, I know that there is collectd that can be used for gathering hardware performance metrics but i was wondering, if there is something else that can be used instead of collectd. Thanks --- ## [How logstash works while running from NFS shared path in LINUX?](https://discuss.elastic.co/t/how-logstash-works-while-running-from-nfs-shared-path-in-linux/123007)
**Author:** [@sivaR](https://discuss.elastic.co/u/sivaR)\ **Replies:** 0\ **Last updated:** [March 8, 2018, 6:09am UTC](https://discuss.elastic.co/t/how-logstash-works-while-running-from-nfs-shared-path-in-linux/123007 "2018-03-08T06:09:27Z")
Does logstash works correctly when running it on the clients from the NFS shared path, below is what i'm running and it works but next very day i see all logstash connections to the clients are stopped and even id don't … --- ## [Message: 403 "Forbidden" while installing logstash plugins](https://discuss.elastic.co/t/message-403-forbidden-while-installing-logstash-plugins/122985)
**Author:** [@saneejmc](https://discuss.elastic.co/u/saneejmc)\ **Replies:** 0\ **Last updated:** [March 8, 2018, 12:39am UTC](https://discuss.elastic.co/t/message-403-forbidden-while-installing-logstash-plugins/122985 "2018-03-08T00:39:52Z")
Hello Gurus, I am trying to install logstash 5.5.2 version in AWS Linux RedHat 4.8.5 system, where I have white listed https://www.elastic.co and https://rubygems.org web sites. though when I am installing plugins usin… --- ## [Trouble filtering out logs](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792)
**Author:** [@Aleita](https://discuss.elastic.co/u/Aleita)\ **Replies:** 6\ **Last updated:** [March 7, 2018, 8:40pm UTC](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792 "2018-03-07T20:40:08Z")
I'm trying to filter out certain types of logs which contain a string message. The error is a log level error which consumes my all events view in kibana. I would like to not have this appear in kibana. I would also b… --- ## [Does csv field order matter when indexing?](https://discuss.elastic.co/t/does-csv-field-order-matter-when-indexing/122952)
**Author:** [@spacepacket](https://discuss.elastic.co/u/spacepacket)\ **Replies:** 1\ **Last updated:** [March 7, 2018, 8:26pm UTC](https://discuss.elastic.co/t/does-csv-field-order-matter-when-indexing/122952 "2018-03-07T20:26:10Z")
If I have logstash reading csv files, with fields in a particular order, and then I change the order in both the new csv files and the logstash config file, does logstash and/or elasticsearch care? If I also add new fie… --- ## [Troubleshooting Unopen Port Issue On Windows 2012](https://discuss.elastic.co/t/troubleshooting-unopen-port-issue-on-windows-2012/121682)
**Author:** [@richl](https://discuss.elastic.co/u/richl)\ **Replies:** 3\ **Last updated:** [March 7, 2018, 8:22pm UTC](https://discuss.elastic.co/t/troubleshooting-unopen-port-issue-on-windows-2012/121682 "2018-03-07T20:22:52Z")
Hi I'm looking for some troubleshooting suggestions to a problem that I can't figure out. I have a problem where no ports are opening after adding them to my config file on Logstash. The ports for Kabana and Elasticsearc… --- ## [CSV Filter Column](https://discuss.elastic.co/t/csv-filter-column/122752)
**Author:** [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Replies:** 4\ **Last updated:** [March 7, 2018, 7:30pm UTC](https://discuss.elastic.co/t/csv-filter-column/122752 "2018-03-07T19:30:53Z")
Hi, Is it possible to expand column from a variable ? EnvironmentFile=-/etc/logstash/env SCALL='"timestamp","id","app","username"' Filter csv { separator =\> "|" columns =\> \[ "${SCALL}"\] } [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=805) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=807)