# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1010 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1011 --- ## [Problem with Logstash logs](https://discuss.elastic.co/t/problem-with-logstash-logs/80276)
**Author:** [@emilio](https://discuss.elastic.co/u/emilio)\ **Replies:** 6\ **Last updated:** [March 28, 2017, 1:38pm UTC](https://discuss.elastic.co/t/problem-with-logstash-logs/80276 "2017-03-28T13:38:44Z")
Dear all, I I would like to get know where is logstash logs. My goal is to have logstash logs in /data01/logstah/logs Please help Here are my configs: logstash.yml ''' Settings file in YAML Settings can be… --- ## [Ingesting json formatted data](https://discuss.elastic.co/t/ingesting-json-formatted-data/80209)
**Author:** [@DigiAngel](https://discuss.elastic.co/u/DigiAngel)\ **Replies:** 3\ **Last updated:** [March 28, 2017, 1:05pm UTC](https://discuss.elastic.co/t/ingesting-json-formatted-data/80209 "2017-03-28T13:05:39Z")
So I'm sure this should be easy, but I thought I'd run it by here first. The plan is to send the contents of a json formatted file to a remote logstash instance listening on a udp or tcp port. Has anyone done this befo… --- ## [\_grokparsefailiure only after date filter has been included](https://discuss.elastic.co/t/-grokparsefailiure-only-after-date-filter-has-been-included/80224)
**Author:** [@rrrtttuuu](https://discuss.elastic.co/u/rrrtttuuu)\ **Replies:** 3\ **Last updated:** [March 28, 2017, 1:02pm UTC](https://discuss.elastic.co/t/-grokparsefailiure-only-after-date-filter-has-been-included/80224 "2017-03-28T13:02:12Z")
I have a filter that looks something like this: filter { grok { match =\> {"message" =\> '%{IPORHOST:remote\_addr}%{SPACE}-%{SPACE}%{HOSTPORT:upstream\_addr}%{SPACE}-%{SPACE}%{USERNAME:remote\_user}%{SPACE}\\\[%{H… --- ## [Logstash is not reading Json file](https://discuss.elastic.co/t/logstash-is-not-reading-json-file/79620)
**Author:** [@udaykirankona](https://discuss.elastic.co/u/udaykirankona)\ **Replies:** 2\ **Last updated:** [March 28, 2017, 11:34am UTC](https://discuss.elastic.co/t/logstash-is-not-reading-json-file/79620 "2017-03-28T11:34:44Z")
HI, Logstash not giving any out put for Json input, my config as follows. input { file { codec =\> json type =\> "json" path =\> "/path/\*" start\_position =\> "beginning" sincedb\_path =\> … --- ## [Updating data which are newly added](https://discuss.elastic.co/t/updating-data-which-are-newly-added/80236)
**Author:** [@Varun\_Patel](https://discuss.elastic.co/u/Varun_Patel)\ **Replies:** 2\ **Last updated:** [March 28, 2017, 10:23am UTC](https://discuss.elastic.co/t/updating-data-which-are-newly-added/80236 "2017-03-28T10:23:49Z")
Hi, I am using jdbc connection for fetching data from database. I have single index. Let say Contacts. I am able to fetch data from database. But what i want is when i run logstash for second time it should only add… --- ## [WARN No appenders could be found for logger (org.apache.kafka.clients.consumer.ConsumerConfig)](https://discuss.elastic.co/t/warn-no-appenders-could-be-found-for-logger-org-apache-kafka-clients-consumer-consumerconfig/76784)
**Author:** [@mostolog](https://discuss.elastic.co/u/mostolog)\ **Replies:** 0\ **Last updated:** [February 28, 2017, 1:41pm UTC](https://discuss.elastic.co/t/warn-no-appenders-could-be-found-for-logger-org-apache-kafka-clients-consumer-consumerconfig/76784 "2017-02-28T13:41:30Z")
Hi During Logstash 5.2.1 boot which uses kafka input, just noticed a few warnings: log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.consumer.ConsumerConfig). log4j:WARN Please initialize the … --- ## [Drop some line when match in logstash](https://discuss.elastic.co/t/drop-some-line-when-match-in-logstash/80270)
**Author:** [@DannyM](https://discuss.elastic.co/u/DannyM)\ **Replies:** 1\ **Last updated:** [March 28, 2017, 9:55am UTC](https://discuss.elastic.co/t/drop-some-line-when-match-in-logstash/80270 "2017-03-28T09:55:38Z")
At the moment i have this filter working to drop some lines ## var\_log\_messages filter filter { if \[type\] == "var\_log\_messages" { grok { patterns\_dir =\> \[… --- ## [How to refer to subfield in logstash](https://discuss.elastic.co/t/how-to-refer-to-subfield-in-logstash/79474)
**Author:** [@scutdk](https://discuss.elastic.co/u/scutdk)\ **Replies:** 2\ **Last updated:** [March 22, 2017, 6:07am UTC](https://discuss.elastic.co/t/how-to-refer-to-subfield-in-logstash/79474 "2017-03-22T06:07:27Z")
I have following logstash conf snippet, but it doesn't work. I got the idea of subfield from this stackoverflow thread. How can I make it to work? I am trying to split the http\_refer field in my access log. Thanks. fil… --- ## [Logstash with multiple pipelines](https://discuss.elastic.co/t/logstash-with-multiple-pipelines/79861)
**Author:** [@Alexander\_Truemper](https://discuss.elastic.co/u/Alexander_Truemper)\ **Replies:** 2\ **Last updated:** [March 28, 2017, 8:19am UTC](https://discuss.elastic.co/t/logstash-with-multiple-pipelines/79861 "2017-03-28T08:19:52Z")
I remember from last elastic on tour that someone said something about having parallel, independent logstash pipelines to avoid conditionals which may lead to Spaghetti Configs. Is this already near to be released or is … --- ## [Index from Elasticsearch to excel](https://discuss.elastic.co/t/index-from-elasticsearch-to-excel/80142)
**Author:** [@wolfghost](https://discuss.elastic.co/u/wolfghost)\ **Replies:** 8\ **Last updated:** [March 28, 2017, 7:48am UTC](https://discuss.elastic.co/t/index-from-elasticsearch-to-excel/80142 "2017-03-28T07:48:40Z")
How can i export data from ES to excel file? --- ## [High CPU Utilization but low memory utilization?](https://discuss.elastic.co/t/high-cpu-utilization-but-low-memory-utilization/80226)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 1\ **Last updated:** [March 28, 2017, 6:23am UTC](https://discuss.elastic.co/t/high-cpu-utilization-but-low-memory-utilization/80226 "2017-03-28T06:23:40Z")
I am running large file with logstash. I see from command top that i have ~100% CPU utilization, but only 5% memory utilization. What ways are there and Should I increase mem utilization? --- ## [Grok Pattern for barnyard2 syslog](https://discuss.elastic.co/t/grok-pattern-for-barnyard2-syslog/80136)
**Author:** [@shookshank](https://discuss.elastic.co/u/shookshank)\ **Replies:** 1\ **Last updated:** [March 28, 2017, 6:04am UTC](https://discuss.elastic.co/t/grok-pattern-for-barnyard2-syslog/80136 "2017-03-28T06:04:07Z")
i have the following syslog alerts. Oct 1 11:59:51 sensor | \[SNORTIDS\[ALERT\]: \[sensor-eth1\] \] || 2014-10-01 11:59:54.967+-04 1 \[1:1000022:1\] LOCAL-RULE Possible login.aspx Brute Force Account Access || web-applicatio… --- ## [Logstash-elasticsearch output plugin issue(UNEXPECTED POOL ERROR)](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-issue-unexpected-pool-error/64899)
**Author:** [@rai](https://discuss.elastic.co/u/rai)\ **Replies:** 12\ **Last updated:** [March 28, 2017, 6:01am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-issue-unexpected-pool-error/64899 "2017-03-28T06:01:46Z")
I have created a docker image based on elk 5.0.0. and used logstash-kafka input plugin. ELK server has started without any issue and but can't see any log on kibana. I am also able to curl elasticsearch (curl "http://\*\*\*… --- ## [Prevent logs from appearing in Kibana](https://discuss.elastic.co/t/prevent-logs-from-appearing-in-kibana/58937)
**Author:** [@scosta](https://discuss.elastic.co/u/scosta)\ **Replies:** 6\ **Last updated:** [March 28, 2017, 5:27am UTC](https://discuss.elastic.co/t/prevent-logs-from-appearing-in-kibana/58937 "2017-03-28T05:27:18Z")
Good Morning. I am a novice in ELK stack. I have problems in my logstash . I get many logs, but I need to keep a log with a specific message to appear on Kibana . How can I keep a log appear in Kibana for your message… --- ## [Logstash getting wrong hostname?](https://discuss.elastic.co/t/logstash-getting-wrong-hostname/80161)
**Author:** [@scutdk](https://discuss.elastic.co/u/scutdk)\ **Replies:** 1\ **Last updated:** [March 28, 2017, 2:29am UTC](https://discuss.elastic.co/t/logstash-getting-wrong-hostname/80161 "2017-03-28T02:29:08Z")
Hi, my current logstash setup always return localhost for host field. My system is CentOS 6.x. I' ve set hostname in /etc/sysconfig/network and also in hostname command. from Google, it says logstash use gethostname f… --- ## [Monitor API curl not allowed?](https://discuss.elastic.co/t/monitor-api-curl-not-allowed/80193)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 1\ **Last updated:** [March 27, 2017, 8:42pm UTC](https://discuss.elastic.co/t/monitor-api-curl-not-allowed/80193 "2017-03-27T20:42:39Z")
I can do curl -XGET url, this give me proper json response similar to the documentation below: { "name" : "Ia\_zMX5", "cluster\_name" : "0-logs", "cluster\_uuid" : "66666666", "version" : { "number" : "5.1.1"… --- ## [Help with Grok pattern with space in field?](https://discuss.elastic.co/t/help-with-grok-pattern-with-space-in-field/80189)
**Author:** [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Replies:** 2\ **Last updated:** [March 27, 2017, 8:13pm UTC](https://discuss.elastic.co/t/help-with-grok-pattern-with-space-in-field/80189 "2017-03-27T20:13:43Z")
I have the following log... 2017-03-27 14:24:50,607 DEBUG :Finalizer thread \[com.company.classname\] ...and the following filter filter { grok { match =\> { "message" =\> \[ "%{TIMESTAMP\_ISO8601:log… --- ## [Data compilation From Netflow & Syslog messages from Cisco Equipment](https://discuss.elastic.co/t/data-compilation-from-netflow-syslog-messages-from-cisco-equipment/80180)
**Author:** [@adrianmc](https://discuss.elastic.co/u/adrianmc)\ **Replies:** 4\ **Last updated:** [March 27, 2017, 7:52pm UTC](https://discuss.elastic.co/t/data-compilation-from-netflow-syslog-messages-from-cisco-equipment/80180 "2017-03-27T19:52:33Z")
Well, Hello everybody, Actually, I'm working with Netflow protocol and have been working fine. But, i would like work with Syslog too. In my case i have some routers Cisco , i get data from Netflow protocol and show i… --- ## [Netflow codec: No matching template](https://discuss.elastic.co/t/netflow-codec-no-matching-template/80150)
**Author:** [@prodrg](https://discuss.elastic.co/u/prodrg)\ **Replies:** 2\ **Last updated:** [March 27, 2017, 7:15pm UTC](https://discuss.elastic.co/t/netflow-codec-no-matching-template/80150 "2017-03-27T19:15:15Z")
Hello everybody, I'm receiving the following issues from the Netflow codec in logstash: \[2017-03-27T09:40:00,696\]\[WARN \]\[logstash.codecs.netflow \] No matching template for flow id 256 \[2017-03-27T09:40:00,707\]\[WARN … --- ## [\_grokparsefailure general exception handling?](https://discuss.elastic.co/t/-grokparsefailure-general-exception-handling/80143)
**Author:** [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Replies:** 1\ **Last updated:** [March 27, 2017, 6:08pm UTC](https://discuss.elastic.co/t/-grokparsefailure-general-exception-handling/80143 "2017-03-27T18:08:34Z")
I've perused the forum regarding \_grokparsefailure, but they all ask about solving specific log messages. I have a general question. As I form my Grok filters for one of our applications, I expect to get \_grokparsefail… --- ## [Grok filter - space not working](https://discuss.elastic.co/t/grok-filter-space-not-working/80141)
**Author:** [@August\_rush](https://discuss.elastic.co/u/August_rush)\ **Replies:** 4\ **Last updated:** [March 27, 2017, 5:28pm UTC](https://discuss.elastic.co/t/grok-filter-space-not-working/80141 "2017-03-27T17:28:06Z")
Hi , I am trying to use grok filter on this log , There is a space after WARN I have tried using %s\* \\s\* and also %{SPACE} but it is not triggering an email. can some one please help me. \[2017-03-25T00:00:07,137\]\[WARN … --- ## [Grok parse failure. Why?](https://discuss.elastic.co/t/grok-parse-failure-why/78738)
**Author:** [@Nikolay\_Shushkin](https://discuss.elastic.co/u/Nikolay_Shushkin)\ **Replies:** 5\ **Last updated:** [March 27, 2017, 4:29pm UTC](https://discuss.elastic.co/t/grok-parse-failure-why/78738 "2017-03-27T16:29:06Z")
Input data: 11.111.111.11 222.222.22.222 - - \[12/Mar/2017:06:26:47 +0000\] "GET /aaa/bbb/ccc/eng/web/listings?sort=ї"&byStartTime=~1401496140000&byEndTime=1401487200000~&mode=summary&byLocationId=21656615412 HTTP/1.1" 4… --- ## [Periodically run elasticsearch query count only new records](https://discuss.elastic.co/t/periodically-run-elasticsearch-query-count-only-new-records/80160)
**Author:** [@Exocomp](https://discuss.elastic.co/u/Exocomp)\ **Replies:** 0\ **Last updated:** [March 27, 2017, 3:35pm UTC](https://discuss.elastic.co/t/periodically-run-elasticsearch-query-count-only-new-records/80160 "2017-03-27T15:35:19Z")
Hi, I'm in need of a running an elasticsearch query periodically, every minute or so, and return a count that is based on new records since the last time the query was executed. I tried the "http\_poller" input plugin… --- ## [SexiLog (ELK) doesnt receive Windows Log Files (\*.log)](https://discuss.elastic.co/t/sexilog-elk-doesnt-receive-windows-log-files-log/80156)
**Author:** [@freichert](https://discuss.elastic.co/u/freichert)\ **Replies:** 0\ **Last updated:** [March 27, 2017, 3:04pm UTC](https://discuss.elastic.co/t/sexilog-elk-doesnt-receive-windows-log-files-log/80156 "2017-03-27T15:04:38Z")
Hello everybody :slight\_smile: since im really struggling with setting up SexiLog to receive Windows Logfiles i decided to ask you - hopefully you can show me the correct way :slight\_smile: so. What i have so far: a fu… --- ## [This is the error getting when i tried to test my logstash config in server](https://discuss.elastic.co/t/this-is-the-error-getting-when-i-tried-to-test-my-logstash-config-in-server/79981)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 2\ **Last updated:** [March 27, 2017, 2:57pm UTC](https://discuss.elastic.co/t/this-is-the-error-getting-when-i-tried-to-test-my-logstash-config-in-server/79981 "2017-03-27T14:57:51Z")
root@logserver:~# tail -f /var/log/logstash/logstash.log {:timestamp=\>"2017-03-24T06:57:54.132000-0400", :message=\>"SIGTERM received. Shutting down the pipeline.", :level=\>:warn} {:timestamp=\>"2017-03-24T06:58:33.872000-… --- ## [SSL communication fails btw Filebeat (5.1.1) and Logstash (5.1.1)](https://discuss.elastic.co/t/ssl-communication-fails-btw-filebeat-5-1-1-and-logstash-5-1-1/80123)
**Author:** [@Gaurav\_Tandon](https://discuss.elastic.co/u/Gaurav_Tandon)\ **Replies:** 2\ **Last updated:** [March 27, 2017, 2:47pm UTC](https://discuss.elastic.co/t/ssl-communication-fails-btw-filebeat-5-1-1-and-logstash-5-1-1/80123 "2017-03-27T14:47:40Z")
Getting below error in Logstash when filebeat tries to publish logs to Logstash “Looks like you either have an invalid key or your private key was not in PKCS8 format.” I followed guidelines in the docs: https://www… --- ## [Order log in kibana based on log timestamp](https://discuss.elastic.co/t/order-log-in-kibana-based-on-log-timestamp/80070)
**Author:** [@manopmk](https://discuss.elastic.co/u/manopmk)\ **Replies:** 9\ **Last updated:** [March 27, 2017, 10:25am UTC](https://discuss.elastic.co/t/order-log-in-kibana-based-on-log-timestamp/80070 "2017-03-27T10:25:32Z")
Hi I want to order my log in kibana based on log timestamp my logstash.conf filter { grok { match =\> \[ "message", "%{DATESTAMP:timestamp}" \] } date { locale =\> "en" match =\> \[ "timestamp", "YYYY… --- ## [By mistaske i deleted the logstash-\* index file in kibana](https://discuss.elastic.co/t/by-mistaske-i-deleted-the-logstash-index-file-in-kibana/80102)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 1\ **Last updated:** [March 27, 2017, 10:23am UTC](https://discuss.elastic.co/t/by-mistaske-i-deleted-the-logstash-index-file-in-kibana/80102 "2017-03-27T10:23:38Z")
by mistaske i deleted the logstash-\* index file in kibana discover page.. --- ## [Update elasticsearch document with logstash](https://discuss.elastic.co/t/update-elasticsearch-document-with-logstash/80116)
**Author:** [@Lemec\_Cinq](https://discuss.elastic.co/u/Lemec_Cinq)\ **Replies:** 0\ **Last updated:** [March 27, 2017, 10:03am UTC](https://discuss.elastic.co/t/update-elasticsearch-document-with-logstash/80116 "2017-03-27T10:03:59Z")
I filled data in elasticsearch via a logstash run with jdbc input plugin: input { jdbc { jdbc\_connection\_string =\> "jdbc:sqlserver://\*.\*.\*.\*;databaseName=DB\_TEST;" jdbc\_user =\> "user" jdbc\_password =\>… --- ## [How to read json file as input and index the data using elasticsearch as output?](https://discuss.elastic.co/t/how-to-read-json-file-as-input-and-index-the-data-using-elasticsearch-as-output/79997)
**Author:** [@Sowji\_Sowjanya](https://discuss.elastic.co/u/Sowji_Sowjanya)\ **Replies:** 1\ **Last updated:** [March 27, 2017, 7:32am UTC](https://discuss.elastic.co/t/how-to-read-json-file-as-input-and-index-the-data-using-elasticsearch-as-output/79997 "2017-03-27T07:32:16Z")
Hi, I have a file having Json data. I have an index "company" and document-type as "employee" present in my local elasticsearch server. I am unable to index the data from the file and send it to elasticsearch server. … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1009) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1011)