I am new to ELK Stack and unable to get basic working. Please help.
Steps Done:
-
Started elasticsearch.bat (Started Fine)
-
Open http://localhost:9200 in Chrome and got:
{ "name" : "SrfDQVN", "cluster_name" : "elasticsearch", "cluster_uuid" : "ZtUOrpL8S8q5gAdh6tCS4g", "version" : { "number" : "6.3.1", "build_flavor" : "default", "build_type" : "zip", "build_hash" : "eb782d0", "build_date" : "2018-06-29T21:59:26.107521Z", "build_snapshot" : false, "lucene_version" : "7.3.1", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } -
Logstash sample.conf file in logstash dir
-sample.conf
input {
beats {
type => beats
port => 5044
host => "localhost"
}
}
filter {
grok {
match => ["message", "%{TIMESTAMP_IS08601:timestamp_string}%{SPACE}%{GREEDYDATA:line}"]
}
}
date {
match => [ "timestamp_string" , "IS08601" ]
}
mutate {
remove_field => [message, timestamp_string]
}
}
output {
stdout {
codec => rubydebug
}
}
-
logstash.yml
http.host: "127.0.0.1" http.port: 5044 -
Started logstash using command from logstash directory
bin\logstash -f sample.conf -
Here is the last line from the command window after starting logstash
[2018-08-01T23:54:00,246][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>5044} -
Filebeat sample.log file
2018-08-01T13:00:00Z I met a traveller from an antique land 1111 2018-08-01T14:00:00Z I met a traveller from an antique land 2222 2018-08-01T15:00:00Z I met a traveller from an antique land 3333 2018-08-01T16:00:00Z I met a traveller from an antique land 4444 2018-08-01T17:00:00Z I met a traveller from an antique land 5555 2018-08-01T18:00:00Z I met a traveller from an antique land 6666 2018-08-01T19:00:00Z I met a traveller from an antique land 7777 2018-08-01T20:00:00Z I met a traveller from an antique land 8888 2018-08-01T21:00:00Z I met a traveller from an antique land 9999 2018-08-01T22:00:00Z I met a traveller from an antique land 11111 2018-08-01T23:00:00Z I met a traveller from an antique land 22222 2018-08-01T00:00:00Z I met a traveller from an antique land 33333 2018-09-01T01:00:00Z I met a traveller from an antique land 44444 2018-09-01T02:00:00Z I met a traveller from an antique land 55555 2018-09-01T03:00:00Z I met a traveller from an antique land 66666 2018-09-01T04:00:00Z I met a traveller from an antique land 77777 -
filebeat.yml file
filebeat.prospectors: - type: log enabled: true paths: - .\sample.log output.logstash: hosts: ["localhost:5044"] -
Running following command to start filebeat through elevated command prompt
PS C:\Program Files\filebeat-6.3.2-windows-x86_64> .\filebeat.exe -c .\filebeat.yml -e -v
I get the following error:(few lines)
2018-08-02T00:25:22.481+0530 INFO registrar/registrar.go:117 Loading registrar data from C:\Program Files\filebeat-6.3.2-windows-x86_64\data\registry
2018-08-02T00:25:22.490+0530 INFO registrar/registrar.go:124 States Loaded from registrar: 2
2018-08-02T00:25:22.492+0530 WARN beater/filebeat.go:354 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-02T00:25:22.493+0530 INFO crawler/crawler.go:48 Loading Inputs: 1
2018-08-02T00:25:22.502+0530 INFO log/input.go:118 Configured paths: [C:\Program Files\filebeat-6.3.2-windows-x86_64\sample.log]
2018-08-02T00:25:22.503+0530 INFO input/input.go:88 Starting input of type: log; ID: 6142389514979083186
2018-08-02T00:25:22.510+0530 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 1
2018-08-02T00:25:22.511+0530 INFO log/harvester.go:228 Harvester started for file: C:\Program Files\filebeat-6.3.2-windows-x86_64\sample.log
2018-08-02T00:25:23.532+0530 ERROR logstash/async.go:235 Failed to publish events caused by: lumberjack protocol error
2018-08-02T00:25:23.535+0530 ERROR logstash/async.go:235 Failed to publish events caused by: client is not connected