HI,
I am using elk stack (elastic , logstash , kibana) version : 6.8.0
in ubuntu 14.04.
when i am trying to send logs from filebeat to logstash there is no error but my logstash is just listening to the port and its not printing any logs.
Filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /home/a.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.logstash:
hosts: ["localhost:9600"]
logging.level: debug
when i run the filebeat its says the configuration ok
filebeat logs :
2019-07-19T15:38:32.719+0530 INFO instance/beat.go:611 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2019-07-19T15:38:32.719+0530 DEBUG [beat] instance/beat.go:648 Beat metadata path: /var/lib/filebeat/meta.json
2019-07-19T15:38:32.719+0530 INFO instance/beat.go:618 Beat UUID: 52f0b2d3-eddb-4fc7-b89f-6ac37b69bd30
2019-07-19T15:38:32.720+0530 INFO [beat] instance/beat.go:931 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "52f0b2d3-eddb-4fc7-b89f-6ac37b69bd30"}}}
2019-07-19T15:38:32.720+0530 INFO [beat] instance/beat.go:940 Build info {"system_info": {"build": {"commit": "c01de6731c401c73d03ed996cdf21f5724ad83f4", "libbeat": "6.8.0", "time": "2019-05-15T19:07:08.000Z", "version": "6.8.0"}}}
2019-07-19T15:38:32.720+0530 INFO [beat] instance/beat.go:943 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.8"}}}
2019-07-19T15:38:32.721+0530 INFO [beat] instance/beat.go:947 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-07-15T11:45:56+05:30","containerized":true,"name":"qolsys-desktop","ip":["127.0.0.1/8","::1/128","192.168.17.128/24","fe80::7254:d2ff:feab:b75d/64","172.17.42.1/16"],"kernel_version":"4.4.0-148-generic","mac":["70:54:d2:ab:b7:5d","02:42:9c:05:a6:43"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"14.04.6 LTS, Trusty Tahr","major":14,"minor":4,"patch":6,"codename":"trusty"},"timezone":"IST","timezone_offset_sec":19800,"id":"3adafe86fcc05c016e767ae85c6124f5"}}}
2019-07-19T15:38:32.721+0530 INFO instance/beat.go:280 Setup Beat: filebeat; Version: 6.8.0
2019-07-19T15:38:32.721+0530 DEBUG [beat] instance/beat.go:301 Initializing output plugins
2019-07-19T15:38:32.721+0530 DEBUG [processors] processors/processor.go:66 Processors:
2019-07-19T15:38:32.722+0530 DEBUG [publish] pipeline/consumer.go:137 start pipeline event consumer
2019-07-19T15:38:32.722+0530 INFO [publisher] pipeline/module.go:110 Beat name: qolsys-desktop
Config OK
filebeat test run also looks fine:
cmd : filebeat test output
logstash: localhost:9600...
connection...
parse host... OK
dns lookup... OK
addresses: 127.0.0.1
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
Logstash.yml
input {
beats {
port => "5044"
}
}
filter {
grok {
match => ["message","%{TIMESTAMP_ISO8601:date}\*\[%{LOGLEVEL:log-level}\]"]
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
when i try to run and print logs on console :
it just listening to the port 9600 but not writing any logs.
logstash logs:
[INFO ] 2019-07-19 15:05:57.002 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.0"}
[INFO ] 2019-07-19 15:06:06.112 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-07-19 15:06:06.666 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2019-07-19 15:06:06.844 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2019-07-19 15:06:06.998 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2019-07-19 15:06:07.002 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO ] 2019-07-19 15:06:07.045 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO ] 2019-07-19 15:06:07.060 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template
[INFO ] 2019-07-19 15:06:07.097 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[INFO ] 2019-07-19 15:06:07.687 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2019-07-19 15:06:07.706 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x77bd26f6 run>"}
[INFO ] 2019-07-19 15:06:07.821 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2019-07-19 15:06:07.828 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-07-19 15:06:08.220 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
Please help me solve it . I'm unable to trace where i am doing it wrong.