HI,
i am using filebeat to get the logs from a file
for filebeat ouput is logstash
and using logstash to filter the logs
for logstash output is elasticsearch
while trying to print logs on a console the logstash is just listening to the port and i cant see any logs on the console.
please help me solve it.
logstash.conf
input {
beats {
port => 5044
}
}
filter {
if [message] =~ /\d{15}/ {
grok {
match => ["message","%{TIMESTAMP_ISO8601:date}\*\[%{LOGLEVEL:log-level}\]\*%{DATA:thread}\*%{DATA:class}\*%{DATA:method}\*%{DATA:imei}\*%{DATA:token}\*%{GREEDYDATA:messagedata}"]
}
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
filebeat.conf
filebeat.inputs:
type: log
paths:
- "/home/Desktop/a.log"
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
path: ${path.config}/modules.d/*.yml
reload.enabled: false
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
when i start filebeat service :
2019-07-01T15:15:00.754+0530 INFO instance/beat.go:611 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2019-07-01T15:15:00.754+0530 INFO instance/beat.go:618 Beat UUID: 52f0b2d3-eddb-4fc7-b89f-6ac37b69bd30
2019-07-01T15:15:00.754+0530 INFO [beat] instance/beat.go:931 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "52f0b2d3-eddb-4fc7-b89f-6ac37b69bd30"}}}
2019-07-01T15:15:00.754+0530 INFO [beat] instance/beat.go:940 Build info {"system_info": {"build": {"commit": "c01de6731c401c73d03ed996cdf21f5724ad83f4", "libbeat": "6.8.0", "time": "2019-05-15T19:07:08.000Z", "version": "6.8.0"}}}
2019-07-01T15:15:00.754+0530 INFO [beat] instance/beat.go:943 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.8"}}}
2019-07-01T15:15:00.755+0530 INFO [beat] instance/beat.go:947 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-06-27T19:54:08+05:30","containerized":true,"name":"qolsys-desktop","ip":["127.0.0.1/8","::1/128","192.168.17.128/24","fe80::7254:d2ff:feab:b75d/64","172.17.42.1/16"],"kernel_version":"4.4.0-148-generic","mac":["70:54:d2:ab:b7:5d","02:42:9b:76:f1:6f"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"14.04.6 LTS, Trusty Tahr","major":14,"minor":4,"patch":6,"codename":"trusty"},"timezone":"IST","timezone_offset_sec":19800,"id":"3adafe86fcc05c016e767ae85c6124f5"}}}
Config OK
when i run filebeat using the command:
sudo filebeat -v -e -d "*" -c /etc/filebeat/filebeat.yml
2019-07-01T16:35:38.151+0530|INFO|[monitoring]|log/log.go:144|Non-zero metrics in the last 30s{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":8}},"total":{"ticks":20,"time":{"ms":32},"value":20},"user":{"ticks":20,"time":{"ms":24}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":5},"info":{"ephemeral_id":"eb776661-1cfa-4f31-a01e-6c065759e1e0","uptime":{"ms":33028}},"memstats":{"gc_next":4194304,"memory_alloc":3209104,"memory_total":4791192,"rss":24674304}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4},"load":{"1":2.95,"15":3.08,"5":3.14,"norm":{"1":0.7375,"15":0.77,"5":0.785}}}}}}|
2019-07-01T16:36:08.150+0530|INFO|[monitoring]|log/log.go:144|Non-zero metrics in the last 30s{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":4}},"total":{"ticks":30,"time":{"ms":4},"value":30},"user":{"ticks":20}},"handles":{"limit":{"hard":4096,"soft":1024},"open":5},"info":{"ephemeral_id":"eb776661-1cfa-4f31-a01e-6c065759e1e0","uptime":{"ms":63028}},"memstats":{"gc_next":4194304,"memory_alloc":3483400,"memory_total":5065488}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":3.01,"15":3.08,"5":3.15,"norm":{"1":0.7525,"15":0.77,"5":0.7875}}}}}}|