Filebeat not receiving logs and error in logstash

Sai_Avinash_Duddupud · July 10, 2020, 7:53am

I am trying to setup filebeat and logstash on my server1 and send data to elasticsearch located on server2 and visualize it using kibana.

Following are filebeat logs and when i run filebeat test output it showed the result as show in image bleow.

As you can observer, filbeat is not receving logs at all....is this because of the error in logstash?

2020-07-10T07:40:14.852Z	DEBUG	[input]	input/input.go:141	Run input
2020-07-10T07:40:14.852Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-07-10T07:40:14.852Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-07-10T07:40:24.853Z	DEBUG	[input]	input/input.go:141	Run input
2020-07-10T07:40:24.853Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-07-10T07:40:24.853Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-07-10T07:40:34.853Z	DEBUG	[input]	input/input.go:141	Run input
2020-07-10T07:40:34.853Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-07-10T07:40:34.853Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-07-10T07:40:44.828Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20},"total":{"ticks":170,"time":{"ms":4},"value":170},"user":{"ticks":150,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":7},"info":{"ephemeral_id":"4f97f60d-b9f4-451c-b9f2-1935988798b1","uptime":{"ms":840027}},"memstats":{"gc_next":10220512,"memory_alloc":5959576,"memory_total":24826152},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.03,"15":0.08,"5":0.06,"norm":{"1":0.015,"15":0.04,"5":0.03}}}}}}
2020-07-10T07:40:44.853Z	DEBUG	[input]	input/input.go:141	Run input
2020-07-10T07:40:44.856Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-07-10T07:40:44.856Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-07-10T07:40:54.856Z	DEBUG	[input]	input/input.go:141	Run input
2020-07-10T07:40:54.856Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-07-10T07:40:54.856Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-07-10T07:41:04.856Z	DEBUG	[input]	input/input.go:141	Run input
2020-07-10T07:41:04.856Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-07-10T07:41:04.856Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-07-10T07:41:14.828Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20},"total":{"ticks":180,"time":{"ms":8},"value":180},"user":{"ticks":160,"time":{"ms":8}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":7},"info":{"ephemeral_id":"4f97f60d-b9f4-451c-b9f2-1935988798b1","uptime":{"ms":870027}},"memstats":{"gc_next":10216448,"memory_alloc":5117872,"memory_total":25101640},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.02,"15":0.08,"5":0.05,"norm":{"1":0.01,"15":0.04,"5":0.025}}}}}}
2020-07-10T07:41:14.856Z	DEBUG	[input]	input/input.go:141	Run input
2020-07-10T07:41:14.856Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-07-10T07:41:14.856Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-07-10T07:41:24.856Z	DEBUG	[input]	input/input.go:141	Run input
2020-07-10T07:41:24.856Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-07-10T07:41:24.856Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-07-10T07:41:34.857Z	DEBUG	[input]	input/input.go:141	Run input
2020-07-10T07:41:34.857Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-07-10T07:41:34.857Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0

Following are logstash logs and I am getting some exception errors

[WARN ] 2020-07-10 07:26:21.802 [nioEventLoopGroup-2-2] DefaultChannelPipeline - An 

exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: -1
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: -1
	at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.9.jar:?]
	at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.9.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	... 8 more

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/mahesh/Documents/refactor/nomi/unity/media/*.log

output.logstash:
  enabled: true
  hosts: ["localhost:5044"]

logstash.conf

input {
beats {
    port => 5044
    ssl => false
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}] %{LOGLEVEL:loglevel}\|%{GREEDYDATA:module}\|%{GREEDYDATA:content}" }
  }
  date {
    locale => "en"
    match => [ "timestamp", "YYYY-MM-dd HH:mm:ss"]
    target => "@timestamp"
    timezone => "America/New_York"
  }
}

output {
  elasticsearch {
    hosts => "elk_server_ip:9200"
    manage_template => false
    index => "blend_test" 
  }
  stdout { codec => rubydebug { metadata => true } }
}

grumo35 · July 10, 2020, 9:02am

First, can you show us logstash and filebeat version ?
Are they up to date ?

Sai_Avinash_Duddupud · July 10, 2020, 9:20am

@grumo35

Filebeat version - 7.8.0
logstash version - 7.8.0

And to my surprise, there are no errors in logstash console, but filebeat logs are the same.... filbeat is not harvesting logs... and I have checked again and again that i have given the correct log file path.

grumo35 · July 10, 2020, 9:23am

What's the rights of the log files ?

Sai_Avinash_Duddupud · July 10, 2020, 9:36am

@grumo35

-rw-rw-r-- 1 ubuntu ubuntu 118366 Jul 10 09:27 django_blend.log

Changed above one to using chmod 777 command

-rwxrwxrwx 1 ubuntu ubuntu 121496 Jul 10 09:43 django_blend.log

but still not receving logs even after restarting filebeat

grumo35 · July 10, 2020, 10:02am

when using tail -f /home/mahesh/Documents/refactor/nomi/unity/media/*.log

Are they any newlines created ?

It's strage that filebeat isnt catchign anything, maybe the directory is not accessible ?

Jiboss · July 10, 2020, 10:11am

Hi,

You don't need to use filebeat if all the data came to the same host. You can add this directly in logstash :

input {
  file {
    path => "/path/to/file"
  }
}

Sai_Avinash_Duddupud · July 10, 2020, 10:17am

@grumo35 no new lines are present.

drwxrwxr-x+ is the permission config for that folder where log file is present

grumo35 · July 10, 2020, 10:21am

If there is no newlines that's why filebeat isnt seeing new content, i guess your log files aren't populated ?

Sai_Avinash_Duddupud · July 10, 2020, 10:22am

@Jiboss, u want me to relpace input node in logstash.conf with above input snippet you provided?

Jiboss · July 10, 2020, 10:29am

Yeah, if you don't need filebeat I would do that. I think you need to receive new data to see your logs. Try to send another file.

Puneeth_S_B_Gowda1 · July 10, 2020, 11:34am

Try with 7.6.2 version as this was the solution for my same issue it might work for you too

Sai_Avinash_Duddupud · July 12, 2020, 4:07pm

Update :- it finally worked after I changed the filebeat.yml to the following config

filebeat.inputs:
- type: log
  paths:
    - /your/path/goes/here/*.log
output.logstash:
  enabled: true
  hosts: ["localhost:5044"]
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

system · August 11, 2020, 2:14am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.