Logstash doesn't open files

mrdiogon · October 19, 2021, 2:00pm

Hi everyone,
I tried to install filebeat on a windows server which send logs to a logstash server on docker. But no indexes are created and it seems logstash doesn't get any input from logstash. Here's my configuration file:

filebeat.inputs:

- type: log
  enabled: false
  paths:
    - C:\Server\Log\batch\*.json
    
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.logstash:
  enabled: true
  hosts: ["logstash.tools.lan:80"]

Does anyone have an idea what's happening here? For what i see, I think filebeat is ignoring the log files.

Thanks

mrdiogon · October 19, 2021, 2:02pm

Here's my log:

2021-10-19T15:32:10.536+0200	INFO	[monitoring]	log/log.go:142	Starting metrics logging every 30s
2021-10-19T15:32:10.538+0200	INFO	memlog/store.go:119	Loading data file of 'C:\ProgramData\filebeat\registry\filebeat' succeeded. Active transaction id=0
2021-10-19T15:32:10.538+0200	INFO	memlog/store.go:124	Finished loading transaction log file for 'C:\ProgramData\filebeat\registry\filebeat'. Active transaction id=0
2021-10-19T15:32:10.538+0200	WARN	beater/filebeat.go:381	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-10-19T15:32:10.538+0200	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
2021-10-19T15:32:10.538+0200	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 1
2021-10-19T15:32:10.539+0200	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2021-10-19T15:32:10.539+0200	INFO	cfgfile/reload.go:164	Config reloader started
2021-10-19T15:32:10.540+0200	INFO	cfgfile/reload.go:224	Loading of config files completed.
2021-10-19T15:32:40.563+0200	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":156,"time":{"ms":156}},"total":{"ticks":312,"time":{"ms":312},"value":312},"user":{"ticks":156,"time":{"ms":156}}},"handles":{"open":217},"info":{"ephemeral_id":"e6df9357-db57-4be9-bd18-7f11aa20cbf1","uptime":{"ms":30135},"version":"7.15.1"},"memstats":{"gc_next":18571648,"memory_alloc":16198416,"memory_sys":32021720,"memory_total":55095984,"rss":55422976},"runtime":{"goroutines":24}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4}}}}}
2021-10-19T15:33:10.573+0200	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":327,"time":{"ms":15},"value":327},"user":{"ticks":171,"time":{"ms":15}}},"handles":{"open":221},"info":{"ephemeral_id":"e6df9357-db57-4be9-bd18-7f11aa20cbf1","uptime":{"ms":60147},"version":"7.15.1"},"memstats":{"gc_next":18571648,"memory_alloc":16397880,"memory_total":55295448,"rss":55451648},"runtime":{"goroutines":24}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2021-10-19T15:33:40.578+0200	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":327,"value":327},"user":{"ticks":171}},"handles":{"open":223},"info":{"ephemeral_id":"e6df9357-db57-4be9-bd18-7f11aa20cbf1","uptime":{"ms":90142},"version":"7.15.1"},"memstats":{"gc_next":18571648,"memory_alloc":16592064,"memory_total":55489632,"rss":55484416},"runtime":{"goroutines":24}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2021-10-19T15:34:10.579+0200	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":327,"value":327},"user":{"ticks":171}},"handles":{"open":228},"info":{"ephemeral_id":"e6df9357-db57-4be9-bd18-7f11aa20cbf1","uptime":{"ms":120165},"version":"7.15.1"},"memstats":{"gc_next":19420848,"memory_alloc":9768376,"memory_total":55759960,"rss":55717888},"runtime":{"goroutines":24}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2021-10-19T15:34:40.556+0200	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":171,"time":{"ms":15}},"total":{"ticks":358,"time":{"ms":31},"value":358},"user":{"ticks":187,"time":{"ms":16}}},"handles":{"open":230},"info":{"ephemeral_id":"e6df9357-db57-4be9-bd18-7f11aa20cbf1","uptime":{"ms":150141},"version":"7.15.1"},"memstats":{"gc_next":19420848,"memory_alloc":9881000,"memory_total":55872584,"rss":55173120},"runtime":{"goroutines":24}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2021-10-19T15:35:10.568+0200	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":171},"total":{"ticks":358,"value":358},"user":{"ticks":187}},"handles":{"open":232},"info":{"ephemeral_id":"e6df9357-db57-4be9-bd18-7f11aa20cbf1","uptime":{"ms":180145},"version":"7.15.1"},"memstats":{"gc_next":19420848,"memory_alloc":10077832,"memory_total":56069416,"rss":55193600},"runtime":{"goroutines":24}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2021-10-19T15:41:11.192+0200	INFO	beater/filebeat.go:515	Stopping filebeat
2021-10-19T15:41:11.192+0200	INFO	beater/crawler.go:148	Stopping Crawler
2021-10-19T15:41:11.192+0200	INFO	beater/crawler.go:158	Stopping 0 inputs
2021-10-19T15:41:11.192+0200	INFO	cfgfile/reload.go:227	Dynamic config reloader stopped
2021-10-19T15:41:11.192+0200	INFO	beater/crawler.go:178	Crawler stopped

leandrojmp · October 19, 2021, 2:12pm

You didn't enabled the input, try to use enabled: true.

mrdiogon · October 21, 2021, 6:09am

Thanks, i forgot. But it still does not work...

grumo35 · October 21, 2021, 9:30am

Hi,

Can you show logs ?

mrdiogon · October 21, 2021, 9:41am

At firts it didn't change but now ther's an explicit error. I'm not sure what's causing it...

2021-10-21T11:33:33.469+0200	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(async(tcp://logstash.tools.lan:80)) established
2021-10-21T11:33:58.271+0200	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8953,"time":{"ms":16}},"total":{"ticks":59843,"time":{"ms":188},"value":59843},"user":{"ticks":50890,"time":{"ms":172}}},"handles":{"open":300},"info":{"ephemeral_id":"b80b724c-d4c8-4611-bfd6-8c1658105122","uptime":{"ms":13260237},"version":"7.15.1"},"memstats":{"gc_next":40082192,"memory_alloc":22078496,"memory_total":2234106352,"rss":84209664},"runtime":{"goroutines":236}},"filebeat":{"harvester":{"open_files":33,"running":78}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":4096,"batches":3,"failed":6144,"total":6144},"read":{"errors":1},"write":{"bytes":263843}},"pipeline":{"clients":1,"events":{"active":4117,"retry":8192}}},"registrar":{"states":{"current":3}}}}}
2021-10-21T11:34:03.515+0200	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: read tcp 10.90.101.37:62964->10.100.153.4:80: i/o timeout
2021-10-21T11:34:03.515+0200	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: read tcp 10.90.101.37:62964->10.100.153.4:80: i/o timeout
2021-10-21T11:34:03.515+0200	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-10-21T11:34:03.515+0200	INFO	[publisher]	pipeline/retry.go:223	  done
2021-10-21T11:34:03.515+0200	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-10-21T11:34:03.515+0200	INFO	[publisher]	pipeline/retry.go:223	  done
2021-10-21T11:34:03.556+0200	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: client is not connected
2021-10-21T11:34:03.556+0200	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-10-21T11:34:03.556+0200	INFO	[publisher]	pipeline/retry.go:223	  done
2021-10-21T11:34:05.442+0200	ERROR	[publisher_pipeline_output]	pipeline/output.go:180	failed to publish events: client is not connected

grumo35 · October 21, 2021, 9:45am

Can you please double check network configuration ? i believe your running it on docker are you able to telnet or curl to the host ?

You can easily check on linux port connectivity with a simple trick

echo > /dev/tcp/[host]/[port] && echo "Port is open"

mrdiogon · October 21, 2021, 10:03am

I tried to test de connection from my windows computer with the Powershell command:

 Test-NetConnection logstash.tools.lan -Port 80

and it succeed.

However, i do use swarm AND traefik for my elk stack. So, with swarm, i redirect the port 80 to the port 5044 of my logstash service...

grumo35 · October 21, 2021, 11:32am

Hum,

I'm sorry i'm not familiar with the stack your using, maybe traefik logs could help on this ?

I'm really not sure what could be your problem here

system · November 18, 2021, 11:32am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.