How to transfer logs from different VM's (Digitalocen) to a single Logstash using Filebeat

Hi all,

From Last two days i got stuck in this task.Task is there are multiple Ubuntu machines are running on cloud (DigitalOcen), I have to take the logs of those machines and ship those logs to Logstash where complete ELK is configured.

I have configured filebeat in one system and my filebeat.yml is like below:

filebeat.prospectors:

  • type: log
    paths:
    • /var/log/nginx/.log
      filebeat.config.modules:
      path: ${path.config}/modules.d/
      .yml
      reload.enabled: true
      output.logstash:
      hosts: ["206.189.129.234:5044"]

Logstash:-

And my simple logstash.conf file is like Below

input {
beats {
port => 5044
}

}

output {

  elasticsearch {
                    hosts => "206.189.129.234:9200"
                    manage_template => false
                    index => "nginx-%{+YYYY.MM.dd}"

}

}

when i start the logstash it is running successfully but I am not able to see any index in elasticsearch. I had tried multiple ways but no results can anyone help me Out of this.

And Is there any particular process is there for above scenario..

Thanks in advance....

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile: You may also want to redact your public IPs.

Do the filebeat logs show it connecting to Logstash?

Hi Thanks for reply and suggestion,

I check my file beat logs /var/log/filebeat folder and the logs are

2020-03-10T03:28:07.717Z INFO instance/beat.go:468 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]

2020-03-10T03:28:07.717Z INFO instance/beat.go:475 Beat UUID: 8a1fc2dc-b9a7-4d1b-ba72-9de650887499

This is the only stuff i have there

I think filebeat is not connected and shipping the logs to logstash

Hi when i again start the filebeat i getting the error like below

<2020-03-10T05:33:22.567Z INFO pipeline/output.go:95 Connecting to backoff(async(tcp://206.189.129.234:5044))/>
<2020-03-10T05:33:31.545Z ERROR fileset/factory.go:105 Error creating input: Can only start an input when all related states are finished: {Id:262590-64513 Finished:false Fileinfo:0xc420518dd0 Source:/var/log/nginx/access.log Offset:0 Timestamp:2020-03-10 05:33:21.544219089 +0000 UTC m=+0.092800296 TTL:-1ns Type:log Meta:map FileStateOS:262590-64513} />
< 2020-03-10T05:33:31.548Z ERROR [reload] cfgfile/list.go:96 Error creating runner from config: Can only start an input when all related states are finished: {Id:262590-64513 Finished:false Fileinfo:0xc420518dd0 Source:/var/log/nginx/access.log Offset:0 Timestamp:2020-03-10 05:33:21.544219089 +0000 UTC m=+0.092800296 TTL:-1ns Type:log Meta:map FileStateOS:262590-64513} />
< 2020-03-10T05:33:41.553Z ERROR fileset/factory.go:105 Error creating input: Can only start an input when all related states are finished: {Id:262590-64513 Finished:false Fileinfo:0xc420518dd0 Source:/var/log/nginx/access.log Offset:0 Timestamp:2020-03-10 05:33:21.544219089 +0000 UTC m=+0.092800296 TTL:-1ns Type:log Meta:map FileStateOS:262590-64513} />
< 2020-03-10T05:33:41.556Z ERROR [reload] cfgfile/list.go:96 Error creating runner from config: Can only start an input when all related states are finished: {Id:262590-64513 Finished:false Fileinfo:0xc420518dd0 Source:/var/log/nginx/access.log Offset:0 Timestamp:2020-03-10 05:33:21.544219089 +0000 UTC m=+0.092800296 TTL:-1ns Type:log Meta:map FileStateOS:262590-64513} />

Can you help me out where i am making mistake?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.