Filebeat is not sending logs to logstash

Elastic Stack Logstash
1

There are some sample apache logs in folder listed in paths, but still they not going to logstash
Here is my configuration,

filebeat:

filebeat:
  prospectors:
    -
      paths:
        - /home/saurabh/sample-log/*.log"
      input_type: log
  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["localhost:5044"]
  console:
    pretty: true

shipper:

logging:
  to_files: true
  files:
    path: /var/log/filebeat
    name: mybeat
    rotateeverybytes: 10485760 # = 10MB

logstash:

input {
  beats {
    port => 5044
  }
  stdin { 
  }
}

filter{
  grok {
    match => ["message", "%{COMBINEDAPACHELOG}"]
  }
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
  }
  geoip {
    source => "clientip"
    target => "geoip"
    database => "/home/saurabh/Documents/GeoLiteCity.dat"
    add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"]
    add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"]
  }
  mutate {
    convert => ["[geoip][coordinates]", "float"]
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    manage_template => false
    **hosts => "localhost:9200"**
  }
}

Thank you.

2

Are they making it to stdout?
Is there anything in the beats logs?
What versions are you on?

3

Nothing is in beats logs as well as stdout.
I am using the latest version - filebeat_1.2.3_amd64

Please correct me if i am wrong, the default ports use by,

  • elasticsearch: 9200
  • kibana: 5601
  • logstash: 5000
  • filebeat 5044
4

Versions, not ports :slight_smile:

5

I am using the latest version -
filebeat_1.2.3_amd64
elasticsearch-2.3.4
logstash_2.3.4-1_all
kibana_4.5.2_amd64

I also want to know that am I using correct ports?

6

@warkolm please tell me why my logs are not going to logstash.
and how to startfeat with stdout.
thanks.

7

This sounds like a beats issue, I'd move the thread there to see if you can get some assitance.

8

Filebeat's output port needs to be the same as Logstash's input port. It doesn't matter if you pick 5000, 5044, or something else, but they must be the same. There might be other problems with your configuration but start by addressing this.

9

@magnusbaeck Thank you for this. I realized that and made changes accordingly.
My config file has same ports as you said.
Sorry I didnt updated this in question. But still it is not working.

10

Perhaps Filebeat is tailing the input files and waiting for more data to be added to them? Also, should there really be a lone double quote at the end of the filename pattern?

    - /home/saurabh/sample-log/*.log"
11

@magnusbaeck Its working.
Thank you. You are lifesaver :slight_smile: