Filebeat not sending data to Elasticsearch or Logstash

[Brandonmcgr (Brandon Mc Grath)]

Hello, Forgive me if this is an issue that is repeated on here.

Originally, I had this working with what must be the exact same process. But, since I rebuilt the virtual machines something is no longer working and data is not making it from Filebeat to Logstash or Elasticsearch.

Also, this set up is to send snort logs.

Here are my versions:

• filebeat version 6.2.3 (amd64), libbeat 6.2.3
• kibana 6.2.2
• elasticsearch 6.2.2
• logstash 6.2.2

Everything starts up fine with no issues:

Logstash:

https://pastebin.com/raw/V7Jxv2VU

Elasticsearch:

https://pastebin.com/raw/mRS2iQDE

Kibana:

https://pastebin.com/raw/XwddpPsE

So, everything is up and in a single-node pointing to 192.168.0.53. Here is the logstash conf file:

input {
beats{
host => "192.168.0.53"
port => 5044
codec => plain {
charset => "US-ASCII"
}
}
}

filter {
csv {
columns => ["datetime","sig_gen","sig_id","sig_rev","msg","proto","src_ip","src_port","dst_ip","dst_port","packet","category","priority","additional"]
}
mutate {
strip => ["datetime"]
}
date {
match => [ "datetime", "MM/dd/yy-HH:mm:ss.SSSSSS" ]
}

if [priority] == "1" {
  mutate {
    add_field => { "severity" => "High" }
  }
}
if [priority] == "2" {
  mutate {
    add_field => { "severity" => "Medium" }
  }
}
if [priority] == "3" {
  mutate {
    add_field => { "severity" => "Low" }
  }
}
if [ids_proto] {
  if [ids_proto] =~ /^GPL/ {
    mutate {
      add_tag => [ "Snort-ET-sig" ]
      add_field => [ "ids_rule_type", "Emerging Threats" ]
    }
  }
  if [ids_proto] =~ /^ET/ {
    mutate {
      add_tag => [ "Snort-ET-sig" ]
      add_field => [ "ids_rule_type", "Emerging Threats" ]
    }
  }
  if "Snort-ET-sig" not in [tags] {
    mutate {
      add_tag => [ "Snort-sig" ]
      add_field => [ "ids_rule_type", "Snort" ]
    }
  }
}
if [src_ip]  {
  geoip {
    source => "src_ip"
    target => "geoip"
    #database => "/etc/logstash/GeoLiteCity.dat"
    add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
    add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
  }
  mutate {
    convert => [ "[geoip][coordinates]", "float" ]
  }
  if ![geoip.ip] {
    if [dest_ip]  {
      geoip {
        source => "dest_ip"
        target => "geoip"
        #database => "/etc/logstash/GeoLiteCity.dat"
        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
      }
      mutate {
        convert => [ "[geoip][coordinates]", "float" ]
      }
    }
  }
}

}

output {
elasticsearch {
hosts => [ "192.168.0.53:9200" ]
}
}

Finally, the entirety of the filebeat log:

2018-04-04T18:02:19.692+0100	INFO	instance/beat.go:468	Home path: [/etc/filebeat] Config path: [/etc/filebeat] Data path: [/etc/filebeat/data] Logs path: [/etc/filebeat/logs]
2018-04-04T18:02:19.692+0100	INFO	instance/beat.go:475	Beat UUID: 4f6379e1-5279-459c-805e-3115ef7234e6

I've tried to supply as much information as possible, but this is the exact same set up as I used before but something is going wrong.

Any ideas?

Thanks!

EDIT: FILEBEAT.YML
https://pastebin.com/raw/iYx0EidU

[adrisr (Adrian Serrano)]

In filebeat.yml, your prospector is not enabled:

- type: log

  # Change to true to enable this prospector configuration.
  enabled: false

need to set it to true.

Also, there is an indentation error a few lines below:

document_type: snort

It needs to be indented by two spaces. Although the document_type setting was removed in 6.0, it will cause anything you add below it to not work as expected.

[Brandonmcgr (Brandon Mc Grath)]

I literally just noticed this myself lol. I was just about to close it.

Thanks anyway