Ingest problem with filebeat and logstash

Elastic Stack Logstash
1

Hi,

I have an ingest problem with filebeat and logstash. I have machines with a configuration on it's own to get different type of log files.
my first filebeat.yml file is like this :

filebeat.prospectors:
- document_type: wowza
  paths:
    - /usr/local/WowzaStreamingEngine-4.0.1/logs/wowzastreamingengine_access.log*

output:
  logstash:
    hosts: ["somepath:5043"]


logging:
  to_files: true
  level: info
  files:
    path: /var/log/filebeat
    name: filebeat.log
    rotateeverybytes: 10485760
    keepfiles: 7

and the second is like this :

filebeat.prospectors:
- document_type: elemental_live
  paths:
    - /opt/elemental_se/web/log/*0000/job_*/*_eme.log


output:
  logstash:
    hosts: ["somepath:5043"]


logging:
  to_files: true
  level: warning
  files:
    path: /var/log
    name: filebeat.log
    rotateeverybytes: 10485760
    keepfiles: 4

my output conf of logstash is this :

output {
  elasticsearch {
    hosts => ["somepath:9200"]
    index => "logstash-%{type}-%{+YYYY.MM.dd}"
  }
}

my problem is, with the 2nd yml file i have indices like this one :
green open logstash-elemental_live-2017.11.22 CvnMPAClRNK9a8MxY63ccQ 1 0 141 0 128.4kb 128.4kb

but with the first file it look like this :
green open logstash-%{type}-2017.11.21 1rOpLykcTBK2RyXkLmZd7A 1 0 3004790 0 756.1mb 756.1mb

and the problem is that with kibana when i try to create a new index pattern it has a bug with "logstash-%{type}" (which is not supposed to be like this at the beginning)

Anyone who has a clue why my indices look like "logstash-%{type}..." and not "logstash-wowza..." ?

Thx in advance.

2

Anyone who has a clue why my indices look like "logstash-%{type}..." and not "logstash-wowza..." ?

Those events clearly don't have the type field set to anything. Do you really have a leading space on the filebeat.prospectors: line for the wowza logs?

3

hi,

no it's just the copy past that did this (just checked it)

4

ok i just checked version,

The wowza config is on filebeat 6.0 which mean that "document_type" is ignored and has to be replaced by "fields"
Is the syntax the same, like we just have to change :

  • document_type: wowza
    to
  • fields: wowza
    ?
5

No, fields points to a dictionary. See https://www.elastic.co/guide/en/beats/filebeat/6.0/configuration-filebeat-options.html#configuration-fields.

6

so if my guessing is correct my filebeat should be like this :

filebeat.prospectors:
 -
    paths:
      - /usr/local/WowzaStreamingEngine-4.0.1/logs/wowzastreamingengine_access.log*
    fields:
      application: "wowza"

and my logstash output file should look like this :

output {
  elasticsearch {
    hosts => ["somepath:9200"]
    index => "logstash-%{application}-%{+YYYY.MM.dd}"
  }
}

?

7

Yeah, that looks okay.

8

tried it, same problem :

green open logstash-%{application}-2017.11.27 JyG-38ilQ6Wcjb8haM1atQ 1 0 469590 0 154.9mb 154.9mb

i don't think i clearly understand how 'fields" work

9

is there a way to configure logstash output to access fields values ?
i want to do something like that :

output {
  if [fields][tags] == "wowza" {
    elasticsearch {
      hosts => ["somepath:9200"]
      index => "logstash-wowza-%{+YYYY.MM.dd}"
    }
  }
  else {
    elasticsearch {
      hosts => ["somepath:9200"]
      index => "logstash-%{type}-%{+YYYY.MM.dd}"
    }
  }
}

my problem is that " if [fields][tags] == "wowza" " don't work and is never reached

EDIT : here is my new filebeat.yml :

filebeat.prospectors:
 -
    paths:
      - /usr/local/WowzaStreamingEngine-4.0.1/logs/wowzastreamingengine_access.log*
    fields:
      application: "wowza"
      tags: ["wowza"]

output:
  logstash:
    hosts: ["somepath:5043"]


logging:
  to_files: true
  level: info
  files:
    path: /var/log/filebeat
    name: filebeat.log
    rotateeverybytes: 10485760
    keepfiles: 7
10

Comment out your elasticsearch outputs and use a stdout { codec => rubydebug } output. What does an example wowza event look like?