Some answers needed on the technical what/how of Filebeat


(Johnny Soetaerts) #1

Hi

As I'm still new on filebeat, can I ask somebody:

Technical what/how ?
• (1) Capture text in log file “error”
• (2) Capture reg.exp in log file “[A-Z]+-[0-9]*/ERR”
• Logical operators : capture ( “pattern-1” || “pattern-2” ) && “pattern-3” && ! “pattern-4”
• Can filebeat “process” a matched log line  “return” parts of matched log line
ORA-[0-9]+
05/03/2016 22:54:12,12 X main()::fucntions | error, Found something wrong Oracle code ORA-01203

and recreate a new “message” / “keyvalue-pairs”

• Enrichment of trapped data with ex. logfile name, host, process, …..  feeding from filebeat configuration (?)

• Bucket concept / counting occurrences ?
React when condition (1) arrived 10 times in 10 minutes

• 1 filebeat per file or 1 filebeat manages mutliple files ?

• Log file “file.log”

  • adding data  trapping
  • remove file and recreate
  • adding data  trapping (does this continue correct)

• - adding data

  • “empty” the file without removing
  • adding data  does trapping continue as expected

Performance ?

Kind regards

Johnny


(Magnus Bäck) #2

Filebeat doesn't parse log files. With the exception of being able to merge multiline events it ships logs as-is. Look into Logstash for such operations.


(Johnny Soetaerts) #3

So, better to use Logstash for all the mentioned questions above, correct?


(ruflin) #4

Or a combination of both. Have filebeat on all the edge servers to fetch the log lines and do the processing on a centralised logstash instance.


(Johnny Soetaerts) #5

Now I'm forwarding my data to elasticSearch directly, can I choose to forward my data to both elasticSearch & Logstash at the same time?


(Magnus Bäck) #6

Yes, but why would you want to do that?


(Johnny Soetaerts) #7

Because some date I need to use XML filter to get the right data (Logstash) and other data can be send directly to ElasticSearch
I do not have the possibility to use Filebeat.yml config file to filter out the needed data from the XML it seems..


(Magnus Bäck) #8

Some lines of input can be excluded from processing by Filebeat, see https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#exclude-lines. But it sounds like what you're really asking for is "send to ES or Logstash", not "send to ES and Logstash", and that's not supported.


(Johnny Soetaerts) #9

OK, thanks for the information. I will have a look on what's going to be best for me, but I guess forwarding the data to Logstash will provide me more possibilities then doing it via the filebeat.yml config file.


(system) #10

This topic was automatically closed after 21 days. New replies are no longer allowed.