Some answers needed on the technical what/how of Filebeat

Elastic Stack Beats
1

Hi

As I'm still new on filebeat, can I ask somebody:

Technical what/how ?
• (1) Capture text in log file “error”
• (2) Capture reg.exp in log file “[A-Z]+-[0-9]*/ERR”
• Logical operators : capture ( “pattern-1” || “pattern-2” ) && “pattern-3” && ! “pattern-4”
• Can filebeat “process” a matched log line  “return” parts of matched log line
ORA-[0-9]+
05/03/2016 22:54:12,12 X main()::fucntions | error, Found something wrong Oracle code ORA-01203

and recreate a new “message” / “keyvalue-pairs”

• Enrichment of trapped data with ex. logfile name, host, process, …..  feeding from filebeat configuration (?)

• Bucket concept / counting occurrences ?
React when condition (1) arrived 10 times in 10 minutes

• 1 filebeat per file or 1 filebeat manages mutliple files ?

• Log file “file.log”

  • adding data  trapping
  • remove file and recreate
  • adding data  trapping (does this continue correct)

• - adding data

  • “empty” the file without removing
  • adding data  does trapping continue as expected

Performance ?

Kind regards

Johnny

2

Filebeat doesn't parse log files. With the exception of being able to merge multiline events it ships logs as-is. Look into Logstash for such operations.

3

So, better to use Logstash for all the mentioned questions above, correct?

4

Or a combination of both. Have filebeat on all the edge servers to fetch the log lines and do the processing on a centralised logstash instance.

5

Now I'm forwarding my data to elasticSearch directly, can I choose to forward my data to both elasticSearch & Logstash at the same time?

6

Yes, but why would you want to do that?

7

Because some date I need to use XML filter to get the right data (Logstash) and other data can be send directly to ElasticSearch
I do not have the possibility to use Filebeat.yml config file to filter out the needed data from the XML it seems..

8

Some lines of input can be excluded from processing by Filebeat, see https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#exclude-lines. But it sounds like what you're really asking for is "send to ES or Logstash", not "send to ES and Logstash", and that's not supported.

9

OK, thanks for the information. I will have a look on what's going to be best for me, but I guess forwarding the data to Logstash will provide me more possibilities then doing it via the filebeat.yml config file.

10

This topic was automatically closed after 21 days. New replies are no longer allowed.