Harvest + Analyze customers application logs

Elastic Stack Beats
1

Hi
I am new with Elastic and Filebeat.
Decided to test harvesting and analyze customers application logs.

  • Installed (Windows) Elastic + Filebeat + Kibana.
  • Configured Filebeat fields.yml to the customized logs fields source:

-- Log row example:

0614_15:00:00.831, "SYSTEM pre-ordering starting, lastRun: <20190613>", [INFO], T@105918, T:ActiveMgr.105918, , com.mm.PreMN, PreMN::orderSystemIntoShadow, ,

-- Fields.yml:

"date":"id": "date"
"error.message":"id": "string"
"process.title":"id": "string"
"event.action":"id": "string"
"process.program":"id": "string"
"event.type":"id": "string"
"event.hash":"id": "string"

Yet, Kibana has created the filebeat new index with the default fields.

Please assist. Thanks!

2

Could you please share your debug logs (output of ./filebeat -e -d "*") and your configuration formatted using </>?

To me it seems that the format of your fields.yml file is incorrect. This is the fields.yml of HAProxy module in Filebeat: https://github.com/elastic/beats/blob/master/filebeat/module/haproxy/_meta/fields.yml You need something like that.

3

Hi Noémi,
thanks for responding to my query.

  • The filebeat debugged output does show that most of the input files didn't change, yet it does not show on Kibana as supposed.

  • Filebeat keeps harvesting the Filebeat+Kibana log files.

  • My fields.yml should be quite short, I just couldn't figure out its structure.
    Can you say how to add just the requested fields to this example load:
    "date":"id": "date"
    "error.message":"id": "string"
    "process.title":"id": "string"
    "event.action":"id": "string"
    "process.program":"id": "string"
    "event.type":"id": "string"
    "event.hash":"id": "string"

Thanks.

4

Do you want to add these fields to the existing fields, or add only these fields without the fields provided by Filebeat?

5

For now, I need to test only those fields.

Next, will need to add fields per log file type.

Wish Filebeat could analyze such log files pattern.

Thanks.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.