Differentiate between logs based on log.file.path

Elastic Stack Beats
1

Hi,

I've a server hosting several sites containing multiple IIS logs from each site in structure like this:

D:\IISLogs\Customer1\logs
D:\IISLogs\Customer2\logs
D:\IISLogs\Customer3\logs

I'm using filebeat with IIS module to send the logs to logstash and then output them to elasticsearch. I would like to differentiate between the logs in kibana, preferably by customer name.

I've seen that filebeat adds log.file.path as metadata. Is there anyway to add a field with the customers name based on the log.file.path?

Thanks in advance.

2

Hi,

With filebeat dissect processor you could extract the customer from the log.file.path field and save it to another field.

3

Hey, thanks for your reply. I did check out the filebeat documentation for dissect, however I'm still not sure how it functions.

I've posted the following in my filebeat.yml configuration file under Processors:

processors:

  • dissect:
    tokenizer: "%{Key1} - %{Key2}"
    field: "log.file.path"
    target_prefix: "NewFieldCustomerName"

But i'm not sure how I should set up the tokenizer pattern to match the customer name in my file structure, any advice would be greatly appreciated.

Thanks in advance.

4

Hi,

I agree the filebeat documentation for the dissect processor is a bit thin.
I can recommend reading the documentation for the logstash dissect filter plugin which explains in more details how it works and how it can be used to extract information from fields/strings.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html

There is also a blog post here:

I'm not saying that filebeat implementation is exactly the same as logstash though.
I don't know.
You should open an issue on github about the documentation for the filebeat dissect processor because it is indeed lacking.
If the implementation is the same, it could include a link to the logstash dissect. It could also mentions the differences if there are any.

5

Your tokenizer should probably be something like
"%{}\%{}\%{customername}\%{}\%{}"

to match a log.file.path value of D:\IISLogs\Customer1\logs\logfile.txt and extract Customer1 into customername

6

Hello,

Thank you for you reply.

I've added the following to my filebeat configuraiton under processors:

processors:

  • add_host_metadata: ~
  • add_cloud_metadata: ~

processors:
- dissect:
tokenizer: "%{}\{customername}\%{}"
field: "log.file.path"
target_prefix: "Customer"

For testing purposes I've a log path which looks like the following: C:\customer\u_ex160121.log
I had to use double \ to esacpe the backslash otherwise filebeat wont start.

I run logging.level: debug but I can't see any trace that any dissection is happening at all.

Thanks yet again, very helpful.

7

Hi, thanks for you reply. I accidentally replied to my own thread instead of your response.

8

You are missing the percentage in front of the {customername}, it should look like below:

tokenizer: "%{}\%{customername}\%{}"

Every field you want to recognise should be of the form %{}

1 Like
9

Wow, my bad. Thank you.

It works like a charm. Now I get a field called "Customer" with the actual customer name as the value.

Cheers!

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.