How to make the changes in filebeat effective

Elastic Stack Beats
1

Hello all,
I am trying to use the filebeat.yml file for the first time. Since, the logs are being logged in a different country and sometimes I see an abrupt jump in the logs visibility. It might be (not sure) because previously those services were not online for the logs to ES clusters. Anyways, I want to get only the date and time and display that as a field in kibana logs.
For the solution, I am trying the script processors as follows:

- script:
          lang: javascript
          id: my_filter
          tag: enable
          source: >
            function process(event) {
                var str = event.Get("event.message");
                var time = str.split(" ").slice(1,3).join("@");
                event.Put("event.start_time",time);
            }
      - timestamp:
       #format the start_time layouts then assign to @timestamp
          field: start_time
          layouts:
            -'2006-01-02@15:04:05.999'

But in Kibana, I am not seeing any field which is "start_time"

Further, It took me a lot of time to write this script and combat with for correct syntax. Since, I am going to write lot of if else and slicing them based on condition, Any easier way to do this?

2

Can you elaborate a little more on your problem, providing examples would be helpful in understanding what you are seeing and what result you want.

3

Thanks for your reply.

So, message in kibana shows [jgsdlkfgasgfy2346-asdsdhfbjsd] [2021-01-05 10:48:36.974] [INFO]..... I am trying to parse this message and get [2021-01-05 10:48:36.974] as a separate field (i.e. start_time) if possible.

Based on my limited understanding, I am using script processors which is posted above but I am not successful. I hope this helps.

4

Ok. And are you referring to https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html when you say script processor?

If so, you probably want to use https://www.elastic.co/guide/en/elasticsearch/reference/current/dissect-processor.html or https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html rather than that one.

5

I am referring to https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html

6

Ahh ok, you posted this in Elasticsearch, hence my links :slight_smile: Let me move it.

7

In that case look at https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html instead. But ultimately you may want to use an ingest pipeline (like what I linked to above) to split things out.

8

Also, one more thing that I noticed was that,
in message : [hsjdfjsdgf-23hjsbdf-2346dsad] [2020-11-25 22:31:16.795] [INFO] but @timestamp is Dec 16, 2020. Hence, the counts of the service hits is very different. This is the main issue I am trying to solve which I do not think can be solved by just parsing.

9

Was Dec the 16th when it was ingested? If so then you will need to use grok/dissect + date ingest processors to properly process your message.

10

can you please share an example based on my log above?

11

There's good examples in each of the various links, it depends on what you want to use.

For https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html it'd be something like;

processors:
  - dissect:
      tokenizer: "%{sometext} %{timestamp} %{loglevel}"
      field: "message"

Then add extra fields as you need to extract them into their own fields.

12 
processors:
  - dissect:
      tokenizer: "%{sometext} %{event.timezone} %{loglevel}"
      field: "message"
  1. still doesn't show event.timezone in fields even though it is ECS searchable field.
  2. Also, I have multiple logs so can I have multiple tokenizer for parsing the dates?
13
  1. Not sure what you mean there.
  2. Yes.
14
  1. event.timezone is ECS field. So I think when I put the above code in the filebeat.yml, I should be able to see the correct time in event.timezone field. However, I do not see that in kibana logs.
15

I would suggest just keeping it simple to start so you get a hang of how it all works.
Get the event processed and structured, then look at using ECS.

17

shouldn't dissect processor just tokenize the message? How will that help in correcting @timestamp?

18

You then need to use something like https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html to map it correctly.

21 
 - if:
     equals.log.file.path: "<filepath>"
    then:
     - dissect:
         tokenizer: '[%{}] [%{start_time}] [%{}] [%{}] %{}'
         field: "message"
    else:
  - timestamp:
   #format the start_time layouts then assign to @timestamp
      field: start_time
      timezone: Europe
      layouts:
        -'2006-01-02 15:04:05.999'

I used the dissect processor and timestamp processor. I checked the date where the data/logs was ingested i.e. Dec 16. It still has the same issue though!

22

It'd be useful if you showed the original event and the output you are seeing.

23

message: [sometext] [2020-11-30 02:10:17.736] [loglevel] [something] text1 - text2
But the @timestamp Dec 16, 2020 @ 18:59:59.715 Since it was recently ingested. Hence, my idea is to parse the message, get the start_time. Using the timezone and parsed value, pass it toas timestamp. Does it make sense now?