How to make the changes in filebeat effective

Elastic Stack Beats
#1

Hello all,
I am trying to use the filebeat.yml file for the first time. Since, the logs are being logged in a different country and sometimes I see an abrupt jump in the logs visibility. It might be (not sure) because previously those services were not online for the logs to ES clusters. Anyways, I want to get only the date and time and display that as a field in kibana logs.
For the solution, I am trying the script processors as follows:

- script:
          lang: javascript
          id: my_filter
          tag: enable
          source: >
            function process(event) {
                var str = event.Get("event.message");
                var time = str.split(" ").slice(1,3).join("@");
                event.Put("event.start_time",time);
            }
      - timestamp:
       #format the start_time layouts then assign to @timestamp
          field: start_time
          layouts:
            -'2006-01-02@15:04:05.999'

But in Kibana, I am not seeing any field which is "start_time"

Further, It took me a lot of time to write this script and combat with for correct syntax. Since, I am going to write lot of if else and slicing them based on condition, Any easier way to do this?

#2

Can you elaborate a little more on your problem, providing examples would be helpful in understanding what you are seeing and what result you want.

#3

Thanks for your reply.

So, message in kibana shows [jgsdlkfgasgfy2346-asdsdhfbjsd] [2021-01-05 10:48:36.974] [INFO]..... I am trying to parse this message and get [2021-01-05 10:48:36.974] as a separate field (i.e. start_time) if possible.

Based on my limited understanding, I am using script processors which is posted above but I am not successful. I hope this helps.

#4

Ok. And are you referring to https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html when you say script processor?

If so, you probably want to use https://www.elastic.co/guide/en/elasticsearch/reference/current/dissect-processor.html or https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html rather than that one.

#5

I am referring to https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html

#6

Ahh ok, you posted this in Elasticsearch, hence my links :slight_smile: Let me move it.

#7

In that case look at https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html instead. But ultimately you may want to use an ingest pipeline (like what I linked to above) to split things out.

#8

Also, one more thing that I noticed was that,
in message : [hsjdfjsdgf-23hjsbdf-2346dsad] [2020-11-25 22:31:16.795] [INFO] but @timestamp is Dec 16, 2020. Hence, the counts of the service hits is very different. This is the main issue I am trying to solve which I do not think can be solved by just parsing.

#9

Was Dec the 16th when it was ingested? If so then you will need to use grok/dissect + date ingest processors to properly process your message.

#10

can you please share an example based on my log above?

#11

There's good examples in each of the various links, it depends on what you want to use.

For https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html it'd be something like;

processors:
  - dissect:
      tokenizer: "%{sometext} %{timestamp} %{loglevel}"
      field: "message"

Then add extra fields as you need to extract them into their own fields.

#12 
processors:
  - dissect:
      tokenizer: "%{sometext} %{event.timezone} %{loglevel}"
      field: "message"
  1. still doesn't show event.timezone in fields even though it is ECS searchable field.
  2. Also, I have multiple logs so can I have multiple tokenizer for parsing the dates?
#13
  1. Not sure what you mean there.
  2. Yes.
#14
  1. event.timezone is ECS field. So I think when I put the above code in the filebeat.yml, I should be able to see the correct time in event.timezone field. However, I do not see that in kibana logs.
#15

I would suggest just keeping it simple to start so you get a hang of how it all works.
Get the event processed and structured, then look at using ECS.

#17

shouldn't dissect processor just tokenize the message? How will that help in correcting @timestamp?

#18

You then need to use something like https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html to map it correctly.

#21 
 - if:
     equals.log.file.path: "<filepath>"
    then:
     - dissect:
         tokenizer: '[%{}] [%{start_time}] [%{}] [%{}] %{}'
         field: "message"
    else:
  - timestamp:
   #format the start_time layouts then assign to @timestamp
      field: start_time
      timezone: Europe
      layouts:
        -'2006-01-02 15:04:05.999'

I used the dissect processor and timestamp processor. I checked the date where the data/logs was ingested i.e. Dec 16. It still has the same issue though!

#22

It'd be useful if you showed the original event and the output you are seeing.

#23

message: [sometext] [2020-11-30 02:10:17.736] [loglevel] [something] text1 - text2
But the @timestamp Dec 16, 2020 @ 18:59:59.715 Since it was recently ingested. Hence, my idea is to parse the message, get the start_time. Using the timezone and parsed value, pass it toas timestamp. Does it make sense now?