# How to make the changes in filebeat effective **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 5, 2021, 1:46am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143 "2021-01-05T01:46:50Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![aagarwal3](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@aagarwal3](https://discuss.elastic.co/u/aagarwal3)\ **Post date:** [January 5, 2021, 1:46am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/1 "2021-01-05T01:46:51Z")
Hello all, I am trying to use the filebeat.yml file for the first time. Since, the logs are being logged in a different country and sometimes I see an abrupt jump in the logs visibility. It might be (not sure) because previously those services were not online for the logs to ES clusters. Anyways, I want to get only the date and time and display that as a field in kibana logs. For the solution, I am trying the script processors as follows: ``` - script: lang: javascript id: my_filter tag: enable source: > function process(event) { var str = event.Get("event.message"); var time = str.split(" ").slice(1,3).join("@"); event.Put("event.start_time",time); } - timestamp: #format the start_time layouts then assign to @timestamp field: start_time layouts: -'2006-01-02@15:04:05.999' ``` But in Kibana, I am not seeing any field which is "start\_time" Further, It took me a lot of time to write this script and combat with for correct syntax. Since, I am going to write lot of `if else` and slicing them based on condition, Any easier way to do this? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2021, 1:50am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/2 "2021-01-05T01:50:16Z")
Can you elaborate a little more on your problem, providing examples would be helpful in understanding what you are seeing and what result you want. ---
**Author:** ![aagarwal3](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@aagarwal3](https://discuss.elastic.co/u/aagarwal3)\ **Post date:** [January 5, 2021, 2:56am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/3 "2021-01-05T02:56:05Z")
Thanks for your reply. So, `message` in kibana shows `[jgsdlkfgasgfy2346-asdsdhfbjsd] [2021-01-05 10:48:36.974] [INFO]....`. I am trying to parse this `message` and get `[2021-01-05 10:48:36.974] ` as a separate field (i.e. `start_time`) if possible. Based on my limited understanding, I am using script processors which is posted above but I am not successful. I hope this helps. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2021, 3:01am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/4 "2021-01-05T03:01:35Z")
Ok. And are you referring to [https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html) when you say script processor? If so, you probably want to use [https://www.elastic.co/guide/en/elasticsearch/reference/current/dissect-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/dissect-processor.html) or [https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html) rather than that one. ---
**Author:** ![aagarwal3](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@aagarwal3](https://discuss.elastic.co/u/aagarwal3)\ **Post date:** [January 5, 2021, 3:04am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/5 "2021-01-05T03:04:04Z")
I am referring to [https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html](https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2021, 3:04am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/6 "2021-01-05T03:04:34Z")
Ahh ok, you posted this in Elasticsearch, hence my links 🙂 Let me move it. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2021, 3:05am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/7 "2021-01-05T03:05:43Z")
In that case look at [https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html](https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html) instead. But ultimately you may want to use an ingest pipeline (like what I linked to above) to split things out. ---
**Author:** ![aagarwal3](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@aagarwal3](https://discuss.elastic.co/u/aagarwal3)\ **Post date:** [January 5, 2021, 3:18am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/8 "2021-01-05T03:18:50Z")
Also, one more thing that I noticed was that, in `message` : `[hsjdfjsdgf-23hjsbdf-2346dsad] [2020-11-25 22:31:16.795] [INFO]` but @timestamp is `Dec 16, 2020`. Hence, the counts of the service hits is very different. This is the main issue I am trying to solve which I do not think can be solved by just parsing. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2021, 3:19am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/9 "2021-01-05T03:19:39Z")
Was Dec the 16th when it was ingested? If so then you will need to use grok/dissect + date ingest processors to properly process your message. ---
**Author:** ![aagarwal3](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@aagarwal3](https://discuss.elastic.co/u/aagarwal3)\ **Post date:** [January 5, 2021, 3:22am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/10 "2021-01-05T03:22:00Z")
can you please share an example based on my log above? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2021, 3:25am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/11 "2021-01-05T03:25:35Z")
There's good examples in each of the various links, it depends on what you want to use. For [https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html](https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html) it'd be something like; ```auto processors: - dissect: tokenizer: "%{sometext} %{timestamp} %{loglevel}" field: "message" ``` Then add extra fields as you need to extract them into their own fields. ---
**Author:** ![aagarwal3](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@aagarwal3](https://discuss.elastic.co/u/aagarwal3)\ **Post date:** [January 5, 2021, 4:03am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/12 "2021-01-05T04:03:29Z")
``` processors: - dissect: tokenizer: "%{sometext} %{event.timezone} %{loglevel}" field: "message" ``` 1. still doesn't show `event.timezone` in fields even though it is ECS searchable field. 2. Also, I have multiple logs so can I have multiple tokenizer for parsing the dates? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2021, 4:18am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/13 "2021-01-05T04:18:25Z")
1. Not sure what you mean there. 2. Yes. ---
**Author:** ![aagarwal3](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@aagarwal3](https://discuss.elastic.co/u/aagarwal3)\ **Post date:** [January 5, 2021, 4:19am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/14 "2021-01-05T04:19:59Z")
1. event.timezone is ECS field. So I think when I put the above code in the filebeat.yml, I should be able to see the correct time in event.timezone field. However, I do not see that in kibana logs. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2021, 4:22am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/15 "2021-01-05T04:22:19Z")
I would suggest just keeping it simple to start so you get a hang of how it all works. Get the event processed and structured, then look at using ECS. ---
**Author:** ![aagarwal3](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@aagarwal3](https://discuss.elastic.co/u/aagarwal3)\ **Post date:** [January 5, 2021, 5:19am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/17 "2021-01-05T05:19:15Z")
shouldn't dissect processor just tokenize the message? How will that help in correcting @timestamp? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2021, 6:07am UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/18 "2021-01-05T06:07:04Z")
You then need to use something like [https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html) to map it correctly. ---
**Author:** ![aagarwal3](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@aagarwal3](https://discuss.elastic.co/u/aagarwal3)\ **Post date:** [January 5, 2021, 8:38pm UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/21 "2021-01-05T20:38:40Z")
```auto - if: equals.log.file.path: "" then: - dissect: tokenizer: '[%{}] [%{start_time}] [%{}] [%{}] %{}' field: "message" else: - timestamp: #format the start_time layouts then assign to @timestamp field: start_time timezone: Europe layouts: -'2006-01-02 15:04:05.999' ``` I used the dissect processor and timestamp processor. I checked the date where the data/logs was ingested i.e. Dec 16. It still has the same issue though! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2021, 8:40pm UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/22 "2021-01-05T20:40:09Z")
It'd be useful if you showed the original event and the output you are seeing. ---
**Author:** ![aagarwal3](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@aagarwal3](https://discuss.elastic.co/u/aagarwal3)\ **Post date:** [January 5, 2021, 8:46pm UTC](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143/23 "2021-01-05T20:46:45Z")
`message: [sometext] [2020-11-30 02:10:17.736] [loglevel] [something] text1 - text2` But the `@timestamp Dec 16, 2020 @ 18:59:59.715` Since it was recently ingested. Hence, my idea is to parse the `message`, get the `start_time`. Using the timezone and parsed value, pass it toas timestamp. Does it make sense now? [Next page](https://discuss.elastic.co/t/how-to-make-the-changes-in-filebeat-effective/260143.md?page=2)