Actual Timestamp should be filtered from message context

Elastic Stack Logstash
1

Dear Team,

Kindly help me with this how to do

{
"_index": "env-2020.03.09",
"_type": "_doc",
"_id": "dsfhdsj2e630",
"_version": 1,
"_score": null,
"_source": {
"type": "Monitor Logs",
"host": "efjf4323",
"message": "2020-03-08 08:55:12,415 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] ",
"@timestamp": "2020-03-09T10:28:56.460Z",
"@version": "1"

},
"fields": {
"@timestamp": [
"2020-03-09T10:28:56.460Z"
]
},
"sort": [
1583749736460
]
}

2

Hi

If you want to capture the date (or any other field) from your message, you'll need to parse your message. You can use the grok{} filter.

Could you please post

  • your pipeline code,
  • a sample of your input
  • and a sample of your expected output

so we can better help you?

Please, use the "</>" tool to format your code, samples, etc.

Thank you

3

Sure @ITIC . Kindly find Below sample json format

{
"_index": "env-2020.03.09",
"_type": "_doc",
"_id": "dsfhdsj2e630",
"_version": 1,
"_score": null,
"_source": {
"type": "Monitor Logs",
"host": "efjf4323",
"message": "2020-03-08 08:55:12,415 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] ",
"@timestamp": "2020-03-09T10:28:56.460Z",
"@version": "1"

},
"fields": {
"@timestamp": [
"2020-03-09T10:28:56.460Z"
]
},
"sort": [
1583749736460
]
}

Thanks for the Support !

4

From the Above JSON format , need to extract Timestamp from Message and replace it with actual
@timestamp in logstash config file

5

Great, thank you.

I´m not sure I understand what you need to do. You need to overwrite a field you extract from your message with logstash's @timestamp? Why not use the @timestamp field itself where needed, instead? I´m afraid I must have misunderstood you.

Now, could you please post

  • your pipeline code,
  • and a sample of your expected output

so we can better help you?

Please, use the "</>" tool to format your code, samples, etc.

Thank you

6

Welcome !

My Input Data which I am receiving is
{
"_index": "env-2020.03.09",
"_type": "_doc",
"_id": "dsfhdsj2e630",
"_version": 1,
"_score": null,
"_source": {
"type": "Monitor Logs",
"host": "efjf4323",
"message": "2020-03-08 08:55:12,415 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] ",
"@timestamp": "2020-03-09T10:28:56.460Z",
"@version": "1"

},
"fields": {
"@timestamp": [
"2020-03-09T10:28:56.460Z"
]
},
"sort": [
1583749736460
]
}

Output Should be Like this
{
"_index": "env-2020.03.09",
"_type": "_doc",
"_id": "dsfhdsj2e630",
"_version": 1,
"_score": null,
"_source": {
"type": "Monitor Logs",
"host": "efjf4323",
"message": "2020-03-08 08:55:12,415 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] ",
"@timestamp": "2020-03-09T10:28:56.460Z",
"@version": "1"

},
"fields": {
"@timestamp": [
"2020-03-08 08:55:12"
]
},
"sort": [
1583749736460
]
}

Kindly check Words with Bold , Easily you can identify.

Cheers,

7

Hi

I get it now, thank you.

To extract the date from your message field you can use a grok{} filter. Info on this filter can be found here: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html and some predefined parameters are here: https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

The next question is why you want to replace logstash's @timestamp. It is not recommended (don't know exactly why, I think it has to do with the relationship with the other elements of the ELK stack) and instead you should have a new field with your time. You'll get this field directly out of the grok{} filter I suggested above.

Hope this helps get you started.

8

Sure , i will do and can you say how to create new field(Time) and assign to extracted timestamp from message body context . if you write sample lines it really helps me

9

HI

Give it a try or five. We will be happy to help you improve your code.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.