How to setup pipline to extract file using grok in Elastic if the timestamp format is "20Aug21 20:36:07.058931 @fsafsasfdasdsadasd"

Elastic Stack Elasticsearch
1

one of my log ,and the format is like as follows: 20Aug21 20:36:07.058931 @fsafsasfdasdsadasd

I want to extract 20Aug21 20:36:07.058931 to a date type field.

I tried:

PUT /_ingest/pipeline/xxx-log
{
  "processors":[
    {
      "grok":{
                    "field":"message",
                    "patterns":[
                        "%{MY_DATE:timestamp_xyz}"
                    ],
                    "pattern_definitions":{
                        "MY_MONTH":"(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|[Mm](?:a|ä)?r(?:ch|z)?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y|i)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|[Oo](?:c|k)?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)",
                        "MY_DATE":"%{MONTHDAY}%{MY_MONTH}%{YEAR} %{TIME}"
                    },
                    "ignore_failure":true
                }
            },
            {
              "date": {
                "field": "timestamp_xyz",
                "formats": ["ddMMMyyyy HH:mm:ss"],
                "target_field": "@timestamp-syz"
              }
            }
        ]
}

but no @timestamp-syz in elastic.

can you kindly give some suggestions how to setup this pipeline?

for the definition of MY_MONTH
I just refer the
logstash-patterns/grok-patterns at master · hpcugent/logstash-patterns · GitHub

MONTH \b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b

only delete the \b in front and rear

2

Hi @chenx319 Welcome to the community

Assuming your message is

20Aug21 20:36:07.058931 @fsafsasfdasdsadasd

PUT /_ingest/pipeline/discuss-timestamp
{
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{timestamp_xyz} @%{other}"
      }
    },
    {
      "date": {
        "field": "timestamp_xyz",
        "formats": [
          "ddMMMyy HH:mm:ss.SSSSSS"
        ],
        "target_field": "@timestamp-syz"
      }
    }
  ]
} 

POST _ingest/pipeline/discuss-timestamp/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "message": "20Aug21 20:36:07.058931 @fsafsasfdasdsadasd"
      }
    }
  ]
}

Result

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "other" : "fsafsasfdasdsadasd",
          "@timestamp-syz" : "2021-08-20T20:36:07.058Z",
          "message" : "20Aug21 20:36:07.058931 @fsafsasfdasdsadasd",
          "timestamp_xyz" : "20Aug21 20:36:07.058931"
        },
        "_ingest" : {
          "timestamp" : "2022-02-24T04:22:35.4143128Z"
        }
      }
    }
  ]
}
3

@stephenb
Thanks for your reply!

Is this need two processors ?
is it possible only in one processors?

and I setup discuss-timestamp to filebeat,

the timestamp-syz in list in Kibana , but the type is string ,not date ?

how to make it is a date type?

Name Type Format Searchable Aggregatable Excluded
@timestamp-syz string
4

2 processors ... They are very efficient.

You need to define a mapping and set your fields type to a date type... Although it should pick it up as a date automatically.

Did you delete the index and try again??

But in general the safe way is to define and mapping.

5

i delete the index, and try again, the type is still string .

my version is :
elasticsearch-7.9.3-windows-x86_64
filebeat-7.16.3-windows-x86_64
kibana-7.9.3-windows-x86_64

6

this is OK for me

Thanks a lot !

PUT indexcx-7.10.0-ghs-2022.02.24
{
  "mappings": {
    "properties": {
      "@timestamp-syz": {
        "type": "date" 
      }
    }
  }
}
7

@stephenb

Is it possible that I define and mapping a field type in filebeat.yml ?

8

No not really, you would be better off adjusting / adding it to the the _index_template

also I noticed it is not best practice to name the field with the @ symbol that is usually reserved for the common @timestamp field... you can but normal practice.

9

@stephenb Thanks very much for your advice

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.