Get filename using beats and filter it in logstash

Elastic Stack Logstash
1

Hi,
I have logfile with date as it's name and time inside it. I'm using filebeats to get logs from remote server. Is it possible to combine date with time somehow and put in @timestamp? I tried something like this:
//
filter {
grok {
match => {
"path" => [
"D:/logs/(?%{YEAR})(?%{MONTHNUM})(?%{MONTHDAY}).log"
]
}
}

    mutate {
        #combine them
        add_field => {"timestamp" => "%{year}-%{month}-%{day} %{logtime}"}
           }


    date {
        match => ["timestamp","ISO8601"]
       timezone => "UTC"
        remove_field => ["timestamp"]
          }

    mutate {
        #throw fields  away
        remove_field => ["logtime","year","day","month"]
           }

}
//
It works fine with local file, but not with beats. Any ideas?

1 Like
2

If the file name were in the [source] field I would expect that to work. That suggests that the file name is not in that field. If you are using Kibana, what does an event look like on the JSON tab in Discover?

3

Sorry, it's not "source", I used "path".
When it's local file then everything is fine, when I'm using beats it shows like %{year}-%{month}-%{day} 15:23:56:122

4

OK, so what does the event look like when you are using beats?

5

{
"_index": "srv01",
"_type": "_doc",
"_id": "VtYu5WsBAHAKgvZocFpf",
"_version": 1,
"_score": 1,
"_source": {
"agent": {
"ephemeral_id": "984319d7-95d2-481a-90b2-44fb1ced0214",
"id": "da2fd808-a61c-46f9-b1e5-89d1912ef38c",
"version": "7.1.0",
"hostname": "srv01",
"type": "filebeat"
},
"@timestamp": "2019-07-12T07:55:52.409Z",
"ecs": {
"version": "1.0.0"
},
"host": {
"hostname": "srv01",
"architecture": "x86_64",
"name": "srv01",
"os": {
"name": "Windows Server 2012 R2 Datacenter",
"family": "windows",
"kernel": "6.3.9600.19228 (winblue_ltsb.181208-0600)",
"version": "6.3",
"platform": "windows",
"build": "9600.19235"
},
"id": "d9d9c909-610c-4089-a277-43e5c3e9e82e"
},
"@version": "1",
"log": {
"offset": 914869,
"file": {
"path": "D:\logs\20190104.log"
}
},
"input": {
"type": "log"
},
"severity": "1",
"tags": [
"beats_input_codec_plain_applied",
"_grokparsefailure",
"_dateparsefailure"
],
"message": [
"some message"
],
"timestamp": "%{year}-%{month}-%{day} 00:03:16.551"
},
"fields": {
"@timestamp": [
"2019-07-12T07:55:52.409Z"
]
}
}

6

Problem solved. Correct filter:
grok {
match => {
"[log][file][path]" => [
"(?%{YEAR})(?%{MONTHNUM})(?%{MONTHDAY})"
]
}
}

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.