Turning a Filebeat Log into a configured index

[SamuelSMendes (Samuel S. Mendes Soares)]

So I've been using Filebeat for a while and so far it worked fined, but is getting into the cluster a single message;

I want to have it setted into the cluster as separate fields in a way that I could manipulate it better in the canvas and etc.

Here is the example of Log I'm dealing with:

2022-07-18 10:47:37,321 2b74cfcc911d7ece [Android] successfully registered device on server

At the moment it goes into the index like this:

message
2022-07-18 10:47:37,321 2b74cfcc911d7ece [Android] successfully registered device on server 

And I want to have it into something like this:

2022-07-18 10:47:37,321 -> timestamp

2b74cfcc911d7ece -> device_id

Android -> OS

successfully registered device on server -> message

So far I've used many things on the stack but I still learning how to work on this type of problem, so if there is anyone who could help I'll be glad.

Best regards!

[sudhagar_ramesh]

Hello @SamuelSMendes

We can use grok pattern to achieve this

%{TIME:TIMESTAMP_ISO8601} %{WORD:device_id} \[%{DATA:OS}\] %{GREEDYDATA:message}

Keep Posted!!! Thanks !!!

[SamuelSMendes (Samuel S. Mendes Soares)]

Thanks @sudhagar_ramesh, that's exactly what I wanted!

But is there a way to have it in my filebeat configuration file?

My inputs are simple but if there is a way to have that grok pattern into it would save my day.

Here is the input:

filebeat.inputs:

- type: filestream
  enabled: true
  paths:
    - /opt/x/xx/log/device_connections.log
  index: "x-devices"

[mtojek (Marcin Tojek)]

I believe that you can achieve it with an ingest pipeline.

[SamuelSMendes (Samuel S. Mendes Soares)]

Thanks, it worked just like I expected!

[SamuelSMendes (Samuel S. Mendes Soares)]

@sudhagar_ramesh Just one last thing about the grok pattern, right now it is like:

%{TIME:timestamp} %{NOTSPACE:device_id} %{DATA:OS} %{GREEDYDATA:message}

The time, device_id and OS is working fine -- So is the message.

The output of message in another particular case prints this return:

  "message": [
    [
      "a:d:l: Notifying connector 'DEVICE DISCONNECTED' [sn=0046750038, version=8.0, model=Moto Z2 Play, manufacture=motorola]"
    ]

So I was wondering if is possible to achieve a grok pattern where 'DEVICE DISCONNECTED' falls into STATUS and 'model' into a MODEL field.

That was the first time I ever see Grok so I'm still quite lost about how to achieve it, been trying a few configurations with what I found on the internet but still couldn't make it work as I described.

[sudhagar_ramesh]

Hello @SamuelSMendes

As far as I understood, your logs having both the messages "DEVICE DISCONNECTED" and also "successfully registered device" cases

Hence we have to use grok and kv processors for "DEVICE DISCONNECTED" case in your ingest pipeline, try the below which has both the cases

grok
{
match => {"message" => ['%{TIME:TIMESTAMP_ISO8601} \[%{WORD:device_id}\] \[%{DATA:OS}\] %{GREEDYDATA:some_msg} \'%{GREEDYDATA:status}\' \[%{GREEDYDATA:device_details}\]','%{TIME:TIMESTAMP_ISO8601} %{WORD:device_id} \[%{DATA:OS}\] %{GREEDYDATA:message}']}
}

kv {
       source => "device_details"
       field_split_pattern => ","
       value_split => "="
   }

Keep posted !!! Thanks !!!

[SamuelSMendes (Samuel S. Mendes Soares)]

Thanks for the reply @sudhagar_ramesh, that seems good but I'm not sure about where I do apply it. For that case in particular all I used was Filebeat, and the first grok I made with your help earlier I configured it inside the ingest pipelines by the DevTools in Kibana. Just like this:

PUT _ingest/pipeline/device_connections_filebeat 
{ 
  "description": "A simple example of using Grok", 
  "processors": [ 
    { 
      "grok": { 
        "field": "message", 
        "patterns": [ 
          "%{TIME:log_time} %{NOTSPACE:device_id} %{DATA:OS} %{GREEDYDATA:message}" 
        ] 
      } 
    } 
  ] 
} 

[sudhagar_ramesh]

Hello Samuel,

You can try like the below

PUT _ingest/pipeline/device_connections_filebeat 
{ 
  "description": "A simple example of using Grok", 
  "processors": [ 
    { 
      "grok": { 
        "field": "message", 
        "patterns": [ 
          "['%{TIME:TIMESTAMP_ISO8601} \[%{WORD:device_id}\] \[%{DATA:OS}\] %{GREEDYDATA:some_msg} \'%{GREEDYDATA:status}\' \[%{GREEDYDATA:device_details}\]','%{TIME:TIMESTAMP_ISO8601} %{WORD:device_id} \[%{DATA:OS}\] %{GREEDYDATA:message}']" 
        ] 
      } ,
      "kv": {
       source => "device_details"
       field_split_pattern => ","
       value_split => "="
       }
    } 
  ] 
}