# Turning a Filebeat Log into a configured index
**URL:** https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926
**Category:** Beats
**Tags:** filebeat
**Created:** [July 18, 2022, 3:43pm UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926 "2022-07-18T15:43:40Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)
#### Post date: [July 18, 2022, 3:43pm UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926/1 "2022-07-18T15:43:40Z")
So I've been using Filebeat for a while and so far it worked fined, but is getting into the cluster a single message;
I want to have it setted into the cluster as separate fields in a way that I could manipulate it better in the canvas and etc.
Here is the example of Log I'm dealing with:
```auto
2022-07-18 10:47:37,321 2b74cfcc911d7ece [Android] successfully registered device on server
```
At the moment it goes into the index like this:
```auto
message
2022-07-18 10:47:37,321 2b74cfcc911d7ece [Android] successfully registered device on server
```
And I want to have it into something like this:
```auto
2022-07-18 10:47:37,321 -> timestamp
2b74cfcc911d7ece -> device_id
Android -> OS
successfully registered device on server -> message
```
So far I've used many things on the stack but I still learning how to work on this type of problem, so if there is anyone who could help I'll be glad.
Best regards!
---
### Author:  [@sudhagar\_ramesh](https://discuss.elastic.co/u/sudhagar_ramesh)
#### Post date: [July 18, 2022, 4:05pm UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926/2 "2022-07-18T16:05:06Z")
Hello @SamuelSMendes
We can use grok pattern to achieve this
`%{TIME:TIMESTAMP_ISO8601} %{WORD:device_id} \[%{DATA:OS}\] %{GREEDYDATA:message}`
Keep Posted!!! Thanks !!!
---
### Author:  [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)
#### Post date: [July 18, 2022, 4:42pm UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926/3 "2022-07-18T16:42:00Z")
Thanks @sudhagar_ramesh, that's exactly what I wanted!
But is there a way to have it in my filebeat configuration file?
My inputs are simple but if there is a way to have that grok pattern into it would save my day.
Here is the input:
```auto
filebeat.inputs:
- type: filestream
enabled: true
paths:
- /opt/x/xx/log/device_connections.log
index: "x-devices"
```
---
### Author:  [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [July 19, 2022, 10:42am UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926/4 "2022-07-19T10:42:25Z")
I believe that you can achieve it with an ingest pipeline.
---
### Author:  [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)
#### Post date: [July 19, 2022, 1:48pm UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926/5 "2022-07-19T13:48:24Z")
Thanks, it worked just like I expected!
---
### Author:  [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)
#### Post date: [July 19, 2022, 3:11pm UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926/6 "2022-07-19T15:11:57Z")
@sudhagar_ramesh Just one last thing about the grok pattern, right now it is like:
```auto
%{TIME:timestamp} %{NOTSPACE:device_id} %{DATA:OS} %{GREEDYDATA:message}
```
The time, device\_id and OS is working fine -- So is the message.
The output of message in another particular case prints this return:
```auto
"message": [
[
"a:d:l: Notifying connector 'DEVICE DISCONNECTED' [sn=0046750038, version=8.0, model=Moto Z2 Play, manufacture=motorola]"
]
```
So I was wondering if is possible to achieve a grok pattern where 'DEVICE DISCONNECTED' falls into STATUS and 'model' into a MODEL field.
That was the first time I ever see Grok so I'm still quite lost about how to achieve it, been trying a few configurations with what I found on the internet but still couldn't make it work as I described.
---
### Author:  [@sudhagar\_ramesh](https://discuss.elastic.co/u/sudhagar_ramesh)
#### Post date: [July 20, 2022, 2:00am UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926/7 "2022-07-20T02:00:13Z")
Hello @SamuelSMendes
As far as I understood, your logs having both the messages "DEVICE DISCONNECTED" and also "successfully registered device" cases
Hence we have to use grok and kv processors for "DEVICE DISCONNECTED" case in your ingest pipeline, try the below which has both the cases
```auto
grok
{
match => {"message" => ['%{TIME:TIMESTAMP_ISO8601} \[%{WORD:device_id}\] \[%{DATA:OS}\] %{GREEDYDATA:some_msg} \'%{GREEDYDATA:status}\' \[%{GREEDYDATA:device_details}\]','%{TIME:TIMESTAMP_ISO8601} %{WORD:device_id} \[%{DATA:OS}\] %{GREEDYDATA:message}']}
}
kv {
source => "device_details"
field_split_pattern => ","
value_split => "="
}
```
Keep posted !!! Thanks !!!
---
### Author:  [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)
#### Post date: [July 20, 2022, 3:12am UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926/8 "2022-07-20T03:12:27Z")
Thanks for the reply @sudhagar_ramesh, that seems good but I'm not sure about where I do apply it. For that case in particular all I used was Filebeat, and the first grok I made with your help earlier I configured it inside the ingest pipelines by the DevTools in Kibana. Just like this:
```auto
PUT _ingest/pipeline/device_connections_filebeat
{
"description": "A simple example of using Grok",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{TIME:log_time} %{NOTSPACE:device_id} %{DATA:OS} %{GREEDYDATA:message}"
]
}
}
]
}
```
---
### Author:  [@sudhagar\_ramesh](https://discuss.elastic.co/u/sudhagar_ramesh)
#### Post date: [July 20, 2022, 4:23am UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926/9 "2022-07-20T04:23:53Z")
Hello Samuel,
You can try like the below
```auto
PUT _ingest/pipeline/device_connections_filebeat
{
"description": "A simple example of using Grok",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"['%{TIME:TIMESTAMP_ISO8601} \[%{WORD:device_id}\] \[%{DATA:OS}\] %{GREEDYDATA:some_msg} \'%{GREEDYDATA:status}\' \[%{GREEDYDATA:device_details}\]','%{TIME:TIMESTAMP_ISO8601} %{WORD:device_id} \[%{DATA:OS}\] %{GREEDYDATA:message}']"
]
} ,
"kv": {
source => "device_details"
field_split_pattern => ","
value_split => "="
}
}
]
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 17, 2022, 6:24am UTC](https://discuss.elastic.co/t/turning-a-filebeat-log-into-a-configured-index/309926/10 "2022-08-17T06:24:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.