# Logstash unable to parse specific format of log
**URL:** https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815
**Category:** Logstash
**Created:** [May 31, 2023, 2:19pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815 "2023-05-31T14:19:31Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@SmoZyNS](https://discuss.elastic.co/u/SmoZyNS)
#### Post date: [May 31, 2023, 2:19pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/1 "2023-05-31T14:19:31Z")
Hello
I am looking for some help since getting some headaches when trying to parse some logs
Raw logs
```auto
cs1Label=username cs1=/test@test.com cn1Label=actionSuccess cn1=1 deviceCustomDate1Label=userActionTime deviceCustomDate1=May 31 2023, 06:42:18 UTC cs2Label=malopID cs2= cs3Label=linkToMalop cs3= cs4Label=actionType cs4=Add cn2Label=affectedIOCCount cn2=1\n
```
What I would like at the end is having this format:
```auto
{
"username" => "test@test.com",
"actionSuccess" => "1",
"userActionTime" => "May 31 2023, 06:42:18 UTC",
"malopID" => "",
"linkToMalop" => "",
"actionType" => "Add",
"affectedIOCCount" => "1"
}
```
Since the field name is the n-1 value and the field data is the n value, I have to use gsub with Kv filter but I don't find a way to achieve it.
Some help to put me on the right track would be very appreciated.
Best regards
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [May 31, 2023, 2:27pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/2 "2023-05-31T14:27:39Z")
> [@SmoZyNS](#):
>
> I am looking for some help since getting some headaches when trying to parse some logs
What is your input? Please share the entire configuration pipeline.
> [@SmoZyNS](#):
>
> `cs1Label=username cs1=/test@test.com cn1Label=actionSuccess cn1=1 deviceCustomDate1Label=userActionTime deviceCustomDate1=May 31 2023, 06:42:18 UTC cs2Label=malopID cs2= cs3Label=linkToMalop cs3= cs4Label=actionType cs4=Add cn2Label=affectedIOCCount cn2=1\n`
This is the entire log or you have anything else? This is a CEF message, the CEF codec would be able to parse this easily.
---
### Author:  [@SmoZyNS](https://discuss.elastic.co/u/SmoZyNS)
#### Post date: [May 31, 2023, 2:48pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/3 "2023-05-31T14:48:11Z")
Hello @leandrojmp , thank you for the reply.
Yes it is indeed CEF
Here is the entire raw logs:
```auto
<150>May 31 06:42:18 cyber-420 auditSyslogLogger CEF:0|Cyber|Cyber||UserAction|Malop Investigation/ManualCustomReputations|0|cs1Label=username cs1=/test@test.com cn1Label=actionSuccess cn1=1 deviceCustomDate1Label=userActionTime deviceCustomDate1=May 31 2023, 06:42:18 UTC cs2Label=malopID cs2= cs3Label=linkToMalop cs3= cs4Label=actionType cs4=Add cn2Label=affectedIOCCount cn2=1\n
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [May 31, 2023, 3:02pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/4 "2023-05-31T15:02:09Z")
> [@SmoZyNS](#):
>
> Yes it is indeed CEF
And what is your configuration? Are you using the CEF codec?
---
### Author:  [@SmoZyNS](https://discuss.elastic.co/u/SmoZyNS)
#### Post date: [May 31, 2023, 3:21pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/5 "2023-05-31T15:21:45Z")
I currently don't have anything in the filter since I don't really know how to start to parse these label and data fields.
Here is my pipeline, nothing fancy atm, and I didn't use the cef codec
```auto
input {
syslog {
port => 5020
type => "edr"
}
}
filter {
if [type] == "edr" {
}
}
output {
if [type] == "edr" {
file {
path => "C:/edr.txt"
}
}
}
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [May 31, 2023, 3:55pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/6 "2023-05-31T15:55:30Z")
Add a CEF codec in your input and see if this help you with the parse.
```auto
input {
syslog {
port => 5020
type => "edr"
codec => "cef"
}
}
```
---
### Author:  [@SmoZyNS](https://discuss.elastic.co/u/SmoZyNS)
#### Post date: [May 31, 2023, 3:58pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/7 "2023-05-31T15:58:29Z")
@leandrojmp I edited the input with the cef codec plugin which managed to format the logs this way for the CEF part:
```auto
"cef": {
"device_custom_string_6": {
"value": "test@test.com",
"label": "username"
},
"device_custom_string_1": {
"value": "",
"label": "malopID"
},
"device_custom_string_2": {
"value": "",
"label": "linkToMalop"
},
"device_custom_string_4": {
"value": "Add",
"label": "actionType"
}
}
....
```
Is there any way to have something like this instead ?
```auto
"username": "test@test.com",
"malopID": "",
"linkToMalop": "",
"actionType": "Add"
```
Thank you in advance
---
### Author:  [@SmoZyNS](https://discuss.elastic.co/u/SmoZyNS)
#### Post date: [June 1, 2023, 9:21am UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/8 "2023-06-01T09:21:46Z")
@leandrojmp maybe writing a ruby script could help ?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [June 1, 2023, 1:11pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/9 "2023-06-01T13:11:01Z")
> [@SmoZyNS](#):
>
> maybe writing a ruby script could help ?
If you have just a couple of fields you could solve this using a mutate filter.
Something like this
```auto
mutate {
rename => {
"[cef][device_custom_string_1][value]" => "malopID"
"[cef][device_custom_string_2][value]" => "linkToMalop"
"[cef][device_custom_string_4][value]" => "actionType"
"[cef][device_custom_string_6][value]" => "username"
}
}
```
But if you have a lot of those fields or they may change, a ruby script would work.
Not sure how the ruby code would look like, but if you search in the forum you will find some examples on how to do this.
---
### Author:  [@SmoZyNS](https://discuss.elastic.co/u/SmoZyNS)
#### Post date: [June 1, 2023, 1:16pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/10 "2023-06-01T13:16:32Z")
> [@leandrojmp](#):
>
> > [@SmoZyNS](#):
> >
> > maybe writing a ruby script could help ?
>
> If you have just a couple of fields you could solve this using a mutate filter.
>
> Something like this
>
> ```auto
> mutate {
> rename => {
> "[cef][device_custom_string_1][value]" => "malopID"
> "[cef][device_custom_string_2][value]" => "linkToMalop"
> "[cef][device_custom_string_4][value]" => "actionType"
> "[cef][device_custom_string_6][value]" => "username"
> }
> }
>
> ```
>
> But if you have a lot of those fields or they may change, a ruby script would work.
>
> Not sure how the ruby code would look like, but if you search in the forum you will find some examples on how to do this.
Thank you for the answer @leandrojmp
I wish I could only have these field, the problem is I have irregular fields name but at the end of the day, they are all in the format "device\_custom\_XXXXXX"
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 1, 2023, 7:53pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/11 "2023-06-01T19:53:36Z")
You could try
```
ruby {
code => '
cef = event.get("cef")
if cef.is_a? Hash
cef.each { |k, v|
if k =~ /^device_custom_(number|string)_\d+/ and v.is_a? Hash
event.set(v["label"], v["value"])
end
}
event.remove("cef")
end
'
}
```
---
### Author:  [@SmoZyNS](https://discuss.elastic.co/u/SmoZyNS)
#### Post date: [June 2, 2023, 1:35pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/12 "2023-06-02T13:35:33Z")
Exactly what I was looking for, thank you very much @Badger !!!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 30, 2023, 1:36pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-specific-format-of-log/334815/13 "2023-06-30T13:36:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.