# Can dissect use a variable number of fields?
**URL:** https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952
**Category:** Logstash
**Created:** [September 25, 2019, 3:35am UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952 "2019-09-25T03:35:37Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@joaociocca](https://discuss.elastic.co/u/joaociocca)
#### Post date: [September 25, 2019, 3:35am UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/1 "2019-09-25T03:35:37Z")
I'm trying to use Logstash to ingest some old, OLD IAS/RRAS logs that seem to be customized. They all begin the same, so I can get dissect to ingest this correctly:
```auto
filter {
dissect {
mapping => {
"message" => "%{ServerName},%{UserName},%{log_timestamp},%{+log_timestamp},%{ServiceType}"
}
}
}
```
And I've read docs and found how I can use `%{?key},%{&key}` to consume a key=value pair that is split like that... but considering the logs seem to be customized, they may not always have the same number of key=value fields.
I haven't found anything that says I CAN do something for this, like mixing regex to make a `%{?key},%{&key}+`, which would indicate the pair can happen more than once. I did see the `Dissect does not use regular expressions` warning on the docs, though. Is there any way to achieve this?
((edit)) still searching, I found this answer... makes me wonder if this would be the way to go...
> [@Iteration in Logstash](https://discuss.elastic.co/t/iteration-in-logstash/26123/2):
>
> filter { ruby { code =\> ' ids = event["id"].split(",") values = event["value"].split(",") if ids.length == values.length ids.each\_index { |i| event["id#{ids[i]}"] = values[i] } end ' } }
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 25, 2019, 4:57am UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/2 "2019-09-25T04:57:35Z")
Dissect is designed to be fast, and that is achieved by having it support a reasonably small number of rules that allow a single pass of the data. I therefore do not think what you are asking for is possible. I would recommend parsing as much as possible using dissect and the use some other filter for the rest.
---
### Author:  [@joaociocca](https://discuss.elastic.co/u/joaociocca)
#### Post date: [September 25, 2019, 5:59am UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/3 "2019-09-25T05:59:39Z")
I'm far away from work now, but since it's still in my head... Yeah, I'm figuring out that maybe I should dissect only the first few fields and treat the keys and values in some other way. Maybe try my hand at Ruby, doesn't seem to hard, even for a non-programmer like me!
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 25, 2019, 6:02am UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/4 "2019-09-25T06:02:55Z")
Have you considered the [kv filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html)?
---
### Author:  [@joaociocca](https://discuss.elastic.co/u/joaociocca)
#### Post date: [September 25, 2019, 12:50pm UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/5 "2019-09-25T12:50:28Z")
I did, but haven't found anything on documentation that would indicate I can use it. File format is only commas, there no other separator character. So I have things like `key1,value1,key2,value2,key3,value3`
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 25, 2019, 1:36pm UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/6 "2019-09-25T13:36:09Z")
Aha. Then it will likely not work.
---
### Author:  [@joaociocca](https://discuss.elastic.co/u/joaociocca)
#### Post date: [September 25, 2019, 1:51pm UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/7 "2019-09-25T13:51:11Z")
Yeah, and as I've said on the initial post, there can be a variable number of kv pairs... Guess I should go for a Ruby solution. Once I get it working I'll post my solution, who knows if anyone else out there will ever need it 😁
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 25, 2019, 1:52pm UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/8 "2019-09-25T13:52:36Z")
> [@joaociocca](#):
>
> key1,value1,key2,value2,key3,value3
```
ruby {
code => '
a = event.get("message").split(",")
h = {}
while a.length > 1
h = h.merge( [a.shift(2)].to_h )
end
event.set("someField", h)
'
}
```
will turn that into
```
"someField" => {
"key1" => "value1",
"key3" => "value3",
"key2" => "value2"
}
```
---
### Author:  [@joaociocca](https://discuss.elastic.co/u/joaociocca)
#### Post date: [September 25, 2019, 2:14pm UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/9 "2019-09-25T14:14:59Z")
AHHHH That's awesome, @Badger!! Thanks!
Now I can just dissect the first few static fields and ruby the rest ❤
---
### Author:  [@joaociocca](https://discuss.elastic.co/u/joaociocca)
#### Post date: [September 25, 2019, 8:21pm UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/10 "2019-09-25T20:21:17Z")
@Badger Looks like I messed up something, forgot about servername value... Sorry about that! Code works =D now to remove that pesky \r at the end of values...
```auto
dissect {
mapping => {
"message" => "%{NASIPAddress},%{UserName},%{log_timestamp},%{+log_timestamp},%{ServiceType},%{ServerName},%{values}"
}
}
ruby {
code => '
a = event.get("values").split(",")
h = {}
while a.length > 1
h = h.merge( [a.shift(2)].to_h )
end
event.set("keyvalues", h)
'
}
mutate {
remove_field => ["values"]
}
```
((edit)) And now I came up with a solution that doesn't use ruby!
```auto
dissect {
mapping => {
"message" => "%{NASIPAddress},%{UserName},%{log_timestamp},%{+log_timestamp},%{ServiceType},%{ServerName},%{values}"
}
}
mutate {
gsub => [
"values", "([^,]+),([^,]+),?", "\1=\2,"
]
}
kv {
field_split => ","
source => "values"
}
```
Only problem left to solve is translate the keys from numbers to their descriptions... i.e: key 4 is NAS-IP-Address, and I'd much rather have "NAS-IP-Address=ip" than "4=ip", right?
---
### Author:  [@joaociocca](https://discuss.elastic.co/u/joaociocca)
#### Post date: [September 26, 2019, 11:47pm UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/11 "2019-09-26T23:47:27Z")
Final configuration file! Thanks @Badger and @Christian_Dahlqvist
>
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 24, 2019, 11:47pm UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952/12 "2019-10-24T23:47:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.