Logstash split fields

Elastic Stack Logstash
1

Actually im trying to split a field's request with & caracter

kv {
      source => "request"
      field_split => "&?"
      target => "params"
 }

For example when the request is:

toto=a&titi=b$tata=c

I get this:

{
    "toto":"a",
    "titi":"b",
    "tata":"c",
}

But same times the value of field contains the & caracter like:

toto=a&titi=b$tata=c..tt&d=lolo

Then with my filter i will get:

{
    "toto":"a",
    "titi":"b",
    "tata":"cc..tt",
    "d":"lolo"
}

Or i want to get:

{
    "toto":"a",
    "titi":"b",
    "tata":"c..tt&d=lolo"
}

Is there any solution to intercept c..tt&d=lolo as a value of tata ?

If no, how can i delete a field contains . or $

B.R

2

Does this string originate from a URL?

3

Yes, and my problem is with the owa_http_referer parameter.

owa_HTTP_REFERER=http://fr.search.yahoo.com/search%3Fp%3Drapports%2Bannuels%2Bde%2Bperformance%26fr%3Diphone%26.tsrc%3Dapple%26pcarrier%3DOrange%2BF%26pmcc%3D208%26pmnc%3D01

You can see it contains the & caracter.

4

Not sure what you mean. Your example string decodes to this:

http://fr.search.yahoo.com/search?p=rapports+annuels+de+performance&fr=iphone&.tsrc=apple&pcarrier=Orange+F&pmcc=208&pmnc=01

Are you saying that there's a "&" in there that's actually supposed to be part of the value?

5

Exactly, the & is a part of the value of owa_HTTP_REFERER field.
I want to get
http://fr.search.yahoo.com/search?p=rapports+annuels+de+performance&fr=iphone&.tsrc=apple&pcarrier=Orange+F&pmcc=208&pmnc=01 as value of owa_HTTP_REFERER.

6

So use the urldecode filter, or am I still not understanding the problem?

7

Sorry if i cant explain clearly.

Suppose we have the following string:
field1=value1$field2=string1=aa&string2=bb&field3=value3

with k filter I get this :
> {

    field1:value1,
    field2=string1=aa,
    string2=bb,
    field3=value3
    }

But i want to get:
> {

    field1:value1,
    field2=string1=aa&string2=bb,
    field3=value3
    }

Im searching a solution to get string1=aa&string2=bb as a valur of fied2.

Are you understand or not clear yet ?

8

Yeah, that's pretty much from the start but I don't see how that applies to the actual URL you sent. I can't find any extra "&" characters.

Anyway, that URL seems to have been produced by broken software that doesn't URL encode the strings properly. Fixing that in a generic way isn't possible. If you can describe, in words, the algorithm to use when joining together the strings I can help you with the Logstash implementation. In other words, following your most recent example, how would Logstash know that it's field2 and string2 that should be joined?

9

There are no way to know that it's field2 or others, i receive requests with an algorithm in a JS file.
And there are no extra & character. I want a solution to tell my filter that all characters of the string (string1=aa&string2=bb) are a single value of the field2.

The problem that the kv (with & split character) will decompose my string in many fields and values because it contains the & character.

At the end, the kv gives me same fileds name starts with dot characters that are not supported on MongoDB. So the solution is to find a filter that gives me this field2 = string1=aa&string2=bb or remove the fields that their name starts with dot character.