How to make 2 separaotrs?

Elastic Stack Logstash
1

Hello guys.
This is a my first question. I want to make 2 separaotrs
For example, this is a example log
ex: a, b, c|| d, f, g|| h
Using logstash, how to declare 2 separators on logstash filter?

filter {
csv {
separator => [",", "|"]

???? Maybe this is not work.

2

I think it won't work.
There is a couple of other way to do so, but first, why do you want 2 separators? (or it's your log format and you can't do anything about it?)

3

Becasue, normally i use a separator ==> |
But. in our log, one column is cookies (Web)
Sometimes cookies have ==> | in the cookie content.
So. we make a special separator ==> |*|!||

So i want to use 2 separators both | and |*|!||.
How can i resolve this problem?

If no more method, i will change log format. But it just my curiousity.

4

Ok I think I see your point.

So typically, your log line looks like:
a, b, c || d, e, f || g, h, ... d, e, f being the cookie, right?

5

Yes. that also right.

Actually a little bit a different.

Your comment is case 1

Case 2 is ==> abc,def,wer,ff1,245,sa,dsada,sadsad <== this case use 5 separators. last column has a "," in a data.
| | | | |

both case1 and case2 are my case that is need to multi separators

6

Maybe, using "grok" , can i resolve my problem?

7

Ok then.

For the 1st case, if there is no "," nor "||" in messages two successive csv filters might do the trick (you can define the source field).

The second case would be, in my opinion, nearly impossible if the field content changes (i mean if sometimes field 1 or 2 has "," in it). I mean, in your case 2, if you always have a log line with 5 separator and only the last one has "," inside it, then it's easily doable. Else, it might be some hell.

If you have the possibility, I recommend you to use specific separators that will not be in any fields or use:
keyA:someContent, keyB:someContent2, ... and the key number is always the same (and the key name doesn't change). In this case, even if the content has ",", it won't be a problem using grok.

Just out of curiosity and see if I can do something, can you post some real log line of both your cases?

8

Log Data
172.16.110.121|64.22.25.14|23424|80|Management1|gogo1525|Yimjunhyeok|0|1|0|0||0x00000000|1|HTTP/1.1|sho2.daumcdn.net|shophow/c/image/content/set/ad4926/20170426110643063_338159|Mozilla/5.0|http://sho2.daumcdn.net|4|48|7|1514732399999|*|!|#||XmQmAfVWqFdZNbPi_yzGgZRufTM=,oy1nvTNUtjAIDbmMr55WTw==,DzQyT3J+z7tv35aWrS9Ilau5c9cP2_IEn31au07+mf8=,7evFCphh3hv31Coi3wx04RDTpEozdNr8P6suEQBuO3Y=,\r\n

Actually, normally, i parse log data with separator "|" except last column.
But last column, |*|!|#|| < == Use this for separator.

above is my real case. (Case1)

According to your first solution,... i can't use. Because |*|!|#|| also include |
If i use | in last column, last data ==> *|!|#||XmQmAfVWqFdZNbPi_yzGgZRufTM=,oy1nvTNUtjAIDbmMr55WTw==,DzQyT3J+z7tv35aWrS9Ilau5c9cP2_IEn31au07+mf8=,7evFCphh3hv31Coi3wx04RDTpEozdNr8P6suEQBuO3Y=,\r\n
But real data is ==> XmQmAfVWqFdZNbPi_yzGgZRufTM=,oy1nvTNUtjAIDbmMr55WTw==,DzQyT3J+z7tv35aWrS9Ilau5c9cP2_IEn31au07+mf8=,7evFCphh3hv31Coi3wx04RDTpEozdNr8P6suEQBuO3Y=,\r\n

If i can't parse with 2 separators, |*|!|#|| will be changed to | . However, sometimes, last-column data has some | in data.(Sometimes) This is Case2

9

Oh, then it is easier than I thought.

I suppose csv if the 1st filter you do.
Try:

filter {
  csv {
    separator => "|*|!|#||"
    columns => ["before", "cookie"]
  }

  if [before] {
    csv {
      separator => "|"
      source => "before"
    }
  }
}

You maybe need to escape some caracters for separator. And if you have other filters, just do the appropriate filter with some if

10

Wow. Fantastic. Very Very Thanks. :slight_smile:

11

This is a my final conf.
filter {
csv {
separator => "|*|!|#||"
columns => ["LastSessionTime", "Cookie"]
}
if [LastSessionTime] {
csv {
separator => "|"
columns => ["SourceIP", "DestinationIP", ....... "AS", "LastSessionTime"]
autogenerate_column_names => false
}
}
}

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.