Filebeat with new line

Elastic Stack Beats
1

Hi

I have following lines, and I am using filebeat to publish this data to logstash and then trying to extract values. Could you please advise?

2017-09-03 16:01:28,574 85732M0=>{"DATALIN": {
"EVA:": "P",
"INTRO": "",
"BACKUP": "",
"NEUTRAL": "",
"ID": "005706",
"FON": 0,
}}
2017-09-03 16:01:28,574 85732M0=>{"DATALIN": {
"EVA:": "P",
"INTRO": "",
"BACKUP": "",
"NEUTRAL": "",
"ID": "005706",
"FON": 0,
}}

2

You mean you would like to combine multiple lines into one message?
That's what the multiline settings are for.

You could use this for example:

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
3

Hi Atira,

I used multiline earlier in filbeat and it generated this message with lot of backslashs and \n characters. Is there a way to eliminate this and generate following format?

"message": "2017-09-03 16:01:28,574 85732M0=\u003e{"DATALIN": {\n "EVA": "P",\n "INTRO": "",\n "BACKUP": "",\n "NEUTRAL": "",\n "ID": 005706,\n "FON": ""\n}}"

target format:

Time: 2017-09-03 16:01:28,574
username: 85732M0
EVA: "P"
INTRO:
BACKUP
NEUTRAL
ID: 005706
FON:

Thanks for your help.

4

I've looked up on it a little.
Taking a better look, your events consist of date + user + json document.
There is a codec plugin that handles json inputs with newlines, the json_lines codec plugin.

However, your events are only partly json documents, so the codec would fail. This makes the solution a little bit, well, uglier. Maybe someone knows better (I'm a beginner at the ELK stack myself).

But this should work.

So, let's take this input:

This, while looks bad for the human eye, is quite standard format, so using the dissect filter should work, which is good because the dissect filter has a much lower performance need.
I added the date filter too, you'll probably need it anyway. I presume the dissect filter will eliminate the space between the date and the time.

filter {
    dissect {
        mapping => {
            "message" => "%{Time} %{+Time} %{username}=\u003e{\"DATALIN\": {\n \"EVA\": \"%{EVA}\",\n \"INTRO\": \"%{INTRO}\",\n \"BACKUP\": \"%{BACKUP}\",\n \"NEUTRAL\": \"%{NEUTRAL}\",\n \"ID\": %{ID},\n \"FON\": \"%{FON}\"\n}}"
            }
        }
    }
    date {
        match => [ "Time", "yyyy-MM-ddHH:mm:ss,SSS" ]
    }
}

I hope it'll work. I can't test it myself, so please come back with the result :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.