Logstash-output-syslog full json in message?

[PandKing (Scot)]

Hi,
I'm sending filebeat data to logstash and from there I send a copy to elastic and another to a 3rd party SIM which only supports syslog.

Is there a way to send the full json line in the message without adding a bunch of specific fields to the message output? I understand there are RFC limits to the message size but mine are pretty small.

Using message => "%{full_log}" but it still missing a lot of data.

Thanks

[Badger]

If you really want the whole event, you can add a field to the event that contains the whole event. Something like

    ruby { code => ' event.set("everything!", event.to_json) ' } 

Then use this on the syslog output

message => "everything!"

It might be possible to use a field within [@metadata] to avoid getting the field in es. Try replacing everything! with [@metadata][everything] and see if that works.

[PandKing (Scot)]

Not getting this to work.

Tied using the same string format as my es output.
output{
elasticsearch {
hosts => "##.##.##.##:9200"
sniffing => false
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
} # End elasticsearch

Event if I put this it doesn't interpret the value just prints the actual word metadata... blaa whatever

message => "%{[@metadata][beat]}-%{[@metadata][version]}-%"

[Badger]

Aren't you using a syslog output to write to the SIM? (Did you mean SEIM?) What does that output look like?

[PandKing (Scot)]

Yea my lazy finger just abbreviated.

At the moment

syslog {
id => "OSSEC_SYSLOG"
host => "#.#.#.#"
port => 514
appname => "OSSEC"
sourcehost => "###-########"
message => "%{full_log}"
}

Which works as extected but does not give me the full json line.

Tried many varients of your example and this one expecting to see "filebeat-6.3.2"

   message => "%{[@metadata][beat]}-%{[@metadata][version]}%"

[Badger]

I do not think you understood my suggestion. I am saying that you convert the entire event to JSON, and insert that as an additional field on the event using

ruby { code => ' event.set("everything!", event.to_json) ' } 

Then instead of sending %{message} to the syslog output, send the %{everything!} field. So instead of using

message => "%{[@metadata][beat]}-%{[@metadata][version]}%"

use this which inserts the entire event as a JSON string

message => "%{everything!}"

I wrote that incorrectly in the first posting. Left out the %{}. Sorry about that.

BTW, what is full_log? Is that a field you parsed out of the original message field?

Also, this will result in your events in elasticsearch having a field called everything!, which doubles the volume of data going into es. I am saying that to avoid that you could try

ruby { code => ' event.set("[@metadata][everything]", event.to_json) ' } 

and this for the syslog output

message => "%{[@metadata][everything]}"

but I am not in a position to test that that works. But I would expect that you would get the normal syslog format (see the source here) with the message part being the JSON string.

[PandKing (Scot)]

Yea I figured that it may be a type-o
Tried %{everything!} and got the word everything also many other variants.

full_log is a field that I selected from kibana.

Msg: Aug 02 01:18:48 ###-####### OSSEC[-]: %{everything!}\0x0a
Msg: Aug 02 01:18:48 ###-####### OSSEC[-]: %{everything!}\0x0a

[PandKing (Scot)]

@Badger

%{[@metadata][everything]}

"%{[@metadata][everything]}"

Msg: Aug 02 01:48:29 ###-####### OSSEC[-]: %{[@metadata][everything]}\0x0a
Msg: Aug 02 01:48:32 ###-####### OSSEC[-]: %{[@metadata][everything]}\0x0a

[Badger]

If %{} does not do the substitution it suggests the field does not exist. If you do not use [@metadata] and put the field on the event do you see it in elasticsearch?

[PandKing (Scot)]

My logs are getting into elastic fine. I assumed based on your feedback %everything% was an internal representation in logstash for the whole line ?

I got the full_log field from looking at the json message in kibana and I can select one or more to put into the message output but nothing I can find that represents the entire message.

[Badger]

Do you see the everything! event in elasticsearch?

everything! is not a logstash thing. It is getting added to the event by this ruby filter.

ruby { code => ' event.set("everything!", event.to_json) ' }