Beats TCP Message Body Decode


(Zeeshan Haider) #1

I just have a Question I am working on a private service and want to integrate beat messages ingestion on my service, I been trying to decode tcp data which is coming from beat, I was successfully decoding the packet itself but unable to decode message body field.

I am using nodejs as a server side framework.

Now as I am new to TCP Am i missing something here?


(Steffen Siering) #2

What exactly do you mean by TCP Message Body? You try to have beats send to your own service (Which output are you using in beats)?


(Zeeshan Haider) #3

I am using logstash as output for my own service.


(Steffen Siering) #4

The message beats send to logstash might be compressed. You can find a go-lang based reference implementation of the lumberjack protocol (look for v2) at https://github.com/elastic/go-lumber


(Zeeshan Haider) #5

Thanks a lot for giving me the let me try and understand it first I will write after if I succeed or not.


(Zeeshan Haider) #6

I tried to use zlib but still can not decript, I Descripted Binary Packaet with binary decripters0 but still data field can not be decripted


(Steffen Siering) #7

Can you add some more details?

Compression in beats can be disabled (set compression: 0). First try to process data with compression disabled. Add compression support later.


(Zeeshan Haider) #8

But I can not control compression option from other people if you have other suggestion ? as I will be using beats as a client for my server from third party clients


(Steffen Siering) #9

That's right, you can not enforce people to disable compression. The idea is to first get protocol support (sans compression) working. Just to remove the chance of some other protocol handling errors. Once you got the protocol working, adding compression is quite simple.


(Zeeshan Haider) #10

I just have one problem can not decode data send from lumberjack client


(Steffen Siering) #11

Without code and extensive debug logs on your server implementation I am not able to help in any way here. The events are encoded in JSON, embedded in the lumberjack framing. If you can not even decode anything I'd guess you have some error in the framing/offset handling.


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.