Packetbeat:How to add a new protocol?

Elastic Stack Beats
1

I know the https://www.elastic.co/guide/en/beats/packetbeat/current/new-protocol.html guide,but the guide is too old for the latest packetbeat version or source code.I find it's too hard to learn how to add a new protocol in the pacektbeat source code.
looking for warding your latest Developer Guide！Thanks a lot！Please help me！

2

Hi @lindsayshow

Great to hear you want to create a new protocol. You are right the guide is definitively not up-to-date and it is on our list to update the guide. Is there something specific you struggle with where we could help out?

3

Oh, I find the example about how to add a new protocol is just too old.
Please give me a new example relating to the packetbeat (latest version) about this task！Thank you very much！

4

Yeah, the examples are outdated, we are aware of that. We are currently changing the structure and are still making changes. We will update the docs as soon as it got more stable. The best examples are actually the existing protocols.

5

Great！So，about when？I need your guide.

6

We are working hard on making all the changes for 5.0 which is priority. I hope we have an updated guide in the next 1-2 months, but no promises.

It would be still interesting for us to know which roadblocks you hit taking the existing protocols as examples so we can focus on these parts by going in more details in the guide or making it easer in the code.

What protocol do you plan to implement?

7

mainly sip /msrp and xmpp

8

I had the same problem and after a few research of the existing http protocol, I found following steps were needed when developing a new protocol. This simple guide is based on packetbeat version 1.2.1

The Register function will register your protocol and call function New which is the second parameter, so you need to implement the function New in next step

  • Create method New or make a copy from http.go in http package, you can find the prototype of function New which is type ProtocolPlugin in beats/packetbeat/protos/registry.go

    type ProtocolPlugin func(
    testMode bool,
    results publish.Transactions,
    cfg *common.Config,
    ) (Plugin, error)

  • Implement plugin interface
    If your protocol is based on TCP, you need to implement the TcpPlugin interface (TcpPlugin is defined in beats/packetbeat/protos/registry.go)
    If your case is UDP, implement UdpPlugin

    type TcpPlugin interface {
    Plugin

      // Called when TCP payload data is available for parsing.
  Parse(pkt *Packet, tcptuple *common.TcpTuple,
  dir uint8, private ProtocolData) ProtocolData

  // Called when the FIN flag is seen in the TCP stream.
  ReceivedFin(tcptuple *common.TcpTuple, dir uint8,
  private ProtocolData) ProtocolData

  // Called when a packets are missing from the tcp stream.
  GapInStream(tcptuple *common.TcpTuple, dir uint8, nbytes int,
  private ProtocolData) (priv ProtocolData, drop bool)

  // ConnectionTimeout returns the per stream connection timeout.
  // Return <=0 to set default tcp module transaction timeout.
  ConnectionTimeout() time.Duration

    }

    type UdpPlugin interface {
    Plugin

      // ParseUdp is invoked when UDP payload data is available for parsing.
  ParseUdp(pkt *Packet)

    }
    Hope this helps

1 Like
9

Sorry, missed this thread so far.

There is a cookiecutter based code-generator implementing most boilerplate with best-practices (so far) employed: https://github.com/urso/old-packetbeat-tcp-generator

10

awesome project, before seeing this I was using copy/paste/replace, during which I can learn much more details though

11

I'd say to have a look at generated code. The style of writing protocol plugins changed somewhat overtime and the code-generator uses some common ideas to reduce allocations and add support for pipelining which some other protocols are still missing. Using a template doesn't free you from understanding what the code is actually doing + understanding if it suits the network protocol at hand (protocols can vary so much).

12

I followed your steps. But as per the github where I downloaded the beats master saying that I have to compile the program using python & go

Is this really necessary??? I want to run this setup in the client server in which I have no permission to install python & go. And after all the additions how to run the code (I cant find any executable file in the packetbeat folder) .

FYI

  1. In beat/packetbeat/main.go I added
    _ "github.com/elastic/beats/packetbeat/protos/tcp" _ "github.com/elastic/beats/packetbeat/protos/udp"
  2. In my zip file I already have the tcp & udp directories along with go files. But there is no init function. So I just added the init function in tcp.go like
    func init() { protos.Register("tcp", New) }
  3. In the beats/packetbeat/protos/registry.go I found that the TcpPlugin & udp plugin so I did nothing here.

What I have to do next??? Please help me to move further

13

If you want to compile packetbeat, GO environment is a must. You can compile your code anywhere like in your dev machine and copy the generated binary to your client server.

Looking at what you did, you tried to add tcp & udp which is not necessary, this confuses me, the thread is about how to add a new protocol, what protocol do you intend to develop ?

14

I want to add tcp & udp..

15

protocol plugins are application layer plugins. TCP and UDP are already handled by packetbeat. What is it you want want to get from TCP/UDP?

16

I am sending tcp/udp packets from my python script. I am getting the packets & sending a chunk packet in return. But in my elastic search I am not getting that data. But Mysql and other are working

FYI

server.py
`#! python

import socket`

host = "172.16.2.143"
port = 12345

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(5)

conn,addr = s.accept()
print 'Connected by', addr

while 1:
nbytes = conn.send("This is a message which contains more than one sixty characters. If we send this to server it has to slice it to 160 characters and send back the remaining junk to the server")
print "Server send ",nbytes," of data"
chunk = conn.recv(1024)
print chunk
break

conn.close()`

client.py
`#! python

import socket

host = "172.16.2.143"
port = 12345

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host,port))
data = s.recv(1024)
chunk = data[160:]
s.send(chunk)
s.close`

17

Please stick to your original thread.