Packetbeat for search query monitoring

I am trying to use packetbeat to monitor users' search queries and store them into elasticsearch using logstash. However, when I am trying to test it out on my local, packetbeat does not capture any of search queries I executed on elasticsearch on localhost.

I have changed the configuration file to listen for HTTP traffic:

 - type: http
 # Configure the ports where to listen for HTTP traffic. You can disable
 # the HTTP protocol by commenting out the list of ports.
 # ports: [80, 8080, 8000, 5000, 8002]
 ports: [9200]
 send_request: true
 include_body_for: ["application/json", "x-www-form-urlencoded"]

It also seems like that packetbeat does not generate any log files at all while running, as in the log folders is always empty.

@Derek_Liu From a quick look at the configuration It look OK, so Lets start packetbeat in console mode and with debugging one, make sure you only have the http module enabled since this could be noisy.

The following command will output all the error and warnings to the active console.

packetbeat -v -e -c youconfig.yml -d "*" 

2018-08-08T09:29:56.033-0700 INFO instance/beat.go:492 Home path: [C:\Program Files\Packetbeat] Config path: [C:\Program Files\Packetbeat] Data path: [C:\Program Files\Packetbeat\data] Logs path: [C:\Program Files\Packetbeat\logs]
2018-08-08T09:29:56.034-0700 DEBUG [beat] instance/beat.go:519 Beat metadata path: C:\Program Files\Packetbeat\data\meta.json
2018-08-08T09:29:56.034-0700 INFO instance/beat.go:499 Beat UUID: 26a601f9-e093-439d-a774-dfe54eb4d122
2018-08-08T09:29:56.034-0700 INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "C:\Program Files\Packetbeat", "data": "C:\Program Files\Packetbeat\data", "home": "C:\Program Files\Packetbeat", "logs": "C:\Program Files\Packetbeat\logs"}, "type": "packetbeat", "uuid": "26a601f9-e093-439d-a774-dfe54eb4d122"}}}
2018-08-08T09:29:56.034-0700 INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "45a9a9e1561b6c540e94211ebe03d18abcacae55", "libbeat": "6.3.2", "time": "2018-07-20T04:13:45.000Z", "version": "6.3.2"}}}
2018-08-08T09:29:56.035-0700 INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.9.4"}}}
2018-08-08T09:29:56.047-0700 INFO instance/beat.go:225 Setup Beat: packetbeat; Version: 6.3.2
2018-08-08T09:29:56.047-0700 DEBUG [beat] instance/beat.go:242 Initializing output plugins
2018-08-08T09:29:56.047-0700 DEBUG [processors] processors/processor.go:49 Processors:
2018-08-08T09:29:56.048-0700 DEBUG [publish] pipeline/consumer.go:120 start pipeline event consumer
2018-08-08T09:29:56.048-0700 INFO pipeline/module.go:81 Beat name: DliuLaptop
2018-08-08T09:29:56.048-0700 INFO procs/procs.go:78 Process matching disabled
2018-08-08T09:29:56.051-0700 DEBUG [main] beater/packetbeat.go:113 Initializing protocol plugins

2018-08-08T09:29:56.053-0700 DEBUG [protos] protos/protos.go:105 registered protocol plugin: http

2018-08-08T09:29:56.053-0700 DEBUG [processors] processors/processor.go:49 Processors:
2018-08-08T09:29:56.054-0700 DEBUG [processors] processors/processor.go:49 Processors:
2018-08-08T09:29:56.054-0700 DEBUG [flows] flows/worker.go:62 new flows worker. timeout=30s, period=10s, tick=10s, ticksTO=3, ticksP=1
2018-08-08T09:29:56.054-0700 DEBUG [sniffer] sniffer/sniffer.go:74 BPF filter: ''
2018-08-08T09:29:56.169-0700 INFO sniffer/device.go:75 Resolved device index 0 to device: \Device\NPF_{8DFAFF5E-3117-4C57-BE22-074636E39623}
2018-08-08T09:29:56.169-0700 DEBUG [sniffer] sniffer/sniffer.go:112 Sniffer type: pcap device: \Device\NPF_{8DFAFF5E-3117-4C57-BE22-074636E39623}
2018-08-08T09:29:56.170-0700 INFO instance/beat.go:315 packetbeat start running.
2018-08-08T09:29:56.170-0700 DEBUG [service] service/service_windows.go:51 Windows is interactive: true
2018-08-08T09:29:56.170-0700 DEBUG [flows] flows/util.go:30 start flows worker
2018-08-08T09:29:56.171-0700 DEBUG [main] beater/packetbeat.go:208 Waiting for the sniffer to finish
2018-08-08T09:29:56.171-0700 DEBUG [flows] flows/worker.go:89 worker wait start(2018-08-08 09:30:00 -0700 PDT): 3.8288311s
2018-08-08T09:29:56.177-0700 DEBUG [tcp] tcp/tcp.go:323 tcp%!(EXTRA string=Port map: %v, map[uint16]protos.Protocol=map[9200:http])
2018-08-08T09:29:56.178-0700 DEBUG [udp] udp/udp.go:94 Port map: map[]
2018-08-08T09:29:56.178-0700 DEBUG [flows] flows/counters.go:134 register flow counter: net_packets_total
2018-08-08T09:29:56.178-0700 DEBUG [flows] flows/counters.go:134 register flow counter: net_bytes_total
2018-08-08T09:29:56.178-0700 DEBUG [decoder] decoder/decoder.go:98 Layer type: Ethernet
2018-08-08T09:29:56.679-0700 DEBUG [sniffer] sniffer/sniffer.go:168 Interrupted
2018-08-08T09:29:57.180-0700 DEBUG [sniffer] sniffer/sniffer.go:168 Interrupted
2018-08-08T09:29:57.680-0700 DEBUG [sniffer] sniffer/sniffer.go:168 Interrupted
2018-08-08T09:29:58.181-0700 DEBUG [sniffer] sniffer/sniffer.go:168 Interrupted
2018-08-08T09:29:58.681-0700 DEBUG [sniffer] sniffer/sniffer.go:168 Interrupted
2018-08-08T09:29:59.182-0700 DEBUG [sniffer] sniffer/sniffer.go:168 Interrupted
2018-08-08T09:29:59.682-0700 DEBUG [sniffer] sniffer/sniffer.go:168 Interrupted
2018-08-08T09:30:00.000-0700 DEBUG [flows] flows/worker.go:98 start flows worker loop
2018-08-08T09:30:00.182-0700 DEBUG [sniffer] sniffer/sniffer.go:168 Interrupted
2018-08-08T09:30:00.683-0700 DEBUG [sniffer] sniffer/sniffer.go:168 Interrupted
2018-08-08T09:30:00.766-0700 DEBUG [service] service/service.go:34 Received sigterm/sigint, stopping
2018-08-08T09:30:00.766-0700 DEBUG [service] service/service.go:41 Received svc stop/shutdown request
2018-08-08T09:30:00.767-0700 INFO beater/packetbeat.go:221 Packetbeat send stop signal
2018-08-08T09:30:01.183-0700 DEBUG [sniffer] sniffer/sniffer.go:168 Interrupted
2018-08-08T09:30:01.183-0700 DEBUG [flows] flows/util.go:39 stop flows worker
2018-08-08T09:30:01.184-0700 DEBUG [flows] flows/util.go:75 stop periodic loop
2018-08-08T09:30:01.184-0700 DEBUG [flows] flows/worker.go:124 exec tick, timeout=false, report=true
2018-08-08T09:30:01.184-0700 INFO flows/util.go:47 flows worker loop stopped
2018-08-08T09:30:01.184-0700 DEBUG [flows] flows/util.go:42 stopped flows worker
2018-08-08T09:30:01.185-0700 INFO instance/beat.go:321 packetbeat stopped.

Still no logs generated.

@Derek_Liu I presume you are running everything on the same machine, packetbeat and elasticsearch?

You might want to change the interface you are listening too to make sure we capture the packet.

I had to use 'lo0' on macos x to correctly capture local traffic.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.