Run Packetbeat in Docker swarm mode?

[shazChaudhry (Shahzad Chaudhry)]

Network capabilities such as "--cap-add=NET_ADMIN" are ignored when deploying a stack in swarm mode with a (version 3)

• Is it possible to start Packetbeat in docker swarm mode? Will be grateful if anyone could share a compose file
• In docker swarm mode, how can traffic be captured from the host system
• Should Packetbeat run as a global service in a docker swarm mode cluster or should it only be constraint to a master node?

[jsoriano (Jaime Soriano)]

I think this is more a question about docker swarm. To capture traffic from host network Packetbeat needs access to host network namespace and to be granted with network admin capabilities. I think that running containers in host network is allowed in swarm, but not sure about giving it capabilities or running it as full privileged, in a quick search it seems that these features are missing by now (see these issues about capabilities and about privileged).

As a workaround to these limitations, you can try to deploy packetbeat in your nodes directly using other configuration management options.

Regarding the deployment options, you'd need to have it deployed in the nodes where you want to capture traffic, if you want to capture traffic from all nodes it'd need to be deployed as a global service, if not, it could be deployed only in the nodes where you want to capture traffic from.

[shazChaudhry (Shahzad Chaudhry)]

Thanks @jsoriano for a quick response. This is what I have already tired but the service still does not start and I get this error task: non-zero exit (1)

• Stack service was started with root privileges in hope that it would have all permissions it needed on each swarm cluster node
• Stack service ran in global mode
• Stack service ran in host network as you suggested

Here is the compose file that I have been trying to stack deploy:

version: "3.6"
services:
  packetbeat:
    image: docker.elastic.co/beats/packetbeat:${ELASTIC_VERSION:-6.2.2}
    hostname: "{{.Node.Hostname}}-packetbeat"
    user: root
    networks:
      - elastic
    configs:
      - source: pb_config
        target: /usr/share/packetbeat/packetbeat.yml
    volumes:
      - packetbeat:/usr/share/packetbeat/data
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST:-node1}
      - KIBANA_HOST=${KIBANA_HOST:-node1}
    command: ["--strict.perms=false"]
    deploy:
      mode: global

networks:
  elastic:
    external: true
    name: host

volumes:
  packetbeat:

configs:
  pb_config:
    file: $PWD/elk/beats/packetbeat/config/packetbeat.yml

Here is my packetbeat config file:

#============================== Network device ================================
packetbeat.interfaces.device: any

#================================== Flows =====================================
packetbeat.flows:
  timeout: 30s
  period: 10s

#========================== Transaction protocols =============================
packetbeat.protocols:
- type: http
  ports: [80, 8080, 5000]
  send_headers: true
  send_all_headers: true

- type: tls
  ports: [443]
  send_certificates: false

#=========================== Monitored processes ==============================
packetbeat.procs:
  enabled: false
  monitored:
    - process: pgsql
      cmdline_grep: postgres

#================================ Processors ===================================
processors:
- include_fields:
   fields: ["cpu"]
- drop_fields:
   fields: ["cpu.user", "cpu.system"]
# The following example drops the events that have the HTTP response code 200:
- drop_event:
   when:
      equals:
          http.code: 200
# The following example enriches each event with metadata from the cloud provider about the host machine.
- add_docker_metadata:
   host: "unix:///var/run/docker.sock"
- add_cloud_metadata: ~
- add_locale: ~

#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
  hosts: ["${ELASTICSEARCH_HOST}:9200"]
  username: elastic
  password: changeme

xpack.monitoring:
  enabled: false
  elasticsearch:

#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index.
# Loading the dashboards is disabled by default and can be enabled either by setting the options here, or by using the `-setup` CLI flag.
setup.dashboards:
  enabled: true
setup.kibana:
  host: "${KIBANA_HOST}:5601"
  username: elastic
  password: changeme

I am really hoping to hear from anyone who actually has tried running Packetbeat successfully in docker swarm mode

[jsoriano (Jaime Soriano)]

Take into account that running a container as root doesn't imply that it has all capabilities, you can check the list of capabilities granted by default in docker documentation, NET_ADMIN is not one of them and is needed by Packetbeat.
If there is no way to run privileged containers in docker swarm then I doubt that Packetbeat can be run there But this is more a question for docker swarm.

Please keep us posted with your discoveries