ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): 401 Unauthorized

Elastic Stack Beats
,
1

I have an express app with nginx that writes the web access logs through filebeat/logstash/Elasticsearch/kibana. I wanted to add packetbeat to this pipeline. The docker-compose is the following:

packetbeat:

    build: ./packetbeat

    command: -e --strict.perms=false

    restart: always

    volumes:

      - type: bind

        source: ./packetbeat/config/packetbeat.yml

        target: /usr/share/packetbeat/config/packetbeat.yml

        read_only: true

    user: packetbeat

    cap_add:

      - NET_RAW

      - NET_ADMIN

    networks:

      - elk

    deploy:

      mode: global

And the packetbeat.yml is the following:

packetbeat.interfaces.device: any

packetbeat.flows:

  timeout: 30s

  period: 10s

packetbeat.protocols.dns:

  ports: [53]

  include_authorities: true

  include_additionals: true

packetbeat.protocols.http:

  ports: [80, 5601, 9200, 8080, 8081, 5000, 8002]

packetbeat.protocols.memcache:

  ports: [11211]

packetbeat.protocols.mysql:

  ports: [3306]

packetbeat.protocols.pgsql:

  ports: [5432]

packetbeat.protocols.redis:

  ports: [6379]

packetbeat.protocols.thrift:

  ports: [9090]

packetbeat.protocols.mongodb:

  ports: [27017]

packetbeat.protocols.cassandra:

  ports: [9042]

processors:

- include_fields:

   fields: ["cpu"]

- drop_fields:

   fields: ["cpu.user", "cpu.system"]

# The following example drops the events that have the HTTP response code 200:

- drop_event:

   when:

      equals:

          http.code: 200

setup.kibana:

  host: "http://kibana:5601"

  username: "elastic"

  password: "changeme"

output.elasticsearch:

  hosts: ["http://elasticsearch:9200"]

  username: "elastic"

  password: "changeme"

setup.dashboards.enabled: true

logging.level: warning

logging.to_files: true

logging.to_syslog: false

logging.files:

  path: /var/log/packetbeat

  name: packetbeat.log

  keepfiles: 2

  permissions: 0644

Any idea of what is going wrong?

2

Could you please post the debug logs?

3

Here are the logs:

2022-05-26T11:12:40.149Z        INFO    instance/beat.go:606    Home path: [/usr/share/packetbeat] Config path: [/usr/share/packetbeat] Data path: [/usr/share/packetbeat/data] Logs path: [/usr/share/packetbeat/logs]
2022-05-26T11:12:40.155Z        INFO    instance/beat.go:614    Beat ID: 98e57dfe-d86f-42b2-81d2-b8dada465fda
2022-05-26T11:12:40.157Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2022-05-26T11:12:40.157Z        INFO    [beat]  instance/beat.go:902    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/packetbeat", "data": "/usr/share/packetbeat/data", "home": "/usr/share/packetbeat", "logs": "/usr/share/packetbeat/logs"}, "type": "packetbeat", "uuid": "98e57dfe-d86f-42b2-81d2-b8dada465fda"}}}
2022-05-26T11:12:40.157Z        INFO    [beat]  instance/beat.go:911    Build info      {"system_info": {"build": {"commit": "a4be71b90ce3e3b8213b616adfcd9e455513da45", "libbeat": "7.3.1", "time": "2019-08-19T19:23:55.000Z", "version": "7.3.1"}}}
2022-05-26T11:12:40.157Z        INFO    [beat]  instance/beat.go:914    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":12,"version":"go1.12.4"}}}
2022-05-26T11:12:40.159Z        INFO    [beat]  instance/beat.go:918    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-05-26T07:16:48Z","containerized":true,"name":"feceb031879e","ip":["127.0.0.1/8","172.26.0.4/16"],"kernel_version":"5.10.16.3-microsoft-standard-WSL2","mac":["02:42:ac:1a:00:04"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2022-05-26T11:12:40.160Z        INFO    [beat]  instance/beat.go:947    Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_admin","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["net_admin","net_raw"],"effective":["net_admin","net_raw"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_admin","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/packetbeat", "exe": "/usr/share/packetbeat/packetbeat", "name": "packetbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-05-26T11:12:38.170Z"}}}
2022-05-26T11:12:40.160Z        INFO    instance/beat.go:292    Setup Beat: packetbeat; Version: 7.3.1
2022-05-26T11:12:40.161Z        INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'packetbeat-7.3.1' as ILM is enabled.
2022-05-26T11:12:40.162Z        INFO    elasticsearch/client.go:170     Elasticsearch url: http://elasticsearch:9200
2022-05-26T11:12:40.163Z        INFO    [publisher]     pipeline/module.go:97   Beat name: feceb031879e
2022-05-26T11:12:40.164Z        INFO    procs/procs.go:105      Process watcher disabled
2022-05-26T11:12:40.164Z        WARN    [cfgwarn]       protos/protos.go:118    DEPRECATED: dictionary style protocols configuration has been deprecated. Please use list-style protocols configuration. Will be removed in version: 7.0.0
2022-05-26T11:12:40.165Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2022-05-26T11:12:40.166Z        INFO    instance/beat.go:421    packetbeat start running.
2022-05-26T11:12:43.157Z        INFO    add_cloud_metadata/add_cloud_metadata.go:347    add_cloud_metadata: hosting provider type not detected.
2022-05-26T11:12:44.158Z        INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elasticsearch:9200))
2022-05-26T11:12:45.403Z        ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
2022-05-26T11:12:45.403Z        INFO    pipeline/output.go:93   Attempting to reconnect to backoff(elasticsearch(http://elasticsearch:9200)) with 1 reconnect attempt(s)
2022-05-26T11:12:47.636Z        ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
2022-05-26T11:12:47.636Z        INFO    pipeline/output.go:93   Attempting to reconnect to backoff(elasticsearch(http://elasticsearch:9200)) with 2 reconnect attempt(s)
2022-05-26T11:12:52.228Z        ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
2022-05-26T11:12:52.228Z        INFO    pipeline/output.go:93   Attempting to reconnect to backoff(elasticsearch(http://elasticsearch:9200)) with 3 reconnect attempt(s)
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.