Packet Beat does not send any data to elastic search

Hi there

I'm running elastic search version 7.8.0 and Kibana 7.8.0 on Win10.
I have installed metricbeat and filebeat - both are running fine.
I have installed packetbeat and runned packetbeat -setup
The index has been created - the index pattern has been created
the timefield was automatically set to @timestamp
packet beat is started with:

C:\Program Files\Elastic\Beats\7.10.1\packetbeat>packetbeat -c C:\ProgramData\Elastic\Beats\packetbeat\packetbeat.yml -e

The relevant parts packetbeat.yml file looks like this:

# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.

**packetbeat.interfaces.device: 2**


  # Set network flow timeout. Flow is killed if no packet is received before being
  # timed out.
  timeout: 30s

  # Configure reporting period. If set to -1, only killed flows will be reported
  period: 10s

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  username: "*******"
  password: "***********"
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"


Packetbeats is starting without any error: 

2020-12-27T17:00:18.884+0100    INFO    instance/beat.go:645    Home path: [C:\Program Files\Elastic\Beats\7.10.1\packetbeat] Config path: [C:\Program Files\Elastic\Beats\7.10.1\packetbeat] Data path: [C:\Program Files\Elastic\Beats\7.10.1\packetbeat\data] Logs path: [C:\Program Files\Elastic\Beats\7.10.1\packetbeat\logs]

2020-12-27T17:00:18.885+0100    INFO    instance/beat.go:653    Beat ID: 895d20ab-c8b4-48e1-b86a-582666d84725

2020-12-27T17:00:18.923+0100    INFO    [beat]  instance/beat.go:981 Beat info       {"system_info": {"beat": {"path": {"config": "C:\\Program Files\\Elastic\\Beats\\7.10.1\\packetbeat", "data": "C:\\Program Files\\Elastic\\Beats\\7.10.1\\packetbeat\\data", "home": "C:\\Program Files\\Elastic\\Beats\\7.10.1\\packetbeat", "logs": "C:\\Program Files\\Elastic\\Beats\\7.10.1\\packetbeat\\logs"}, "type": "packetbeat", "uuid": "895d20ab-c8b4-48e1-b86a-582666d84725"}}}

2020-12-27T17:00:18.923+0100    INFO    [beat]  instance/beat.go:990  Build info      {"system_info": {"build": {"commit": "1da173a9e716715a7a54bb3ff4db05b5c24fc8ce", "libbeat": "7.10.1", "time": 

"2020-12-04T22:46:34.000Z", "version": "7.10.1"}}}

2020-12-27T17:00:18.924+0100    INFO    [beat]  instance/beat.go:993  Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":56,"version":"go1.14.12"}}}

2020-12-27T17:00:18.945+0100    INFO    [beat]  instance/beat.go:997 Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-12-14T16:25:41.25+01:00","name":"ALEX","ip":["fe80::8839:7d83:2926:fa27/64","192.168.1.182/24","fe80::d9a3:6d7c:d20d:91a8/64","192.168.1.183/24","fe80::1969:44b6:58b1:7792/64","169.254.119.146/16","::1/128","127.0.0.1/8"],"kernel_version":"10.0.19041.685 (WinBuild.160101.0800)","mac":["00:25:90:5e:b0:4e","00:25:90:5e:b0:4f","02:00:4c:4f:4f:50"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"19041.685"},"timezone":"CET","timezone_offset_sec":3600,"id":"bac0ba19-9660-488d-b185-c9ce446c8807"}}}

2020-12-27T17:00:18.945+0100    INFO    [beat]  instance/beat.go:1026 Process info    {"system_info": {"process": {"cwd": "C:\\Program Files\\Elastic\\Beats\\7.10.1\\packetbeat", "exe": "C:\\Program Files\\Elastic\\Beats\\7.10.1\\packetbeat\\packetbeat.exe", "name": "packetbeat.exe", "pid": 31616, "ppid": 10468, "start_time": "2020-12-27T17:00:18.458+0100"}}}

2020-12-27T17:00:18.946+0100    INFO    instance/beat.go:299    Setup Beat: packetbeat; Version: 7.10.1

2020-12-27T17:00:18.946+0100    INFO    [index-management]      idxmgmt/std.go:184    Set output.elasticsearch.index to 'packetbeat-7.10.1' as ILM is enabled.

2020-12-27T17:00:18.948+0100    INFO    eslegclient/connection.go:99 elasticsearch url: http://localhost:9200

2020-12-27T17:00:18.948+0100    INFO    [publisher]     pipeline/module.go:113        Beat name: ALEX

2020-12-27T17:00:18.949+0100    INFO    procs/procs.go:105      Process watcher disabled

2020-12-27T17:00:18.957+0100    WARN    [cfgwarn]       sip/plugin.go:64      BETA: packetbeat SIP protocol is used

2020-12-27T17:00:19.050+0100    INFO    sniffer/device.go:92    Resolved device index 2 to device: \Device\NPF_{AA95C0C9-969F-4024-B88F-253ED08AB0B6}

2020-12-27T17:00:19.050+0100    INFO    instance/beat.go:455    packetbeat start running.

2020-12-27T17:00:19.051+0100    INFO    [monitoring]    log/log.go:118        Starting metrics logging every 30s

Every 30 seconds I get the following message type: 

2020-12-27T17:00:49.060+0100    INFO    [monitoring]    log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":453,"time":{"ms":453}},"total":{"ticks":546,"time":{"ms":546},"value":546},"user":{"ticks":93,"time":{"ms":93}}},"handles":{"open":281},"info":{"ephemeral_id":"a27b78ff-62c4-4f8f-9318-e7dc91654c82","uptime":{"ms":30234}},"memstats":{"gc_next":37860880,"memory_alloc":25717160,"memory_total":31237056,"rss":63066112},"runtime":{"goroutines":41}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":16,"events":{"active":0}}},"system":{"cpu":{"cores":56}}}}}

The index Config for the packetbeat Index looks like this:

{
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "stats": {
    "uuid": "sFwG0npwSYOtvkJ1mtuTng",
    "primaries": {
      "docs": {
        "count": 0,
        "deleted": 0
      },
      "store": {
        "size_in_bytes": 208
      },
      "indexing": {
        "index_total": 0,
        "index_time_in_millis": 0,
        "index_current": 0,
        "index_failed": 0,
        "delete_total": 0,
        "delete_time_in_millis": 0,
        "delete_current": 0,
        "noop_update_total": 0,
        "is_throttled": false,
        "throttle_time_in_millis": 0
      },
      "get": {
        "total": 0,
        "time_in_millis": 0,
        "exists_total": 0,
        "exists_time_in_millis": 0,
        "missing_total": 0,
        "missing_time_in_millis": 0,
        "current": 0
      },
      "search": {
        "open_contexts": 0,
        "query_total": 18,
        "query_time_in_millis": 7,
        "query_current": 0,
        "fetch_total": 16,
        "fetch_time_in_millis": 13,
        "fetch_current": 0,
        "scroll_total": 0,
        "scroll_time_in_millis": 0,
        "scroll_current": 0,
        "suggest_total": 0,
        "suggest_time_in_millis": 0,
        "suggest_current": 0
      },
      "merges": {
        "current": 0,
        "current_docs": 0,
        "current_size_in_bytes": 0,
        "total": 0,
        "total_time_in_millis": 0,
        "total_docs": 0,
        "total_size_in_bytes": 0,
        "total_stopped_time_in_millis": 0,
        "total_throttled_time_in_millis": 0,
        "total_auto_throttle_in_bytes": 20971520
      },
      "refresh": {
        "total": 2,
        "total_time_in_millis": 0,
        "external_total": 2,
        "external_total_time_in_millis": 0,
        "listeners": 0
      },
      "flush": {
        "total": 1,
        "periodic": 0,
        "total_time_in_millis": 0
      },
      "warmer": {
        "current": 0,
        "total": 1,
        "total_time_in_millis": 0
      },
      "query_cache": {
        "memory_size_in_bytes": 0,
        "total_count": 0,
        "hit_count": 0,
        "miss_count": 0,
        "cache_size": 0,
        "cache_count": 0,
        "evictions": 0
      },
      "fielddata": {
        "memory_size_in_bytes": 0,
        "evictions": 0
      },
      "completion": {
        "size_in_bytes": 0
      },
      "segments": {
        "count": 0,
        "memory_in_bytes": 0,
        "terms_memory_in_bytes": 0,
        "stored_fields_memory_in_bytes": 0,
        "term_vectors_memory_in_bytes": 0,
        "norms_memory_in_bytes": 0,
        "points_memory_in_bytes": 0,
        "doc_values_memory_in_bytes": 0,
        "index_writer_memory_in_bytes": 0,
        "version_map_memory_in_bytes": 0,
        "fixed_bit_set_memory_in_bytes": 0,
        "max_unsafe_auto_id_timestamp": -1,
        "file_sizes": {}
      },
      "translog": {
        "operations": 0,
        "size_in_bytes": 55,
        "uncommitted_operations": 0,
        "uncommitted_size_in_bytes": 55,
        "earliest_last_modified_age": 0
      },
      "request_cache": {
        "memory_size_in_bytes": 20439,
        "evictions": 0,
        "hit_count": 7,
        "miss_count": 11
      },
      "recovery": {
        "current_as_source": 0,
        "current_as_target": 0,
        "throttle_time_in_millis": 0
      }
    },
    "total": {
      "docs": {
        "count": 0,
        "deleted": 0
      },
      "store": {
        "size_in_bytes": 208
      },
      "indexing": {
        "index_total": 0,
        "index_time_in_millis": 0,
        "index_current": 0,
        "index_failed": 0,
        "delete_total": 0,
        "delete_time_in_millis": 0,
        "delete_current": 0,
        "noop_update_total": 0,
        "is_throttled": false,
        "throttle_time_in_millis": 0
      },
      "get": {
        "total": 0,
        "time_in_millis": 0,
        "exists_total": 0,
        "exists_time_in_millis": 0,
        "missing_total": 0,
        "missing_time_in_millis": 0,
        "current": 0
      },
      "search": {
        "open_contexts": 0,
        "query_total": 18,
        "query_time_in_millis": 7,
        "query_current": 0,
        "fetch_total": 16,
        "fetch_time_in_millis": 13,
        "fetch_current": 0,
        "scroll_total": 0,
        "scroll_time_in_millis": 0,
        "scroll_current": 0,
        "suggest_total": 0,
        "suggest_time_in_millis": 0,
        "suggest_current": 0
      },
      "merges": {
        "current": 0,
        "current_docs": 0,
        "current_size_in_bytes": 0,
        "total": 0,
        "total_time_in_millis": 0,
        "total_docs": 0,
        "total_size_in_bytes": 0,
        "total_stopped_time_in_millis": 0,
        "total_throttled_time_in_millis": 0,
        "total_auto_throttle_in_bytes": 20971520
      },
      "refresh": {
        "total": 2,
        "total_time_in_millis": 0,
        "external_total": 2,
        "external_total_time_in_millis": 0,
        "listeners": 0
      },
      "flush": {
        "total": 1,
        "periodic": 0,
        "total_time_in_millis": 0
      },
      "warmer": {
        "current": 0,
        "total": 1,
        "total_time_in_millis": 0
      },
      "query_cache": {
        "memory_size_in_bytes": 0,
        "total_count": 0,
        "hit_count": 0,
        "miss_count": 0,
        "cache_size": 0,
        "cache_count": 0,
        "evictions": 0
      },
      "fielddata": {
        "memory_size_in_bytes": 0,
        "evictions": 0
      },
      "completion": {
        "size_in_bytes": 0
      },
      "segments": {
        "count": 0,
        "memory_in_bytes": 0,
        "terms_memory_in_bytes": 0,
        "stored_fields_memory_in_bytes": 0,
        "term_vectors_memory_in_bytes": 0,
        "norms_memory_in_bytes": 0,
        "points_memory_in_bytes": 0,
        "doc_values_memory_in_bytes": 0,
        "index_writer_memory_in_bytes": 0,
        "version_map_memory_in_bytes": 0,
        "fixed_bit_set_memory_in_bytes": 0,
        "max_unsafe_auto_id_timestamp": -1,
        "file_sizes": {}
      },
      "translog": {
        "operations": 0,
        "size_in_bytes": 55,
        "uncommitted_operations": 0,
        "uncommitted_size_in_bytes": 55,
        "earliest_last_modified_age": 0
      },
      "request_cache": {
        "memory_size_in_bytes": 20439,
        "evictions": 0,
        "hit_count": 7,
        "miss_count": 11
      },
      "recovery": {
        "current_as_source": 0,
        "current_as_target": 0,
        "throttle_time_in_millis": 0
      }
    }
  }
}

I don't get any documents in that index.

Can somebody tell me whats wrong here.

Thanks in advance
Alex

In Dev Tools what do you get when you do GET packet-*/_count? Assuming you didn't rename the default packetbeat index name.

Thanks for Your reply. Id did not rename the generated index name.
The count showed me 0 documents.

I have stated the process on shell using the admin user with -e option - so i saw the events that have been recorded from the network interfaces.
The process trace on the windows machine showed me that the packetbeat process on the machine was working - but at the end I didn't see any entries in the index.

So I did a last desperate try. I deleted the index and the index pattern.
I runned the packetbeat -setup process again - this is automatically creating
the index and the index pattern on a windows machine.

I enhanced the elasticsearch port definition with a valid admin user name and a pwd in the .yml file.

I restarted the packetbeat again ------- AND it worked - Problem solved!

But anyway many thanks for taking care on my request!

Have good and of the year and all the best for 2021!
stay healthy
Best
Alex

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.