World Map not displaying geodata

Elastic Stack Kibana
1

Hello Team,
I am using kibana 8.3.3 and the world map is not displaying geo data via packetbeat

I am getting this err. can anyone please help.
{
"error": {
"root_cause": [
{
"type": "index_not_found_exception",
"reason": "no such index [my-index]",
"resource.type": "index_or_alias",
"resource.id": "my-index",
"index_uuid": "na",
"index": "my-index"
}
],
"type": "index_not_found_exception",
"reason": "no such index [my-index]",
"resource.type": "index_or_alias",
"resource.id": "my-index",
"index_uuid": "na",
"index": "my-index"
},
"status": 404
}

2

Hi @Raj_test Welcome to the community.

Packetbeat data is ingested into an index / datastream should be packetbeat-8.3.3 if you did not change default configuration.

That error. Looks like you're looking for an index called my-index which does not exist exactly as the error says.

So it's unclear if you tried to make packetbeat write to another index and it failed or you're just not looking at the right index.

If you're just getting started, I would highly recommend not trying to change the index names in the very beginning until you understand how all the parts work.

If you're trying to change what index packetbeat writes into you should open a separate thread on that. There's more to it than you suspect.

If you go to Kibana - Dev Tools and run

GET _cat/indices/?

What is the result?

3

Thanks Stephen.

Here is my config from packetbeat yaml file. I did not create any index for packetbeat, it is still by deafult.
hosts: ["localhost:9200"]
pipeline: geoip-info

Result of GET _cat/indices/?

yellow open .items-default-000001                                                   
yellow open .ds-metrics-system.fsstat-default-2022.08.01-000001                   
yellow open .ds-packetbeat-8.3.3-2022.08.01-000001                            
yellow open .lists-default-000001                                               
yellow open .ds-logs-elastic_agent-default-2022.08.01-000001                   
yellow open .ds-logs-elastic_agent.metricbeat-default-2022.08.01-000001        
yellow open .ds-logs-elastic_agent.filebeat-default-2022.08.01-000001          
yellow open .ds-metrics-elastic_agent.filebeat-default-2022.08.01-000001       
yellow open .ds-logs-system.application-default-2022.08.01-000001              
yellow open .ds-metrics-elastic_agent.elastic_agent-default-2022.08.01-000001 
yellow open .internal.alerts-security.alerts-default-000001                    
yellow open publications                                                       
yellow open .internal.alerts-observability.metrics.alerts-default-000001      
yellow open .ds-metrics-system.filesystem-default-2022.08.01-000001          
green  open .ds-.fleet-actions-results-2022.08.01-000001                       
yellow open .ds-filebeat-8.3.3-2022.08.01-000001                               
yellow open packetbeat                                                         
yellow open .ds-winlogbeat-8.3.3-2022.08.01-000001                             1
yellow open my_ip_locations                                      
yellow open .ds-metrics-system.process.summary-default-2022.08.01-000001     
yellow open .ds-logs-system.system-default-2022.08.01-000001                 
yellow open .ds-metrics-system.process-default-2022.08.01-000001             
yellow open .ds-metrics-system.uptime-default-2022.08.01-000001               
yellow open .ds-metrics-system.socket_summary-default-2022.08.01-000001      
yellow open .internal.preview.alerts-security.alerts-default-000002           
yellow open .ds-metrics-elastic_agent.metricbeat-default-2022.08.01-000001  
yellow open .ds-metrics-system.memory-default-2022.08.01-000001               
yellow open .ds-metricbeat-8.3.3-2022.08.04-000001                        
yellow open .ds-metrics-system.diskio-default-2022.08.01-000001             
yellow open .ds-auditbeat-8.3.3-2022.08.03-000001                           
yellow open .internal.preview.alerts-security.alerts-default-000003        
yellow open .ds-metrics-elastic_agent.fleet_server-default-2022.08.01-000001   
green  open metrics-endpoint.metadata_current_default                   
yellow open .ds-logs-system.security-default-2022.08.01-000001              
yellow open .ds-logs-elastic_agent.fleet_server-default-2022.08.01-000001     
yellow open .ds-metrics-system.network-default-2022.08.01-000001               
yellow open .ds-metrics-system.cpu-default-2022.08.01-000001
4

Hi @Raj_test

!st please format you code / results in the future using the format button </>

2nd that is not the complete output of the cat indices you left out all the important info

please rerun this and provide the complete output

GET _cat/indices/*packet*/?v

It should look something like, you did not post the additional columns

health status index                                uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   filebeat                             4k-j37gcTEGCbXoQeOvcyg   1   1          1            0      3.8kb          3.8kb
yellow open   .ds-filebeat-8.3.3-2022.08.07-000001 KTfXTHdpSjuCyH7RaFP4-Q   1   1          0            0       225b           225b

2nd what version of the stack are you on?

3rd did you follow these instructions exactly?
If not clean up the data packetbeat stream, delete any packetbeat indices and follow the instruction exactly.

Finally what you posted above is not your full packetbeat.yml

Please start with these

5

Hi Stephen,

  1. GET _cat/indices/*packet*/?v
health status index                                  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .ds-packetbeat-8.3.3-2022.08.01-000001 QYKH7sDdS_6SuLmWpQN0Eg   1   1    1589177            0    578.5mb        578.5mb
yellow open   packetbeat                             1mqVaN_WTE6k0uiLDdH5aA   1   1          0            0       225b           225b

  1. stack version 8.3.3. I have tested it with 7.17 and it worked fine. I only have the issue with 8.3.3.

  2. I will reinstall packetbeat and revert back

6

Exactly what issues you you have a Data Stream name packetbeat-8.3.3 under stack management, index management? Do you have a data view for the packetbeat data?

I formatted for you again :frowning: please going forward.
Looks like the datastream index is there
not sure why you have the actual packetbeat index that should not be there.

OK

BTW I just installed packetbeat 8.3.3
Added geoip processor
ran setup
sudo packetbeat setup -e
Then started
packetbeat setup -e

Working for me... I added destination.geo.location to the map to see

Screen Shot 2022-08-08 at 11.42.09 AM
Screen Shot 2022-08-08 at 11.42.09 AM1920×926 139 KB

7

Just installed 8.3.3, still no luck.

here is my conf for packetbeat.yml

#################### Packetbeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The packetbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/packetbeat/index.html

# =============================== Network device ===============================

# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: 4

# The network CIDR blocks that are considered "internal" networks for
# the purpose of network perimeter boundary classification. The valid
# values for internal_networks are the same as those that can be used
# with processor network conditions.
#
# For a list of available values see:
# https://www.elastic.co/guide/en/beats/packetbeat/current/defining-processors.html#condition-network
packetbeat.interfaces.internal_networks:
  - private

# =================================== Flows ====================================

# Set `enabled: false` or comment out all options to disable flows reporting.
packetbeat.flows:
  # Set network flow timeout. Flow is killed if no packet is received before being
  # timed out.
  timeout: 30s

  # Configure reporting period. If set to -1, only killed flows will be reported
  period: 10s

# =========================== Transaction protocols ============================

packetbeat.protocols:
- type: icmp
  # Enable ICMPv4 and ICMPv6 monitoring. The default is true.
  enabled: true

- type: amqp
  # Configure the ports where to listen for AMQP traffic. You can disable
  # the AMQP protocol by commenting out the list of ports.
  ports: [5672]

- type: cassandra
  # Configure the ports where to listen for Cassandra traffic. You can disable
  # the Cassandra protocol by commenting out the list of ports.
  ports: [9042]

- type: dhcpv4
  # Configure the DHCP for IPv4 ports.
  ports: [67, 68]

- type: dns
  # Configure the ports where to listen for DNS traffic. You can disable
  # the DNS protocol by commenting out the list of ports.
  ports: [53]

- type: http
  # Configure the ports where to listen for HTTP traffic. You can disable
  # the HTTP protocol by commenting out the list of ports.
  ports: [80, 8080, 8000, 5000, 8002]

- type: memcache
  # Configure the ports where to listen for memcache traffic. You can disable
  # the Memcache protocol by commenting out the list of ports.
  ports: [11211]

- type: mysql
  # Configure the ports where to listen for MySQL traffic. You can disable
  # the MySQL protocol by commenting out the list of ports.
  ports: [3306,3307]

- type: pgsql
  # Configure the ports where to listen for Pgsql traffic. You can disable
  # the Pgsql protocol by commenting out the list of ports.
  ports: [5432]

- type: redis
  # Configure the ports where to listen for Redis traffic. You can disable
  # the Redis protocol by commenting out the list of ports.
  ports: [6379]

- type: thrift
  # Configure the ports where to listen for Thrift-RPC traffic. You can disable
  # the Thrift-RPC protocol by commenting out the list of ports.
  ports: [9090]

- type: mongodb
  # Configure the ports where to listen for MongoDB traffic. You can disable
  # the MongoDB protocol by commenting out the list of ports.
  ports: [27017]

- type: nfs
  # Configure the ports where to listen for NFS traffic. You can disable
  # the NFS protocol by commenting out the list of ports.
  ports: [2049]

- type: tls
  # Configure the ports where to listen for TLS traffic. You can disable
  # the TLS protocol by commenting out the list of ports.
  ports:
    - 443   # HTTPS
    - 993   # IMAPS
    - 995   # POP3S
    - 5223  # XMPP over SSL
    - 8443
    - 8883  # Secure MQTT
    - 9243  # Elasticsearch

- type: sip
  # Configure the ports where to listen for SIP traffic. You can disable
  # the SIP protocol by commenting out the list of ports.
  ports: [5060]

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# A list of tags to include in every event. In the default configuration file
# the forwarded tag causes Packetbeat to not add any host fields. If you are
# monitoring a network tap or mirror port then add the forwarded tag.
#tags: [forwarded]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Packetbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  pipeline: geoip-info

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "j2VgXOOy21xi3pf4Lr5+"
  ssl:
    enabled: true
    ca_trusted_fingerprint: "c0845ac708500474b0be6045586b4f4558da5a7715f3cf69f44a9d2460196c9e"


# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

processors:
  - # Add forwarded to tags when processing data from a network tap or mirror.
    if.contains.tags: forwarded
    then:
      - drop_fields:
          fields: [host]
    else:
      - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - detect_mime_type:
      field: http.request.body.content
      target: http.request.mime_type
  - detect_mime_type:
      field: http.response.body.content
      target: http.response.mime_type

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Packetbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Packetbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the packetbeat.
#instrumentation:
    # Set to true to enable instrumentation of packetbeat.
    #enabled: false

    # Environment in which packetbeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
8

I followed this

copied & pasted this below on the console
PUT _ingest/pipeline/geoip-info {....}

GET _ingest/geoip/stats
{
"stats": {
"successful_downloads": 0,
"failed_downloads": 0,
"total_download_time": 0,
"databases_count": 0,
"skipped_updates": 0,
"expired_databases": 0
},
"nodes": {}
}

Geodata is not being loaded on the Map

map
map1593×684 182 KB

while analysing the packetbeat logs: it says geoip-info does not exists.
Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [geoip-info] does not exist"}, dropping event!","service.name":"packetbeat","ecs.version":"1.6.0"}

under ingest/ingest_pipelines, i can see the pipeline:

geoip-info
geoip-info1600×57 2.05 KB

9

Is packetbeat and elastic both on versions

What does this return?

GET _ingest/pipeline/geoip-info

Also this is an issue

GET _ingest/geoip/stats

should list at least 1 node perhaps you have a firewall blocking the update of the geoip database?
Did you every disable the GeoIP Database?
Did change one of these settings?

{
  "stats": {
    "successful_downloads": 0,
    "failed_downloads": 1,
    "total_download_time": 0,
    "databases_count": 0,
    "skipped_updates": 0,
    "expired_databases": 0
  },
  "nodes": {.  <!----- You are missing this?
    "niw5OGJMRTm0rZIF-aM_Pg": {
      "databases": [
        {
          "name": "GeoLite2-City.mmdb"
        },
        {
          "name": "GeoLite2-ASN.mmdb"
        },
        {
          "name": "GeoLite2-Country.mmdb"
        }
      ],
      "files_in_temp": [
        "GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-ASN.mmdb_LICENSE.txt",
        "GeoLite2-City.mmdb_LICENSE.txt",
        "GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-ASN.mmdb",
        "GeoLite2-City.mmdb_COPYRIGHT.txt",
        "GeoLite2-City.mmdb",
        "GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-Country.mmdb_LICENSE.txt",
        "GeoLite2-Country.mmdb",
        "GeoLite2-ASN.mmdb_COPYRIGHT.txt",
        "GeoLite2-Country.mmdb_COPYRIGHT.txt",
        "GeoLite2-City.mmdb_README.txt"
      ]
    }
  }
}

Note you pasted your password so make sure to change it.

10

Thanks for the notice, I did change the password right away.

Is packetbeat and elastic both on versions

yes - both are 8.3.3

What does this return GET _ingest/pipeline/geoip-info ?

{
  "geoip-info": {
    "description": "Add geoip info",
    "processors": [
      {
        "geoip": {
          "field": "client.ip",
          "target_field": "client.geo",
          "ignore_missing": true
        }
      },
      {
        "geoip": {
          "database_file": "GeoLite2-ASN.mmdb",
          "field": "client.ip",
          "target_field": "client.as",
          "properties": [
            "asn",
            "organization_name"
          ],
          "ignore_missing": true
        }
      },
      {
        "geoip": {
          "field": "source.ip",
          "target_field": "source.geo",
          "ignore_missing": true
        }
      },
      {
        "geoip": {
          "database_file": "GeoLite2-ASN.mmdb",
          "field": "source.ip",
          "target_field": "source.as",
          "properties": [
            "asn",
            "organization_name"
          ],
          "ignore_missing": true
        }
      },
      {
        "geoip": {
          "field": "destination.ip",
          "target_field": "destination.geo",
          "ignore_missing": true
        }
      },
      {
        "geoip": {
          "database_file": "GeoLite2-ASN.mmdb",
          "field": "destination.ip",
          "target_field": "destination.as",
          "properties": [
            "asn",
            "organization_name"
          ],
          "ignore_missing": true
        }
      },
      {
        "geoip": {
          "field": "server.ip",
          "target_field": "server.geo",
          "ignore_missing": true
        }
      },
      {
        "geoip": {
          "database_file": "GeoLite2-ASN.mmdb",
          "field": "server.ip",
          "target_field": "server.as",
          "properties": [
            "asn",
            "organization_name"
          ],
          "ignore_missing": true
        }
      },
      {
        "geoip": {
          "field": "host.ip",
          "target_field": "host.geo",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "server.as.asn",
          "target_field": "server.as.number",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "server.as.organization_name",
          "target_field": "server.as.organization.name",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "client.as.asn",
          "target_field": "client.as.number",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "client.as.organization_name",
          "target_field": "client.as.organization.name",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "source.as.asn",
          "target_field": "source.as.number",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "source.as.organization_name",
          "target_field": "source.as.organization.name",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "destination.as.asn",
          "target_field": "destination.as.number",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "destination.as.organization_name",
          "target_field": "destination.as.organization.name",
          "ignore_missing": true
        }
      }
    ]
  }
}

Did you every disable the GeoIP Database?

While troubleshooting the issue (before posting here), i followed this : https://www.elastic.co/guide/en/kibana/7.17/maps-connect-to-ems.html

To disable EMS, change your [kibana.yml]
1. Set `map.includeElasticMapsService` to `false` to turn off the EMS connection.

I turned it to true again and disbaled it

Did change one of settings?

No
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.