Issues with packetbeat geo Location

Elastic Stack Beats
1

Hello,

I have configured the host to push network traffic using packet beat. On kibana, I was able to see packet beat dashboard too. But in my dashboard, i was not able to see the geolocation. When I check what packet beat sending to elastic search, I see it missing geolocation field. Can someone help me how can I configure my packet beat to include geolocation? Thanks for your help!

2 
shipper:
  geoip:
    paths:
      # Path to where you extracted the .dat file:
      - /usr/share/GeoIP/GeoLiteCity.dat
3

@andrewkroh Thanks you, for the quick response!
I have tried geoip configuration. Still I did not see Geolocation in kibana dashboard.
This is my configuration

shipper:
  # The name of the shipper that publishes the network data. It can be used to group
  # all the transactions sent by a single shipper in the web interface.
  # If this options is not defined, the hostname is used.
  #name:

  # The tags of the shipper are included in their own field with each
  # transaction published. Tags make it easy to group servers by different
  # logical properties.
  #tags: ["service-X", "web-tier"]

  # Uncomment the following if you want to ignore transactions created
  # by the server on which the shipper is installed. This option is useful
  # to remove duplicates if shippers are installed on multiple servers.
  #ignore_outgoing: true

  # How often (in seconds) shippers are publishing their IPs to the topology map.
  # The default is 10 seconds.
  #refresh_topology_freq: 10

  # Expiration time (in seconds) of the IPs published by a shipper to the topology map.
  # All the IPs will be deleted afterwards. Note, that the value must be higher than
  # refresh_topology_freq. The default is 15 seconds.
  #topology_expire: 15

  # Internal queue size for single events in processing pipeline
  #queue_size: 1000

  # Configure local GeoIP database support.
  # If no paths are not configured geoip is disabled.
  geoip:
    paths:
       - /usr/share/GeoIP/GeoLiteCity.dat
    #  - "/usr/local/var/GeoIP/GeoLiteCity.dat"

On packet beat debugs logs I see below JSON

 Publish: {
  "@timestamp": "2016-03-18T17:03:50.993Z",
  "beat": {
    "hostname": "abc",
    "name": "abc"
  },
  "bytes_in": 2501,
  "bytes_out": 780,
  "client_ip": "10.81.x.x",
  "client_port": 44569,
  "client_proc": "",
  "client_server": "abc",
  "count": 1,
  "direction": "out",
  "http": {
    "code": 200,
    "content_length": 693,
    "phrase": "OK"
  },
  "ip": "40.78.x.x",
  "method": "POST",
  "params": "",
  "path": "/_bulk",
  "port": 443,
  "proc": "",
  "query": "POST /_bulk",
  "responsetime": 502,
  "server": "",
  "status": "OK",
  "type": "http"
}

On my host, i am running Docker container which has the private IP address. On eth0, i see public IP.Any help is appreciated.

Thx,
Padmaja

4

The client IP cannot be private.

5

@andrewkroh Thanks! I will check with my Devop team.
Thanks for you help!

6

@andrewkroh My host is running on Azur. The host does not know its public IP. How do we configure the packet beat to know host pubic IP?
Is there any configuration setting I can use?

Thx,
Padmaja

7

What are you trying to monitor?

A typical use case for geo-ip is monitoring requests to a your web servers. In this case the client IPs will be public. Often in this scenario the web-servers are located behind a load balancer so the client IP becomes the private IP of the load balancer. When this happens you can tell Packetbeat to not use the client IP from the request, but instead use the IP from the X-Forwarded-For header added to the request by the load balancer.

8

@andrewkroh

This is my use case.
I have two VM hosted on Azur at the different location. We are running our application on each host as docker container. docker container can receive packets from the internet and send packets to another container on the different host. I am running packet beat on both the VM host to capture the network traffic. Which gives us the pretty good view of latency, errors and Response times. But the only issue I was facing, I can not see geolocation on kibana due to private IP vs public IP. Kindly help me on this.

1020×694 39.1 KB

9

Where is the "client" in the diagram? What are the protocols? It seems odd that traffic flows into one VM but the return path is from a different VM.

When packets from the Internet hit the "worker docker container" they will pass through DNAT and this will modify their destination address to be the private IP of the docker container, but it will not modify the source address so it should remain as a public IP.

10

@andrewkroh

Clients can be in different locations. For example, Source is in US and Destination is in Belgium. Source and destination are sitting in a private network. VM's are providing tunneling service between source and destination. One VM close to the Source and other VM close to the destination. we are using Http (80) or Https (443). In packet beat, I am capturing traffic on 80 and 443 ports.

11

If this is the case, then the HTTP packets never have a public client IP address. What IP address did you expect to be able to geolocate?