How can ı drawing ip based map

Elastic Stack Kibana
1

Hi everyone
I want to draw ip map but it gives an error. I use model ip type for srcIp.

"csIp": {
      "type": "ip"
    }

The error I received is as follows.

Screenshot
Screenshot.png1729×826 262 KB

2

IPs do not contain location information so you need to enrich the events with this information when indexing the data, e.g. through the Logstash Geoip plugin or an ingest node processor.

3

Hi Christian
I installed ingest geo_ip for elasticsearch plugin. I use filter as follows.

filter {
  if [fileset][module] == "nginx" {
    if [fileset][name] == "access" {
      grok {
        match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\""] }
        remove_field => "message"
      }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
        remove_field => "[nginx][access][time]"
      }
      useragent {
        source => "[nginx][access][agent]"
        target => "[nginx][access][user_agent]"
        remove_field => "[nginx][access][agent]"
      }
      geoip {
        source => "[nginx][access][remote_ip]"
        target => "[nginx][access][geoip]"
      }
    }
    else if [fileset][name] == "error" {
      grok {
        match => { "message" => ["%{DATA:[nginx][error][time]} \[%{DATA:[nginx][error][level]}\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\*%{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}"] }
        remove_field => "message"
      }
      mutate {
        rename => { "@timestamp" => "read_timestamp" }
      }
      date {
        match => [ "[nginx][error][time]", "YYYY/MM/dd H:m:s" ]
        remove_field => "[nginx][error][time]"
      }
    }
  }
}

I don't understand any progress :frowning:

4

I use json model

{
"csIp": "92.123.105.156",
}
5

None of the configuration you posted enrich IP based on the csIp field in your data. As your field is not what is assumed by the config you posted you need to change it.

6

How can ı any progress your idea? Can you help me ? ı need to use logstash when you first sent

7

Try something like this:

geoip {
  source => "[csIp]"
}
8

Hi Christian
I can't. I don't know problem where is it. I can tell you approach

  1. I have created my model, it is name akamai
  2. I send data post to http://....:9092/akaami { json exmpler}
  3. IMy model csIp: ip but system can't find ip area for visualization
  4. I want to design ip map for my model
    Can you help me?
9

You need to enrich and add geo location (see my example) fields to your event at indexing time and plot based on these added fields.

Have a look at this blog post:

1 Like
10

I don't find correct solution. I don't know where ı made a mistake. ı use bellow model

PUT :9092/xxx 
{
      "mappings": {
        "dtakamai": {
          "properties": {
            "csCookie": {
              "type": "text"
            },
            "csIp": {
              "type": "ip"
            },
            "csMethod": {
              "type": "keyword"
            },
            "csReferer": {
              "type": "text"
            }
          }
        }
      }
    }

I use data

POST akamai/dtakamai
{
  "dateTime": "2019-03-27T15:06:50",
  "csIp": "104.194.203.69",
  "csMethod": "DELETE",
  "csUri": "/bxxx.com",
  "scStatus": 200,
}

Log stash conf

filter {
	geoip {
	  source => "[csIp]"
	}
}

I direct send data to elasticsearch:9092/akamai. Now I think where is the error?

11

Hi
found as a below exampler mapping

PUT _ingest/pipeline/geoip
{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "ip"
      }
    }
  ]
}
PUT my_index/my_type/my_id?pipeline=geoip
{
  "ip": "8.8.8.8"
}
GET my_index/my_type/my_id

But ı want to designe my own mapping. How Can I do similary mapping? I'd like to use field csUseragent, csReferer vb...

12

Transforming data can be done in Logstash, so I would recommend you have a look at this blog post which describes the process. Once you have understood how this works, you should be able to transform this into a more complete ingest node pipeline if you want to as a lot of the filters are similar (although the configuration does differ).

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.