Correct src_ip field datatype to 'ip'

Elastic Stack Elasticsearch
1

Hello,
I have only recently discovered the field datatype 'ip' I have most of the my IP addresses labeled as 'src_ip' looking at the current data type it's a 'string'.

  1. Can i force the change somehow?
  2. How do i fix it?

Deep down I realise the answer will be re-index the data. Unfortunately I can't just delete all the indexes on a production system. Is there a way to reindex the current data from elastic itself - Is the process documented anywhere.

Thanks
Cam

2

Hi,

AFAIK, you need to reindex your data with a new mapping. You can use the following API:

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/docs-reindex.html

bye,
Xavier

3

So knowing that I need to reindex my data, what else do I need to do to achieve the correct mapping moving forward?

  1. I feel I need to update the 'index template' and assign src_ip as type 'ip'.
    Whats the best way to go about this?
  • Should I update the default 'dynamic template' ?
  • Should I overwrite the default template?
  • Should I overwrite the logstash template?
  • Should I append to the logstash template with an ordering of "1"?

Considering that I have 4 current index patterns, eg. logstash-, firewall-, syslog-, docker-

I'm tempted to set the default index-template. Does this have any repercussions?

So to correct the fieldtype is this the order of events?

  • Update default index-template
  • Reindex data using 'reindex API'

Anything else?

Thanks in advance

4

Hi,

I would suggest to modify only templates you need. If they all depend of the default one, why not, but you have to be careful. Mabye you can test on the logstash one in a first time ?

The plan is:

  • update your templates
  • create new index and verify the mapping
  • call the reindex API

Hope it helps

5

So I'm attempting to use a dynamic template, this is it here:

GET /_template/mylogstash
    {
  "my_logstash": {
    "order": 1000,
    "index_patterns": [
      "logstash-*",
      "firewall-*"
    ],
    "settings": {},
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "dest_ip": {
              "path_match": "dest_ip",
              "mapping": {
                "type": "ip"
              }
            }
          },
          {
            "dst_ip": {
              "path_match": "dst_ip",
              "mapping": {
                "type": "ip"
              }
            }
          },
          {
            "src_ip": {
              "path_match": "src_ip",
              "mapping": {
                "type": "ip"
              }
            }
          }
        ]
      }
    },
    "aliases": {}
  }
}

Even using a regular non-dynamic template, Kibana? fails to set the type as 'ip'
Example: normal template:

{
"order": 1000,
"index_patterns": [
  "logstash-*",
  "firewall-*"
],
"mappings": {
  "_default_": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "dest_ip": {
        "type": "ip"
      },
      "dst_ip": {
        "type": "ip"
      },
      "src_ip": {
        "type": "ip"
      },
      "zsrc_ip": {
        "type": "ip"
      },      
      "@version": {
        "type": "keyword"
      }
    }
  }
},
"aliases": {}
}

It's inserted into Elastic no worries. And I index some test data into an index where the fieldname does not exist yet.
I then search for the data, I see the new field name is not known, and Kibana asks me to refresh field list, I refresh the field list, at which point kibana reports the field name eg. 'dst_ip' as a string.

What am I doing wrong? I want kibana to recognise this as type: 'ip'

6

Sorry but I'm not very aware of the version 6.x and default templating... :frowning:

7

I got the template working.
This was the template I used:

PUT /_template/mylogstash
{
"order": 1000,
"index_patterns": [
  "logstash-*",
  "docker-*",
  "syslog-*",
  "ironport-*",
  "firewall-*"
],
"mappings": {
  "doc": {
    "dynamic": "true",
    "properties": {
      "host": {
        "type": "keyword"
      },
      "program": {
        "type": "keyword"
      },
      "logsource": {
        "type": "keyword"
      },
      "bytes": {
        "type": "integer"
      },
      "bytes_in": {
        "type": "integer"
      },
      "bytes_out": {
        "type": "integer"
      },
      "dest_port": {
        "type": "integer"
      },
      "src_port": {
        "type": "integer"
      },
      "dest_ip": {
        "type": "ip"
      },
      "src_ip": {
        "type": "ip"
      },
      "geoip": {
        "properties": {
          "ip": {
            "type": "ip"
          },
          "location": {
            "type": "geo_point"
          },
          "latitude": {
            "type": "half_float"
          },
          "longitude": {
            "type": "half_float"
          }
        }
      }
    }
  }
}
}

Not sure if i'm going to re-index my data, as the indexes roll over daily. Just the 'conflict' in Kibana makes me sad.

8

I actually had to fix some things regarding 'host.keyword' disappearing and not 'aggreagatable' which i had to fix with an another template update, documented here:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.