Help for a Newbie

Elastic Stack Elasticsearch
1

Hi All,

I just start with ES to collect data from a netflow sensor for our company. I have implement nprobe to send data, everything is ok but the template given by nprobe is not compatible with the last version of ES. I need some help to adapt the template, i hope everyone can help me. Here the original template that is compatible with an older version of ES :

{
  "template" : "nprobe-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true, "omit_norms" : true},
       "dynamic_templates" : [ {
             "string_fields" : {
               "match" : "*",
               "match_mapping_type" : "string",
               "mapping" : {
                 "type" : "string", "index" : "analyzed", "omit_norms" : true,
                   "fields" : {
                     "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
                   }
               }
             }
       }, {
             "geo_fields" : {
               "match" : "*_IP_LOCATION",
               "mapping": {
                      "type": "geo_point"
                }
             }
       }, {
             "ip_fields" : {
               "match" : "IPV4_*",
               "match_mapping_type" : "string",
               "mapping": {
                      "type": "ip"
                }
             }
       } ],
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" }
       }
    }
  }

and my adaptation :

PUT nprobe
{
  "mappings" : {
    "_default_" : {
       "_all" : {
         "norms" : true
       },
       "dynamic_templates" : [ 
         {
             "strings" : {
               "match_mapping_type": "string",
               "match" : "*",
               "mapping" : {
                 "type" : "text", "index" : true , "norms" : true,
                   "fields" : {
                     "raw" : {"type": "keyword", "index" : false, "ignore_above" : 256}
                   }
               }
             }
       }, {
             "geo_fields" : {
               "match" : "*_IP_LOCATION",
               "mapping": {
                      "type": "geo_point"
                }
             }
       }, {
             "ip_fields" : {
               "match" : "IPV4_*",
               "match_mapping_type" : "string",
               "mapping": {
                      "type": "ip"
                }
             }
       } ],
       "properties" : {
         "@version": { "type": "text", "index": false }
       }
    }
  }
}

the result :

#! Deprecation: [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
#! Deprecation: [_default_] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "nprobe"
}

I don't know about the _copyto and where to put it.
Other, my nprobe create each day a new index : nprobe-%d-%m-%Y i have try to create a mapping with :
`

PUT nprobe*

but don't work, thank you in advance for any help.

`

2

Replace _default_ by doc and remove the _all part of the mapping and you should be good.

3

To collect, store and analyze Netflow (as well as IPFIX and sFlow) you may also want to take a look at ElastiFlow...

4

Thank you rcowart & dadoonet, i think i will try it after accomplish nprobe with ES.

dadoonet, i do what you tell me and i think it's ok. The netflow is sended to ES and ES create an index with good mapping. But since i have a template, it's impossible to view something in the visualize, kibana ask me to create an index but there is already an index created.

Here my commande GET _all:

{
  "fol57-25-02-2018": {
    "aliases": {},
    "mappings": {
      "doc": {
        "dynamic_templates": [
          {
            "strings": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "fields": {
                  "raw": {
                    "ignore_above": 256,
                    "index": false,
                    "type": "keyword"
                  }
                },
                "index": true,
                "norms": true,
                "type": "text"
              }
            }
          },
          {
            "geo_fields": {
              "match": "*_IP_LOCATION",
              "mapping": {
                "type": "geo_point"
              }
            }
          },
          {
            "ip_fields": {
              "match": "IPV4_*",
              "match_mapping_type": "string",
              "mapping": {
                "type": "ip"
              }
            }
          }
        ],
        "properties": {
          "@version": {
            "type": "text",
            "index": false
          }
        }
      }
    },
    "settings": {
      "index": {
        "refresh_interval": "5s",
        "number_of_shards": "5",
        "provided_name": "fol57-25-02-2018",
        "creation_date": "1519524968159",
        "number_of_replicas": "1",
        "uuid": "NSPdPtNBQvmwGID70KbzEw",
        "version": {
          "created": "6020299"
        }
      }
    }
  }

We can see that there is a new index with the good mapping. What's the next step to see the data now ?

If i delete the template, kibana give me the possibility to add a new index with fol57*

Thank you in advance

5

Index data in this index I guess...

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.