WHY index stops accepting any documents

nskalis · May 24, 2016, 10:32am

Hi guys,

I would really need your advice on this one;

There is 1 elastic search node with 4 indices. kopf shows

After 22h, the __10 index stops indexing documents, while the other ones operate as expected

The data are getting indexed using bulk requests via HTTP and this the log when "not working"
Responce is 200

2016-05-24 12:26:36.910 : DEBUG : IPacc-datastore : bin_packing_cache : > process_id "31922768" [IPacc:cache] found data "IPacc:cache:__10:pmacct" and calculated "2" bins of size "150000" plus "41974" messages
2016-05-24 12:27:05.100 : DEBUG : IPacc-datastore : bin_packing_cache : > {'req': <Response [200]>, 'pid': 31922768, 'rc': True}
2016-05-24 12:27:05.631 : DEBUG : IPacc-datastore : bin_packing_cache : > {'req': <Response [200]>, 'pid': 31922768, 'rc': True}
2016-05-24 12:27:05.886 : DEBUG : IPacc-datastore : bin_packing_cache : > {'req': <Response [200]>, 'pid': 31922768, 'rc': True}

Could you please please help me and try to find what is going wrong ?

Nikos

Christian_Dahlqvist · May 24, 2016, 10:40am

It looks like you are hitting the Lucene limit of 2 billion documents per shard. Overall the shards look very large, so I would recommend using a higher number of shards for each index. If you tell us a bit more about your use case we may be able to provide better advice.

nskalis · May 24, 2016, 11:09am

Thanks a lot @Christian_Dahlqvist
I was not aware of that limit. And I reach that in ~22h.
The objective is to hold data for ~1w or so.
I have defined also a _ttl equal to 7d.

I guess setting the number of shards to 15 would suffice ?
Could you possibly advise on any other setting that may needs tweaking ?

Below are the settings and the mapping:

nskalis · May 24, 2016, 11:10am

{
  "IPacc": {
    "_routing": {
      "required": true
    },
    "_ttl": {
      "default": 604800000,
      "enabled": true
    },
    "dynamic": "true",
    "_all": {
      "enabled": false
    },
    "properties": {
      "dev_name": {
        "index": "not_analyzed",
        "type": "string"
      },
      "dst_as": {
        "type": "integer"
      },
      "dst_mask": {
        "type": "integer"
      },
      "local_pref": {
        "type": "integer"
      },
      "time_stamp": {
        "format": "epoch_second",
        "type": "date"
      },
      "peer_dst_as": {
        "type": "integer"
      },
      "peer_src_as": {
        "type": "integer"
      },
      "src_mask": {
        "type": "integer"
      },
      "dst_ip": {
        "index": "not_analyzed",
        "type": "string"
      },
      "packets": {
        "type": "long"
      },
      "peer_src_ip": {
        "index": "not_analyzed",
        "type": "string"
      },
      "src_ip": {
        "index": "not_analyzed",
        "type": "string"
      },
      "peer_dst_ip": {
        "index": "not_analyzed",
        "type": "string"
      },
      "med_metric": {
        "type": "integer"
      },
      "bytes": {
        "type": "long"
      },
      "ifindex_in": {
        "type": "integer"
      },
      "ifindex_out": {
        "type": "integer"
      },
      "proto": {
        "index": "not_analyzed",
        "type": "string"
      },
      "tos": {
        "type": "integer"
      },
      "as_path": {
        "type": "string"
      },
      "bgp_comm": {
        "type": "string"
      },
      "src_as": {
        "type": "integer"
      }
    }
  },
  "_default_": {
    "_routing": {
      "required": true
    },
    "_ttl": {
      "default": 604800000,
      "enabled": true
    },
    "dynamic": "true",
    "_all": {
      "enabled": false
    },
    "properties": {
      "dev_name": {
        "index": "not_analyzed",
        "type": "string"
      },
      "dst_as": {
        "type": "integer"
      },
      "dst_mask": {
        "type": "integer"
      },
      "local_pref": {
        "type": "integer"
      },
      "time_stamp": {
        "format": "epoch_second",
        "type": "date"
      },
      "peer_dst_as": {
        "type": "integer"
      },
      "peer_src_as": {
        "type": "integer"
      },
      "src_mask": {
        "type": "integer"
      },
      "dst_ip": {
        "index": "not_analyzed",
        "type": "string"
      },
      "packets": {
        "type": "long"
      },
      "peer_src_ip": {
        "index": "not_analyzed",
        "type": "string"
      },
      "src_ip": {
        "index": "not_analyzed",
        "type": "string"
      },
      "peer_dst_ip": {
        "index": "not_analyzed",
        "type": "string"
      },
      "med_metric": {
        "type": "integer"
      },
      "bytes": {
        "type": "long"
      },
      "ifindex_in": {
        "type": "integer"
      },
      "ifindex_out": {
        "type": "integer"
      },
      "proto": {
        "index": "not_analyzed",
        "type": "string"
      },
      "tos": {
        "type": "integer"
      },
      "as_path": {
        "type": "string"
      },
      "bgp_comm": {
        "type": "string"
      },
      "src_as": {
        "type": "integer"
      }
    }
  }
}

nskalis · May 24, 2016, 11:10am

{
  "index": {
    "cache": {
      "field": {
        "type": "soft"
      },
      "filter": {
        "type": "node"
      },
      "query": {
        "enable": "true"
      }
    },
    "memory": {
      "index_buffer_size": "20%"
    },
    "refresh_interval": "1m",
    "fielddata": {
      "cache": {
        "type": "node"
      }
    },
    "number_of_shards": "1",
    "merge": {
      "scheduler": {
        "type": "concurrent"
      },
      "async": "true",
      "policy": {
        "type": "log_byte_size",
        "merge_factor": "15"
      }
    },
    "creation_date": "1463945724550",
    "store": {
      "throttle": {
        "type": "none"
      }
    },
    "number_of_replicas": "0",
    "uuid": "qbzWKQW4TMyKq_H1AirNXQ",
    "version": {
      "created": "2030299"
    }
  }
}

Christian_Dahlqvist · May 24, 2016, 11:16am

You may want to look into using time-based indices instead as this is significantly more efficient than using TTL (which is being deprecated).

nskalis · May 24, 2016, 11:22am

Thanks . Indeed I read the guideline. But time-based indices do not 'fit' well in my data pipeline because of the bulk insertion. According to the documentation they will replace it with another mechanism (retention policy) instead of the _ttl.

Taking advantage of your prompt replies;
May you have any suggestions in regards to any shards settings, etc ? That need to be setup in a better way ?

Christian_Dahlqvist · May 24, 2016, 11:26am

I do not see how bulk inserts are related to whether you use time based indices or not? Could you please elaborate?

nskalis · May 24, 2016, 11:34am

Because in order to post documents in a time-based index, the index has to be created first.
I am not using log stash that can create the index automatically.
And not all of the docs when I do a bulk request have the same timestamp.
I would like to avoid to set up indices before hand with specific dates.
Or there is another way of doing it ?

Christian_Dahlqvist · May 24, 2016, 11:39am

Indices are created when you first write data to them, so you do not need to explicitly create them. Logstash uploads an index template that specifies how it is to be created and then just writes to the correct index pattern. If the index does not exist, it is created automatically based on the template.

nskalis · May 24, 2016, 11:58am

But how do you set the mapping of the data ? I know that you can do that also per request. But is more efficient if I create the index and just post the data. In my case, on every index there is 1 mapping.
So what will happen when is close to midnight and there is a bulk of 150k docs to be pushed ?
The only option is to group them by date before i do post, and if the mapping is not there to create it, correct ?

Christian_Dahlqvist · May 24, 2016, 12:05pm

The index template contains the mapping as well as the index settings. If you send a bulk request inserting documents that do not already exist, these will be automatically created according to the index template, so you do not need to group the data and batch it up based on timestamp.

This is what the default logstash index template looks like, and as you can see it contains index settings as well as mappings to be used.

nskalis · May 24, 2016, 12:16pm

Am sorry am not getting it..

Practically speaking, let's say that I have an index named xxx10
And I would like to post documents using time-based indices.
How the request would look like ? What will be the name of the index ?

magnusbaeck · May 29, 2016, 1:01pm

Practically speaking, let's say that I have an index named xxx10
And I would like to post documents using time-based indices.
How the request would look like ? What will be the name of the index ?

Your indexing request would post the data to an index named something like xxx10-YYYY.MM.DD (e.g. xxx10-2016.05.29). If that index doesn't exist it'll be created. If there's an index template that matches the name of the index it'll get applied. You'd have to figure out the date to put in the index name.

nskalis · May 29, 2016, 2:41pm

thanks a lot @magnusbaeck i had the same impression as well. so
either i have to create the index before hand
or make the aggregate post by separating the data based on timestamp and create the index if necessary.

magnusbaeck · May 29, 2016, 3:14pm

No, I still don't think you're getting it. You never have to explicitly create the index. All you have to do is change the index name in your requests (bulk or not) to include the current date.