Hello!
What JVM and OS are you using?
We'r using Centos 7
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)**
What does the mapping look like?
{
"http_template" : {
"order" : 0,
"template" : "http*",
"settings" : {
"index" : {
"analysis" : {
"analyzer" : {
"string_analyzer" : {
"filter" : [ "lowercase", "asciifolding" ],
"tokenizer" : "keyword"
}
}
},
"number_of_shards" : "4",
"number_of_replicas" : "1"
}
},
"mappings" : {
"default" : {
"dynamic_templates" : [ {
"string_template" : {
"mapping" : {
"analyzer" : "string_analyzer",
"type" : "string"
},
"match_mapping_type" : "string",
"match" : "*"
}
} ],
"_all" : {
"enabled" : false
}
}
},
"aliases" : { }
}
}
What are the aggs?
We have many different aggs, one example is:
{
"size": 0,
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"range": {
"fireEventDate": {
"from": "{{startDate}}",
"to": "{{endDate}}",
"include_lower": true,
"include_upper": false
}
}
}
]
}
}
}
},
"aggregations": {
"host": {
"terms": {
"field": "host",
"size": 0,
"order": {
"_term": "asc"
}
},
"aggregations": {
"ipDst": {
"terms": {
"field": "ipDst",
"size": 0,
"order": {
"_term": "asc"
}
},
"aggregations": {
"country": {
"terms": {
"field": "country",
"size": 0,
"order": {
"_term": "asc"
}
},
"aggregations": {
"min": {
"min": {
"field": "fireEventDate"
}
},
"max": {
"max": {
"field": "fireEventDate"
}
}
}
}
}
}
}
}
}
}
What is the node config?
...
cluster.name: myApp
node.name: node-1
path.data: /var/custom/elastic/data
bootstrap.mlockall: true
http.port: 9201
transport.tcp.port: 9301
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9399"]
discovery.zen.ping.multicast.enabled: false
script.inline: true
script.indexed: true
cluster.routing.allocation.disk.threshold_enabled: True
cluster.routing.allocation.disk.watermark.low: 30gb
cluster.routing.allocation.disk.watermark.high: 20gb
indices.fielddata.cache.size: 75%
indices.breaker.fielddata.limit: 85%
node.master: false
node.data: true
http.enabled: false
We tried too without cache.size and limit or with very low percentage
indices.fielddata.cache.size: 10%
indices.breaker.fielddata.limit: 15%
We tried two data nodes with 16GB the heap or four data nodes with 6GB or 8GB for the heap
We have one dedcated master node with 1GB and one dedicated client node. We tried 1G to 8GB of RAM for the dedicated client
How much data do you have? How many indices, shards?
We have two log types (http and DNS), we create one index per day (with about 80.000.000 hits and 15GB per index, sometimes is less),
but we tried diffferent configurations with similar results:
- 3 Shards and one replica for index, 1 index per day (about 20GB-30GB per index)
- 4 Shards and one replica for index, 1 index per day (about 20GB-30GB per index)
- 5 Shards and one replica for index, 1 index per day (about 20GB-30GB per index)
- 3 Shards and one replica for index, 4 index per day (about 5GB-6GB per index)
- 4 Shards and one replica for index, 4 index per day (about 5GB-6GB per index)
- 4 Shards and one replica for index, 24 index per day (about 1GB per index)
We have other kind of logs with less shards (1 and 1 replica).
In this moment:
{
"cluster_name" : "myApp",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 6,
"number_of_data_nodes" : 4,
"active_primary_shards" : 1047,
"active_shards" : 2094,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
This is more or less 20 days, but at least we need four times more retention.
thank you very much!
this time) :