Hi everybody,
I have currently massive problems with my elasticsearch setup. I store aggregated google analytics data in elastic search for further processing. There are currently around 450mio documents in this index.
The problem I have that the aggregate queries that I need to perform on the index sometimes take longer than 60sec which results in a "Gateway Timeout".
But the much bigger problem is that the spark-jobs that I use to add data once a day as a batch started failing on me because it seems that the index can't keep up inserting documents. This basically makes the data inconsistent and of not that much worth.
When I started I didn't configure much as shown below. But after the first problems occurred I started researching indexes and shards and so on... also I read that it is a good idea to split the index in monthly indexes.
So that was my next and current try. But that performs even worse for lookup. Haven't tested it for writing tho.
Since the amount of data is quite large and copying data from one index to another takes quite some time. There are not that many shots that I can take. That's why some help would really be appreciated.
I started with a very basic index without much configuration:
{
"kpi_dashboard" : {
"aliases" : { },
"mappings" : {
"properties" : {
"date" : {
"type" : "date"
},
"fullVisitorId" : {
"type" : "keyword",
"eager_global_ordinals" : true
},
"page_hostname" : {
"type" : "keyword"
},
"portfolio" : {
"type" : "keyword",
"eager_global_ordinals" : true
},
"session_id" : {
"type" : "keyword"
},
"time_on_host" : {
"type" : "long"
},
"visitId" : {
"type" : "keyword"
}
}
},
"settings" : {
"index" : {
"creation_date" : "1583354059301",
"number_of_shards" : "5",
"number_of_replicas" : "1",
"uuid" : "XSHICZ1gQICzHfbSmBKajA",
"version" : {
"created" : "7010199"
},
"provided_name" : "kpi_dashboard"
}
}
}
}
Then used a monthswise index like this:
{
"kpi-dashboard-traffic-2019-01-00001" : {
"aliases" : {
"kpi-dashboard-traffic-read" : { }
},
"mappings" : {
"properties" : {
"date" : {
"type" : "date"
},
"fullVisitorId" : {
"type" : "keyword",
"eager_global_ordinals" : true
},
"page_hostname" : {
"type" : "keyword"
},
"portfolio" : {
"type" : "keyword",
"eager_global_ordinals" : true
},
"session_id" : {
"type" : "keyword",
"eager_global_ordinals" : true
},
"time_on_host" : {
"type" : "long"
},
"visitId" : {
"type" : "keyword"
}
}
},
"settings" : {
"index" : {
"number_of_shards" : "1",
"provided_name" : "kpi-dashboard-traffic-2019-01-00001",
"creation_date" : "1583998849339",
"sort" : {
"field" : [
"date",
"portfolio",
"session_id"
],
"order" : [
"desc",
"desc",
"desc"
]
},
"number_of_replicas" : "1",
"uuid" : "T1ZMx5DHSEie8y7qC_OlgA",
"version" : {
"created" : "7010199"
}
}
}
}
}
The Elasticsearch Cluster is currently running on AWS with 1 master and 1 data node in each of 3 availability zones. (so 6 servers)
masters are r5.large
datanodes are r5.2xlarge
Thank you very much.