Hi I have created rollup Job as per the specification given below. This will roll up data on an hourly basis. I have also created a rollup index pattern combining the rollup index and source/original index pattern. Please note ILM has not been implemented, so the original index still contains all data, no data is deleted.
Following are my queries
-
while creating a visualization (Data Table), the default "count" metrics is shown as 0. However if I select a "term" aggregation (eg: operationName) in buckets for splitting, it shows the count. Why count is shown "0" by default?
-
If I select "Average" metric for field "duration" without any bucketing (Split Rows) it shows the result. However if I apply bucketing (eg: "term" aggregation on field "operationName") elastic throws error "Rollup search Error: An internal server error occurred". However instead of "average" if I select "Min", "Max" for field "Duration" it shows the result.
-
While rolling up I can select Avg,Min, Max etc for a filed in Metrics section. I have also requirement for percentile of values, say 90th, 95th percentile of "duartion" field. How I can do that in elastic rollup job?
-
Currently I am rolling up data hourly, I have also requirement for rolling up data for Daily, weekly,Monthly etc also. If I create different roll-up jobs for the same with different index how I can combine them all in rollup index pattern? As per the elastic documentation while creating "Rollup index Pattern" you can select only one "roll up index" and rest should be the normal index.
Currently I am working on kibana 7.5.2 version.
Please find below the rollup job details, Let me know if I have make any change in the job.
{
"config": {
"id": "rollup_15jun-UTC",
"index_pattern": "jaeger-*",
"rollup_index": "rollup_test_bri_15jun",
"cron": "0 0 * * * ?",
"groups": {
"date_histogram": {
"fixed_interval": "60m",
"field": "startTimeMillis",
"delay": "4d",
"time_zone": "UTC"
},
"terms": {
"fields": [
"process.serviceName",
"operationName"
]
}
},
"metrics": [
{
"field": "duration",
"metrics": [
"avg",
"max",
"min",
"sum",
"value_count"
]
}
],
"timeout": "20s",
"page_size": 1000
},
"status": {
"job_state": "stopped",
"current_position": {
"operationName.terms": "ras dashboard",
"process.serviceName.terms": "ui-web",
"startTimeMillis.date_histogram": 1591873200000
},
"upgraded_doc_id": true
},
"stats": {
"pages_processed": 162,
"documents_processed": 10715691,
"rollups_indexed": 159060,
"trigger_count": 2,
"index_time_in_ms": 19975,
"index_total": 160,
"index_failures": 0,
"search_time_in_ms": 81818,
"search_total": 162,
"search_failures": 0
}
}