Querying slow on kibana after re installation of docker

Elastic Stack Elasticsearch
1

Hi All,

After doing a clean reinstall of my ELK stack, querying the same reindexed data takes much longer, it now with saved queries that would be a few seconds previously now pop up with this:

Your query is taking awhile

Run beyond timeout

an example query: "corona virus" OR "covid" OR "covid-19" OR "coronavirus"

I havent changed anything in any of the config files other than i did try to upgrade from 7.7.1 > 7.8 before reinstalling again and going back to 7.7.1

Any ideas on what to check?

thankyou!

Tom

2

RAM & Heap are the same? This was all on Docker, so you used same Volumes? How did you 'go back' to 7.7.1? Just older container on same volume?

So now on the same version as before, just slower? Strikes me as RAM/Heap settings to the containers. Maybe swapping, etc.

3

Thanks Steve!

I havent changed the yml/yaml files and so the heapsize will be the same. I deleted all containers and volumes and then just did a fresh install of 7.7.1.

Essentially everything is exactly the same theoretically. I imagine its something to do with docker?

Thankyou

Tom

4

Hmm, no idea - and it was fine again going back to 7.7.1? All using volumes in Docker, so storage the same? Both versions were in Docker? Just looking for what can be different.

5

Yep both versions were in docker, but i did reinstall docker. yeah it was fine going to 7.7.1 because i deleted everything and then reinstalled (including the indexes)· the site is currently public if you want the direct url?

here's my yaml:

version: '3.2'

services:

elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.7.1
ports:
- 9200:9200
environment:
- node.name=elasticsearch
- discovery.type=single-node
- "ES_JAVA_OPTS=-Xms8000m -Xmx8000m"
- "LS_JAVA_OPTS=-Xmx3g"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- ./config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
- esdata:/usr/share/elasticsearch/data
networks:
- elk
restart: always

kibana:
image: docker.elastic.co/kibana/kibana:7.7.1
depends_on:
- elasticsearch
ports:
- 5601:5601
volumes:
- ./config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
- ./config/kibana-plugins:/usr/share/kibana/plugins:ro
- ./config/certs:/etc/certs
- kbndata:/usr/share/kibana/data
environment:
- ES_JAVA_OPTS="-Xmx8000m -Xms8000m"

6

Hmm, looks okay and simple enough - But you are saying the NEW 7.7.1 is slower than the old, i.e. BOTH 7.8 and new 7.7 are much worse than your old 7.7?

You are sure old container not have 16GB of HEAP or something? Or swap or other stuff going on with this VM? Makes no sense two identical 7.7 installs on same machine have vastly different performance - just can't be unless there is some big history, config, index/shards difference.

Was thinking about history like caching, though, i.e. old system was caching a lot - you of course have to run your query several times to get good results. Also maybe you query right after loading and have huge number of segments that have not yet merged, slowing things down.

If it's just 7.8 and you have disk space, you can setup up BOTH versions at once, load same data, and then query via curl (no need for Kibana) to just test apples to apples - would be nice test bed to help Elastic folks also compare and see 7.7 vs. 7.8

How much data, how many indexes, how many shards?

7

Hey - Yes both are worse.

I imagine it could be caching, but i didnt think it would slow down queries this much. is there anyway to force caching so i can see an increase?

The old container was built out of this exact yaml so it wouldnt be different.

The data is quite complex json, but there's only 38k documents on this particular index and each isnt that big. the index pattern has about 200 mappings. 1 shard, one replica.

I'd love to do a comparison but dont think i have the space unfortunately :frowning:
thanks alot for your help, any other ideas?

Tom

8

Hmm, no idea - for caching, just do the query once and then do it again, though if you are also writing it'll invalidate the cache on that shard/index. Note you won't get replicas on a one node system, so I'd suspect you'll be yellow?

9

yes its yellow, would another node improve performance?

10

No idea, as went from seconds to 10X longer or worse, it seems, with no obvious changes, so something sure changed, a lot. I wonder if old cluster had lots of settings tuning, on refresh, mappings, not sure what else that made it much faster; all would have been lost on new cluster.

Normally two nodes and thus splitting an index over twice as many CPUs, drives, etc. will improve performance, but you'd have to test.

11

thanks! it doesnt look like my queries are getting cached (the cache is empty) is there a way to check this properly?

12 
  "indices" : {
"open-opps-comp-house-post-mar-awards" : {
  "uuid" : "OCXC7jesSSmPdZ-SJLdeJA",
  "primaries" : {
    "docs" : {
      "count" : 1573,
      "deleted" : 0
    },
    "store" : {
      "size_in_bytes" : 23376734
    },
    "indexing" : {
      "index_total" : 0,
      "index_time_in_millis" : 0,
      "index_current" : 0,
      "index_failed" : 0,
      "delete_total" : 0,
      "delete_time_in_millis" : 0,
      "delete_current" : 0,
      "noop_update_total" : 0,
      "is_throttled" : false,
      "throttle_time_in_millis" : 0
    },
    "get" : {
      "total" : 0,
      "time_in_millis" : 0,
      "exists_total" : 0,
      "exists_time_in_millis" : 0,
      "missing_total" : 0,
      "missing_time_in_millis" : 0,
      "current" : 0
    },
    "search" : {
      "open_contexts" : 1,
      "query_total" : 2,
      "query_time_in_millis" : 76,
      "query_current" : 0,
      "fetch_total" : 1,
      "fetch_time_in_millis" : 737,
      "fetch_current" : 1,
      "scroll_total" : 0,
      "scroll_time_in_millis" : 0,
      "scroll_current" : 0,
      "suggest_total" : 0,
      "suggest_time_in_millis" : 0,
      "suggest_current" : 0
    },
    "merges" : {
      "current" : 0,
      "current_docs" : 0,
      "current_size_in_bytes" : 0,
      "total" : 0,
      "total_time_in_millis" : 0,
      "total_docs" : 0,
      "total_size_in_bytes" : 0,
      "total_stopped_time_in_millis" : 0,
      "total_throttled_time_in_millis" : 0,
      "total_auto_throttle_in_bytes" : 20971520
    },
    "refresh" : {
      "total" : 2,
      "total_time_in_millis" : 0,
      "external_total" : 2,
      "external_total_time_in_millis" : 1,
      "listeners" : 0
    },
    "flush" : {
      "total" : 0,
      "periodic" : 0,
      "total_time_in_millis" : 0
    },
    "warmer" : {
      "current" : 0,
      "total" : 1,
      "total_time_in_millis" : 1
    },
    "query_cache" : {
      "memory_size_in_bytes" : 0,
      "total_count" : 0,
      "hit_count" : 0,
      "miss_count" : 0,
      "cache_size" : 0,
      "cache_count" : 0,
      "evictions" : 0
    },
    "fielddata" : {
      "memory_size_in_bytes" : 0,
      "evictions" : 0
    },
    "completion" : {
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 1,
      "memory_in_bytes" : 106116,
      "terms_memory_in_bytes" : 76384,
      "stored_fields_memory_in_bytes" : 520,
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory_in_bytes" : 10048,
      "points_memory_in_bytes" : 0,
      "doc_values_memory_in_bytes" : 19164,
      "index_writer_memory_in_bytes" : 0,
      "version_map_memory_in_bytes" : 0,
      "fixed_bit_set_memory_in_bytes" : 0,
      "max_unsafe_auto_id_timestamp" : -1,
      "file_sizes" : { }
    },
    "translog" : {
      "operations" : 0,
      "size_in_bytes" : 55,
      "uncommitted_operations" : 0,
      "uncommitted_size_in_bytes" : 55,
      "earliest_last_modified_age" : 0
    },
    "request_cache" : {
      "memory_size_in_bytes" : 8897,
      "evictions" : 0,
      "hit_count" : 0,
      "miss_count" : 2
    },
    "recovery" : {
      "current_as_source" : 0,
      "current_as_target" : 0,
      "throttle_time_in_millis" : 0
    }
  },
  "total" : {
    "docs" : {
      "count" : 1573,
      "deleted" : 0
    },
    "store" : {
      "size_in_bytes" : 23376734
    },
    "indexing" : {
      "index_total" : 0,
      "index_time_in_millis" : 0,
      "index_current" : 0,
      "index_failed" : 0,
      "delete_total" : 0,
      "delete_time_in_millis" : 0,
      "delete_current" : 0,
      "noop_update_total" : 0,
      "is_throttled" : false,
      "throttle_time_in_millis" : 0
    },
    "get" : {
      "total" : 0,
      "time_in_millis" : 0,
      "exists_total" : 0,
      "exists_time_in_millis" : 0,
      "missing_total" : 0,
      "missing_time_in_millis" : 0,
      "current" : 0
    },
    "search" : {
      "open_contexts" : 1,
      "query_total" : 2,
      "query_time_in_millis" : 76,
      "query_current" : 0,
      "fetch_total" : 1,
      "fetch_time_in_millis" : 737,
      "fetch_current" : 1,
      "scroll_total" : 0,
      "scroll_time_in_millis" : 0,
      "scroll_current" : 0,
      "suggest_total" : 0,
      "suggest_time_in_millis" : 0,
      "suggest_current" : 0
    },
    "merges" : {
      "current" : 0,
      "current_docs" : 0,
      "current_size_in_bytes" : 0,
      "total" : 0,
      "total_time_in_millis" : 0,
      "total_docs" : 0,
      "total_size_in_bytes" : 0,
      "total_stopped_time_in_millis" : 0,
      "total_throttled_time_in_millis" : 0,
      "total_auto_throttle_in_bytes" : 20971520
    },
    "refresh" : {
      "total" : 2,
      "total_time_in_millis" : 0,
      "external_total" : 2,
      "external_total_time_in_millis" : 1,
      "listeners" : 0
    },
    "flush" : {
      "total" : 0,
      "periodic" : 0,
      "total_time_in_millis" : 0
    },
    "warmer" : {
      "current" : 0,
      "total" : 1,
      "total_time_in_millis" : 1
    },
    "query_cache" : {
      "memory_size_in_bytes" : 0,
      "total_count" : 0,
      "hit_count" : 0,
      "miss_count" : 0,
      "cache_size" : 0,
      "cache_count" : 0,
      "evictions" : 0
    },
    "fielddata" : {
      "memory_size_in_bytes" : 0,
      "evictions" : 0
    },
    "completion" : {
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 1,
      "memory_in_bytes" : 106116,
      "terms_memory_in_bytes" : 76384,
      "stored_fields_memory_in_bytes" : 520,
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory_in_bytes" : 10048,
      "points_memory_in_bytes" : 0,
      "doc_values_memory_in_bytes" : 19164,
      "index_writer_memory_in_bytes" : 0,
      "version_map_memory_in_bytes" : 0,
      "fixed_bit_set_memory_in_bytes" : 0,
      "max_unsafe_auto_id_timestamp" : -1,
      "file_sizes" : { }
    },
    "translog" : {
      "operations" : 0,
      "size_in_bytes" : 55,
      "uncommitted_operations" : 0,
      "uncommitted_size_in_bytes" : 55,
      "earliest_last_modified_age" : 0
    },
    "request_cache" : {
      "memory_size_in_bytes" : 8897,
      "evictions" : 0,
      "hit_count" : 0,
      "miss_count" : 2
    },
    "recovery" : {
      "current_as_source" : 0,
      "current_as_target" : 0,
      "throttle_time_in_millis" : 0
    }
  }
}

}
}

13

I don't know; it may not cache everything, especially if too large - you can see settings from GET /_cluster/settings and the defaults with:

GET /_cluster/settings?include_defaults=true&flat_settings=true

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.