Will kibana work if i am trying to index a file of 6.5 GB having 3 million records

Elastic Stack Kibana
1

Hi Team,

Will kibana work if i am trying to index a file of 6.5 GB having 3 million records.
When i try to query the data i get 3000ms request timed out error and application status goes down having status as red

2

Shouldn't be a problem. What does the query look like? What happens if you run the same query directly against elasticsearch?

3

I get the following error
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Trying to query 1469 shards, which is over the limit of 1000. This limit exists because querying many shards at the same time can make the job of the coordinating node very CPU and/or memory intensive. It is usually a better idea to have a smaller number of larger shards. Update [action.search.shard_count.limit] to a greater value if you really want to query that many shards at the same time."
}
],
"type": "illegal_argument_exception",
"reason": "Trying to query 1469 shards, which is over the limit of 1000. This limit exists because querying many shards at the same time can make the job of the coordinating node very CPU and/or memory intensive. It is usually a better idea to have a smaller number of larger shards. Update [action.search.shard_count.limit] to a greater value if you really want to query that many shards at the same time."
},
"status": 400
}

query is:
GET _search
{
"query": {
"match_all": {}
}
}

4

Seems like a lot of shards. How many indices are you creating, and how many shards per index?

5

I recommend reducing the number of shards using _shrink and then seeing if the query works.

6

Hi,

I am not sure about shards..i have my data in logstash which is filtered and output config file is as follows:
output {
if [type] == "fsimagedaily" {
elasticsearch {
hosts => ["localhost:9200" ]
index => "fsimage-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["localhost:9200" ]
}
}

When i navigate to kibana UI it opens up but in management index patter tab i see the below:

image
image.png1164×743 62 KB

and in dev tools tab i get the above error pasted not sure where i am going wrong..

7

Just to clarify, are you saying that when you visit the index pattern creation page in Kibana, you get the error pasted above? Where do you see the error? In your browser's dev tools?

Or are you getting that error when running a query in kibana's Dev Tools app?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.