Elasticsearch input plugin size

[JFO_90]

Hi,

I have a logstash configuration from which I'm trying to extract elasticsearch information with the elasticsearch input plugin. The problem that i have is that despite indicating a size, it always returns all the documents from the index has instead of the amount indicated in the index.

input {
elasticsearch {
hosts => ["localhost:9200"]
index => "myindex"
query => '{ "query": {"match_all": {}} }'
size => 100
docinfo => true
}
}

In the configuration you can see how I indicate a size of 100 but ignore this value and return all the documents of the index.

Can someone help me with this problem?

Best regards

[Badger]

I think the size option only takes effect if scroll is enabled. In which case the input will repeatedly make calls to the search API until all of the documents that match the query are returned.

If you want to limit the number of documents returned by the query you need to tell elasticsearch that. I think that would be

query => '{ "query": {"match_all": {}}, "size": 100 }'

[JFO_90]

I have tried with the scroll field and with the size in the query and it keeps returning all the docs.

input {
elasticsearch {
hosts => "localhost:9200"
index => "myindex"
query => '{ "query": {"match_all": {}}, "size": 100 }'
docinfo => true
}
}

input {
elasticsearch {
hosts => ["localhost:9200"]
index => "myindex"
query => '{ "query": {"match_all": {}} }'
size => 100
scroll => "5m"
docinfo => true
}
}

Both return all the docs of the index.

[Badger]

The second one is expected to return all documents. The input will loop fetching 100 documents at a time until it has gotten them all.

I think I had size in the wrong place. Try

query => '{ "query": {"match_all": {}, "size": 100}}'

[JFO_90]

Y tried
elasticsearch {
hosts => "localhost:9200"
index => "myindex"
query => '{ "query": {"match_all": {}, "size": 100}}'
docinfo => true
}

And the result was

E Error: [400] {"error":{"root_cause":[{"type":"parsing_exception","reason":"[match_all] malformed query, expected [END_OBJECT] but found [FIELD_NAME]","line":1,"col":26}],"type":"parsing_exception","reason":"[match_all] malformed query, expected [END_OBJECT] but found [FIELD_NAME]","line":1,"col":26},"status":400}

[Badger]

Looking at an example from the input documentation

query => '{ "query": { "match": { "statuscode": 200 } }, "sort": [ "_doc" ] }'

"sort" is at the same level as "query" so I had it right the first time. I don't know DSL well enough to know if size can be used in the same way. I guess not.

[JFO_90]

I was trying with different queries like this

query => '{
"query": {
"bool": {
"filter": [
{
"terms": {
"enviroment": [
"pre"
]
}
}
]
}
},
"size": 100
}'

This DSL query return results, but not 100 as i said on the size, this query return more than 100 results.
It seems that the size field does not work as indicated in the beginning of the topic

[JFO_90]

Has anyone encountered this problem or knows how to solve it?

Best regards