[ElasticSearch Query] Select all not null fields

[amoussoubaruch]

Hi evryone,

I face some issue and i don't really know how i can solve that.
I have one index with more than 600 fields. Sometimes i can have arround 500 fields with null values. I want to make a select query and have a result without all null fields.

For exemple, suppose i have this index :

{
"took": 11,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 4392822,
"max_score": 1,
"hits": [
{
"_index": "test_index",
"_type": "test_indextype",
"_id": "MHbBSGwBfQlFPBBvF79Y",
"_score": 1,
"_source": {
"field1": "Value1",
"f_id": "17AEBA",
"code": "DT10",
"l_uc": "INJ1",
"d_id": 23765,
"Info1": null,
"Info2": null,
"Info3": null,
"Info4": null,
"Info5": null,
"Info6": null,
"Info7": null,
"Info8": null,
"Info9": null,
"Info10": null,
"Info11": null,
"Info12": null,
"Info13": null,
"Info14": null,
"Info15": null,
"Info16": null
}}

I want a query to have only this :

{
"took": 11,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 4392822,
"max_score": 1,
"hits": [
{
"_index": "test_index",
"_type": "test_indextype",
"_id": "MHbBSGwBfQlFPBBvF79Y",
"_score": 1,
"_source": {
"field1": "Value1",
"f_id": "17AEBA",
"code": "DT10",
"l_uc": "INJ1",
"d_id": 23765}}

Thanks in advance

[gabriel_tessier]

Hi @amoussoubaruch,

I think the more easy way to remove the fields that are null form the result is to not index them.
It can save disk space.

You can add an ingest with remove processor that will drop the null fields with condition.
https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html
Or depend on how you get your data it maybe more easy to just drop them before indexing.

[amoussoubaruch]

Hi @gabriel_tessier,

Thanks for your reply. I will test it and get back to you as soon as.