Limiting data in object properties coming to browser from elastic search

Elastic Stack Elasticsearch
1

Hi,

We are currently pulling large amounts of data from Elasticsearch for reporting products in our software. For our larger clients this means sending a large amount of data to the browser which is then loaded into a react component. One possibility we'd like to investigate to improve performance is to bring in less data from Elasticsearch, but not via filtering. We have module objects that come in with attributes. Meaning that there is, for example, an object that represents a an item or an order etc… In these module objects there is a property called "attributes" this property contains all kinds of attributes that relates to the module. These attributes, while necessary in the actual software, are not necessary in the reports. However in Elasticsearch, while we have the possibility to filter on the attributes (which would filter in or out certain items), we do not know how to exclude attribute data from being sent to the browser. Any solution to limit the data being sent in the objects themselves would be greatly appreciated!

2

You could explore some options here:

Otherwise you need to solve that in your app which is probably in the middle, right?
I mean: are you connecting your end users directly to elasticsearch?

3

Ok great, thanks for that! So I tried implementing but I'm having a bit of trouble . I'm trying to see how to get something to work notionally. In the following I modified an existing query to add _source=false and added fields for id and number and it's still returning exactly the same response as the unmodified query. Any idea on how I could get it to return a response with only id and number fields? The help is greatly appreciated!

POST transactions/_search?filter_path=hits.hits._source,_scroll_id&scroll=1m&size=100 {
"_source": false,
"query":{
"bool":{
"should":[{
"bool":{
"filter":[
{"term":{"elastic_tag_module":"invoice"}},
{"term":{"elastic_tag_subdomain":"utilizetestdev"}}
]
} }] } },
"fields": [
"id",
"number"
],
"sort":[{"hits.hits._source.quantity": "asc"}]
}

This is an example of a response to the unmodified query:

{
  "_scroll_id" : "FGluY2x1ZGVfY29udGV4dF91dWlkDXF1ZXJ5QW5kRmV0Y2gBFmF1dm4tb0hLUkFtNWQ0U3JiNE9CdUEAAAAAAAJznhZERGVLVE9mM1QyQ1ByYUwyY3BORTZR",
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "id" : 23502,
          "number" : "WB21-023484",
          "display" : null,
          "status" : "complete",
          "company" : 1,
          "organization" : "D702DAAB-249D-4486-84EE-7A6B96A1C76C",
          "currency" : "EUR",
          "transaction_date" : "2021-10-21T12:11:15.489Z",
          "source" : null,
          "shipping_addresses" : {
            "to" : {
              "id" : 2520,
              "description" : "",
              "display" : """Ankerweg 4
1041 AT Amsterdam
NL""",
              "street_number" : null,
              "route" : null,
              "neighborhood" : null,
              "sublocality" : null,
              "locality" : "Amsterdam",
              "administrative_area_one" : "Ankerweg 4",
              "administrative_area_one_code" : "",
              "administrative_area_two" : null,
              "country" : "Netherlands",
              "country_code" : "NL",
              "postal_code" : "1041 AT Amsterdam"
            },
            "from" : {
              "id" : 2520,
              "description" : "",
              "display" : """Ankerweg 4
1041 AT Amsterdam
NL""",
              "street_number" : null,
              "route" : null,
              "neighborhood" : null,
              "sublocality" : null,
              "locality" : "Amsterdam",
              "administrative_area_one" : "Ankerweg 4",
              "administrative_area_one_code" : "",
              "administrative_area_two" : null,
              "country" : "Netherlands",
              "country_code" : "NL",
              "postal_code" : "1041 AT Amsterdam"
            }
          },
          "billing_addresses" : {
            "to" : {
              "id" : 2520,
              "organization" : "D702DAAB-249D-4486-84EE-7A6B96A1C76C",
              "owner" : "BA0BDAE8-C360-443E-A2D1-E33621E54A01",
              "description" : "",
              "street_number" : null,
              "route" : null,
              "display" : """Ankerweg 4
1041 AT Amsterdam
NL""",
              "neighborhood" : null,
              "sublocality" : null,
              "locality" : "Amsterdam",
              "administrative_area_one" : "Ankerweg 4",
              "administrative_area_one_code" : "",
              "administrative_area_two" : null,
              "postal_code" : "1041 AT Amsterdam",
              "country" : "Netherlands",
              "country_code" : "NL"
            },
            "from" : {
              "id" : 2520,
              "organization" : "D702DAAB-249D-4486-84EE-7A6B96A1C76C",
              "owner" : "BA0BDAE8-C360-443E-A2D1-E33621E54A01",
              "description" : "",
              "street_number" : null,
              "route" : null,
              "display" : """Ankerweg 4
1041 AT Amsterdam
NL""",
              "neighborhood" : null,
              "sublocality" : null,
              "locality" : "Amsterdam",
              "administrative_area_one" : "Ankerweg 4",
              "administrative_area_one_code" : "",
              "administrative_area_two" : null,
              "postal_code" : "1041 AT Amsterdam",
              "country" : "Netherlands",
              "country_code" : "NL"
            }
          },
          "attributes" : {
            "currency" : [
              "EUR"
            ],
            "notepad" : [
              """Motorwagen
0610765188 Louis"""
            ],
            "ship_date" : [
              "2021-10-22T12:00:00.000Z"
            ],
            "division" : [
              "VEP"
            ],
            "minox_debtor_number" : [
              115080
            ],
            "klant" : [
              "POEL BLOEMBOLLEN B.V."
            ],
            "ordersoort" : [
              "Productie"
            ],
            "transactietype" : [
              "Werkorder"
            ],
            "ordernummer" : [
              "ORD21-025810"
            ],
            "icp_zone" : [
              "NL"
            ],
            "distribution_status" : [
              "Not Distributed"
            ],
            "validatie" : [
              "CLEAR"
            ],
            "shipment_complete" : [
              true
            ]
          },
          "lines" : [
            {
              "id" : 42840,
              "iid" : 1,
              "source" : {
                "id" : 217087,
                "type" : "order_detail"
              },
              "item" : 5404,
              "unit_price" : 26.77,
              "quantity" : 30,
              "discount" : {
                "lines" : [ ],
                "total" : 0
              },
              "total" : 803.19,
              "attributes" : {
                "description" : [
                  "Potgrond Tulpen plus 15% zand +5cc RID"
                ],
                "verpakkingsoort" : [
                  "Bulk"
                ],
                "zeefstand" : [
                  "Middel"
                ],
                "streefgewicht_per_m_en" : [
                  400
                ],
                "streefwaarde_p_h" : [
                  6
                ],
                "streefwaarde_ec" : [
                  1
                ],
                "en_factor" : [
                  20
                ]
              },
              "bom" : [
                {
                  "id" : 355674,
                  "item" : 1491,
                  "type" : "material",
                  "total_cost" : 0.0143,
                  "total_quantity" : 6300,
                  "uom" : "kilo",
                  "attributes" : { },
                  "created_on" : "2021-10-21T10:11:16.282Z",
                  "created_by" : "3ECA62F3-1EFF-46F7-92C4-BA56D5F738D3",
                  "modified_on" : "2021-10-23T23:54:32.888Z",
                  "modified_by" : "B7E19DD4-E17C-4B55-BE35-DBF02A16263A"
                },
                {
                  "id" : 355673,
                  "item" : 484,
                  "type" : "material",
                  "total_cost" : 21.4,
                  "total_quantity" : 6,
                  "uom" : "m³ WM",
                  "attributes" : { },
                  "created_on" : "2021-10-21T10:11:16.282Z",
                  "created_by" : "3ECA62F3-1EFF-46F7-92C4-BA56D5F738D3",
                  "modified_on" : "2021-10-23T23:54:32.888Z",
                  "modified_by" : "B7E19DD4-E17C-4B55-BE35-DBF02A16263A"
                },
                {
                  "id" : 355672,
                  "item" : 470,
                  "type" : "material",
                  "total_cost" : 493,
                  "total_quantity" : 0.15,
                  "uom" : "liter",
                  "attributes" : { },
                  "created_on" : "2021-10-21T10:11:16.282Z",
                  "created_by" : "3ECA62F3-1EFF-46F7-92C4-BA56D5F738D3",
                  "modified_on" : "2021-10-23T23:54:32.888Z",
                  "modified_by" : "B7E19DD4-E17C-4B55-BE35-DBF02A16263A"
                },
                {
                  "id" : 355671,
                  "item" : 1501,
                  "type" : "material",
                  "total_cost" : 1.0E-4,
                  "total_quantity" : 3,
                  "uom" : "Stuk",
                  "attributes" : { },
                  "created_on" : "2021-10-21T10:11:16.282Z",
                  "created_by" : "3ECA62F3-1EFF-46F7-92C4-BA56D5F738D3",
                  "modified_on" : "2021-10-23T23:54:32.888Z",
                  "modified_by" : "B7E19DD4-E17C-4B55-BE35-DBF02A16263A"
                },
                {
                  "id" : 355670,
                  "item" : 36,
                  "type" : "material",
                  "total_cost" : 0.43,
                  "total_quantity" : 15,
                  "uom" : "kilo",
                  "attributes" : { },
                  "created_on" : "2021-10-21T10:11:16.282Z",
                  "created_by" : "3ECA62F3-1EFF-46F7-92C4-BA56D5F738D3",
                  "modified_on" : "2021-10-23T23:54:32.888Z",
                  "modified_by" : "B7E19DD4-E17C-4B55-BE35-DBF02A16263A"
                },
                {
                  "id" : 355669,
                  "item" : 31,
                  "type" : "material",
                  "total_cost" : 1.03,
                  "total_quantity" : 24,
                  "uom" : "kilo",
                  "attributes" : { },
                  "created_on" : "2021-10-21T10:11:16.282Z",
                  "created_by" : "3ECA62F3-1EFF-46F7-92C4-BA56D5F738D3",
                  "modified_on" : "2021-10-23T23:54:32.888Z",
                  "modified_by" : "B7E19DD4-E17C-4B55-BE35-DBF02A16263A"
                },
                {
                  "id" : 355668,
                  "item" : 29,
                  "type" : "material",
                  "total_cost" : 0.04,
                  "total_quantity" : 210,
                  "uom" : "kilo",
                  "attributes" : { },
                  "created_on" : "2021-10-21T10:11:16.282Z",
                  "created_by" : "3ECA62F3-1EFF-46F7-92C4-BA56D5F738D3",
                  "modified_on" : "2021-10-23T23:54:32.888Z",
                  "modified_by" : "B7E19DD4-E17C-4B55-BE35-DBF02A16263A"
                },
                {
                  "id" : 355667,
                  "item" : 13,
                  "type" : "material",
                  "total_cost" : 20.03,
                  "total_quantity" : 6,
                  "uom" : "m³ WM",
                  "attributes" : { },
                  "created_on" : "2021-10-21T10:11:16.282Z",
                  "created_by" : "3ECA62F3-1EFF-46F7-92C4-BA56D5F738D3",
                  "modified_on" : "2021-10-23T23:54:32.888Z",
                  "modified_by" : "B7E19DD4-E17C-4B55-BE35-DBF02A16263A"
                },
                {
                  "id" : 355666,
                  "item" : 124,
                  "type" : "material",
                  "total_cost" : 19.5,
                  "total_quantity" : 18,
                  "uom" : "m³ WM",
                  "attributes" : { },
                  "created_on" : "2021-10-21T10:11:16.282Z",
                  "created_by" : "3ECA62F3-1EFF-46F7-92C4-BA56D5F738D3",
                  "modified_on" : "2021-10-23T23:54:32.888Z",
                  "modified_by" : "B7E19DD4-E17C-4B55-BE35-DBF02A16263A"
                }
              ],
              "source_display" : "Order : ORD21-025810"
            }
          ],
          "quantity" : 30,
          "subtotal" : 803.19,
          "total" : 803.19,
          "created_on" : "2021-10-21T10:11:15.507Z",
          "created_by" : "3ECA62F3-1EFF-46F7-92C4-BA56D5F738D3",
          "modified_on" : "2021-10-23T23:53:16.588Z",
          "modified_by" : "",
          "source_display" : "Order : ORD21-025810",
          "elastic_tag_module" : "workorder",
          "elastic_tag_subdomain" : "egmondpotgronddev"
        }
      }
    ]
  }
}
4 
POST transactions/_search?filter_path=_scroll_id,hits.hits._source._id,hits.hits._source.number&scroll=1m&size=100
1 Like
5

Thanks! This is what's needed, I guess there's no need for the fields functionality after all.

A follow up question: Is it possible to exclude instead of include? Meaning, if I want everything except id and number.

6

The doc says it all:

GET /_count?filter_path=-_shards

So it should be:

GET transactions/_search?filter_path=-hits.hits._source.number,-hits.hits._source.id
1 Like
7

Ah right, I hadn't looked at the second link. Thanks very much for this, I just read through it.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.