Upgrade to NEST 5.0.0.0 and Elastic search 5.0.0.0 issue error 400 but with Kibana it is working fine and extracts result

[dibyajyoti_dev]

Hi ,

I got upgraded my MVC app to NEST and elastic version 5.0.0.0 and as I see the documentation about size parameter should be greater than 0 i did put Size(10). as below.

var response = Elastic
.Search(selector => selector
.Index(indexesToQuery)
.Aggregations(aggs => aggs
.Terms("app", appTerm => appTerm
.Size (10)
.Field(l => l.ApplicationCode)
.Aggregations(appAggs => appAggs
.Terms("module", moduleTerm => moduleTerm
.Size(10)
.Field(l => l.Module)
.Aggregations(moduleAggs => moduleAggs
.Terms("log-levels", t => t
.Size(10)
.Field(l => l.LogLevel)
.Aggregations(lla => lla
.DateRange("by-date-range",
dateRange => dateRange
.Field(l => l.EventUtcTimestamp)
.Ranges(r => r.From(new DateMathExpression(dateRangeStart)).To(new DateMathExpression(dateRangeEnd))))
)
)
)
)
)
)
)
.Size(10)
);

The error stack is below with error 400

Invalid NEST response built from a unsuccessful low level call on POST: /log-messages-20171205%2A/logmessage/_search
# Audit trail of this API call:
** - [1] BadResponse: Node: http://test.application.companyName.com:9200/ Took: 00:00:00.6054258**
# ServerError: ServerError: 400Type: search_phase_execution_exception Reason: "all shards failed"
# OriginalException: System.Net.WebException: The remote server returned an error: (400) Bad Request.
** at System.Net.HttpWebRequest.GetResponse()**
** at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData)**
# Request:
{"size":10,"aggs":{"app":{"terms":{"field":"applicationCode","size":10},"aggs":{"module":{"terms":{"field":"module","size":10},"aggs":{"log-levels":{"terms":{"field":"logLevel","size":10},"aggs":{"by-date-range":{"date_range":{"field":"eventUtcTimestamp","ranges":[{"from":"2017-12-05T08:42:48.9641734Z","to":"2017-12-05T09:42:49.3565214Z"}]}}}}}}}}}}
# Response:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [applicationCode] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"log-messages-20171205","node":"7hBicvU8Seeyxn-yAT7Fcw","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [applicationCode] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}]},"status":400}

The node cannot be accessed as it is organisation specific and has to be kept private as per policies. Hope the stack trace rings a bell.

Any help would be much appreciated!

[forloop (Russ Cam)]

The error reason provides some good details on what the issue is

@dibyajyoti_dev

Fielddata is disabled on text fields by default. Set fielddata=true on [applicationCode] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

The ApplicationCode property is mapped as a text field datatype and the query is attempting to perform a terms aggregation on it. In order for this to work, fielddata needs to be enabled on the field mapping as it is disabled by default, as fielddata on a large text field over many documents can consume a lot of memory and impede cluster performance.

As per the message and documentation, you may want to consider mapping the field also as a keyword datatype, using a multi_field, and performing the aggregation on that field.

[dibyajyoti_dev]

Thanks Russ Cam for your reply. Just wanted to re-visit on the same as in Kibana it all looks perfect and data is extracted.

The problem is when I use the NEST dll (version 5.0.0.0 which in turn uses elastic search 5.0.0.0). The same is not a problem with the lower version of NEST (in my MVC .Net App)neither in Kibana even with upgrade.

it clearly implies it is owing to NEST and elastic search upgrade. Hope my performance is not degraded with setting field data to true

[forloop (Russ Cam)]

@dibyajyoti_dev

it clearly implies it is owing to NEST and elastic search upgrade.

It is not NEST, but a change in mapping following an upgrade. I would recommend changing over to using keyword type if you can.

[dibyajyoti_dev]

Thanks.. So I changed the code as below with inclusion of keyword term => term.Field(l => l.Machine.Suffix("keyword")) . is this the way ? Please note the field "Machine" is of type string

public static List GetMachinesList(string applicationCode)
{
var result = Elastic.Search(s => s
.Index("log-messages-*")
.Query(q => q.Term(l => l.ApplicationCode, applicationCode))
.Aggregations(aggs => aggs
.Terms("machine-codes",
term => term.Field(l => l.Machine.Suffix("keyword")).OrderAscending("_term").Size(int.MaxValue))));

        return result.Aggs.Terms("machine-codes").Buckets.Select(b => b.Key).ToList();
    }