Raw JSON ( unstructured data ) handling with elastic C# API

Elastic Stack Elasticsearch
1

We are working on an application, where we receive data in json format from different sources and different format and user can also add such sources there own, So we don't know json format what properties will be there

Our application presents that that data to a grid format and grid provides filtering, sorting, paging, grouping etc. kind of standard operation on that data.

We decided to use elastic search to store such large and unstructured data.On the backend, we use .NET ( C#).

Started to use bulk API to index data.
Here is sample json:
myJson = @"{""index"":{""_index"":""abc"",""_type"":""abc"",""_id"":1}},
{""Name"":""ApplicationFrameHost"",""CPU"":1.25,""Company"":null,""Product"":null,""Path"":null}
{""index"":{""_index"":""abc"",""_type"":""abc"",""_id"":2}},
{""Name"":""audiodg"",""CPU"":1.5625,""Company"":null,""Product"":null,""Path"":null}
{""index"":{""_index"":""abc"",""_type"":""abc"",""_id"":3}},
{""Name"":""Calculator"",""CPU"":0.5,""Company"":null,""Product"":null,""Path"":null}
{""index"":{""_index"":""abc"",""_type"":""abc"",""_id"":4}},
{""Name"":""chrome"",""CPU"":144.109375,""Company"":null,""Product"":null,""Path"":null}
{""index"":{""_index"":""abc"",""_type"":""abc"",""_id"":5}},
{""Name"":""chrome"",""CPU"":3384.609375,""Company"":null,""Product"":null,""Path"":null}

";

On C# code side:

var connectionSettings = new ConnectionSettings(new Uri("http://localhost:9200/"));
        connectionSettings.DisableDirectStreaming(true);
        var client2 = new ElasticClient(connectionSettings);

var jsonPostData = new PostData<object>(myJson);
        var bulkRequestParameters = new BulkRequestParameters
        {

        };

        Func<BulkRequestParameters, BulkRequestParameters> convert = delegate (BulkRequestParameters s)
        {
            s.ErrorTrace(true);
            return s.Refresh(Refresh.True);
        };

        ElasticsearchResponse<VoidResponse> response = client2.LowLevel.Bulk<VoidResponse>("abc", "abc", jsonPostData, convert);

It does successfully insert data to elastic.

Just wanted to confirm is this the optimal way to perform this?

Problem we are facing
We need grouping on any field on index data ( user can group on any field while presenting data, So we need aggregated data at runtime.
How can this be achieved with C# API?

Also, How filtering can be done on multiple fields with C# API?

I am new to elastic search so not have much idea, what capabilities it has and what are the ways to achieve this with C# API though I have played a bit with Kibana console on grouping, I am getting following error:
Fielddata is disabled on text fields by default. Set fielddata=true on [Name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

Thanks in advanced.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.