# Best practice for a newbie

**URL:** <https://discuss.elastic.co/t/best-practice-for-a-newbie/228030>\
**Category:** Elasticsearch\
**Created:** [April 15, 2020, 4:37am UTC](https://discuss.elastic.co/t/best-practice-for-a-newbie/228030 "2020-04-15T04:37:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rhh](https://avatars.discourse-cdn.com/v4/letter/r/e9a140/32.png) [@Rhh](https://discuss.elastic.co/u/Rhh)\
**Post date:** [April 15, 2020, 4:37am UTC](https://discuss.elastic.co/t/best-practice-for-a-newbie/228030/1 "2020-04-15T04:37:03Z")

</div>

Hi

I am totaly new to elastic and tries to figure out how to use elastic with serilog as provider.

I have a lot of different log sources (100 +), and they all generate a JSON I need to log. I know I can log the  
different JSON's in one field, but I thing for searching purposes this will not fly well. Most of the JSON's are  
different in the respect of common fields, maybe only 5-10% in some of them where common fields will occure.

So my questions is :

1. 

Should I save the complete JSON in one field or should I split the JSON's into separate fields for every data inside it ?

1. 

If I create separate fields for every data in the JSON's, due to the different structures, should I have them all in the same index ?  
If not, should I create one index pr JSON type ? Will this not influence performance ?

Regards

Sample 1 :

{  
"message": "my custom data 1"  
}

Sample 2 :

{  
"message": "my custom data 2",  
"data": {  
"A": "valueA",  
"B": {  
"B1": "valueB1"  
},  
"C": {  
"C1": "valueC1",  
"C2": {  
"C2\_1": "valueC2\_1",  
"C2\_2": "valueC2\_2"  
}  
}  
}  
}

Sample 3 :

{  
"message": "my custom data 3",  
"data1": {  
"D": {  
"D1": {  
"D1\_1": "valueC1\_1",  
"D1\_2": "valueC1\_2"  
}  
},  
"data2": {  
"X": {  
"X1\_1": "valueX1\_1",  
"X1\_2": "valueX1\_2"  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [April 15, 2020, 11:56am UTC](https://discuss.elastic.co/t/best-practice-for-a-newbie/228030/2 "2020-04-15T11:56:13Z")

</div>

Hey @Rhh,

You may want to check out [`Elastic.CommonSchema.Serilog`](https://www.nuget.org/packages/Elastic.CommonSchema.Serilog/) nuget package, which is part of the suite of integrations for the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html) with .NET.

Elastic Common Schema as the name implies, is a specification for a common set of fields when storing logs and metrics in Elasticsearch. `Elastic.CommonSchema.Serilog` contains an `ITextFormatter` implementation that formats a Serilog event into a JSON representation that adheres to Elastic Common Schema. You can read more about it in the [GitHub repository](https://github.com/elastic/ecs-dotnet/tree/master/src/Elastic.CommonSchema.Serilog) and [blog post](https://www.elastic.co/blog/elastic-common-schema-dotnet-library-and-integrations-released-for-elasticsearch).

To answer your questions

> [@Rhh](#):
>
> Should I save the complete JSON in one field or should I split the JSON's into separate fields for every data inside it ?

Ideally, each log line in the JSON file is a separate log event that would be indexed as a separate document in Elasticsearch.

> [@Rhh](#):
>
> If I create separate fields for every data in the JSON's, due to the different structures, should I have them all in the same index ?

A typical approach is to define a unified logging format, similar to what Elastic Common Schema does, that all log events adhere to, making it easier to analyze and correlate across log events. If logs are already different structures, it may make sense to put them into separate, time-based indices.

> [@Rhh](#):
>
> If not, should I create one index pr JSON type ? Will this not influence performance ?

An index is made up of one or more shards. It is really more the number of shards rather than the number of indices that have an impact i.e. one index with 5 shards or five indices each with one shard has about the same impact.

---

<div class="post-metadata">

**Author:** ![Rhh](https://avatars.discourse-cdn.com/v4/letter/r/e9a140/32.png) [@Rhh](https://discuss.elastic.co/u/Rhh)\
**Post date:** [April 16, 2020, 10:03am UTC](https://discuss.elastic.co/t/best-practice-for-a-newbie/228030/3 "2020-04-16T10:03:26Z")

</div>

OK, I will try to use `Elastic.CommonSchema.Serilog`...

I try to use it this way 😑

LoggerConfiguration.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(  
new Uri(TmpURL))  
{  
AutoRegisterTemplate  
= true ,

```
            CustomFormatter
                = new EcsTextFormatter()
        });

```

But I get :

Failed to create the template

Any ide why this does not work ?

Regards

---

<div class="post-metadata">

**Author:** ![Rhh](https://avatars.discourse-cdn.com/v4/letter/r/e9a140/32.png) [@Rhh](https://discuss.elastic.co/u/Rhh)\
**Post date:** [April 17, 2020, 6:34am UTC](https://discuss.elastic.co/t/best-practice-for-a-newbie/228030/4 "2020-04-17T06:34:42Z")

</div>

After som more work on this I have decided to create my own custom template.

When I request the data I get Norwegian characters like this with the curl -X GET "localhost:9200/\_search?pretty" command from my cms windows under Windows 10.

"message" : "{"Message":"5 fors├©k", …

If should have been

"message" : "{"Message":"5 forsøk", …

I am quite sure the text is added correctly to ES.

Why is this ?

Regards

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 15, 2020, 6:34am UTC](https://discuss.elastic.co/t/best-practice-for-a-newbie/228030/5 "2020-05-15T06:34:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.