I am totaly new to elastic and tries to figure out how to use elastic with serilog as provider.
I have a lot of different log sources (100 +), and they all generate a JSON I need to log. I know I can log the
different JSON's in one field, but I thing for searching purposes this will not fly well. Most of the JSON's are
different in the respect of common fields, maybe only 5-10% in some of them where common fields will occure.
So my questions is :
Should I save the complete JSON in one field or should I split the JSON's into separate fields for every data inside it ?
If I create separate fields for every data in the JSON's, due to the different structures, should I have them all in the same index ?
If not, should I create one index pr JSON type ? Will this not influence performance ?
Elastic Common Schema as the name implies, is a specification for a common set of fields when storing logs and metrics in Elasticsearch. Elastic.CommonSchema.Serilog contains an ITextFormatter implementation that formats a Serilog event into a JSON representation that adheres to Elastic Common Schema. You can read more about it in the GitHub repository and blog post.
To answer your questions
Ideally, each log line in the JSON file is a separate log event that would be indexed as a separate document in Elasticsearch.
A typical approach is to define a unified logging format, similar to what Elastic Common Schema does, that all log events adhere to, making it easier to analyze and correlate across log events. If logs are already different structures, it may make sense to put them into separate, time-based indices.
An index is made up of one or more shards. It is really more the number of shards rather than the number of indices that have an impact i.e. one index with 5 shards or five indices each with one shard has about the same impact.
After som more work on this I have decided to create my own custom template.
When I request the data I get Norwegian characters like this with the curl -X GET "localhost:9200/_search?pretty" command from my cms windows under Windows 10.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.