Best practice for the creation of an index

Hello.

I'm working on a project with Elastic for my company.
I have to export some logs from the test software of our solution to an elastic database.
Every night, the test software will create many logs with Java, using the JavaAPI, but I don't know what is the best practice to do. Should it be an unique entry with many tests, many time updates during the night or should every test be an independent entry in the index?

The project is an inside project of the company and will be used by a single person at a time. We don't need extraordinary performances and we will exploit the data with an Angular app on front later.

We will create an unique index for all the test and all the result. Is it the good practice or should we create many index?

thanks and have a good day


indepandant entry:
In this case, everytime a test is done, we push it in the data base with the same idOfTheTestSerie1 for all test in this night.

{
    "idOfTheTestSerie1": "********",
    "idOfTheTest1": "********",
    "someInformationAboutTheTest": "********",
    "commentaires": 
    [
        {
            "part1Test1": "*****",
            "informations": "*******"
        },
        {
            "part2Test1": "*****",
            "informations": "*******"
        }
    ]
}
{
    "idOfTheTestSerie1": "********",
    "idOfTheTest2": "********",
    "someInformationAboutTheTest": "********",
    "commentaires": 
    [
        {
            "part1Test2": "*****",
            "informations": "*******"
        },
        {
            "part2Test2": "*****",
            "informations": "*******"
        }
    ]
}

or unique entry
In this case, we push an unique data in the database and we update it everytime a test is finished.

{
    "idOfTheTestSerie1": "********",
     "test": 
     [
        {
           "idOfTheTest1": "********",
           "someInformationAboutTheTest": "********",
           "commentaires": 
           [
                {
                    {
                        "part1Test1": "*****",
                        "informations": "*******"
                    },
                    {
                        "part2Test1": "*****",
                        "informations": "*******"
                    }
                }
            ]
        },
        {
            "idOfTheTest2": "********",
            "someInformationAboutTheTest": "********",
            "commentaires": 
            [
                 {
                     {
                         "part1Test2": "*****",
                         "informations": "*******"
                     },
                     {
                         "part2Test2": "*****",
                         "informations": "*******"
                     }
                 }
             ]
        }
    ]
}

All the test, all the log will created hundreds of index and hundreds of shard.

one node in cluster has limit of 1000 shard. beyond that it will work but it can create lot of problem.

rule of thumb is you create a shard worth of 20-30gig if it is possible.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.