Saving the content of a file in an elasticsearch index using springboot RESTAPI

Elastic Stack Elasticsearch
1

So I am building a Spring Boot rest api that it takes a file ( Multipart file ) ( and it is a log file ) as an argument and saves its content in a unique elasticsearch index ! Each line of the file will be in a document. so my plan was : 1- create the index. 2- read each line of the file 3- save this line in the index.

the version elasticsearch that I am using is : 8.6

the problem : I am dealing with big files ( their size can reach to 200mb) and when I tested my code, it took hours just for an average file ( size way too smaller than 100mb )

this is my code ( I used the buffer reader and elasticsearch java api client ) :

// Creating the index 
String indexName = "newIndex";

CreateIndexResponse createResponse =client.indices().create(
                new CreateIndexRequest.Builder()
                .index(indexName)
                .build()
                );
// read each line and save it 
try (BufferedReader br = new BufferedReader(new InputStreamReader(file.getInputStream()))){
            
            String line ="";
            while((line = br.readLine()) != null) {
                if (!line.trim().isEmpty()) {
                    LogFile log = new LogFile(); // the code is below this
                    log.setMessage(line);
                    IndexResponse resp = client.index(i -> i
                            .index(indexName)
                            .document(log));
                        } 
            }
}

// the code of the class LogFile 
@Document(indexName = "log-model")
@Data
public class LogFile {

    @Id
    private String id;
    
    @Field(type = FieldType.Text)
    private String message;

}

What I am looking for : How can I improve my code ? is there a library or another tool like logstach or something that i can use to load theese type of files in a shorter time ! I ve been searching for more than one week and I didn't find anything helpful ! is there an api to directly upload a file or something .... please I really need HELP.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.