So I am building a Spring Boot rest api that it takes a file ( Multipart file ) ( and it is a log file ) as an argument and saves its content in a unique elasticsearch index ! Each line of the file will be in a document. so my plan was : 1- create the index. 2- read each line of the file 3- save this line in the index.
the version elasticsearch that I am using is : 8.6
the problem : I am dealing with big files ( their size can reach to 200mb) and when I tested my code, it took hours just for an average file ( size way too smaller than 100mb )
this is my code ( I used the buffer reader and elasticsearch java api client ) :
// Creating the index
String indexName = "newIndex";
CreateIndexResponse createResponse =client.indices().create(
new CreateIndexRequest.Builder()
.index(indexName)
.build()
);
// read each line and save it
try (BufferedReader br = new BufferedReader(new InputStreamReader(file.getInputStream()))){
String line ="";
while((line = br.readLine()) != null) {
if (!line.trim().isEmpty()) {
LogFile log = new LogFile(); // the code is below this
log.setMessage(line);
IndexResponse resp = client.index(i -> i
.index(indexName)
.document(log));
}
}
}
// the code of the class LogFile
@Document(indexName = "log-model")
@Data
public class LogFile {
@Id
private String id;
@Field(type = FieldType.Text)
private String message;
}
What I am looking for : How can I improve my code ? is there a library or another tool like logstach or something that i can use to load theese type of files in a shorter time ! I ve been searching for more than one week and I didn't find anything helpful ! is there an api to directly upload a file or something .... please I really need HELP.