Send raw logs to elastic server

Hi
I have 2 question about elastic

1-how can i send all files located in specific path with random name via filebeat like this:
/var/log/app/serverlog.log
/var/log/app/applog.log
/var/log/app/exceptionlog.log
...

to elastic directly or logstash without define how save them in destination, exactly save them like in source (with exact name and pattern) I need raw file store in elastic server.

2-what is the different if i send file directly to elastic vs logstash? (I mean from performance,security, filtering ... point of view)

Any idea?
Thanks

I not clear with your question.

Do you want to send the data inside those log files to Elasticsearch?

Is that what you are looking out for?

1 Like

When you store logs in Elasticsearch you do not generally store the whole file as a unit (unless they are very small), but rather store lines individually as separate documents. You can use a multiline processor to group related lines into a single document though.

Elasticsearch accepts JSON documents, so you need something, e.g. Filebeat or Logstash, that converts log lines into documents that can be indexed.

1 Like

@dadiasish @Christian_Dahlqvist need to store raw log with exact file structure on elastic server after that index them.

e.g
on client side
/var/log/app/serverlog.log
/var/log/app/applog.log
/var/log/app/exceptionlog.log
...

server side
/var/log/app/serverlog.log
/var/log/app/applog.log
/var/log/app/exceptionlog.log
...

without define exactly on server how i expect to receive them. (mirror files)

I do not understand what you mean. You need to explain in more detail and provide an example or show the flow you are expecting. Where are logs coming from? Where are they moved and how are they processed?

1 Like

@Christian_Dahlqvist simple scenario:
have client and server

client generate logs that belong to my app

server revived and store log files on the same structure.

after that index them in elastic.

Client (raw file) > Server(raw file) > Elastic(index)

If the client sends the raw file to the server, set up Filebeat and/or Logstash on the server and index the data into Elasticsearch as I described earlier. For a practical guide on how to do this with Logstash, have a look at this blog post. It is old but I think it is still generally valid.

1 Like

@Christian_Dahlqvist you miss that part that I told need store raw file on server side just like as it was in client side.

You will not store whole log files in Elasticsearch, so if you need to do this I suspect you need to copy the files into appropriate storage, e.g. using a custom script. That is not something that Logstash or Filebeat does as far as i know.

If you are looking for Elasticsearch to store complete log files, I suspect you are looking at the wrong tool.

1 Like

Write a script that transforms the log into one big string and index that into a single field in the index. Possible but pretty useless because searching through that or performing any aggregation becomes really slow.
Probably better to structure that log into specific fields and as @Christian_Dahlqvist pointed out split it into multiple fields. One log per document, every document different fields like "LogCreationTime; LogErrorText; user; ip; errorcode" etc. Whats the use in having all the logs in there and not beeing able to use them.

Elasticsearch does not generally work well with huge documents and there are size limits to what can be stored. Storing full log as a string may be possible for small files but does IMHO not work.

I indexed the whole bible in a single document field (With fscrawler) and that worked and is still searchable. So its possible, but performance wise probably not very smart.

I believe the bible has less than 800000 words and takes up around 4MB in text form. Log files can often be a lot longer than that.

Yea sure, logs can become pretty big. But I am on "your side" anyway :wink: Its stupid to store logs in one huge chunk that slow to search and impossible to visualize. Data is useless if its not stored nice and tidy.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.