Inject line based log/text files

(Luke) #1

Hi

I have some logs files which are currently unable to send directly to logstash. Those logs are consists of multiple lines of data as data blocks.

For example;

CompanyName: XYZ Pvt Ltd
Date & Time: 2018-09-28 00:03:47.312
Some value: xxx
Some text: abcd
So on, it has multiple lines
END:

CompanyName: ABC Pvt Ltd
Date & Time: 2018-09-28 00:02:20.312
Some value: xxx
Some text: abcdddd
So on, it has multiple lines
END:

I can find a starting point and an end point.

How can I inject these type of logs to logstash? Can somebody help me, please?

Thanks in advance,
Luke.

(Lewis Barclay) #2

We need more information about the source, how are you reading these into Elastic? Are they stored in files, shipped through beats or?

(Luke) #3

Hi Lewis,

These types of files are generating daily basis and every 1MB file will be roll out for the next file. Files having a unique file ID so that we can filtered out the latest file has generated.

I can download the latest file to elasticsearch node using a cron job.

Once downloaded it, I want to send it to logstsh --> elasticsearch for rest of data analyzing process.

Thank you
Luke.

(Lewis Barclay) #4

Hi Luke,

Would using filebeat be an option rather than copying them manually?

Lewis

(Luke) #5

Hi Lewis,

Thanks for the suggestion. Suppose I use the filebeat to sending files. But then how can I insert these types of multi-line logs files to elasticsearch? How can I get the " Date & Time:" field to the x-axis to draw the graphs?

Most of these log files are a couple of month old. So the timestamp for the x-axis must be the " Date & Time:" which comes with the log data.

Thank you
Luke.