Need advice on AWS logs indexing using logstash

(Subbu v) #1

i have elastic search + logstash installed on one server. i am trying to index the logs from AWS and push into elastic search. can any one help with the format i have to use in logstash conf file input.

(Magnus B├Ąck) #2

If you can give an example of the logs of logs you want to parse there's a better chance that someone will be able to help.

(Subbu v) #3

thanks for your response.

Not sure if i understood your question.
the logs are usually random unstructured server logs residing in AWS.

(Mark Walkom) #4

Right, but how can we help with the config unless we can see the format.

What have you tried so far, or are you expecting us to write it all for you?

(Subbu v) #5

ok got it.

i am indexing the logs from my local windows machine and below is the input i am using.

input {
file {
path => "C:/web/apps/wslogs/*"
start_position => beginning
type => syslog

below are some of the lines from logs.

Voltage Margin: Enabled
Time: 06:07:42 Aug 11 2014 Temp: 27 C: 27 C: 28 C: 27 C: 27 C: 27 C
Last Status: NONE

ESS Test Run 1
Starts at 06:07:44 Aug 11 2014 Temp: 27 C: 27 C: 28 C: 28 C: 27 C: 27 C
rst# x^x^x^0^x^x^x^x^H^0^06:07:44^0^27^27^28^28^27^27^1.06^3.28^3.25^3.25^2.50^1.80^1.50^1.19^1.20^1.00^0.98^1.01

(Mark Walkom) #6

It looks like these are multiline, ie a single event over more than one line, is that right?

(Subbu v) #7

yes, thats right.

(system) #8