Issue in parsing the file which has # character at the beginning of the line

Elastic Stack Logstash
1

Hi All,

I am new to logstash and recently started scripiting in logstash. we have a requirement where some logs files have # in the beginning of the line. Say
####<Aug 11, 2016 11:40:53 AM CDT><Info> <Security> <HostName>
####<Aug 11, 2016 11:40:54 AM CDT> <Info> <Security> <HostName>
now, I want to parse each line separately if it has #### in the beginning, but i tried couple of scripts and all are failing with ""Expected one of #, input, filter, output at line 28, column 1 (byte 302) after ", :level=>:error}" error:

Scripts i used:

1:
input
{
file
{
path => "/tmp/testXML1.log"
codec => plain { charset => "US-ASCII"}
type => "OSBLog"
}
}
output {
elasticsearch {
hosts => ["192.168.99.100:9200"]
codec => plain {charset => "US-ASCII"}
}
stdout { codec => rubydebug }
}
2:
input
{
file
{
path => "/tmp/testXML1.log"
codec => multiline
{
pattern => "^/####"
}
type => "OSBLog"
}
}
output {
elasticsearch {
hosts => ["192.168.99.100:9200"]
}
stdout { codec => rubydebug }
}

Any help will be highly appreciated.

2

The error message you posted indicates that your configuration doesn't pass logstash --configtest. You're not telling us which of these configurations you had problems with, but when testing myself it seems the first one is okay but the second configuration given the following error with Logstash 2.3.4:

Missing a required setting for the multiline codec plugin:

  codec {
    multiline {
      what => # SETTING MISSING
      ...
    }
  } {:level=>:error}
The given configuration is invalid. Reason: Something is wrong with your configuration. {:level=>:fatal}

And indeed, a multiline codec with just pattern set doesn't make sense. I'm not sure what you're trying to do there. What does

I want to parse each line separately if it has #### in the beginning

mean? Please give more context to the problem. Concrete examples are better than descriptions.

3

Thanks for your reply Magnus, looks like hash characters that I included are wiped off, may be I should have been clear.

I am trying to process the log files (oracle) using logstash and sending it to Kibana dashboard. The logs are of following format:
####<Aug 11, 2016 11:40:53 AM CDT> <Info> <Security> <hostname> <>
####<Aug 11, 2016 11:40:54 AM CDT> <Info> <Security> <hostname> <>

The requirement is whenever logstash sees a line with #### then it should treat it like a new line, from the above case it should index two times on kibana dashboard (one for each line).

I had a similar kind of requirement where each line starts with '[', where I wrote the following script and it is working fine. I tried to use the same replacing '[' by '####'
input
{
file
{
path => "/tmp/testXML.log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline
{
pattern => "^["
negate => "true"
what => previous
}
type => "OSBLog"
}
}
output {
elasticsearch {
hosts => ["192.168.99.100:9200"]
}
stdout { codec => rubydebug }
}
I was of the view that Logstash is treating lines starting with '####' as commented lines.

4

I understand what you're trying to do, and I'm telling you that the configuration you showed us is invalid and obviously doesn't pass logstash --configtest.

I had a similar kind of requirement where each line starts with '[', where I wrote the following script and it is working fine. I tried to use the same replacing '[' by '####'

That should work just fine.

I was of the view that Logstash is treating lines starting with '####' as commented lines.

In configuration files yes, in log files no.

5

Can you give me some suggestion, where it will work through

6

Suggestion of what, exactly?

7

Thanks for answering my questions with patience. Looks like I did something wrong from my side, now it is working fine with the following code:
input
{
file
{
path => "/tmp/testXML1.log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline
{
pattern => "^####"
negate => "true"
what => previous
}
type => "OSBLog"
}
}
output {
elasticsearch {
hosts => ["192.168.99.100:9200"]
}
stdout { codec => rubydebug }
}

thanks once again.
~
~
~
~