Parse log file with two formats in it

[Abhilash_B]

I have a log file with two formats in it. The metadata is more like a key value format where the delimiter is '=', the actual data is in a csv format. Both these formats have been bundled into a single file. Can I generate each event as individual document with metadata as part of every document?

The log file is like this.

Appreciate any help on this.
Thanks

[Badger]

This should give you an idea of how to do it. It drops comments and lines that are just whitespace. Then it parses key=value and stashes it in a class variable. Then it parses anything with multiple commas as a csv. That leaves you with odds and ends to handle, such as

The RTC is running 0 hours, 0 mins and 2 secs behind real time.
The flash last updated 1/3/2018 at 2:04
The last .tab file placed 1/3/2018 at 14:01

If you need to get data out of those lines use grok. Make sure you anchor your patterns using ^.

    if [message] =~ /^#/ {
        drop {}
    } else if [message] =~ /^\s*$/ {
        drop {}
    } else if [message] =~ /^[A-Za-z0-9]+=/ {
        ruby {
            init => '
                @@metadata = {}
            '
            code => '
                msg = event.get("message")
                matches = msg.scan(/^([A-Za-z0-9]+)=(.*)/)
                m = matches[0]
                @@metadata[m[0]] = m[1]
            '
        }
        drop {}
    } else if [message] =~ /,.*,.*,/ {
        csv {
            autodetect_column_names => true
        }
        ruby {
            code => '
                event.set("metadata", @@metadata)
            '
        }
    }

[Abhilash_B]

@Badger, Any ideas on what my input plugin should be?

[Badger]

You probably want either a file input or filebeat connecting to a beats input.

[Abhilash_B]

@Badger, My metadata looks like this. Any idea how should I remove the trailing "\r" in the values of each key.

Thanks.

[Badger]

\r counts as whitespace, so

 mutate { strip => [ "message" ] }

will remove it and the space before it.