How to split a single line message, into parts

Elastic Stack Logstash
1

Hi, I've created a logstash to filter a log and output a single message. Then I want to split that message into several fields. What filter plugin should I use? I try to use split filter and it doesn't work

The example messege
0200F338400988E08000000000000500000416603494608702093001100000004000000002241005250000000042290710052502246011011100110345106450005000481422907ATM35802BSMCUST451ABCFHUSJOKKLMNA360107102901902000003451

The Field i want:

field 1: 0200
field 2: F3384
etc..

Logstash

input {
  beats {
    port => 5044
  }
}

filter {
     ruby {
	    # old nw     code =>  "  event.set('msgfilter', event.get('message').scan(/\s{7}[a-zA-Z0-9]+(?=\r\n|\s+\r\nReceived)/) ) "
	    
		#code =>  "  event.set('msgfilter', event.get('message').scan(/\s{7}[a-zA-Z0-9\s]+(?=\r\n)/) ) "
	    code =>  "  event.set('msgfilter', event.get('message').scan(/\s{7}[a-zA-Z0-9\s]+(?=\n)/) ) "
	    id => "ruby-counter"
     }

       # merge array of strings
       mutate { join => { "msgfilter" => "" }  }
	   # remove spaces
	   mutate { gsub => [ "msgfilter", " " , "" ] }
	   
	   mutate {  
	   remove_field => [ "message", "tags", "path"]  
	   }
	   
	   #split {
        #add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
      #}

}

output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

Thank you

2

"etc." does not tell us enough. Do you want a string broken into 9 character chunks which are then split into two fields of 4 and 5 characters? If so, ruby and .scan look like a good way to go. But the nature of .scan is that you get arrays of arrays, so you may need to rearrange them. See here and here.

4

I've followed the way you suggested, and it still doesn't work. Which part did I go wrong?

filter {
     ruby {
	    # old nw     code =>  "  event.set('msgfilter', event.get('message').scan(/\s{7}[a-zA-Z0-9]+(?=\r\n|\s+\r\nReceived)/) ) "
	    
		code =>  "  event.set('msgfilter', event.get('message').scan(/\s{7}[a-zA-Z0-9\s]+(?=\r\n)/) )"
	    id => "ruby-counter"
     }

       # merge array of strings
       mutate { join => { "msgfilter" => "" }  }
	   # remove spaces
	   mutate { gsub => [ "msgfilter", " " , "" ] }
	   
	   mutate {  
	   remove_field => [ "message", "tags", "path"]  
	   }
	   
	ruby {
		code => '
            s = event.get("msgfilter")
            if s
                event.set("matches", s.scan(([a-zA-Z0-9\s]{4}/);([a-zA-Z0-9\s]{5}/))
            end
        '
     }
}
5

You have not explained what results you are getting or what you do not like about them.

6

It' doesn't work when i running the logstash

image
image1899×769 69 KB

7

Please do not post pictures of text, just post the text, they are impossible to search, some people will be unable to view them, and we cannot copy and paste information to try and reproduce and diagnose the problem.

.scan take a regexp, so you need // around the regexp

s.scan(/([a-zA-Z0-9]{5})/)

Not sure why you have added () to create a capture group when that pattern only matches one thing.

8

I just tried it for one value, after it works I will add several other values.

I've tried it and it works in powershell but it doesn't show up in kibana, I've refreshed the index pattern but it's still the same.

image

9

Hi, it's working now. But how do I retrieve with several different number of digits like the example in the photo? And how do I give each field a name. Sorry I'm new to elastic. Thank you
image

10

Please do not post pictures of text. They are not searchable, some people cannot even see them, and nobody can copy and paste them to try to reproduce and diagnose the issue.

That said, you might want something like this, or this, or this. Or maybe not, you have not explained what you are trying to do.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.