How create a filter to my rails log?


(Cândido Sales Gomes) #1

My stack is using Lograge (Gem) -> Filebeat -> Logstash -> Elastic -> KIbana.

Elastic is indexing this in format:

{ "_index": "filebeat-2016.03.09", "_type": "log", "_id": "AVNbq1ImHUX-l-CtdIqR", "_score": null, "_source": { "message": "{\"method\":\"GET\",\"path\":\"/extract_json\",\"format\":\"html\",\"controller\":\"extracts\",\"action\":\"show_json\",\"status\":200,\"duration\":12809.6,\"view\":0.11,\"db\":22.38,\"params\":{},\"env\":\"development\",\"mdc\":\"33bcbbeb896379c07f309024e1a9c810\",\"host\":\"localhost\",\"@timestamp\":\"2016-03-09T13:57:41.119Z\",\"@version\":\"1\",\"message\":\"[200] GET /extract_json (extracts#show_json)\"}", "@version": "1", "@timestamp": "2016-03-09T13:57:42.550Z", "beat": { "hostname": "brasilct-Aspire-E5-573G", "name": "brasilct-Aspire-E5-573G" }, "count": 1, "fields": { "origin": "development", "technology": "Ruby on Rails" }, "input_type": "log", "offset": 67638, "source": "/home/brasilct/dev/ruby-projects/bonusesfera/log/lograge_development.log", "type": "log", "host": "brasilct-Aspire-E5-573G", "tags": [ "beats_input_codec_plain_applied" ] }, "fields": { "@timestamp": [ 1457531862550 ] }, "sort": [ 1457531862550 ] }

However I do not want that "message" to be a string.

How do I make the field "message" in searchable fields, which are not strings? Where should I set up? Are the filters of Logstash or need to add some configuration in filebeat.yml?


(Cândido Sales Gomes) #2

I solved stopped filebeat service and reconfigured logstash.conf like above:

input {
file {
type => "rails"
path => ["/home/brasilct/dev/ruby-projects/bonusesfera/log/lograge_development.log"]
codec => json {
charset => "UTF-8"
}
}
}
filter{
grok{
match => [
"message",
"Started %{WORD:method} (?[^ ]+) for.*%{IP:ip} at %{TIMESTAMP_ISO8601:time}"
]
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
}
}

Now as the Elastic indexes my log:

{
"_index": "logstash-2016.03.09",
"_type": "rails",
"_id": "AVNb61syHUX-l-CtdIqZ",
"_score": null,
"_source": {
"method": "GET",
"path": "/extract_json",
"format": "html",
"controller": "extracts",
"action": "show_json",
"status": 200,
"duration": 11923.5,
"view": 0.41,
"db": 228.28,
"params": {},
"env": "development",
"mdc": "649326dc6cda3247c2dcc1c10531822b",
"host": "localhost",
"@timestamp": "2016-03-09T15:07:43.614Z",
"@version": "1",
"message": "[200] GET /extract_json (extracts#show_json)",
"type": "rails",
"tags": [
"_grokparsefailure"
]
},
"fields": {
"@timestamp": [
1457536063614
]
},
"sort": [
1457536063614
]
}

But still I want to set the filebeat.

Anyone can me help?


(Magnus Bäck) #3

But still I want to set the filebeat.

You mean use Filebeat instead of Logstash? Use the json codec for the beats input just like you did with the file input.


(Cândido Sales Gomes) #4

Do you say change logstash configuration as below? (Sorry for my english)

input {
beats {
port => 5044
codec => json {
charset => "UTF-8"
}
}
}


(Magnus Bäck) #5

Yes, that should work (assuming all messages sent to that port are JSON).


(Cândido Sales Gomes) #6

Great, it's working! Thanks! :smile:


(Cândido Sales Gomes) #7

@magnusbaeck How to configure logstash which have two inputs, one with json codec for rails logs and other without codec for topbeat or packetbeat logs?


(Magnus Bäck) #8

Use two beats inputs with different ports and different codec settings.


(Cândido Sales Gomes) #9

input {
beats {
port => 5044
codec => json {
charset => "UTF-8"
}
}
}

input {
beats {
port => 5045
}
}

But how do I configure the "filebeat.yml" that there will be two outputs in logstash one on port 5044 and another 5045?


(Magnus Bäck) #10

That's a good point. I'm not sure you can do that. You might want to start a new thread in the Filebeat category to get the attention of the Filebeat experts.


(system) #11