How create a filter to my rails log?

Elastic Stack Kibana
1

My stack is using Lograge (Gem) -> Filebeat -> Logstash -> Elastic -> KIbana.

Elastic is indexing this in format:

{ "_index": "filebeat-2016.03.09", "_type": "log", "_id": "AVNbq1ImHUX-l-CtdIqR", "_score": null, "_source": { "message": "{\"method\":\"GET\",\"path\":\"/extract_json\",\"format\":\"html\",\"controller\":\"extracts\",\"action\":\"show_json\",\"status\":200,\"duration\":12809.6,\"view\":0.11,\"db\":22.38,\"params\":{},\"env\":\"development\",\"mdc\":\"33bcbbeb896379c07f309024e1a9c810\",\"host\":\"localhost\",\"@timestamp\":\"2016-03-09T13:57:41.119Z\",\"@version\":\"1\",\"message\":\"[200] GET /extract_json (extracts#show_json)\"}", "@version": "1", "@timestamp": "2016-03-09T13:57:42.550Z", "beat": { "hostname": "brasilct-Aspire-E5-573G", "name": "brasilct-Aspire-E5-573G" }, "count": 1, "fields": { "origin": "development", "technology": "Ruby on Rails" }, "input_type": "log", "offset": 67638, "source": "/home/brasilct/dev/ruby-projects/bonusesfera/log/lograge_development.log", "type": "log", "host": "brasilct-Aspire-E5-573G", "tags": [ "beats_input_codec_plain_applied" ] }, "fields": { "@timestamp": [ 1457531862550 ] }, "sort": [ 1457531862550 ] }

However I do not want that "message" to be a string.

How do I make the field "message" in searchable fields, which are not strings? Where should I set up? Are the filters of Logstash or need to add some configuration in filebeat.yml?

2

I solved stopped filebeat service and reconfigured logstash.conf like above:

input {
file {
type => "rails"
path => ["/home/brasilct/dev/ruby-projects/bonusesfera/log/lograge_development.log"]
codec => json {
charset => "UTF-8"
}
}
}
filter{
grok{
match => [
"message",
"Started %{WORD:method} (?[^ ]+) for.*%{IP:ip} at %{TIMESTAMP_ISO8601:time}"
]
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
}
}

Now as the Elastic indexes my log:

{
"_index": "logstash-2016.03.09",
"_type": "rails",
"_id": "AVNb61syHUX-l-CtdIqZ",
"_score": null,
"_source": {
"method": "GET",
"path": "/extract_json",
"format": "html",
"controller": "extracts",
"action": "show_json",
"status": 200,
"duration": 11923.5,
"view": 0.41,
"db": 228.28,
"params": {},
"env": "development",
"mdc": "649326dc6cda3247c2dcc1c10531822b",
"host": "localhost",
"@timestamp": "2016-03-09T15:07:43.614Z",
"@version": "1",
"message": "[200] GET /extract_json (extracts#show_json)",
"type": "rails",
"tags": [
"_grokparsefailure"
]
},
"fields": {
"@timestamp": [
1457536063614
]
},
"sort": [
1457536063614
]
}

But still I want to set the filebeat.

Anyone can me help?

3

But still I want to set the filebeat.

You mean use Filebeat instead of Logstash? Use the json codec for the beats input just like you did with the file input.

4

Do you say change logstash configuration as below? (Sorry for my english)

input {
beats {
port => 5044
codec => json {
charset => "UTF-8"
}
}
}

5

Yes, that should work (assuming all messages sent to that port are JSON).

6

Great, it's working! Thanks! :smile:

7

@magnusbaeck How to configure logstash which have two inputs, one with json codec for rails logs and other without codec for topbeat or packetbeat logs?

8

Use two beats inputs with different ports and different codec settings.

9

input {
beats {
port => 5044
codec => json {
charset => "UTF-8"
}
}
}

input {
beats {
port => 5045
}
}

But how do I configure the "filebeat.yml" that there will be two outputs in logstash one on port 5044 and another 5045?

10

That's a good point. I'm not sure you can do that. You might want to start a new thread in the Filebeat category to get the attention of the Filebeat experts.