How to find JSON-Properties from LogFile as Fields in Elasticsearch?

Elastic Stack Logstash
1

Hi,

I have to create an ELK-Showcase for my team.

The task is to take Log(files) from an ASP.NET-Application with Filebeat, forward them via Logstash to Elasticsearch and create a few sample-dashboards in Kibana.

I have a basic setup up and runnig now. But I am not really happy about what I´m seeing in elasticsearch. Right now I find basically the whole Log-message in the message-field in ES. But I want to see all JSON-Properties separated, to query them easily. Example:

I have created a new ASP.NET Core WebApplication, with Serilog as a logger. One Log-entry can look like this:

{
"Timestamp":"2018-01-02T17:15:37.8033208+01:00",
"Level":"Information",
"MessageTemplate":"{ActionName} has been invoked. Output is {Output}",
"Properties":
{
"ActionName":"GetAction",
"Output":["value1","value2"],
"SourceContext":"ValuesLogger",
"ActionId":"459f66df-299e-44e7-813e-45749f4f8d48",
"RequestId":"0HLAHVG7IOG5M:00000001",
"RequestPath":"/api/values",
"Scope":["GetValues"]
}
}

I want to see something similar to this JSON in ES, to find e.g. every Message from 2017/12 with above message-template. What is the best way to achieve that?

Right now I have installed ES with MSI-Installer (without any Extras), Logstash is configured like this:
input {
beats {
port => "5044"
}
}

output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}

And here is the filebeat configuration:

output.logstash:
hosts: ["localhost:5044"]

filebeat.prospectors:

  • type: log
    paths:
    • C:\myPath*.log

I´ve read something about logstashs JSON-Filter but I don´t really understand if this is what I am looking for and if so, how to configure it correctly.

Does anyone have an idea what I´m missing, doing wrong etc.?

Thanks in advance,
Josh

2

One Log-entry can look like this:

On one line or spread across multiple lines as in your example?

3

On one line.. Just formatted it like this for better readability.

4

Hi @magnusbaeck,

I´ve just seen that my initial answer is not correct: If I log any Microsoft-ExceptionMessages, the message contains line break characters \r\n. Does this have any implications regarding your question?

5

After a long day I´ve found out, that logstash was not the right place for my configuration. My answer was here: Parse JSON data with filebeat - #3 by calmandniceperson

This configuration for filebeat.prospectors finally solved my problem.

@magnusbaeck nevertheless thank you for your answer!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.