I've spent quite a while reworking my apache/nginx Logstash filter to be ECS-compliant, mostly using the url and http field sets. The newly-formated events are indexing correctly and look fine in Kibana Discover, but in the Infra/Logs interface most of the message field displays "undefined".
I'm really not sure what's going on here. I assumed the logs interface would just display the message field but it seems to be trying to parse some other fields in the event.
The settings in the Logs interface are pretty standard. Just a customised index name (which it can read fine) and some columns. The only thing of note is that this is in a particular user's space, but it's the only place we currently have http events.
How do I fix this, and continue using ECS fields for my events? At the moment I've had to roll back to my old filters which doesn't bode well for converting the rest of my pipeline to ECS.
I think it's great that you're aiming for ECS-compatibility. And thanks for providing such helpful examples for debugging.
In this case you're falling victim to some peculiarities of the Log UI's message formatting, which we introduced to stay compatible with old versions of various filebeat access log modules. The filebeat modules used to leave out the message field, which is why we try to reconstruct it from certain pre-parsed fields if they're available.
The first undefined you're seeing is the event.module field, which you could set to something informative like apache or nginx or whatever your source is.
The /?undefined means that the url.path is correctly represented ("/" here), but the url.query is not set.
The HTTP/undefined indicates that http.version is not set.
The last undefined is due to the lack of http.response.body.bytes.
Obviously we've not done a great job supporting partial documents and this is something we should improve. I've created https://github.com/elastic/kibana/issues/39575 a while ago to allow overriding the heuristics. Please feel free to weigh in with your perspective. I'll think about whether we can do something more immediate in the short term.
That was all very helpful, especially generic_webserver.ts as it showed I was incorrectly using client.ip instead of source.ip in my filter.
One thing that is still a problem though is infra/logs is still expecting url.query to contain a value,
which isn't always the case. In this case it will still print ?undefined even for a plain URL, e.g., GET /.
The filter I'm using will resolve url.query if a query is present in the URL, but there isn't one here so the field doesn't exist.
Is there any value I can set to avoid this, or is it a limitation of the interface?
[edit] As a work-around I've taken out the ecs.version field so the message formatting isn't invoked, and we have "normal" http log messages again. Hopefully there's a nicer way to resolve this though.
Glad to hear you're making progress. Would it be possible to set an empty string for url.query in the document source instead of not setting the field at all?
That's not ideal - we should definitely be smarter about that. Unfortunately I can't think of anything I could offer as an immediate solution beyond what you already figured out. Maybe we can manage to introduce a quick way to disable the message reconstruction during upcoming releases.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
And thanks for providing such helpful examples for debugging.
Unfortunately I can't think of anything I could offer as an immediate solution beyond what you already figured out. Maybe we can manage to introduce a quick way to disable the message reconstruction during upcoming releases.