Elastic Common Schema and Logstash and Ingest node processing tagging

Elastic Stack Logstash
1

Hi,
I'm working on setting up my custom sources to be more in line with ECS. In my filebeat configurations I had been adding the fields
environment (ex devl,qual,cert,prod)
building(ex buildingA)
location(ex thirdfloor)
ingest (pipeline)

Reading the ECS doc, I think the first three belong under the "host" top level.
host.environment
host.location
host.building
The last one I was using to track whether a message was processed through an ingest node (pipeline) or logstash.
I'm trying to figure out how this field would be better represented in ECS. Ideally, I want to record
Processed by logstash and which pipeline
or
Processed by ingest node and which pipeline
Does anyone have suggestions on how this should be represented? Or I could just use tags?
Thanks,
Tim

2

I don't know if you will get an exact answer here.

One of the goals of ECS is the affordance of sharing Kibana visualisations, across teams and in "solutions" offered by Elastic and others.

If you need to use these visualisations then they probably will not have elements that visualise the mechanism the data was ingested - if you pick an ECS field to carry this data, be sure that any reused visualisations do not expect (and therefore use) the ECS field you picked for a different purpose.

In this doc I see...

  • If no relevant ECS Extended field exists, consider keeping your field with its original details, or possibly renaming it using ECS naming guidelines and attempt to map one or more of your original event fields to it.

ECS Naming guidelines

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

4

Currently ECS doesn't define a whole lot, in terms of user's custom environments.

There's a few things you can use, in line with ECS.

  • The "geo" field set can be nested in side host, and "geo.name" is for a free-form value (not the result of ip geolocation, for example. So concretely, perhaps you can capture the location in host.geo.name
  • The rest I see should be in custom fields. The simplest place to add those is under labels, so perhaps labels.environment: prod & so on