Using certain fields to visualize

[yeaustin]

Hello, I'm using the ELK stack for the first time and I'm trying to visualize some fields.

Let's say that I have a very large number of fields each containing a value. Each of the fields has a different name, but all have the same postfix (e.g. field-1000-postfix=1, field-1001-postfix=3, field-1945-postfix=32, foo-1843-postfix=22). I have been able to make Elasticsearch recognize them as fields with int values using Logstash. As the log files update every few seconds, more fields are added.

Is there a way to find and display 5, for example, of the fields with the largest value with that postfix? I understand that there's a way to find documents with fields using the console in the DevTools menu of Kibana, but I haven't been able to find clear guidance on how to take something like that over to Kibana's Visualize.

I've also seen that using terms can be a way to go. I'm not sure how to add my fields to be associated with a term.

I would be grateful for any insight. Thanks!

[tsullivan (Tim Sullivan)]

It seems like you are trying to use field names as terms. Elasticsearch can't aggregate on field names. I think what you really what is to have a field for the term. It looks like the terms in your data are "type" and "category", maybe. So maybe you could map the data like:

{
  "type": "field",
  "category": "1000",
  "value": 1
}

{
  "type": "field",
  "category": "1001",
  "value": 3
}

{
  "type": "field",
  "category": "1294",
  "value": 32
}

{
  "type": "foo",
  "category": "1843",
  "value": 22
}

Having the number of fields update constantly is going to be a bad situation. Every time a new field appears, Elasticsearch has to update the mapping for the index. In Kibana, the mapping is stored in an Index Pattern document, along with your scripted fields and field formatters. If the mapping changes, then you need to manually refresh the mapping in Kibana. So Kibana will really not be able to do anything with this scheme.

@yeaustin

I’m not sure how to add my fields to be associated with a term.

If you are using Logtash, you can use a filter plugin to break up the data strings that you are reading into multiple fields. This is where you would add the fields for the terms you need. If you're using an input plugin, you could also use add_field to add something common to all the data going through a pipeline.

Hope this helps!