Custom fields are empty

Elastic Stack Beats
1

Testing the functionality of using custom fields in order to better sort data for users, but I'v not been able to get it to work properly. In my test case, I have a simple Ubuntu client with metricbeat installed locally and my metrcibeat.yml updated to include:

# Optional fields that you can specify to add additional information to the
# output.
fields_under_root: true
fields:
  environment: "test"
  owner: "Me"

with everything set I then pointed it directly at my elasticsearch:9200 and can see the index created (good). However when it comes to visualize that data in kibana, the fields are hidden because they are all empty? To make things more interesting (less?), if i simply watch the logs in /var/log/metricbeat/metricbeat it does regularly post "Total non-zero values" ... but never posts info on those fields?

So I feel like I'm overlooking something obvious ... but really the goal is simple: I want my documents to have custom fields so departments/environments can easily be filtered without being "forced" to create lots of indices.

2

The fields shouldn't be empty, but they might be missing from the Kibana index pattern. If you go to the Discover page and look at an event, do you see the custom fields?

It could be that you need to refresh the index pattern (in Kibana, go to Management / Index patterns. However, make sure you have data, because refreshing removes the fields for which there are no values.

3

First off, thanks for your feedback. Still don't have it working, but Ill try to give a bit more info:

In kibana: I do see the fields when I check in Management --> Index Patterns. Both custom fields (from the above example) are "search-able" but the other fields are cleared. Each field also has an accompanying "environment.keyword" and "owner.keyword". These are "aggregate-able". However, If I go into discover-->available fields ... I can only find them if i clear the "hide missing fields" checkbox as the system says they are not present in any documents.

Now I did delete the test index, then created a new template that has these fields added so that is likely why they are "present" in elaticsearch despite no data coming in ... but I can't figure out why metricbeat is not providing these fields in the documents it sends.

Is there a simple way to have metricbeat output to a file first just so I can validate it's providing the fields and rule out if the client is even sending the correct data?

4

There is a file output.

5

Thanks. Turned that on and, annoyingly enough, it looks like the fields re populated.

Yet I dont see them in kibana.

I guess the next step is to try to query them directly in elasticsearch and see if the data is there and I just have kibana issues ...

6

OK so there's a larger ingest problem here. simple curl for the doc in the index shows basically an empty doc that looks nothing like that metricbeat is commiting.

Maybe I need to go back to to ELK 101 ... but I'm pretty sure I should see a LOT more than this when I query as a sample:

curl -u elastic:changeme http://es-demo:9200/system-2017.07.24/doc/AV11pX-PhgyiYRXj32uy?pretty
{
  "_index" : "system-2017.07.24",
  "_type" : "doc",
  "_id" : "AV11pX-PhgyiYRXj32uy",
  "_version" : 1,
  "found" : true,
  "_source" : { }
}

compared to what metric beats is sending I'm losing every meaningful piece of data and being left with ... nothing.

7

Could you please share the full beat config file that you are using.

8

It's pretty dry as it's a test run:

quick gist share

9

Customizing the index name is a bit of an advanced task. I'd leave it as the default if you are getting started.

When you customize the index then you need to modify the index template to apply to the new index name. Also you will need to create a custom kibana index pattern to match your index name. The dashboards are likely affected as well because they are looking for data in metricbeat-*.

10

problem is "metricbeats" is a technology term ... naming the indexes around business/resource terms shortens the learning curve for the consumers. Down the road.

I can flip it back for the purpose of the test, but it doesn't answer why the index is created and yet all the fields are effectively blank unless you feel it's hitting some kind of filter?

11

I was thinking that you might be encountering a mapping issue. Have you checked the elasticsearch logs?

12

Going to double check those but as I do ... a quick validation of the index templates I think is in order as well. Ill let you kow what I dig up.

13

OK guys, this has finally been solved and I'm posting thesolution here for reference by others:

Root cause the wrong index template was being applied to the index which was stripping the extra data.

As this was a test box, another tech had added a template index with the query "*" at a higher order than my own index. This was mangling the data coming in (for everyone ... luckily .. test box). Removing the offending template solved the issue, and I my index is being created properly with all data and cutom fields.

Thank you everyone for giving ideas and input that helped me lead to this problem.

1 Like
14

This topic was automatically closed after 21 days. New replies are no longer allowed.