Metricbeat: upload output file to elasticsearch manually

Elastic Stack Beats
1

Hi,

I'd like to post-process the metricbeat output, so I am using the "File output" module. However, when uploading the data to elasticsearch, I am getting an "illegal_argument_exception":

A single example message taken from metricbeat.1 file

msg = '{"@timestamp":"2017-02-21T21:33:35.659Z","beat":{"hostname":"elk1","name":"elk1","version":"5.2.1"},"metricset":{"module":"system","name":"cpu","rtt":53},"system":{"cpu":{"cores":1,"idle":{"pct":0.000000},"iowait":{"pct":0.000000},"irq":{"pct":0.000000},"nice":{"pct":0.000000},"softirq":{"pct":0.000000},"steal":{"pct":0.000000},"system":{"pct":0.000000},"user":{"pct":0.000000}}},"type":"metricsets"}'

PUT on a new index fails:

In [1]: requests.put('http://elk:9200/metricbeat-1/', data=msg).text
Out[1]: u'{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"unknown setting [index.@timestamp] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"}],"type":"illegal_argument_exception","reason":"unknown setting [index.@timestamp] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"},"status":400}'

Additional:

  • Using version 5.2.2 of elasticsearch and metricbeat
  • No indices, mappings, index templates exist in elasticsearch (clean installation)
  • Whether the default metricbeat.template.json is loaded or not, does not seem to make a difference.

How can I manually upload the metricbeat output file to the elasticsearch ingest node, so that it will look like a regular metricbeat client?

Kind regards,
Wiem DW

2

TBH I never tried that and the file output was no designed with this in mind. I'm not sure why you get a plugin error above but one thing that is probably missing from the event is the type.

What is the client you are using above?

3

The type seems to be set to "metricsets" (see the end of the example 'msg=').

I am using python-requests for testing, so basically just a plain HTTP PUT call.

4

Can you remove the @timestamp from the event again and see if that works? Are you using any ingest pipeline in ES?

5

When I remove the '@timestamp' field, the error remains the same but about the 'unknown setting [index.beat.hostname]" field.

I am not using any pipeline or mapping, just a auto-created index.

6

Could you try to put the document into elasticsearch with just using curl and see what happens? Which ES version are you using?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.