Can't put http_endpoint messages to root

Elastic Stack Beats
1

Here is my config:

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 127.0.0.1
  listen_port: 20000
  response_code: 204
  response_body: null
  prefix: null
  fields:
    app_name: "MedService.UI"
  fields_under_root: true

processors:
  - timestamp:
      field: Timestamp
      layouts:
        - '2006-01-02T15:04:05.999999999Z07:00'
  - drop_fields:
      fields: [Timestamp]

setup.ilm:
  enabled: false
  rollover_alias: "filebeat"
  pattern: "{now/M{yyyy.MM}}-000001"
  policy_name: "filebeat"

setup.template:
  type: index
  name: "filebeat"
  pattern: "filebeat-*"

logging.level: warning

I'm trying to transfer log messages from a filebeat's http endpoint to Elastic instance.
The messages transfers fine, except they are all prefixed with "json" which I wanted to avoid and place them at the root level.
This is how it works with log input:

- type: log
  enabled: true
  paths:
    - logs\*.log
  json:
    keys_under_root: true
    json.overwrite_keys: true
  fields:
    app_name: "MedService.API"
  fields_under_root: true

Using this config my messages are all placed at the root level.
However, when using http_endpoint input there is no such parameter.

When setting prefix to "" or '' the messages just stop showing up in Kibana.
When changing prefix to something like "logs" the messages are correctly prefixed with "logs".
When setting prefix to something like null or just leave it empty, then default "json" appears in messages as a prefix.

Is it possible to place my logs at the root level using http_endpoint input?

2

Taking a wild guess here since I'm not a Golang expert in any way but could it be that this is the source of the issue?

func (h *httpHandler) publishEvent(obj common.MapStr, headers common.MapStr) {
	event := beat.Event{
		Timestamp: time.Now().UTC(),
		Fields: common.MapStr{
			h.messageField: obj, // when setting `prefix` (aka `messageField`) to empty this might not work since the field will be empty, or does Go allow this?
		},
	}
	if h.preserveOriginalEvent {
		event.PutValue("event.original", obj.String())
	}
	if len(headers) > 0 {
		event.PutValue("headers", headers)
	}

	h.publisher.Publish(event)
}
3

Ur correct, see beats/config.go at master · elastic/beats · GitHub. if u don't set a field name it will default to json. It doesn't look like you'll be able to have the fields at the root unless you move each field individually using processors.

4

Could it be a bug or just by design?
I think it would be logical to make all inputs with similar functionality to work, well, similarly.

5

Its by design. All the other inputs put the message/data initially in the message field prior to being parsed. This is essentially the same thing. You can use Filebeat or Elasticsearch ingest processors to move the fields to where you want them.

6

Thanks.
I guess "this is the way".

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.