Error indexing with httpjson

Elastic Stack Elasticsearch
1

Hi,
Im having a issue while indexing via filebeat httpjson to elastic

I have this configuration

filebeat.inputs:
- type: httpjson
  enabled: true
  config_version: 2
  fields_under_root: true
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  processors:
    - decode_json_fields:
        fields: ["message"]
  interval: 5m
  request.url: https://endpoint/slowQueries
  request.proxy_url: http://proxy:8080
  request.transforms:
    - set:
        target: header.api-key
        value: 'key'
output.elasticsearch:
  enabled: true
  hosts: ["elastic:9200"]
  index: "mongoatlas-%{+yyyy.MM.dd}"
  username: "elastic"
  password: "xxxxxx"
logging.level: info
logging.selectors: ["*"]
setup.template.name: "mongoatlas"
setup.template.pattern: "mongoatlas-*"

The result from the endpoint is a valid JSON

{
  "process": "atlas-k2w8aa-shard-00-02.l48fp.mongodb.net:27017",
  "queries": [ ]
}

This is the error that Im getting at ES

2022-03-08T17:06:47.236Z WARN [Elasticsearch] Elasticsearch/client.go:405 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc08201298c05b3cd, ext:2576639853, loc:(*time.Location)(0x55f47600d320)}, Meta:null, Fields:{"agent":{"ephemeral_id":"ab754087-f546-47df-b0bc-f4747179be3e","hostname":"filebeat-mongodb-7c8dc6b4bc-gpgkj","id":"af2be906-c358-4bad-ab25-e18208150fbe","name":"filebeat-mongodb-7c8dc6b4bc-gpgkj","type":"filebeat","version":"7.14.1"},"ecs":{"version":"1.10.0"},"event":{"created":"2022-03-08T17:06:46.201Z"},"host":{"name":"filebeat-mongodb-7c8dc6b4bc-gpgkj"},"input":{"type":"httpjson"},"message":{"process":"atlas-k2w8aa-shard-00-02.l48fp.mongodb.net:27017","queries":}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [message] of type [text] in document with id '-qt_an8BK1CtvcLQYWlD'. Preview of field's value: '{process:27017=atlas-k2w8aa-shard-00-02.l48fp.mongodb.net, queries=}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:361"}}

Help would be appreciated

2

Can't get text on a START_OBJECT has some places to start on this.

3

Mark, the json is valid i can make the query to the remote endpoint and parse it with jq or any other json parser, filebeat has an issue reading It.
And if I send the stream to logstash and It didn't complain about It.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.