Simply push json -> elasticsearch

[AaronNBrock]

I have a simple application that logs to a file in single complete json strings. Example

/tmp/my.log

{ "user": "bob", "event":"speak", "message":"Hello, world!" }
{ "user": "bill", "event":"sleep", "duration":8 }

I'd like to be able to push this directly to elasticsearch under a "my_app_logs" index so that it can be visualized in Kibana.

Attempt 1

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/my.log

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "changeme"

This does in fact push something to elastic search, but comes out as a string in the "message" field, I.e.:

{
  ...
  "message": "{ \"user\": \"bob\", \"event\":\"speak\", \"message\":\"Hello, world!\" }"
  ...
}

Attempt 2

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /mnt/sdb1/hyperloop/log.txt
  json.keys_under_root: true
  json.add_error_key: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "changeme"

This simply produces a bunch of errors when trying to push, specifically:

...
"stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: object mapping for [user] tried to parse field [user] as object, but found a concrete value"
...

Which from some googling seems to be a complaint about trying to push the wrong type to a field. With this information I noticed that the index created has some 500+ fields for many tools I'm not using "apache", "mysql" etc.! WHAT? Why is this default behavior?

Anyway, how can I go about getting my desired results? (preferable w/o all those fields I don't need)

[jsoriano (Jaime Soriano)]

Hi @AaronNBrock and welcome to discuss

You are on the good way, but take into account that some fields are reserved for the use of filebeat modules, for example user is intended to contain an object with user information according to ECS. The error you see is caused by that, you are trying to store a string in a field where Elasticsearch expects an object.

I'd recommend you to try to store your known fields in ECS-compliant fields, and keep the rest under json, you could do it with a config similar to this one (not tested):

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /mnt/sdb1/hyperloop/log.txt
  json.add_error_key: true
  processors:
  - rename:
      fields:
        - from: json.user
          to: user.name
        - from: json.event
          to: event.action
        - from: json.duration
          to: event.duration
        - from: json.message
          to: log.message

You can find the ECS fields in the documentation: https://www.elastic.co/guide/en/ecs/1.0/ecs-field-reference.html