Hi,
Is there such an option to stop publishing (anything new) when output fails for whatever reason?
To give an example
My Filebeat config (minimal) is like this:
filebeat.prospectors:
- type: log
json.keys_under_root: true
paths:
- /path/to/file.log
output.elasticsearch:
hosts: ["127.0.0.1:9200"]
template.enabled: false
index: "log-%{[index]}"
My log file is already in json format (one json per line)
For different reasons (eg: field type mismatch, missing mappings, elastic down, etc) publish action may fail. I don't want to loose that log line. I'd like Filebeat to stop harvesting and publishing and alert me somehow (lock file, email, smoke signals, anything I can monitor). Then I can fix the issue and restart Filebeat.
All I have now is a WARN in the log like this (I know what is is, it's missing mapping, it's intentional for this test):
2018-09-25T23:21:05.999+0300 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee2c40c3a701471, ext:15139988, loc:(*time.Location)(0x52ca960)}, Meta:common.MapStr(nil), Fields:common.MapStr{"beat":common.MapStr{"name":"localhost", "hostname":"localhost", "version":"6.2.4"}, "source":"/path/to/file.log", "offset":127, "index":"2018-08-31", "message":"test log for demo", "lines":[]interface {}{common.MapStr{"line":0, "message":"message1"}, common.MapStr{"line":1, "message":"message2"}}, "prospector":common.MapStr{"type":"log"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4200bdd40), Source:"/path/to/file.log", Offset:127, Timestamp:time.Time{wall:0xbee2c40c3a60e101, ext:14143782, loc:(*time.Location)(0x52ca960)}, TTL:-1, Type:"log", FileStateOS:file.StateOS{Inode:0x23d3f8, Device:0x100000a}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"object mapping [lines] can't be changed from nested to non-nested"}
Thank you