Logstash json ouput choose keys and values

[Sous_Lesquels]

Say I have a pipeline that writes events to two files. Both files should be in json format, but they should have different fields from the event and in a different format.

For example, say the event is like this:

{"a": 1.3, "b": "Jul 15 2019", "c": false, "d": "cn4339b371"}

I might want to store these two jsons in two separate files:

{"b1": "07/15/2019", "d": "cn4339b371"}
{"a": 1.3, "b2": "20190715", "c": false, "d": "cn4339b371"}

where key b goes to b1 and b2 and are formatted differently and a & c are dropped from the first file.

Is there an example of how this can be done with logstash that I can look into?

[Badger]

I suggest you look at the forked path pattern for pipeline to pipeline communication.

[Sous_Lesquels]

Thanks Badger - I'll look into it. However, right now we're starting one logstash instance and specifying path.config. I.e. we're sharing the settings, thus pipeline.yml as well between different instances, so this will be problematic to implement. Is it possible to do this within one pipeline in order to preserve this setup? Or should we consider splitting this up into multiple instances? Or other suggestions?

[Badger]

@Sous_Lesquels

Is it possible to do this within one pipeline

The documentation for the forked path pattern mentions that it can be done in one pipeline 'through creative use of the clone filter and if/else rules'.

[Sous_Lesquels]

Thanks Badger - I'll try that.

The other part of the question - any reference configs for how I can define output json fields from the event?