Hi there,
first of all please make use of the code formatter tool ( ) when pasting non plain text (such as pipeline conf file lines) cause otherwise it'll be more difficult to read and go through.
Now, to my understanding you'd like to process that array of json of yours from a file and send each json as a separate document, each one with its fields extracted.
Now, first thing I suggest you should do is edit your source file (if possible) to have the whole json on a single line and be careful to leave a empty line at the end of the file. So your file should look something like this:
Also, please note that the one you posted is not a valid json. In fact, pasting it in any json validator, it'll highlight the useless commas after the "timestamp" values and the quotes around the nested KEY4 value. To be a valid json, yours should look something like this:
[
{
"KEY1": "ABC",
"KEY2": "ABC",
"KEY3": "ABC",
"KEY4": {
"region":"11",
"UserSessionId":"222",
"UserId":"gllexie"
},
"host": "ABC",
"timestamp": 1595411041516
},
{
"KEY1": "ABC2",
"KEY2": "ABC2",
"KEY3": "ABC2",
"KEY4": {
"region":"22",
"UserSessionId":"No%20CPM%20Profile",
"UserId":"gllexie"
},
"host": "ABC2",
"timestamp": 1595411041516
}
]
Now, having said that, what you could do (after you managed to have your json shrinked in one line with a trailing empty line in the file) is a pipeline like the following:
input {
file {
path => "path/to/json/file"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => "json"
}
}
filter {
mutate {
remove_field => ["host", "KEY4"]
}
}
output {
stdout{}
}
Having your source json file on one line will avoid you that multiline and all the hassles that come with it.
Obviously you can replace the standard output with your ES instance.