Help with JSON output format

Elastic Stack Logstash
1

Can someone help me with writing correct configuration file such that JSON output

  1. Does not contain metadata like source file name, time stamp, host name etc..
  2. Each column in input does not become a JSON document

Listed below is a line from input file and 2 rows from output file.

Input file format:
1|Toy Story (1995)|01-Jan-1995||http://us.imdb.com/M/title-exact?Toy%20Story%20(1995)|0|0|0|1|1|1|0|0|0|0|0|0|0|0|0|0|0|0

Current JSON output:
{"path":"C:\ml-100k\ml-100k\u.item","@timestamp":"2017-06-18T10:08:49.210Z","@version":"1","host":"WIN-KALHJUSVJJU","message":"1"}{"path":"C:\ml-100k\ml-100k\u.item","@timestamp":"2017-06-18T10:08:49.225Z","@version":"1","host":"WIN-KALHJUSVJJU","message":"Toy Story (1995)"}

Thank you,
Amit

2

1 - You can use remove_field for some of this.
2 - Depends on what your config looks like now.

If you are having issues then posting the specifics would be good.

3

Listed is the config file. Any pointers or examples on how to implement the remove_fields ?

input{
file {
path => "C:\ml-100k\ml-100k\u.item"
start_position => ["beginning"]
delimiter => "|"
sincedb_path => "nul"
}
}

filter {
}

output {
file {
codec => json
path => "C:\items.json"
}

}

4

You should really use something like the csv filter to properly structure the event.

That has remove field too. https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html