Logstash always outputting only 9 documents

[mohsin106 (Mohsin)]

I have a json file with 4K json objects in it. Each object is on its own line. I'm using the file plugin with the multiline codec to have Logstash parse all the objects. I'm also flattening the json objects and then sending the output to standard out using the stdout plugin. However, I'm only seeing 9 records print to the screen.

I want to do some aggregation on the data and need to be able to see all the records print to the screen. How can I accomplish this?

After reading a couple of articles in the community I came across @Badger's solution to read in JSON content and I configured my Logstash config like this:

input {
	file { 
		path => "/a/b/file1.json" 
		sincedb_path => "/dev/null" 
		start_position => beginning 
		codec => multiline { 
			pattern => "^Spalanzani" 
			negate => true 
			what => previous 
			auto_flush_interval => 1 
			multiline_tag => "" 
		} 
	}
}

filter { 
	json { 
		source => "message" 
		remove_field => [ "message" ] 
	}
	if [fields] {
        ruby {
          code => '
            event.get("fields").each { |k, v|
              event.set(k,v)
            }
            event.remove("fields")
          '
        }
    }

    if [tags] {
        ruby {
          code => '
            event.get("tags").each { |k, v|
              event.set(k,v)
            }
             event.remove("tags")
          '
        }
    }
	
	date {
        match => [ "timestamp", "UNIX" ]
    }

}

output { 
	stdout { codec => rubydebug { metadata => false }} 
}

[Badger]

@mohsin106

Each object is on its own line.

Then why not use a json codec instead of a multiline codec?

The multiline codec that you quote is attempting to combine every line in the file into a single event. The multiline codec has a max_lines option which defaults to 500, so it will create an event for every 500 lines. That is probably why you are getting 9 events.

[mohsin106 (Mohsin)]

@Badger

Then why not use a json codec instead of a multiline codec?

That worked. Thanks for the direction.
Updated my file filter as follows:

file { 
		path => "/a/b/file1.json" 
		sincedb_path => "/dev/null" 
		start_position => beginning 
		codec => json
	}