# Logstash always outputting only 9 documents

**URL:** <https://discuss.elastic.co/t/logstash-always-outputting-only-9-documents/281751>\
**Category:** Logstash\
**Created:** [August 17, 2021, 9:48pm UTC](https://discuss.elastic.co/t/logstash-always-outputting-only-9-documents/281751 "2021-08-17T21:48:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [August 17, 2021, 9:48pm UTC](https://discuss.elastic.co/t/logstash-always-outputting-only-9-documents/281751/1 "2021-08-17T21:48:34Z")

</div>

I have a json file with 4K json objects in it. Each object is on its own line. I'm using the file plugin with the multiline codec to have Logstash parse all the objects. I'm also flattening the json objects and then sending the output to standard out using the stdout plugin. However, I'm only seeing 9 records print to the screen.

I want to do some aggregation on the data and need to be able to see all the records print to the screen. How can I accomplish this?

After reading a couple of articles in the community I came across @Badger's [solution](https://discuss.elastic.co/t/parsing-json-file-with-logstash/214080/11) to read in JSON content and I configured my Logstash config like this:

```auto
input {
	file { 
		path => "/a/b/file1.json" 
		sincedb_path => "/dev/null" 
		start_position => beginning 
		codec => multiline { 
			pattern => "^Spalanzani" 
			negate => true 
			what => previous 
			auto_flush_interval => 1 
			multiline_tag => "" 
		} 
	}
}

filter { 
	json { 
		source => "message" 
		remove_field => ["message"] 
	}
	if [fields] {
        ruby {
          code => '
            event.get("fields").each { |k, v|
              event.set(k,v)
            }
            event.remove("fields")
          '
        }
    }

    if [tags] {
        ruby {
          code => '
            event.get("tags").each { |k, v|
              event.set(k,v)
            }
             event.remove("tags")
          '
        }
    }
	
	date {
        match => ["timestamp", "UNIX"]
    }

}

output { 
	stdout { codec => rubydebug { metadata => false }} 
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 17, 2021, 10:01pm UTC](https://discuss.elastic.co/t/logstash-always-outputting-only-9-documents/281751/2 "2021-08-17T22:01:21Z")

</div>

> [@mohsin106](#):
>
> Each object is on its own line.

Then why not use a json codec instead of a multiline codec?

The multiline codec that you quote is attempting to combine every line in the file into a single event. The multiline codec has a max\_lines option which defaults to 500, so it will create an event for every 500 lines. That is probably why you are getting 9 events.

---

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [August 17, 2021, 10:19pm UTC](https://discuss.elastic.co/t/logstash-always-outputting-only-9-documents/281751/3 "2021-08-17T22:19:24Z")

</div>

> [@Badger](#):
>
> Then why not use a json codec instead of a multiline codec?

That worked. Thanks for the direction.  
Updated my file filter as follows:

```auto
file { 
		path => "/a/b/file1.json" 
		sincedb_path => "/dev/null" 
		start_position => beginning 
		codec => json
	}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 14, 2021, 10:19pm UTC](https://discuss.elastic.co/t/logstash-always-outputting-only-9-documents/281751/4 "2021-09-14T22:19:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.