# S3 output creating invalid json from CSV **URL:** \ **Category:** Logstash\ **Created:** [May 7, 2018, 1:52pm UTC](https://discuss.elastic.co/t/s3-output-creating-invalid-json-from-csv/130843 "2018-05-07T13:52:17Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![mememe](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@mememe](https://discuss.elastic.co/u/mememe)\ **Post date:** [May 7, 2018, 1:52pm UTC](https://discuss.elastic.co/t/s3-output-creating-invalid-json-from-csv/130843/1 "2018-05-07T13:52:17Z")
Hi, using s3 output to read from a csv file and store it as json. My logstash conf looks like below. Problem: The output is not a valid json array consisting of json objects but a file having a list of json objects in it. Any idea if that can be changed? Instead of a file looking like this {jsonobject\_a} {jsonobject\_b} {jsonobject\_c} I want a file like [{jsonobject\_a},{jsonobject\_b},{jsonobject\_c}] my logstash.conf input { tcp { port =\> 4560 codec =\> json\_lines add\_field =\> { "logstash\_input" =\> "tcp\_4560" } } beats { port =\> 5044 add\_field =\> { "logstash\_input" =\> "beats" } } } filter { ... } else { csv { columns =\> [ "date", "level", "server", "log\_message" ] separator =\> "|" } } } output { s3{ region =\> "us-east-1" bucket =\> "....mybucket-logdata" codec =\> "json" } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 7, 2018, 2:24pm UTC](https://discuss.elastic.co/t/s3-output-creating-invalid-json-from-csv/130843/2 "2018-05-07T14:24:45Z")
> The output is not a valid json array consisting of json objects but a file having a list of json objects in it. > Any idea if that can be changed? To avoid misunderstandings please show examples intead of describing the situations. What do you get now? What would you like to get instead? ---
**Author:** ![mememe](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@mememe](https://discuss.elastic.co/u/mememe)\ **Post date:** [May 7, 2018, 2:54pm UTC](https://discuss.elastic.co/t/s3-output-creating-invalid-json-from-csv/130843/3 "2018-05-07T14:54:00Z")
Hi Magnus, logstash turns each line into a json object and puts all of those objects in the file it creates in s3. hwoever, these objects are not embedded in a json array. they are simle added one after the otehr to the file. instead of that output in the s3 file (what is simply some list of json objects) { "date": "2018-04-18T09:26:35.0039150-05:00", "server": "abcd", "offset": 74, "level": "error", "prospector": { "type": "log" }, "source": "/var/log/x\_debug/x\_debugyy.log", "message": "2018-04-18T09:26:35.0039150-05:00|error|yyy|Checking Product..", "logstash\_input": "beats", "tags": ["beats\_input\_codec\_plain\_applied"], "@timestamp": "2018-05-07T12:52:15.481Z", "@version": "1", "beat": { "name": "ip-xxxx", "hostname": "ip-xxxx", "version": "6.1.2" }, "host": "ip-xxxx", "log\_message": "Checking Product..", "fields": { "source\_system\_id": "x\_debug" } } { "date": "2018-04-18T09:42:48.0478973-05:00", "server": "yyy", "offset": 154, "level": "Information", "prospector": { "type": "log" }, "source": "/var/log/x\_debug/x\_debugyy.log", "message": "2018-04-18T09:42:48.0478973-05:00|Information|yyy|Checking Product..", "logstash\_input": "beats", "tags": ["beats\_input\_codec\_plain\_applied"], "@timestamp": "2018-05-07T12:52:15.481Z", "@version": "1", "beat": { "name": "ip-xxx", "hostname": "ip-xxx", "version": "6.1.2" }, "host": "ip-xxx", "log\_message": "Checking Product..", "fields": { "source\_system\_id": "x\_debug" } } I want below structure to be the output of the file generated by logstash in s3 [{ "date": "2018-04-18T09:26:35.0039150-05:00", "server": "abcd", "offset": 74, "level": "error", "prospector": { "type": "log" }, "source": "/var/log/x\_debug/x\_debugyy.log", "message": "2018-04-18T09:26:35.0039150-05:00|error|yyy|Checking Product..", "logstash\_input": "beats", "tags": ["beats\_input\_codec\_plain\_applied"], "@timestamp": "2018-05-07T12:52:15.481Z", "@version": "1", "beat": { "name": "ip-xxxx", "hostname": "ip-xxxx", "version": "6.1.2" }, "host": "ip-xxxx", "log\_message": "Checking Product..", "fields": { "source\_system\_id": "x\_debug" } } , { "date": "2018-04-18T09:42:48.0478973-05:00", "server": "yyy", "offset": 154, "level": "Information", "prospector": { "type": "log" }, "source": "/var/log/x\_debug/x\_debugyy.log", "message": "2018-04-18T09:42:48.0478973-05:00|Information|yyy|Checking Product..", "logstash\_input": "beats", "tags": ["beats\_input\_codec\_plain\_applied"], "@timestamp": "2018-05-07T12:52:15.481Z", "@version": "1", "beat": { "name": "ip-xxx", "hostname": "ip-xxx", "version": "6.1.2" }, "host": "ip-xxx", "log\_message": "Checking Product..", "fields": { "source\_system\_id": "x\_debug" } }] ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 7, 2018, 5:37pm UTC](https://discuss.elastic.co/t/s3-output-creating-invalid-json-from-csv/130843/4 "2018-05-07T17:37:08Z")
Surely each JSON object is on a line of its own rather than pretty-printed like this? Use the json\_lines codec to make sure there's a linebreak between each event. There are two very good reasons why Logstash works like this: - Logstash doesn't know when the file is "done", so it doesn't know when to write the final `]`. - Reading the kind of logfile you're asking for would be very costly if the file is big. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 4, 2018, 5:37pm UTC](https://discuss.elastic.co/t/s3-output-creating-invalid-json-from-csv/130843/5 "2018-06-04T17:37:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.