# Logstash breaking my data structure

**URL:** <https://discuss.elastic.co/t/logstash-breaking-my-data-structure/364565>\
**Category:** Elastic Search\
**Created:** [August 8, 2024, 12:57am UTC](https://discuss.elastic.co/t/logstash-breaking-my-data-structure/364565 "2024-08-08T00:57:06Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![joe\_El\_Yahchouchi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_el_yahchouchi/32/131912_2.png) [@joe\_El\_Yahchouchi](https://discuss.elastic.co/u/joe_El_Yahchouchi)\
**Post date:** [August 8, 2024, 12:57am UTC](https://discuss.elastic.co/t/logstash-breaking-my-data-structure/364565/1 "2024-08-08T00:57:06Z")

</div>

Hello,

I have defined my own index in elasticsearch

```auto
PUT donor
{
"mappings": {
   "properties": {
{...}
"donations": {"type": "nested"}
}

```

Basically I have a donor user in mysql and the donor has many donations, I have built a json array of object donations, and I wish to feed all this data to elasticsearch through logstash. Mysql builds the json array perfectly, however, when logstash does the query to send it to elastic, weird stuff occurs and I get this:

```auto
{:status=>400, :action=>["index", {:_id=>"137347", :_index=>"donors", :routing=>nil}, { "donations"=>"[\"{\\\"donation_id\\\": 13378812, \\\"donation_type_id\\\": 4, \\\"donation_center_id\\\": 1, \\\"donation_status_id\\\": 1, \\\"donation_sponsor_id\\\": 334},{\\\"donation_id\\\": 33432523, \\\"donation_type_id\\\": 4, \\\"donation_center_id\\\": 2, \\\"donation_status_id\\\": 1, \\\"donation_sponsor_id\\\": 300}\"]"}], :response=>{"index"=>{"status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:36] object mapping for [donations] tried to parse field [donations] as object, but found a concrete value"}}}}

```

Why is it finding a concrete value? If I manually post my result from mysql through Kibana it works...  
Sample result as returned by mysql:

```auto
[{"donation_id": 13334300, "donation_type_id": 2, "donation_center_id": 2, "donation_status_id": 3, "donation_sponsor_id": 393}, {"donation_id": 33388011, "donation_type_id": 4, "donation_center_id": 3, "donation_status_id": 3, "donation_sponsor_id": 387}]

```

The logstash config is pretty bare bones, no filters, no processing no nothing. Input mysql, output elasticsearch. That is all

The TLDR is, I want logstash to not do anything to my entity, just pass it through as is. Why is it so hard? I have been stuck on this issue for hours. Please help.

---

<div class="post-metadata">

**Author:** ![Alex\_Salgado-Elastic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_salgado-elastic/32/103081_2.png) [@Alex\_Salgado-Elastic](https://discuss.elastic.co/u/Alex_Salgado-Elastic)\
**Post date:** [August 8, 2024, 3:04am UTC](https://discuss.elastic.co/t/logstash-breaking-my-data-structure/364565/2 "2024-08-08T03:04:14Z")

</div>

Hi @joe_El_Yahchouchi

The error likely indicates that Elasticsearch is receiving the `donations` field as a JSON string instead of an array of objects. This usually happens because Logstash is not correctly deserializing the JSON data received from MySQL.

Try to handle this in the Logstash configuration file, as suggested below:

```auto
input {
  jdbc {
    jdbc_connection_string => "jdbc:mysql://localhost:3306/your_database"
    jdbc_user => "your_user"
    jdbc_password => "your_password"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    statement => "SELECT id, name, donations FROM donors"
  }
}

filter {
  mutate {
    # If necessary, convert the donations field to a string
    convert => { "donations" => "string" }
  }

  json {
    # Parse the donations field as JSON
    source => "donations"
    target => "donations"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "donors"
    document_id => "%{id}"
  }
}

```

* * *

This configuration should ensure that Logstash correctly parses the `donations` field as a JSON array of objects before sending it to Elasticsearch.

---

<div class="post-metadata">

**Author:** ![joe\_El\_Yahchouchi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_el_yahchouchi/32/131912_2.png) [@joe\_El\_Yahchouchi](https://discuss.elastic.co/u/joe_El_Yahchouchi)\
**Post date:** [August 8, 2024, 1:04pm UTC](https://discuss.elastic.co/t/logstash-breaking-my-data-structure/364565/3 "2024-08-08T13:04:43Z")

</div>

> [@Alex\_Salgado-Elastic](#):
>
> ```auto
> mutate {
> # If necessary, convert the donations field to a string
> convert => { "donations" => "string" }
> }
> 
> json {
> # Parse the donations field as JSON
> source => "donations"
> target => "donations"
> }
> 
> ```

Hello, thank you for the reply, but no this did not work. Still getting the same error

```auto
{"donations"=>["{\"donation_id\": 1, \"donation_type_id\": 1, \"donation_center_id\": 1, \"donation_status_id\": 2, \"donation_sponsor_id\": 128},{\"donation_id\": 20053712, \"donation_type_id\": 3, \"donation_center_id\": 2, \"donation_status_id\": 1, \"donation_sponsor_id\": 135}"], "@version"=>"1", "prefix"=>"", "@timestamp"=>2024-08-08T13:02:00.512527998Z,}], :response=>{"index"=>{"status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:15] object mapping for [donations] tried to parse field [null] as object, but found a concrete value"}

```

There are fewer escape slashes now. Are these a factor at all? It's also throwing a quote around my brackets { I'm going to see about trying to remove those.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 8, 2024, 1:36pm UTC](https://discuss.elastic.co/t/logstash-breaking-my-data-structure/364565/4 "2024-08-08T13:36:31Z")

</div>

Can you add a file output in your logstash pipeline and share some sample of what is the output you are getting from Logstash?

From what you shared it looks that the `donations` fields is not being parsed correctly, which is probably because its value is not recognized as a valid json.

You can add a file output with this:

```auto
file {
    paht => "/tmp/sample-output.log"
}

```

Then get some sample lines and share them.

---

<div class="post-metadata">

**Author:** ![joe\_El\_Yahchouchi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_el_yahchouchi/32/131912_2.png) [@joe\_El\_Yahchouchi](https://discuss.elastic.co/u/joe_El_Yahchouchi)\
**Post date:** [August 8, 2024, 1:50pm UTC](https://discuss.elastic.co/t/logstash-breaking-my-data-structure/364565/5 "2024-08-08T13:50:35Z")

</div>

> [@leandrojmp](#):
>
> ```auto
> file {
> paht => "/tmp/sample-output.log"
> }
> 
> ```

Hello, it ended up working. I fiddled a bit with the spacing inside the mysql query in the file itself and that somehow had an impact?  
And thank you for teaching me about file output, very handy.

Thank you all, this can now be closed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 5, 2024, 1:51pm UTC](https://discuss.elastic.co/t/logstash-breaking-my-data-structure/364565/6 "2024-09-05T13:51:14Z")

</div>

Closed on September 5, 2024, 1:51pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.