# Not able to read the data from external json file in logstash config

**URL:** https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257
**Category:** Logstash
**Created:** [December 13, 2023, 12:52pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257 "2023-12-13T12:52:27Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![subash\_k](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subash_k/32/121314_2.png) [@subash\_k](https://discuss.elastic.co/u/subash_k)
#### Post date: [December 13, 2023, 12:52pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/1 "2023-12-13T12:52:27Z")

</div>

I'm trying to search the host value from current event and looking for same value in json file. If Json block has the host value I'm just converting the block into struct value and inserting as a new column in index.

output.json file (External json file)

```auto
[
  {
    "datacenter": "aa",
    "port": "444",
    "decommissioned_date": "",
    "environment": "int",
    "hostname": "hello.net",
    "provisioned_date": "01.01.2020",
    "current_status": "active",
    "Cluster_name": "blue",
    "ip": "10.44.44.44",
  },
  {
    "datacenter": "aws-e1",
    "port": "333",
	"decommissioned_date": "",
    "environment": "integration",
	"provisioned_date": "01.01.2020",
    "hostname": "google.com",
	"current_status": "active",
    "cluster_name": "black",
    "ip": "96.94.44.22",
  }
]

```

Logstash.config

```auto
input {
  #stdin { }
  tcp {
    codec => json_lines { charset => "UTF-8" }
    port => 4560
  }
}
filter {
    json {
        source => "payload_raw"
        target => "payload"
   }

  translate {
    dictionary_path => "/app/output.json"
    field => "hostname"
    destination => "external_host_data"
    refresh_interval => 3600
    override => true
    }

    if [external_host_data] {
    mutate {
      add_field => {
        "struct_field" => "%{[external_host_data][0]}"
      }
    }
  }

}
output {
   elasticsearch {
           hosts => ["host.net:9200"]
           index => "logstash-testyml-details-%{+YYYY.MM.dd}"
           pipeline => "query_default_pipeline"
           user => "admin"
           password => " ****"
           }
stdout { codec => rubydebug }
}

```

error :

```auto
[2023-12-13T12:30:13,942][ERROR][logstash.javapipeline][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::Filters::Dictionary::DictionaryFileError: Tr
anslate: no implicit conversion of Array into Hash when loading dictionary file at /app/output.json>

```

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 13, 2023, 1:06pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/2 "2023-12-13T13:06:59Z")

</div>

> [@subash\_k](#):
>
> ```auto
> [2023-12-13T12:30:13,942][ERROR][logstash.javapipeline][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::Filters::Dictionary::DictionaryFileError: Tr
> anslate: no implicit conversion of Array into Hash when loading dictionary file at /app/output.jso
> 
> ```

You have an error related to your dictionary file, please share how the file `/app/output.json` looks like:

---

<div class="post-metadata">

### Author: ![subash\_k](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subash_k/32/121314_2.png) [@subash\_k](https://discuss.elastic.co/u/subash_k)
#### Post date: [December 13, 2023, 1:09pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/3 "2023-12-13T13:09:54Z")

</div>

Here it is

> [@subash\_k](#):
>
> output.json file (External json file)

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 13, 2023, 1:11pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/4 "2023-12-13T13:11:49Z")

</div>

> [@subash\_k](#):
>
> Here it is

You need to share the content of the file, or at least some sample lines of it.

---

<div class="post-metadata">

### Author: ![subash\_k](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subash_k/32/121314_2.png) [@subash\_k](https://discuss.elastic.co/u/subash_k)
#### Post date: [December 13, 2023, 1:14pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/5 "2023-12-13T13:14:32Z")

</div>

Below content from output.json file

```auto
[
  {
    "datacenter": "aa",
    "port": "444",
    "decommissioned_date": "",
    "environment": "int",
    "hostname": "hello.net",
    "provisioned_date": "01.01.2020",
    "current_status": "active",
    "Cluster_name": "blue",
    "ip": "10.44.44.44"
  },
  {
    "datacenter": "aws-e1",
    "port": "333",
	"decommissioned_date": "",
    "environment": "integration",
	"provisioned_date": "01.01.2020",
    "hostname": "google.com",
	"current_status": "active",
    "cluster_name": "black",
    "ip": "96.94.44.22"
  }
]

```

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 13, 2023, 1:32pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/6 "2023-12-13T13:32:14Z")

</div>

Oh I see, I thought this was your input.

Your dictionary cannot be in this format, the dictionaries for the translate filter needs to be a key-value pair file, this will not work, you will need to change your dictionary.

It needs to be something like this:

```auto
"google.com": '{"datacenter": "aws-e1","port": "333","decommissioned_date": "","environment": "integration","provisioned_date": "01.01.2020","current_status": "active","cluster_name": "black","ip": "96.94.44.22" }'
"hello.net": '{"datacenter": "aa","port": "444","decommissioned_date": "","environment": "int","provisioned_date": "01.01.2020","current_status": "active","Cluster_name": "blue","ip": "10.44.44.44"}'

```

I made an old post a couple of years ago explaining how to use the translate filter, you can check it [here](https://web.leandrojmp.com/posts/en/2021/02/logstash-translate) if you have any doubts.

---

<div class="post-metadata">

### Author: ![subash\_k](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subash_k/32/121314_2.png) [@subash\_k](https://discuss.elastic.co/u/subash_k)
#### Post date: [December 14, 2023, 11:07am UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/7 "2023-12-14T11:07:22Z")

</div>

Thanks @leandrojmp  
If i have external file as this format it's working. I can see `{"datacenter": "aws-e1","port": "333","decommissioned_date": "","environment": "integration","provisioned_date": "01.01.2020","current_status": "active","cluster_name": "black","ip": "96.94.44.22" }` as separate column. But it's falling on varchar. How can i push it as object to in index (so that i can easily query the nested data (

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 14, 2023, 12:59pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/8 "2023-12-14T12:59:49Z")

</div>

> [@subash\_k](#):
>
> But it's falling on varchar

It is not clear what you mean with that.

---

<div class="post-metadata">

### Author: ![subash\_k](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subash_k/32/121314_2.png) [@subash\_k](https://discuss.elastic.co/u/subash_k)
#### Post date: [December 14, 2023, 1:06pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/9 "2023-12-14T13:06:00Z")

</div>

I'm able to see the below value in index as separate columns and the column type is varchar. How can i make the column type as object or rowtype ? Is it possible ?

```auto
Index name : testing
Column name in index : struct_field
Column type in index : varchar
Column value : 
{"datacenter": "aws-e1","port": "333","decommissioned_date": "","environment": "integration","provisioned_date": "01.01.2020","current_status": "active","cluster_name": "black","ip": "96.94.44.22" }

```

In any way i can query the value of datacenter directly ? (i.e **select struct\_field.datacenter from testing** )

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 14, 2023, 1:09pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/10 "2023-12-14T13:09:01Z")

</div>

> [@subash\_k](#):
>
> ```auto
> Column name in index : struct_field
> Column type in index : varchar
> 
> ```

Where is this from? There is no such thing as `varchar` in Elasticsearch.

But your issue is that the value in the dictionary is added as a string, you need to parse it using the json filter.

Did you read the blog post I shared? There is an explanation and an example on how to do that when you have a json as the value of your dictionary.

You will need to use a `json` filter on the field `external_host_data` to parse the json value.

---

<div class="post-metadata">

### Author: ![subash\_k](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subash_k/32/121314_2.png) [@subash\_k](https://discuss.elastic.co/u/subash_k)
#### Post date: [December 14, 2023, 1:26pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/11 "2023-12-14T13:26:32Z")

</div>

Sorry for the confusion I'm using trino elastic connector to read the data.  
[Elasticsearch connector — Trino 434 Documentation](https://trino.io/docs/current/connector/elasticsearch.html?highlight=Type%20mapping#type-mapping).  
Between I read your blog that helps much 🙂 Thanks

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 11, 2024, 1:27pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257/12 "2024-01-11T13:27:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
