# Is Logstash useful for structured data?

**URL:** <https://discuss.elastic.co/t/is-logstash-useful-for-structured-data/226883>\
**Category:** Logstash\
**Created:** [April 7, 2020, 11:38am UTC](https://discuss.elastic.co/t/is-logstash-useful-for-structured-data/226883 "2020-04-07T11:38:42Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Robert\_Ga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_ga/32/51129_2.png) [@Robert\_Ga](https://discuss.elastic.co/u/Robert_Ga)\
**Post date:** [April 7, 2020, 11:38am UTC](https://discuss.elastic.co/t/is-logstash-useful-for-structured-data/226883/1 "2020-04-07T11:38:43Z")

</div>

Hello everyone,

Recently I started to migrate some data into Elasticsearch using the Python client.

The data is already structured in json files and needs some processing like:

- lowercase all the values
- handle value inconsistencies for country field( ex: "United States of America" and "USA")
- add some extra fields

In this case, is still worth it to use Logstash( from what I have seen in documentation and tutorials is mainly used for formatting log files using grok) or is better to make all the processing in python and then add the data into Elasticsearch?

Thank you!

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [April 7, 2020, 12:38pm UTC](https://discuss.elastic.co/t/is-logstash-useful-for-structured-data/226883/2 "2020-04-07T12:38:35Z")

</div>

Hi,

for me it perfectly makes sense to use logstash to add fields and modify event data going through pipelines.

Maybe confirm with an elastic team member as i'm not an expert 😉

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [April 7, 2020, 1:00pm UTC](https://discuss.elastic.co/t/is-logstash-useful-for-structured-data/226883/3 "2020-04-07T13:00:12Z")

</div>

may be we can help each other :). I am good in logstash but zero in python.

but yes you can do all three bullet point you listed in logstash.  
in sort version  
mutate - lowercase =\> [fieldname]  
if ([country] /United Stats of America to USA/ ) { change to USA }  
mutate - add\_field

---

<div class="post-metadata">

**Author:** ![Robert\_Ga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_ga/32/51129_2.png) [@Robert\_Ga](https://discuss.elastic.co/u/Robert_Ga)\
**Post date:** [April 7, 2020, 1:17pm UTC](https://discuss.elastic.co/t/is-logstash-useful-for-structured-data/226883/4 "2020-04-07T13:17:44Z")

</div>

Sure, if you have any Python questions you can reach me out.

Regarding the country field. I will have to write in the config file a line for each country (which seems a lot of hard-coding).

Is it possible to have something like a pre-defined key-value variable and use it to map each coutry name to its shortened version?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 7, 2020, 2:43pm UTC](https://discuss.elastic.co/t/is-logstash-useful-for-structured-data/226883/5 "2020-04-07T14:43:05Z")

</div>

A translate filter could do that.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [April 7, 2020, 10:05pm UTC](https://discuss.elastic.co/t/is-logstash-useful-for-structured-data/226883/6 "2020-04-07T22:05:35Z")

</div>

> [@Robert\_Ga](#):
>
> ey-value variable and us

here is example

```
  translate {
      field => "[username]"
      destination => "[manager]"
      dictionary_path => "/tmp/user_manager.csv"
   }

```

and this is how your /tmp/user\_manager.csv looks like

"sam","zulu"  
"zulu","younameit"

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 5, 2020, 10:12pm UTC](https://discuss.elastic.co/t/is-logstash-useful-for-structured-data/226883/7 "2020-05-05T22:12:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.