# How to add optional json field in Logstash

**URL:** <https://discuss.elastic.co/t/how-to-add-optional-json-field-in-logstash/296650>\
**Category:** Logstash\
**Created:** [February 8, 2022, 6:25pm UTC](https://discuss.elastic.co/t/how-to-add-optional-json-field-in-logstash/296650 "2022-02-08T18:25:58Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wonder\_Garance](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wonder_garance/32/91180_2.png) [@Wonder\_Garance](https://discuss.elastic.co/u/Wonder_Garance)\
**Post date:** [February 8, 2022, 6:25pm UTC](https://discuss.elastic.co/t/how-to-add-optional-json-field-in-logstash/296650/1 "2022-02-08T18:25:58Z")

</div>

Hi, I wrote a Logstash pipeline to modify some fields of json input files. The keys introduction.text, introduction.html and body in json are optional, I don't want to create empty fields in json output files. How to add checks in my pipeline?

```auto
input {
  file {
    mode => "read"
    path => "/documents/*" 
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline { pattern => "^Spalanzani" negate => true what => "previous" auto_flush_interval => 2 }
    file_chunk_size => 409600
  }
}  
filter {
  json {
    source => "message"
    id => "%{[message][id]}"
    add_field => {
      title => "%{[title][text]}"
      date => "%{[editorial_date]}"
      topic => "%{[topic][title]}"
      text => "%{[introduction][text]} %{[body]}"
      text_html => "%{[introduction][html]} %{[body]}"
    }
  }
...

```

As I couldn't write a valid pipeline with if condition, I tried gsub, but it didn't work. %{[introduction][text]} are in the output json file.

```auto
filter {
  json {
    source => "message"
    id => "%{[message][id]}"
    add_field => {
      title => "%{[title][text]}"
      date => "%{[editorial_date]}"
      text => "%{[introduction][text]} %{[body]}"
      text_html => "%{[introduction][html]} %{[body]}"
    }
  }
  mutate {
    gsub => [
      "texte", "<.*?>", " ",
      "texte", "(%{[introduction][text]}|%{[body]})", " ",
      "texte_html", "(%{[introduction][html]}|%{[body]})", " ",
      "date", "T.*$", ""
    ]
  }
}

```

If I escape [ with \ in gsub:

```auto
"texte", "(%{\[introduction]\[text]}|%{\[body]})", " ",

```

I get error:  
Exception caught while applying mutate filter {:exception=\>"Invalid FieldReference: `\\[introduction][\\[text]`](file://[text]%60)"}

here is a source json file:

```auto
{
  "id": "1111",
  "editorial_date": "2015-10-19T00:00:00Z",
    "title": {
    "text": "test title",
    "html": "<h1>test title</h1>"
  },
"body": "<video data-els-src-id=\"\"..."
}

```

---

<div class="post-metadata">

**Author:** ![Wonder\_Garance](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wonder_garance/32/91180_2.png) [@Wonder\_Garance](https://discuss.elastic.co/u/Wonder_Garance)\
**Post date:** [February 10, 2022, 12:38pm UTC](https://discuss.elastic.co/t/how-to-add-optional-json-field-in-logstash/296650/2 "2022-02-10T12:38:52Z")

</div>

Wait for a response, I found a way to add optional JSON fields. Does someone have a better solution?

```auto
filter {
  json {
    source => "message"
    id => "%{[message][id]}"
    add_field => {
      title => "%{[title][text]}"
      date => "%{[editorial_date]}"
      topic => "%{[topic][title]}"
      intro_text => "%{[introduction][text]}"
      intro_html => "%{[introduction][html]}"
      body => "%{[body]}"
    }
  }
  if [intro_text] != "%{[introduction][text]}" {
    if [body] != "%{[body]}" {
      mutate {
        add_field => { "texte" => "%{[intro_text]} %{[body]}" }
      }
    } else {
      mutate {
        add_field => { "texte" => "%{[intro_text]}" }
      }
    }
  } else if [body] != "%{[body]}" {
    mutate {
      add_field => { "texte" => "%{[body]}" }
    }
  }
  if [intro_html] != "%{[introduction][html]}" {
    if [body] != "%{[body]}" {
      mutate {
        add_field => { "texte_html" => "%{[intro_html]} %{[body]}" }
      }
    } else {
      mutate {
        add_field => { "texte_html" => "%{[intro_html]}" }
      }
    }
  } else if [body] != "%{[body]}" {
    mutate {
      add_field => { "texte_html" => "%{[body]}" }
    }
  }
}
...

```

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 12, 2022, 7:57pm UTC](https://discuss.elastic.co/t/how-to-add-optional-json-field-in-logstash/296650/3 "2022-02-12T19:57:48Z")

</div>

If the keys are optional and when the keys are not contained in the json input file, no fields for the keys might be created.

What is the output with empty fields? Could you share sample output using [rubydebug output plugin](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html)?

Anyway, using ruby filter could be good for you.

---

<div class="post-metadata">

**Author:** ![Wonder\_Garance](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wonder_garance/32/91180_2.png) [@Wonder\_Garance](https://discuss.elastic.co/u/Wonder_Garance)\
**Post date:** [February 15, 2022, 5:00pm UTC](https://discuss.elastic.co/t/how-to-add-optional-json-field-in-logstash/296650/4 "2022-02-15T17:00:27Z")

</div>

With filter.json.add\_field, I got some fields which contain only %{[xxx]}  
For example, if the field introduction isn't in json, the output (stdout) is:

```auto
      intro_text => "%{[introduction][text]}"
      intro_html => "%{[introduction][html]}"

```

@Tomo_M Thank your for your advise, now I use Ingest pipeline for the json transformation, it's easier.

```auto
    {
      "set": {
        "field": "intro_html",
        "value": "{{introduction.html}}",
        "ignore_missing": true
      }
    }

```

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 15, 2022, 5:13pm UTC](https://discuss.elastic.co/t/how-to-add-optional-json-field-in-logstash/296650/5 "2022-02-15T17:13:47Z")

</div>

I see those fields were created by the "add\_field" block. thanks. I'm glad to hear you find the way.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 15, 2022, 5:14pm UTC](https://discuss.elastic.co/t/how-to-add-optional-json-field-in-logstash/296650/6 "2022-03-15T17:14:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.