# Field list

**URL:** <https://discuss.elastic.co/t/field-list/214863>\
**Category:** Logstash\
**Created:** [January 13, 2020, 3:48pm UTC](https://discuss.elastic.co/t/field-list/214863 "2020-01-13T15:48:48Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 13, 2020, 3:48pm UTC](https://discuss.elastic.co/t/field-list/214863/1 "2020-01-13T15:48:49Z")

</div>

I have this entry in logstash.conf

```auto
mutate {
       split => ["message","Employee"]
       add_field => {"part1" =>"%{[message][0]}"}
       add_field => {"part2" =>"%{[message][1]}"}      
}

mutate {
       split => ["part2","#"]
       add_field => {"part2_1" =>"%{[part2][0]}"}
       add_field => {"part2_2" =>"%{[part2][1]}"}

```

}

This adds part1 , part2 , part2\_1 and part2\_2 fields in Kibana's Available field list.

But my requirement is to add only part2\_2 field . rest of the fields are not required in Kibana.

What changes I should make here so that only part2\_2 field is added in Kibana's Available field list.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 13, 2020, 6:29pm UTC](https://discuss.elastic.co/t/field-list/214863/2 "2020-01-13T18:29:16Z")

</div>

If you do not want the other fields then do not add them If you need to remove existing fields then use mutate+remove\_field.

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 14, 2020, 4:13am UTC](https://discuss.elastic.co/t/field-list/214863/3 "2020-01-14T04:13:52Z")

</div>

> [@Badger](#):
>
> If you do not want the other fields then do not add them

How ? I dont want field part1

You mean change this

```
mutate {
       split => ["message","Employee"]
       add_field => {"part1" =>"%{[message][0]}"}
       add_field => {"part2" =>"%{[message][1]}"}      
}

```

to this

```
mutate {
       split => ["message","Employee"]
       part1 =>"%{[message][0]}"}
       add_field => {"part2" =>"%{[message][1]}"}      
}

```

look at part1 . Is this correct syntax not to add the field? I am stuck at this spot.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 14, 2020, 2:13pm UTC](https://discuss.elastic.co/t/field-list/214863/4 "2020-01-14T14:13:25Z")

</div>

Change

```
mutate {
   split => ["message","Employee"]
   add_field => {"part1" =>"%{[message][0]}"}
   add_field => {"part2" =>"%{[message][1]}"}      
}

```

to

```
mutate {
   split => ["message","Employee"]
   add_field => {"part2" =>"%{[message][1]}"}      
}
```

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 14, 2020, 2:32pm UTC](https://discuss.elastic.co/t/field-list/214863/5 "2020-01-14T14:32:23Z")

</div>

> [@volcano](#):
>
> mutate { split =\> ["message","Employee"] add\_field =\> {"part1" =\>"%{[message][0]}"} add\_field =\> {"part2" =\>"%{[message][1]}"} } mutate { split =\> ["part2","#"] add\_field =\> {"part2\_1" =\>"%{[part2][0]}"} add\_field =\> {"part2\_2" =\>"%{[part2][1]}"}

No. It is not that easy. I need the field in next mutate for **further splitting.**

see this , you'll understand the intent.

```
mutate {
       split => ["message","Employee"]
       add_field => {"part1" =>"%{[message][0]}"} // No need to send this to Output
       add_field => {"part2" =>"%{[message][1]}"} // No need to send this to Output    
}

mutate {
       split => ["part2","#"]
       add_field => {"part2_1" =>"%{[part2][0]}"} // No need to send this to Output
       add_field => {"part2_2" =>"%{[part2][1]}"} // No need to send this to Output
}

mutate {
       split => ["part2_2","="]
       add_field => {"X" =>"%{[part2_2][0]}"} // This is required in output
       add_field => {"Y" =>"%{[part2_2][1]}"} // This is required in output
}

```

tell me what change I should do here so that only X , Y goes to output

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 14, 2020, 2:37pm UTC](https://discuss.elastic.co/t/field-list/214863/6 "2020-01-14T14:37:24Z")

</div>

It is unclear what you want, but you might find it useful to add fields inside [[@metadata]](https://www.elastic.co/blog/logstash-metadata). Those are attached to the event, but are not added to the document by the output.

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 14, 2020, 2:45pm UTC](https://discuss.elastic.co/t/field-list/214863/7 "2020-01-14T14:45:50Z")

</div>

Yes. I tried that but it gives error.  
Please see this  
[https://pastebin.com/65WNbSck](https://pastebin.com/65WNbSck)

Is it wrong syntactically ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 14, 2020, 3:44pm UTC](https://discuss.elastic.co/t/field-list/214863/8 "2020-01-14T15:44:25Z")

</div>

```
split => [[@metadata][qravsmanual],"="]

```

This should be

```
split => { "[@metadata][qravsmanual]" => "=" }

```

Edited to fix syntax...

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 14, 2020, 4:25pm UTC](https://discuss.elastic.co/t/field-list/214863/9 "2020-01-14T16:25:11Z")

</div>

I guess you missed "}" here . Is it a typo ?  
split =\> { "[@metadata][qravsmanual]" =\> "="] **}**

here is the latest config after correction

[https://pastebin.com/6Pufu6pF](https://pastebin.com/6Pufu6pF)

Is this syntactically okay now ?

---

<div class="post-metadata">

**Author:** ![andres-perez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andres-perez/32/136461_2.png) [@andres-perez](https://discuss.elastic.co/u/andres-perez)\
**Post date:** [January 14, 2020, 5:13pm UTC](https://discuss.elastic.co/t/field-list/214863/10 "2020-01-14T17:13:09Z")

</div>

> [@volcano](#):
>
> Is this syntactically okay now ?

You can just [run logstash from command line](https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html) with **`--config.test_and_exit`** to check syntax.

If you are comfortable using docker containers on your own you can use this tool [Web-UI for Logstash filter development](https://discuss.elastic.co/t/web-ui-for-logstash-filter-development/209748)

Either way you can test it and if it throws an error, **paste here all the details** :

- log contents used as input
- logstash configuration
- errors, desired or unexpected output

In fact, if you paste here a sample log line it would be really useful. I presume that, instead of splitting a message in two by a word separator, splitting the result in two again by another word, splitting again by other separators... is a contrieved way to extract the desired information. Maybe grok or kv filters are a simpler solution for your use case.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 14, 2020, 5:27pm UTC](https://discuss.elastic.co/t/field-list/214863/11 "2020-01-14T17:27:22Z")

</div>

> [@volcano](#):
>
> Is this syntactically okay now ?

No, remove the ] from line 19.

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 15, 2020, 2:18pm UTC](https://discuss.elastic.co/t/field-list/214863/12 "2020-01-15T14:18:38Z")

</div>

> [@andres-perez](#):
>
> You can just [run logstash from command line](https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html) with **`--config.test_and_exit`** to check syntax.

when I run this  
`/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit`

I get this error

> **[\[root@ip-10-0-0-27 logstash\]# /usr/share/logstash/bin/logstash -f...](https://pastebin.com/2P8V6CDR)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

Could you please tell , which syntax I am making wrong ?

Here is my logstash.conf

> **[input { file { path =\> "/opt/codedeploy/pricingservice.log" start\_p...](https://pastebin.com/XNpeGme7)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 15, 2020, 2:21pm UTC](https://discuss.elastic.co/t/field-list/214863/13 "2020-01-15T14:21:58Z")

</div>

@Badger I did that. But there is still syntax error.

Here is the latest logstash.conf ... I have appended the @metadata snippet at the end block.

latest config file  
[https://pastebin.com/XNpeGme7](https://pastebin.com/XNpeGme7)

What I am missing ?

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [January 15, 2020, 2:33pm UTC](https://discuss.elastic.co/t/field-list/214863/14 "2020-01-15T14:33:15Z")

</div>

Hi,  
end of the line 135 }} should be }"} ?

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 15, 2020, 3:02pm UTC](https://discuss.elastic.co/t/field-list/214863/15 "2020-01-15T15:02:41Z")

</div>

@grumo35

> [@grumo35](#):
>
> end of the line 135 }} should be }"} ?

corrected this part.

when I run this now  
`/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit`

This is still giving syntax error

> **[\[root@ip-10-0-0-27 logstash\]# /usr/share/logstash/bin/logstash -f...](https://pastebin.com/4www1DHK)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

Here is the latest config file

> **[input { file { path =\> "/opt/codedeploy/pricingservice.log" start\_p...](https://pastebin.com/gtv8RrBS)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

what I am missing ?

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [January 15, 2020, 3:06pm UTC](https://discuss.elastic.co/t/field-list/214863/16 "2020-01-15T15:06:44Z")

</div>

Line 145  
split =\> [[@metadata][request\_id],"="] should be ["[ at the begging

I'm not sure of your strange syntax.

You should go for :

split =\> ["[FIELDNAME]" , "=" ]

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 15, 2020, 3:19pm UTC](https://discuss.elastic.co/t/field-list/214863/17 "2020-01-15T15:19:00Z")

</div>

I just change this to

`split => { "[@metadata][request_id]" => "=" }`

This seems working .

I run again  
`/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit`

Result :  
**runner - Using config.test\_and\_exit mode. Config Validation Result: OK. Exiting Logstash**

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 15, 2020, 3:34pm UTC](https://discuss.elastic.co/t/field-list/214863/18 "2020-01-15T15:34:21Z")

</div>

@grumo35

could you please tell how you are identifying line no of issue ? I dont see that in the error message.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 15, 2020, 3:40pm UTC](https://discuss.elastic.co/t/field-list/214863/19 "2020-01-15T15:40:43Z")

</div>

The error message contains the line number

```
Reason: Expected one of #, ", ', -, [, {,] at line 145, column 19
```

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [January 15, 2020, 3:53pm UTC](https://discuss.elastic.co/t/field-list/214863/20 "2020-01-15T15:53:29Z")

</div>

Thanks....that helped a lot.

Now I'm getting the fields successfully in the output and also in Kibana.

**Kibana screen**  
 ![](https://us1.discourse-cdn.com/elastic/original/3X/7/1/71ff3dd5aa2a97732ea8ce9aa9a289e346779d8d.png)

But I do not see these fields in the **filter** (arrow marked) in Kibana.

Is there anything I require to do so that these fields are visible in filter ?

This is because , I want to apply a filter condition for these fields.

[Next page](https://discuss.elastic.co/t/field-list/214863.md?page=2)