# Parse syslog from Syslog

**URL:** <https://discuss.elastic.co/t/parse-syslog-from-syslog/168291>\
**Category:** Logstash\
**Created:** [February 13, 2019, 8:58pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291 "2019-02-13T20:58:37Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Digital](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digital/32/41054_2.png) [@Digital](https://discuss.elastic.co/u/Digital)\
**Post date:** [February 13, 2019, 8:58pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/1 "2019-02-13T20:58:37Z")

</div>

Hi all,

I am new in ELK solution 😃 and currently I am working on Logstash -\> Elasticsearch -\> Kibana.  
I send the syslog from a checkpoint to logstash however I would like to parse correctly the syslog with logstash.

I think I didn't understand well how to parse a character in string element. Currently in Kibana I have this result:  
 ![Capture22](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fc4c8eedd2d1f5fca217fcb7a501705139952483.png)

We can see one field "message" and on this one I have:  
`<134>1 2019-02-12T21:47:03Z info CheckPoint 2105 - [action:"Drop"; flags:"393216"; ifdir:"inbound"; ifname:"bond2"; loguid:"{0x5c633gd8,0x6,0x24bv703e,0x47f4985}"; origin:"9.1.0.1"; originsicname:"CN=name-1,O=name.domain.com.com"; sequencenum:"201"; time:"1550008023"; version:"5"; __policy_id_tag:"test vpn & FireWall-1[db_tag={E53339C-8456-1C43-AF36-F12340F57FDA};mgmt=name1;date=1549371733;policy_name=office\]"; dst:"3.2.1.2"; message_info:"Missing OS route"; product:"VPN-1 & FireWall-1"; proto:"17"; s_port:"1633"; service:"1633"; src:"192.168.2.3"; ]`

You can see other parameters and I would like to parse this parameters is it possible?  
Can you help me on that?

Thank you very much for your help and your time!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 14, 2019, 2:44pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/2 "2019-02-14T14:44:28Z")

</div>

Try this

```
    dissect { mapping => { "message" => "<%{level}>%{} %{ts} %{loglevel} %{something} - [%{kvString}" } }
    date { match => ["ts", "ISO8601"] }
    kv { source => "kvString" field_split => ";" value_split => ":" trim_key => " " }
```

---

<div class="post-metadata">

**Author:** ![Digital](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digital/32/41054_2.png) [@Digital](https://discuss.elastic.co/u/Digital)\
**Post date:** [February 16, 2019, 9:38pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/3 "2019-02-16T21:38:53Z")

</div>

Hi Badger,

Thank you for your reply. However, do you have a documentation on that? I would like to understand how he works 😛  
I will try your solution next Monday.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 16, 2019, 10:02pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/4 "2019-02-16T22:02:50Z")

</div>

[dissect](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html), [kv](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html) and [date](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html) are all standard filters.

---

<div class="post-metadata">

**Author:** ![Digital](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digital/32/41054_2.png) [@Digital](https://discuss.elastic.co/u/Digital)\
**Post date:** [February 18, 2019, 4:59pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/5 "2019-02-18T16:59:10Z")

</div>

> [@Badger](#):
>
> dissect { mapping =\> { "message" =\> "\<%{level}\>%{} %{ts} %{loglevel} %{something} - [%{kvString}" } } date { match =\> ["ts", "ISO8601"] } kv { source =\> "kvString" field\_split =\> ";" value\_split =\> ":" trim\_key =\> " " }

Thank you for your help. However I added your proposition in my "GROK" and when I would like to execute logstash I receive an error (failed to execute action (grok...))

Maybe my configuration file on my logstash is not correct?! 😕

```
input {
  tcp {
port => 5140
type => syslog
  }
  udp {
port => 5140
type => syslog
  }
}

filter {
  kv {
  }
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

    dissect { mapping => { "message" => "<%{level}>%{} %{ts} %{loglevel} %{something} - [%{kvString}" } }
    date { match => ["ts", "ISO8601"] }
    kv { source => "kvString" field_split => ";" value_split => ":" trim_key => " " }
  }
  mutate {
     gsub => ["timestamp"," "," "]
  }
  date {
     match => ["timestamp", "ddMMMYYYY HH:mm:ss"]
  }
}

output {
  elasticsearch {
hosts => "127.0.0.1"
index => "syslog-%{+YYYY.MM.dd}"
  }
}

```

Can I have your support on the filter "grok"?

Thank you very much!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 18, 2019, 5:09pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/6 "2019-02-18T17:09:39Z")

</div>

Get rid of grok and also the kv{} filter with no options. Just use

```
filter {
    dissect { mapping => { "message" => "<%{level}>%{} %{ts} %{loglevel} %{something} - [%{kvString}" } }
    date { match => ["ts", "ISO8601"] }
    kv { source => "kvString" field_split => ";" value_split => ":" trim_key => " " }
}
```

---

<div class="post-metadata">

**Author:** ![Digital](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digital/32/41054_2.png) [@Digital](https://discuss.elastic.co/u/Digital)\
**Post date:** [February 18, 2019, 5:18pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/7 "2019-02-18T17:18:03Z")

</div>

> [@Badger](#):
>
> filter { dissect { mapping =\> { "message" =\> "\<%{level}\>%{} %{ts} %{loglevel} %{something} - [%{kvString}" } } date { match =\> ["ts", "ISO8601"] } kv { source =\> "kvString" field\_split =\> ";" value\_split =\> ":" trim\_key =\> " " } }

Han ok. It's strange it's not necessary to specify the syslog format? Now I receive:  
`Feb 18 18:15:44 bcb-chp-mgmt2.localdomain logstash[13078]: [2019-02-18T18:15:44,241][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"syslog-2019.02.18", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x6db15994>], :response=>{"index"=>{"_index"=>"syslog-2019.02.18", "_type"=>"doc", "_id"=>"lYabAWkBSNN2StILLB_S", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [1000] in index [syslog-2019.02.18] has been exceeded"}}}}`

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 18, 2019, 6:18pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/8 "2019-02-18T18:18:01Z")

</div>

> [@Digital](#):
>
> Limit of total fields [1000] in index [syslog-2019.02.18] has been exceeded

If you go to Kibana and look at the index mapping, do the field names look reasonable? You may need to update the mapping so as not to autoindex, or you may just need to start over with a new daily index.

---

<div class="post-metadata">

**Author:** ![Digital](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digital/32/41054_2.png) [@Digital](https://discuss.elastic.co/u/Digital)\
**Post date:** [February 20, 2019, 11:56am UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/9 "2019-02-20T11:56:47Z")

</div>

Hi Badger,

thank you very much for your support your solution is working now!  
However I am little lost on the filter xD

```
filter {
dissect { mapping => { "message" => "<%{level}>%{} %{ts} %{loglevel} %{something} - [%{kvString}" } }
date { match => ["ts", "ISO8601"] }
kv { source => "kvString" field_split => ";" value_split => ":" trim_key => " " }

```

}

On this step: "message" =\> "\<%{level}\>%{} %{ts} %{loglevel} %{something} - [%{kvString}"

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 20, 2019, 12:28pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/10 "2019-02-20T12:28:05Z")

</div>

The [dissect](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html) filter parses a field using delimiters. In that case the delimiters are \<, \>, space, space, space, and " - [".

---

<div class="post-metadata">

**Author:** ![Digital](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digital/32/41054_2.png) [@Digital](https://discuss.elastic.co/u/Digital)\
**Post date:** [February 25, 2019, 2:41pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/11 "2019-02-25T14:41:04Z")

</div>

Is it possible to use one field present in the syslog in the output element?  
I would like to sort the syslog before to send all in the DB.

For exemple

```
> output {
> If (origin = 192.168.1.1){
> hosts => "127.0.0.1"
> index => "index1"
> }
> If (origin = 192.168.1.2){
> hosts => "127.0.0.1"
> index => "index2"
> }
> If (origin = 192.168.1.3){
> hosts => "127.0.0.1"
> index => "index3"
> }
> }
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 25, 2019, 2:42pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/12 "2019-02-25T14:42:53Z")

</div>

Try

```
if [origin] == "192.168.1.1" {
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 25, 2019, 2:42pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291/13 "2019-03-25T14:42:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.