# How to parse logs

**URL:** <https://discuss.elastic.co/t/how-to-parse-logs/67705>\
**Category:** Logstash\
**Created:** [December 1, 2016, 4:18am UTC](https://discuss.elastic.co/t/how-to-parse-logs/67705 "2016-12-01T04:18:50Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [December 1, 2016, 4:18am UTC](https://discuss.elastic.co/t/how-to-parse-logs/67705/1 "2016-12-01T04:18:50Z")

</div>

Hello All,

I have a amazon redshift logging been stored on amazon s3. I want to use the data from this s3 bucket to visualize. Below are some sample logs which is generated from redshift.

```
'2016-11-29T00:00:04Z UTC [db=dev user=rdsdb pid=29788 userid=1 xid=283300]' LOG: SELECT 1

```

'2016-11-29T00:00:04Z UTC [db=dev user=rdsdb pid=29788 userid=1 xid=283301]' LOG: SET statement\_timeout TO 120000  
'2016-11-29T00:00:04Z UTC [db=dev user=rdsdb pid=29788 userid=1 xid=283302]' LOG: select 'ConnectionCheckQuery'  
'2016-11-29T00:00:04Z UTC [db=dev user=rdsdb pid=29836 userid=1 xid=283303]' LOG: SELECT 1  
'2016-11-29T00:00:04Z UTC [db=dev user=rdsdb pid=29836 userid=1 xid=283304]' LOG: SET statement\_timeout TO 120000  
'2016-11-29T00:00:04Z UTC [db=dev user=rdsdb pid=29836 userid=1 xid=283305]' LOG: select \* from STV\_FDISK\_STATS where name = 'blocks\_to\_backup'  
'2016-11-29T00:00:04Z UTC [db=dev user=rdsdb pid=29836 userid=1 xid=283305]' LOG: SELECT pg\_catalog.stv\_fdisk\_stats.node\_num AS node\_num, pg\_catalog.stv\_fdisk\_stats.name AS name, pg\_catalog.stv\_fdisk\_stats.value AS value FROM pg\_catalog.stv\_fdisk\_stats WHERE pg\_catalog.stv\_fdisk\_stats.name = 'blocks\_to\_backup'::Char(16);

I tried using grokconstructor for this and below is what i constructed.

\A'%{SYSLOGBASE2:timestamp}%{CRON\_ACTION}%{NOTSPACE}%{CRON\_ACTION}%{EMAILLOCALPART:db}%{CRON\_ACTION}%{EMAILLOCALPART:user}%{CRON\_ACTION}%{EMAILLOCALPART:pid}%{CRON\_ACTION}%{EMAILLOCALPART:userid}%{CRON\_ACTION}%{EMAILLOCALPART:xid}%{CRON\_ACTION}%{NOTSPACE}%{SPACE}%{GREEDYDATA:sql}

As of now i am able to get some data but i still see that grokparse failure in logstash output.

```
{
      "message" => "'2016-11-28T03:18:16Z UTC [db=dev user=rdsdb pid=29788 userid=1 xid=270214]' LOG: SET statement_timeout TO 120000\n",
     "@version" => "1",
   "@timestamp" => "2016-12-01T04:13:33.620Z",
         "type" => "redshift-access-log",
    "timestamp" => "2016-11-28T03:18:16Z UTC",
"timestamp8601" => "2016-11-28T03:18:16Z",
    "logsource" => "UTC",
           "db" => "db=dev",
         "user" => "user=rdsdb",
          "pid" => "pid=29788",
       "userid" => "userid=1",
          "xid" => "xid=270214",
          "sql" => "LOG: SET statement_timeout TO 120000\n",
         "tags" => [
    [0] "_dateparsefailure"
]

```

}

Can someone guide a help on this.

--  
Niraj

---

<div class="post-metadata">

**Author:** ![eperry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eperry/32/551_2.png) [@eperry](https://discuss.elastic.co/u/eperry)\
**Post date:** [December 1, 2016, 5:43am UTC](https://discuss.elastic.co/t/how-to-parse-logs/67705/2 "2016-12-01T05:43:33Z")

</div>

Try using [https://grokdebug.herokuapp.com](https://grokdebug.herokuapp.com) to check your regex

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 5, 2016, 6:46am UTC](https://discuss.elastic.co/t/how-to-parse-logs/67705/3 "2016-12-05T06:46:49Z")

</div>

> As of now i am able to get some data but i still see that grokparse failure in logstash output.

No, but you have a `_dateparsefailure` so it's your date filter that doesn't work.

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [December 5, 2016, 6:58am UTC](https://discuss.elastic.co/t/how-to-parse-logs/67705/4 "2016-12-05T06:58:43Z")

</div>

@magnusbaeck , Thanks for the reply. Sorry i didn't updated the thread. I managed to fix it by removing the additional date pattern i had. It works perfectly now.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 2, 2017, 6:59am UTC](https://discuss.elastic.co/t/how-to-parse-logs/67705/5 "2017-01-02T06:59:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.