# Logstash configuration

**URL:** <https://discuss.elastic.co/t/logstash-configuration/74982>\
**Category:** Logstash\
**Created:** [February 14, 2017, 7:46am UTC](https://discuss.elastic.co/t/logstash-configuration/74982 "2017-02-14T07:46:30Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 15, 2017, 7:06am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/21 "2017-02-15T07:06:05Z")

</div>

Sure, there are other filters you could use (perhaps the csv filter if you make " | " the column separator) but I don't think it'll be much easier.

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 15, 2017, 7:10am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/22 "2017-02-15T07:10:30Z")

</div>

can i use aggregate filter here?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 15, 2017, 7:11am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/23 "2017-02-15T07:11:07Z")

</div>

What? No.

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 15, 2017, 7:14am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/24 "2017-02-15T07:14:11Z")

</div>

okk.if i use csv filter then is it too lengthy?

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 15, 2017, 7:16am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/25 "2017-02-15T07:16:00Z")

</div>

can u suggest appropriate filter for this? Because i m getting confuse now

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 15, 2017, 7:20am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/26 "2017-02-15T07:20:20Z")

</div>

My suggestion is to use the grok filter as described earlier. Start with the very simplest expression, `^%{TIMESTAMP_ISO8601:timestamp}`. Does that work? If yes, continue adding more to the expression, each time validating that it continues to work. Be systematic.

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 15, 2017, 7:21am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/27 "2017-02-15T07:21:30Z")

</div>

ok.i will try it

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 15, 2017, 7:55am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/28 "2017-02-15T07:55:49Z")

</div>

it not working. it says that "incorrect config file"

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 15, 2017, 8:36am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/29 "2017-02-15T08:36:08Z")

</div>

Please try to understand that it's impossible to help with the small amount of details that you give. I need to see the exact configuration you tried (copy/paste the text and format what you paste as preformatted text using the `</>` toolbar button) and the exact error message.

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 15, 2017, 11:58am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/30 "2017-02-15T11:58:00Z")

</div>

The solution is working now.I m able to parse logs now

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 16, 2017, 11:11am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/31 "2017-02-16T11:11:51Z")

</div>

hi i want to filter logs as per the time taken by particular query to perform.

February 15th 2017, 18:03:37.133 2017-02-15T17:59:12.258+0530 I COMMAND [conn2] command narendra.$cmd command: delete { delete: "inventory", deletes: [{ q: { status: "A" }, limit: 0.0 }], ordered: true } keyUpdates:0 writeConflicts:0 numYields:0 reslen:25 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } }, Metadata: { acquireCount: { W: 1 } } } protocol:op\_command 3ms  
February 15th 2017, 18:03:37.133 2017-02-15T17:59:12.257+0530 I WRITE [conn2] remove narendra.inventory query: { status: "A" } ndeleted:7 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } } 2ms  
February 15th 2017, 18:03:37.128 2017-02-15T17:48:53.581+0530 I COMMAND [conn2] command narendra.inventory command: find { find: "inventory", filter: { status: { $in: ["A", "D"] } } } planSummary: COLLSCAN keysExamined:0 docsExamined:10 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:5 reslen:684 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op\_command 41ms  
February 15th 2017, 18:03:37.127 2017-02-15T17:46:29.632+0530 I - [conn2] Creating profile collection: narendra.system.profile  
February 15th 2017, 18:03:37.126 2017-02-15T17:46:29.632+0530 I COMMAND [conn2] command narendra.inventory command: insert { insert: "inventory", documents: [{ \_id: ObjectId('58a4469db28fdcc74588e721'), item: "canvas", qty: 100.0, tags: [ "cotton"], size: { h: 28.0, w: 35.5, uom: "cm" } } ], ordered: true } ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:25 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 1, W: 1 } }, Collection: { acquireCount: { W: 1 } } } protocol:op\_command 206ms  
February 15th 2017, 18:03:37.123 2017-02-15T17:45:57.812+0530 I COMMAND [conn2] command admin.system.users command: saslStart { saslStart: 1, mechanism: "SCRAM-SHA-1", payload: "xxx" } keyUpdates:0 writeConflicts:0 numYields:0 reslen:164 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op\_command 4ms

these are filtered logs.Now the challenge is sort out those by time  
means time taken by query less tan 10ms and greater than 10ms

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 20, 2017, 6:51am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/32 "2017-02-20T06:51:30Z")

</div>

February 15th 2017, 18:03:37.133 2017-02-15T17:59:12.258+0530 I COMMAND [conn2] command narendra.$cmd command: delete { delete: "inventory", deletes: [{ q: { status: "A" }, limit: 0.0 }], ordered: true } keyUpdates:0 writeConflicts:0 numYields:0 reslen:25 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } }, Metadata: { acquireCount: { W: 1 } } } protocol:op\_command 3ms

can i split this message in different fields as timestamp in different field , that command in different field and query in different field and time taken by query in different field?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2017, 7:22am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/33 "2017-02-20T07:22:11Z")

</div>

Yes, use the grok filter. The grok constructor web site should be helpful when creating the expression.

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 20, 2017, 7:27am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/34 "2017-02-20T07:27:16Z")

</div>

filter {  
grok {  
match =\> {"message" =\> "%{DATA:timestamp} | %(COMMAND|NETWORK) %{GREEDYDATA:message}"}

}  
}

mutate{  
split =\> ["message"]  
}

i used this file.But it not working

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 21, 2017, 5:51am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/35 "2017-02-21T05:51:39Z")

</div>

input {  
file {  
path =\> "C:/Data/log/mongolog.log"  
start\_position =\> "beginning"  
}  
}  
filter {  
grok {  
match =\> {"message" =\> "[%{DATA:timestamp} | %(COMMAND|NETWORK) %{GREEDYDATA:message}]"}  
}  
}  
mutate { timestamp =\> { "@timestamp" =\> "timestamp","dd/MM/yy HH:mm:ss:Z"}}  
mutate { command =\> { "Instruction" =\> "I COMMAND"}}  
mutate { message =\> { "message"=\> "query"}}  
mutate { query\_time =\> { op\_command =\> "time in ms"}}  
date {  
match =\> ["timestamp","dd/MM/yy HH:mm:ss:Z"]  
}  
}  
output {  
elasticsearch{ hosts =\> ["localhost:9200"] index =\> "log" }  
stdout{codec =\> "rubydebug" }  
}

i used this config file. it gives error of logstash pipeline aborted.  
I will add the error

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 21, 2017, 5:58am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/36 "2017-02-21T05:58:50Z")

</div>

now its showing differet error

configuration at  
default config whi  
h::Runner] ERROR l  
file {\npath =\> "  
\n}\nfilter {\ngro  
(COMMAND|NETWORK)  
timestamp" =\> "t  
"Instruction" =\>  
"}}\nmutate { quer  
["timestamp","

> ["localhost:920  
> n", :reason=\>"Expe  
> e 203) after "}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 21, 2017, 6:30am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/37 "2017-02-21T06:30:50Z")

</div>

As I've said before: Copy/paste the text and format what you paste as preformatted text using the \</\> toolbar button. Another general advice is to use the preview pane on the right when posting. Is the text I'm about to post complete and correctly formatted?

> ```
> %(COMMAND|NETWORK)
> 
> ```

This is incorrect. Remove the % sign.

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 21, 2017, 6:35am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/38 "2017-02-21T06:35:27Z")

</div>

ok.i will correct it

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 21, 2017, 7:03am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/39 "2017-02-21T07:03:55Z")

</div>

the format you are gave me is works but it not parse as per [command.it](http://command.it) shows \_ grokparse success for all the queries.

February 20th 2017, 10:45:29.087 2017-02-20T10:38:13.437+0530 I COMMAND [conn2] command narendra.inventory command: insert { insert: "inventory", documents: 5, ordered: true } ninserted:5 keyUpdates:0 writeConflicts:0 numYields:0 reslen:25 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 1, W: 1 } }, Collection: { acquireCount: { w: 1, W: 1 } } } protocol:op\_command 165ms  
February 20th 2017, 10:45:29.084 2017-02-20T10:35:21.892+0530 I COMMAND [conn4] command narendra.restaurants command: insert { insert: "restaurants", ordered: false, documents: 1000 } ninserted:1000 keyUpdates:0 writeConflicts:0 numYields:0 reslen:40 locks:{ Global: { acquireCount: { r: 17, w: 17 } }, Database: { acquireCount: { w: 16, W: 1 } }, Collection: { acquireCount: { w: 16, W: 1 } } } protocol:op\_query 202ms  
February 20th 2017, 10:45:29.084 2017-02-20T10:34:46.189+0530 I COMMAND [conn3] command narendra.grades command: insert { insert: "grades", ordered: false, documents: 287 } ninserted:287 keyUpdates:0 writeConflicts:0 numYields:0 reslen:40 locks:{ Global: { acquireCount: { r: 6, w: 6 } }, Database: { acquireCount: { w: 5, W: 1 } }, Collection: { acquireCount: { w: 5, W: 1 } } } protocol:op\_query 196ms  
February 20th 2017, 10:45:29.083 2017-02-20T10:33:40.970+0530 I COMMAND [conn2] CMD: drop narendra.users  
February 20th 2017, 10:45:29.082 2017-02-20T10:33:20.437+0530 I COMMAND [conn2] CMD: drop narendra.restaurants  
February 20th 2017, 10:45:29.075 2017-02-20T10:33:01.582+0530 I COMMAND [conn2] CMD: drop narendra.inventory  
February 20th 2017, 10:45:29.072 2017-02-20T10:32:20.895+0530 I COMMAND [conn2] command narendra.grades command: drop { drop: "grades" } keyUpdates:0 writeConflicts:0 numYields:0 reslen:63 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { W: 1 } } } protocol:op\_command 104ms  
February 20th 2017, 10:45:29.067 2017-02-20T10:31:59.015+0530 I COMMAND [conn2] CMD: drop narendra.[object Object]  
February 20th 2017, 10:45:29.067 2017-02-20T10:32:20.790+0530 I COMMAND [conn2] CMD: drop narendra.grades  
February 20th 2017, 10:45:29.066 2017-02-20T10:31:34.639+0530 I ACCESS [conn3] Successfully authenticated as principal deepak on narendra  
February 20th 2017, 10:45:29.065 2017-02-20T10:31:34.637+0530 I ACCESS [conn4] Successfully authenticated as principal deepak on narendra

---

<div class="post-metadata">

**Author:** ![Nikparab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikparab/32/17493_2.png) [@Nikparab](https://discuss.elastic.co/u/Nikparab)\
**Post date:** [February 21, 2017, 7:05am UTC](https://discuss.elastic.co/t/logstash-configuration/74982/40 "2017-02-21T07:05:23Z")

</div>

I want to skip those system log.and just want query logs.  
can u pls help me write filter for that.  
I tried as per i understand

[Previous page](https://discuss.elastic.co/t/logstash-configuration/74982.md?page=1)

[Next page](https://discuss.elastic.co/t/logstash-configuration/74982.md?page=3)