# Processing solr logs using logstash **URL:** \ **Category:** Logstash\ **Created:** [December 3, 2015, 5:09am UTC](https://discuss.elastic.co/t/processing-solr-logs-using-logstash/36241 "2015-12-03T05:09:15Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Maddy](https://avatars.discourse-cdn.com/v4/letter/m/838e76/32.png) [@Maddy](https://discuss.elastic.co/u/Maddy)\ **Post date:** [December 3, 2015, 5:09am UTC](https://discuss.elastic.co/t/processing-solr-logs-using-logstash/36241/1 "2015-12-03T05:09:15Z")
Hi, I am using solr which has multiple collections in it. Solr dumps the log of all collections into a single file(solr.log) and I want the logs of one particular collection. Here is a sample log structure, INFO - 2015-11-19 10:55:36.389; org.apache.solr.core.SolrCore; [example1\_shard1\_replica1] webapp=/solr path=/select params={some params} hits=3219 status=0 QTime=361 INFO - 2015-11-19 10:55:37.085; org.apache.solr.core.SolrCore; [example1\_shard2\_replica1] webapp=/solr path=/select params={some params} status=0 QTime=20 INFO - 2015-11-19 10:55:37.218; org.apache.solr.core.SolrCore; [example2\_shard1\_replica1] webapp=/solr path=/select params={some params} status=0 QTime=153 INFO - 2015-11-19 10:55:37.310; org.apache.solr.core.SolrCore; [example1\_shard1\_replica1] webapp=/solr path=/select params={some params} hits=9612 status=0 QTime=1286 INFO - 2015-11-19 10:56:08.597; org.apache.solr.core.SolrCore; [example3\_shard1\_replica1] webapp=/solr path=/select params={some params} hits=84 status=0 QTime=577 INFO - 2015-11-19 10:56:09.002; org.apache.solr.core.SolrCore; [example2\_shard2\_replica1] webapp=/solr path=/select params={some params} hits=26 status=0 QTime=318 INFO - 2015-11-19 10:56:09.390; org.apache.solr.core.SolrCore; [example3\_shard2\_replica1] webapp=/solr path=/select params={some params} status=0 QTime=36 INFO - 2015-11-19 10:56:09.407; org.apache.solr.core.SolrCore; [example1\_shard1\_replica1] webapp=/solr path=/select params={some params} hits=84 status=0 QTime=727 I want lines where collection contains "example1" Please let me know any ideas/suggestion ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 3, 2015, 5:10am UTC](https://discuss.elastic.co/t/processing-solr-logs-using-logstash/36241/2 "2015-12-03T05:10:42Z")
You need to send it to Logstash and then filter out the ones you want ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 3, 2015, 7:10am UTC](https://discuss.elastic.co/t/processing-solr-logs-using-logstash/36241/4 "2015-12-03T07:10:33Z")
- Be very careful about using DATA or GREEDYDATA in more than one place. It can create unexpected matches that are non-obvious to debug. - You should be able to use the kv filter to parse the key/value data. - Use conditionals to apply filters and outputs depending on the contents of fields: ```auto if "example1" in [collection] { ... } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 7, 2015, 6:36pm UTC](https://discuss.elastic.co/t/processing-solr-logs-using-logstash/36241/6 "2015-12-07T18:36:45Z")
The key is to pick patterns that aren't too wide (i.e. could match too much) but still matches all the possible values. Let's look at some of the strings that you wanted to match with DATA: - org.apache.solr.core.SolrCore: Use `%{JAVACLASS}`. - example3\_shard1\_replica1: I don't know what this string means, but it's a reasonable assumption that it can't contain a closing square bracket so we can use that as a delimiter: `(?[^\]]+)`. - /solr: It's probably safe to assume that there can't be any spaces here (and if there are spaces in the URL they're probably still encoded with %20 or +). So: `%{NOTSPACE:webapp}` Actually, the latter part of the string contains a bunch of key=value pairs. Using the kv filter for parsing them should be convenient. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 13, 2015, 6:49pm UTC](https://discuss.elastic.co/t/processing-solr-logs-using-logstash/36241/8 "2015-12-13T18:49:19Z")
What _do_ you get? Does it extract the `collection` field properly? So it's the second grok filter that doesn't work? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 14, 2015, 8:24am UTC](https://discuss.elastic.co/t/processing-solr-logs-using-logstash/36241/10 "2015-12-14T08:24:22Z")
The grok expression works fine to extract the `collection` field. I have no idea why it doesn't work for you. ```auto $ echo 'INFO - 2015-11-19 10:55:36.389; org.apache.solr.core.SolrCore; [example1_shard1_replica1] webapp=/solr path=/select params={some params} hits=3219 status=0 QTime=361' | /opt/logstash/bin/logstash -f test.config { "message" => "INFO - 2015-11-19 10:55:36.389; org.apache.solr.core.SolrCore; [example1_shard1_replica1] webapp=/solr path=/select params={some params} hits=3219 status=0 QTime=361", "@version" => "1", "@timestamp" => "2015-12-14T08:22:16.151Z", "host" => "lnxolofon", "collection" => "example1_shard1_replica1" } Logstash startup completed Logstash shutdown completed ``` Next time, do copy/paste command output. Do not attach screenshots for text that can be copy/pasted. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:18am UTC](https://discuss.elastic.co/t/processing-solr-logs-using-logstash/36241/11 "2017-07-06T05:18:38Z")