# Querying only logs coming from source x **URL:** \ **Category:** Elasticsearch\ **Created:** [June 18, 2017, 1:39pm UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849 "2017-06-18T13:39:22Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\ **Post date:** [June 18, 2017, 1:39pm UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/1 "2017-06-18T13:39:22Z")
Hi, I am trying to search only logs that come from a certain source. Here is what I tried to do: GET filebeat-\*/log/\_search { "query": { "bool":{ "must": [ { "match": { "source": "/opt/modulo/smsc/cdr/traffic.cdr" } } ] } } } From some reason this retrieves me all logs also logs that doent come from this exact path. How can I retrieve only logs from this path? ---
**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\ **Post date:** [June 19, 2017, 1:52am UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/2 "2017-06-19T01:52:29Z")
I could not reproduce this using a vanilla setup. Maybe the template is wrong? Please post the output of `GET _template/filebeat`. ---
**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\ **Post date:** [June 20, 2017, 7:33am UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/3 "2017-06-20T07:33:31Z")
Hi I get an empty reply, what is this query? ---
**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\ **Post date:** [June 20, 2017, 1:50pm UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/4 "2017-06-20T13:50:13Z")
There should have been a template for your index. Probably the absence of it is the reason why you are querying and more is coming. What about `GET _template`? Can you post the output? ---
**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\ **Post date:** [June 22, 2017, 10:16am UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/5 "2017-06-22T10:16:24Z")
Ummm... from some reason also this gives me a blank message ![](https://us1.discourse-cdn.com/elastic/original/3X/6/0/60fae37fa1982485a51f18728e87c5efc7016a24.png) ---
**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\ **Post date:** [June 22, 2017, 10:52am UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/6 "2017-06-22T10:52:46Z")
I think I know what's going on, but I need one last confirmation. Expand the `filebeat-*` index pattern into an existing index, for instance, `filebeat-2017.06.22` (you can get a list of existing indices with `GET _cat/indices`) and then get the output of `GET filebeat-2027.06.22/_mapping`. Post this output here please. ---
**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\ **Post date:** [June 25, 2017, 6:50am UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/7 "2017-06-25T06:50:48Z")
> [@thiago](#): > > GET filebeat-2027.06.22/\_mapping Hi, Sorry about the delays, The query is: `GET filebeat-2017.06.14/_mapping` The output is pretty long. Since the the reply is limited to 7000 chars I will split the output: ``` { "filebeat-2017.06.14": { "mappings": { "log": { "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Jms_Lai_TS": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "apiMedTimeGot": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "apiMedTimeSent": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "authCBGot": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "authCBSent": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "beat": { "properties": { "hostname": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "billable": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "blackListCBGot": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "blackListCBSent": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, ``` ---
**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\ **Post date:** [June 25, 2017, 6:51am UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/8 "2017-06-25T06:51:21Z")
Second part (probably not an useful part: ``` "calledImsi": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "calledMsc": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "calledMscNpi": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "calledMscTon": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "calledNumber": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "callingImsi": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "callingMsc": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "callingNumber": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "campignId": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "cdrType": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "channel": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "clientId": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "concatenated": { "type": "boolean" }, "concatenatedFrom": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "deferred": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "destinationProtocol": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "gmt1": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "gmt2": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "host": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "input_type": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, ``` ---
**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\ **Post date:** [June 25, 2017, 12:35pm UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/9 "2017-06-25T12:35:43Z")
The issue here is that the proper `filebeat` template was not created. If you are _not_ indexing directly in elasticsearch from filebeat then you need to [load the index template](https://www.elastic.co/guide/en/beats/filebeat/5.4/filebeat-template.html) before indexing any data. ---
**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\ **Post date:** [June 27, 2017, 7:32am UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/10 "2017-06-27T07:32:43Z")
Hi, Thanks, I didnt completely understand what does this mean, but I tried to fix any way. since I am sending from Filebeat to Logstash to ES I understood I have to do this manually. To this manually in Centos I understood I wrote something like this: > curl -H 'Content-Type: application/json' -XPUT '[http://192.168.1.116:9200/\_template/filebeat](http://192.168.1.116:9200/_template/filebeat)' -d@/etc/filebeat/filebeat.template.json Is this right? I still get for the query: ``` GET filebeat-*/log/_search { "query": { "bool":{ "must": [ { "match": { "source": "\/path_to_files" } } ] } } } ``` all the logs and not only the logs in the path I wanted. On the other I dont get now for: ` GET _template` an empty answer. ---
**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\ **Post date:** [June 27, 2017, 7:51am UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/11 "2017-06-27T07:51:27Z")
Maybe there is another way all together to obtain logs only from a certain source? By the way I have control to put the logs on different directories if this can help ---
**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\ **Post date:** [June 27, 2017, 10:10am UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/12 "2017-06-27T10:10:45Z")
The command is wrong, there is a missing space, it should be: ```auto curl -H 'Content-Type: application/json' -XPUT 'http://192.168.1.116:9200/_template/filebeat' -d @/etc/filebeat/filebeat.template.json ``` You need to get it working, this is crucial. You may verify it worked by running `GET _template` and verifying that it is not an empty response. Lastly, after you properly create the template, for the template to be effective, you will need to delete all `filebeat-*` indices and reindex everything again. Note: if you can not reindex externally, then let me know, since there is a way to ask Elasticsearch to reindex internally ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 25, 2017, 10:10am UTC](https://discuss.elastic.co/t/querying-only-logs-coming-from-source-x/89849/13 "2017-07-25T10:10:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.