# Question about FSRiver 0.0.3

**URL:** <https://discuss.elastic.co/t/question-about-fsriver-0-0-3/11174>\
**Category:** Elasticsearch\
**Created:** [March 15, 2013, 11:22pm UTC](https://discuss.elastic.co/t/question-about-fsriver-0-0-3/11174 "2013-03-15T23:22:52Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![shadow](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@shadow](https://discuss.elastic.co/u/shadow)\
**Post date:** [March 15, 2013, 11:22pm UTC](https://discuss.elastic.co/t/question-about-fsriver-0-0-3/11174/1 "2013-03-15T23:22:52Z")

</div>

FS River seems like a great plugin!  
I tried to play around with some data. For one, it was quite fast (y).

But I ran into some issues/quirks, or maybe I didn't use it correctly. I'm  
using version 0.0.3, with 0.20.5 of elasticsearch and using filesystem.

1. When I create river with following setting:  
curl -XDELETE 127.0.0.1:9200/\_river/foo  
curl -XDELETE 127.0.0.1:9200/foo  
curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
"type": "fs",  
"fs": {  
"name": "Foo Data",  
"url": "/Users/slodha/foo/content",  
"update\_rate": 60000,  
"includes": "\*.json",  
"json\_support" : true  
},  
"index": {  
"index": "foo",  
"type": "foo",  
"bulk\_size": 50  
}  
}'

and search on this with this query:  
{  
"query": {  
"query\_string": {  
"default\_field": "\_all",  
"query": "slodha"  
}  
}  
}

I get results like:

hits: [

- {
  - \_index: foo
  - \_type: foo
  - \_id: 18156b6b5a6b3a8e1ec5984f185e18
  - \_score: 6.9584246
  - \_source: {
    - sunnyVal: slodha  
}  
}

- {
  - \_index: foo
  - \_type: foo
  - \_id: d7b4df4222e0d075d74ffde8aaa04a56
  - \_score: 6.901722
  - \_source: {
    - fileNameTest: slodha  
}  
}

]

-- I never get which file it belonged to - which I would definitely need to  
be able to search in the filesystem eventually.

When I do this:

curl -XDELETE 127.0.0.1:9200/\_river/foo  
curl -XDELETE 127.0.0.1:9200/foo  
curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
"type": "fs",  
"fs": {  
"name": "Foo Data",  
"url": "/Users/slodha/foo/content",  
"update\_rate": 60000  
},  
"index": {  
"index": "city",  
"type": "city",  
"bulk\_size": 50  
}  
}'

Notice here I do not use any json restriction..  
I get results like this:

hits: [

- {
  - \_index: foo
  - \_type: foo
  - \_id: 18156b6b5a6b3a8e1ec5984f185e18
  - \_score: 2.0015228
  - \_source: {
    - name: slodha\_1.json
    - postDate: 1363384941000
    - pathEncoded: 44d22b925f562f4e8d1d847253493336
    - rootpath: 948cd64d775db4119962b5a36dd530
    - virtualpath: t/sunnyTest
    - file: {
      - \_name: slodha\_1.json
      - content: ewoic3VubnlWYWwiIDogInNsb2RoYSIgCn0K  
}  
}  
}

- {
  - \_index: foo
  - \_type: foo
  - \_id: d7b4df4222e0d075d74ffde8aaa04a56
  - \_score: 1.7533717
  - \_source: {
    - name: file1.json
    - postDate: 1363388628000
    - pathEncoded: 99d79f46f1ce275b6b9152a0de54d5
    - rootpath: 948cd64d775db4119962b5a36dd530
    - virtualpath: t/sunnyTest/sunnyTest2
    - file: {
      - \_name: file1.json
      - content: ewoiZmlsZU5hbWVUZXN0IiA6ICJzbG9kaGEiCn0K  
}  
}  
}

]

Now, here I get to know the exact file paths, but this time the content is  
all a hash sum, and not readable..

I'm sure there should be a way for me to see both content and file paths as  
human readable with just one river. Can you suggest what I'm doing wrong?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![shadow](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@shadow](https://discuss.elastic.co/u/shadow)\
**Post date:** [March 16, 2013, 12:44am UTC](https://discuss.elastic.co/t/question-about-fsriver-0-0-3/11174/2 "2013-03-16T00:44:51Z")

</div>

An update, I tried with Latest Elasticsearch and FS River plugin - 0.1.0,  
but still issue persists. The only difference was with 0.1.0, if I use  
{"filename\_as\_id": true} in the definition of "fs", then I do get the  
local file name (Not the complete URL of file/ relative url of the file  
from the url given in definition). I would definitely not use  
"filename\_as\_id" flag , because then chances of conflict are quite high, I  
would have expected with that flas, I get absolute/relative file path from  
the url given. can we get that?

Also, still looking for a solution where I see the content of the files  
completely in search result and also the full/relative file path where the  
search hit a match.

Thanks!

On Friday, March 15, 2013 4:22:52 PM UTC-7, Shadow wrote:

> FS River seems like a great plugin!  
> I tried to play around with some data. For one, it was quite fast (y).
> 
> But I ran into some issues/quirks, or maybe I didn't use it correctly. I'm  
> using version 0.0.3, with 0.20.5 of elasticsearch and using filesystem.
> 
> 1. When I create river with following setting:  
> curl -XDELETE 127.0.0.1:9200/\_river/foo  
> curl -XDELETE 127.0.0.1:9200/foo  
> curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
> "type": "fs",  
> "fs": {  
> "name": "Foo Data",  
> "url": "/Users/slodha/foo/content",  
> "update\_rate": 60000,  
> "includes": "\*.json",  
> "json\_support" : true  
> },  
> "index": {  
> "index": "foo",  
> "type": "foo",  
> "bulk\_size": 50  
> }  
> }'
> 
> and search on this with this query:  
> {  
> "query": {  
> "query\_string": {  
> "default\_field": "\_all",  
> "query": "slodha"  
> }  
> }  
> }
> 
> I get results like:
> 
> hits: [
> 
> - {
> - \_index: foo
> - \_type: foo
> - \_id: 18156b6b5a6b3a8e1ec5984f185e18
> - \_score: 6.9584246
> - \_source: {
> - sunnyVal: slodha  
> }  
> }
> 
> - {
> - \_index: foo
> - \_type: foo
> - \_id: d7b4df4222e0d075d74ffde8aaa04a56
> - \_score: 6.901722
> - \_source: {
> - fileNameTest: slodha  
> }  
> }
> 
> ]
> 
> -- I never get which file it belonged to - which I would definitely need  
> to be able to search in the filesystem eventually.
> 
> When I do this:
> 
> curl -XDELETE 127.0.0.1:9200/\_river/foo  
> curl -XDELETE 127.0.0.1:9200/foo  
> curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
> "type": "fs",  
> "fs": {  
> "name": "Foo Data",  
> "url": "/Users/slodha/foo/content",  
> "update\_rate": 60000  
> },  
> "index": {  
> "index": "city",  
> "type": "city",  
> "bulk\_size": 50  
> }  
> }'
> 
> Notice here I do not use any json restriction..  
> I get results like this:
> 
> hits: [
> 
> - {
> - \_index: foo
> - \_type: foo
> - \_id: 18156b6b5a6b3a8e1ec5984f185e18
> - \_score: 2.0015228
> - \_source: {
> - name: slodha\_1.json
> - postDate: 1363384941000
> - pathEncoded: 44d22b925f562f4e8d1d847253493336
> - rootpath: 948cd64d775db4119962b5a36dd530
> - virtualpath: t/sunnyTest
> - file: {
> - \_name: slodha\_1.json
> - content: ewoic3VubnlWYWwiIDogInNsb2RoYSIgCn0K  
> }  
> }  
> }
> 
> - {
> - \_index: foo
> - \_type: foo
> - \_id: d7b4df4222e0d075d74ffde8aaa04a56
> - \_score: 1.7533717
> - \_source: {
> - name: file1.json
> - postDate: 1363388628000
> - pathEncoded: 99d79f46f1ce275b6b9152a0de54d5
> - rootpath: 948cd64d775db4119962b5a36dd530
> - virtualpath: t/sunnyTest/sunnyTest2
> - file: {
> - \_name: file1.json
> - content: ewoiZmlsZU5hbWVUZXN0IiA6ICJzbG9kaGEiCn0K  
> }  
> }  
> }
> 
> ]
> 
> Now, here I get to know the exact file paths, but this time the content is  
> all a hash sum, and not readable..
> 
> I'm sure there should be a way for me to see both content and file paths  
> as human readable with just one river. Can you suggest what I'm doing wrong?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 16, 2013, 2:58am UTC](https://discuss.elastic.co/t/question-about-fsriver-0-0-3/11174/3 "2013-03-16T02:58:27Z")

</div>

This is by design.

If you want to inject JSON files (json\_support: true), it's injected as is without any modification.  
When you disable json\_support (default), files (whatever the format) are sent to the mapper attachment plugin and content is extracted with Tika and indexed.

What you see in file.content is the raw file base64 encoded.

A nice option would be to add path in metadata section as we have \_index, \_type, \_id.  
I should try to do it with [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/mapping/meta.html).

Could you open an issue for that?

## Thanks for the feedback.

David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 mars 2013 à 01:44, Shadow [suryaveersingh.lodha@gmail.com](mailto:suryaveersingh.lodha@gmail.com) a écrit :

An update, I tried with Latest Elasticsearch and FS River plugin - 0.1.0, but still issue persists. The only difference was with 0.1.0, if I use {"filename\_as\_id": true} in the definition of "fs", then I do get the local file name (Not the complete URL of file/ relative url of the file from the url given in definition). I would definitely not use "filename\_as\_id" flag , because then chances of conflict are quite high, I would have expected with that flas, I get absolute/relative file path from the url given. can we get that?

Also, still looking for a solution where I see the content of the files completely in search result and also the full/relative file path where the search hit a match.

Thanks!

On Friday, March 15, 2013 4:22:52 PM UTC-7, Shadow wrote:

> FS River seems like a great plugin!  
> I tried to play around with some data. For one, it was quite fast (y).
> 
> But I ran into some issues/quirks, or maybe I didn't use it correctly. I'm using version 0.0.3, with 0.20.5 of elasticsearch and using filesystem.
> 
> 1. When I create river with following setting:  
> curl -XDELETE 127.0.0.1:9200/\_river/foo  
> curl -XDELETE 127.0.0.1:9200/foo  
> curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
> "type": "fs",  
> "fs": {  
> "name": "Foo Data",  
> "url": "/Users/slodha/foo/content",  
> "update\_rate": 60000,  
> "includes": "\*.json",  
> "json\_support" : true  
> },  
> "index": {  
> "index": "foo",  
> "type": "foo",  
> "bulk\_size": 50  
> }  
> }'
> 
> and search on this with this query:  
> {  
> "query": {  
> "query\_string": {  
> "default\_field": "\_all",  
> "query": "slodha"  
> }  
> }  
> }
> 
> I get results like:
> 
> hits: [  
> {  
> \_index: foo  
> \_type: foo  
> \_id: 18156b6b5a6b3a8e1ec5984f185e18  
> \_score: 6.9584246  
> \_source: {  
> sunnyVal: slodha  
> }  
> }  
> {  
> \_index: foo  
> \_type: foo  
> \_id: d7b4df4222e0d075d74ffde8aaa04a56  
> \_score: 6.901722  
> \_source: {  
> fileNameTest: slodha  
> }  
> }  
> ]
> 
> -- I never get which file it belonged to - which I would definitely need to be able to search in the filesystem eventually.
> 
> When I do this:
> 
> curl -XDELETE 127.0.0.1:9200/\_river/foo  
> curl -XDELETE 127.0.0.1:9200/foo  
> curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
> "type": "fs",  
> "fs": {  
> "name": "Foo Data",  
> "url": "/Users/slodha/foo/content",  
> "update\_rate": 60000  
> },  
> "index": {  
> "index": "city",  
> "type": "city",  
> "bulk\_size": 50  
> }  
> }'
> 
> Notice here I do not use any json restriction..  
> I get results like this:
> 
> hits: [  
> {  
> \_index: foo  
> \_type: foo  
> \_id: 18156b6b5a6b3a8e1ec5984f185e18  
> \_score: 2.0015228  
> \_source: {  
> name: slodha\_1.json  
> postDate: 1363384941000  
> pathEncoded: 44d22b925f562f4e8d1d847253493336  
> rootpath: 948cd64d775db4119962b5a36dd530  
> virtualpath: t/sunnyTest  
> file: {  
> \_name: slodha\_1.json  
> content: ewoic3VubnlWYWwiIDogInNsb2RoYSIgCn0K  
> }  
> }  
> }  
> {  
> \_index: foo  
> \_type: foo  
> \_id: d7b4df4222e0d075d74ffde8aaa04a56  
> \_score: 1.7533717  
> \_source: {  
> name: file1.json  
> postDate: 1363388628000  
> pathEncoded: 99d79f46f1ce275b6b9152a0de54d5  
> rootpath: 948cd64d775db4119962b5a36dd530  
> virtualpath: t/sunnyTest/sunnyTest2  
> file: {  
> \_name: file1.json  
> content: ewoiZmlsZU5hbWVUZXN0IiA6ICJzbG9kaGEiCn0K  
> }  
> }  
> }  
> ]
> 
> Now, here I get to know the exact file paths, but this time the content is all a hash sum, and not readable..
> 
> I'm sure there should be a way for me to see both content and file paths as human readable with just one river. Can you suggest what I'm doing wrong?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![shadow](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@shadow](https://discuss.elastic.co/u/shadow)\
**Post date:** [March 16, 2013, 6:48am UTC](https://discuss.elastic.co/t/question-about-fsriver-0-0-3/11174/4 "2013-03-16T06:48:07Z")

</div>

Thanks, opened an issue - [Can we add feature to be able to to see both file content and file paths as human readable format (text) with just one river for text files · Issue #11 · dadoonet/fscrawler · GitHub](https://github.com/dadoonet/fsriver/issues/11)

I think the idea behind such a design would have been to be able to just  
search across files and see what files changed, after a given time or  
something like that, but I assume, if we want to do search using this on  
file system for text files - json, xml, any custom text format (instead of  
grepping around), we should be able show the content & location of that  
content along with the result of the search too.

Cheers  
Sunny

On Friday, March 15, 2013 7:58:27 PM UTC-7, David Pilato wrote:

> This is by design.
> 
> If you want to inject JSON files (json\_support: true), it's injected as is  
> without any modification.  
> When you disable json\_support (default), files (whatever the format) are  
> sent to the mapper attachment plugin and content is extracted with Tika and  
> indexed.
> 
> What you see in file.content is the raw file base64 encoded.
> 
> A nice option would be to add path in metadata section as we have \_index,  
> \_type, \_id.  
> I should try to do it with  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/meta.html).
> 
> Could you open an issue for that?
> 
> ## Thanks for the feedback.
> 
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 16 mars 2013 à 01:44, Shadow \<[suryaveers...@gmail.com](mailto:suryaveers...@gmail.com) \<javascript:\>\> a  
> écrit :
> 
> An update, I tried with Latest Elasticsearch and FS River plugin - 0.1.0,  
> but still issue persists. The only difference was with 0.1.0, if I use  
> {"filename\_as\_id": true} in the definition of "fs", then I do get the  
> local file name (Not the complete URL of file/ relative url of the file  
> from the url given in definition). I would definitely not use  
> "filename\_as\_id" flag , because then chances of conflict are quite high, I  
> would have expected with that flas, I get absolute/relative file path from  
> the url given. can we get that?
> 
> Also, still looking for a solution where I see the content of the files  
> completely in search result and also the full/relative file path where the  
> search hit a match.
> 
> Thanks!
> 
> On Friday, March 15, 2013 4:22:52 PM UTC-7, Shadow wrote:
> 
> > FS River seems like a great plugin!  
> > I tried to play around with some data. For one, it was quite fast (y).
> > 
> > But I ran into some issues/quirks, or maybe I didn't use it correctly.  
> > I'm using version 0.0.3, with 0.20.5 of elasticsearch and using filesystem.
> > 
> > 1. When I create river with following setting:  
> > curl -XDELETE 127.0.0.1:9200/\_river/foo  
> > curl -XDELETE 127.0.0.1:9200/foo  
> > curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
> > "type": "fs",  
> > "fs": {  
> > "name": "Foo Data",  
> > "url": "/Users/slodha/foo/content",  
> > "update\_rate": 60000,  
> > "includes": "\*.json",  
> > "json\_support" : true  
> > },  
> > "index": {  
> > "index": "foo",  
> > "type": "foo",  
> > "bulk\_size": 50  
> > }  
> > }'
> > 
> > and search on this with this query:  
> > {  
> > "query": {  
> > "query\_string": {  
> > "default\_field": "\_all",  
> > "query": "slodha"  
> > }  
> > }  
> > }
> > 
> > I get results like:
> > 
> > hits: [
> > 
> > - {
> > - \_index: foo
> > - \_type: foo
> > - \_id: 18156b6b5a6b3a8e1ec5984f185e18
> > - \_score: 6.9584246
> > - \_source: {
> > - sunnyVal: slodha  
> > }  
> > }
> > 
> > - {
> > - \_index: foo
> > - \_type: foo
> > - \_id: d7b4df4222e0d075d74ffde8aaa04a56
> > - \_score: 6.901722
> > - \_source: {
> > - fileNameTest: slodha  
> > }  
> > }
> > 
> > ]
> > 
> > -- I never get which file it belonged to - which I would definitely need  
> > to be able to search in the filesystem eventually.
> > 
> > When I do this:
> > 
> > curl -XDELETE 127.0.0.1:9200/\_river/foo  
> > curl -XDELETE 127.0.0.1:9200/foo  
> > curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
> > "type": "fs",  
> > "fs": {  
> > "name": "Foo Data",  
> > "url": "/Users/slodha/foo/content",  
> > "update\_rate": 60000  
> > },  
> > "index": {  
> > "index": "city",  
> > "type": "city",  
> > "bulk\_size": 50  
> > }  
> > }'
> > 
> > Notice here I do not use any json restriction..  
> > I get results like this:
> > 
> > hits: [
> > 
> > - {
> > - \_index: foo
> > - \_type: foo
> > - \_id: 18156b6b5a6b3a8e1ec5984f185e18
> > - \_score: 2.0015228
> > - \_source: {
> > - name: slodha\_1.json
> > - postDate: 1363384941000
> > - pathEncoded: 44d22b925f562f4e8d1d847253493336
> > - rootpath: 948cd64d775db4119962b5a36dd530
> > - virtualpath: t/sunnyTest
> > - file: {
> > - \_name: slodha\_1.json
> > - content: ewoic3VubnlWYWwiIDogInNsb2RoYSIgCn0K  
> > }  
> > }  
> > }
> > 
> > - {
> > - \_index: foo
> > - \_type: foo
> > - \_id: d7b4df4222e0d075d74ffde8aaa04a56
> > - \_score: 1.7533717
> > - \_source: {
> > - name: file1.json
> > - postDate: 1363388628000
> > - pathEncoded: 99d79f46f1ce275b6b9152a0de54d5
> > - rootpath: 948cd64d775db4119962b5a36dd530
> > - virtualpath: t/sunnyTest/sunnyTest2
> > - file: {
> > - \_name: file1.json
> > - content: ewoiZmlsZU5hbWVUZXN0IiA6ICJzbG9kaGEiCn0K  
> > }  
> > }  
> > }
> > 
> > ]
> > 
> > Now, here I get to know the exact file paths, but this time the content  
> > is all a hash sum, and not readable..
> > 
> > I'm sure there should be a way for me to see both content and file paths  
> > as human readable with just one river. Can you suggest what I'm doing wrong?
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![shadow](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@shadow](https://discuss.elastic.co/u/shadow)\
**Post date:** [March 18, 2013, 5:52pm UTC](https://discuss.elastic.co/t/question-about-fsriver-0-0-3/11174/5 "2013-03-18T17:52:44Z")

</div>

Also, when using JSON support and "filename as ID" is it possible to get  
the absolute/relative "filepath" as ID, and not just the filename? maybe  
another key "filepath\_as\_id" ? I personally feel that would be quite  
helpful as well.

On Friday, March 15, 2013 11:48:07 PM UTC-7, Shadow wrote:

> Thanks, opened an issue - [Can we add feature to be able to to see both file content and file paths as human readable format (text) with just one river for text files · Issue #11 · dadoonet/fscrawler · GitHub](https://github.com/dadoonet/fsriver/issues/11)
> 
> I think the idea behind such a design would have been to be able to just  
> search across files and see what files changed, after a given time or  
> something like that, but I assume, if we want to do search using this on  
> file system for text files - json, xml, any custom text format (instead of  
> grepping around), we should be able show the content & location of that  
> content along with the result of the search too.
> 
> Cheers  
> Sunny
> 
> On Friday, March 15, 2013 7:58:27 PM UTC-7, David Pilato wrote:
> 
> > This is by design.
> > 
> > If you want to inject JSON files (json\_support: true), it's injected as  
> > is without any modification.  
> > When you disable json\_support (default), files (whatever the format) are  
> > sent to the mapper attachment plugin and content is extracted with Tika and  
> > indexed.
> > 
> > What you see in file.content is the raw file base64 encoded.
> > 
> > A nice option would be to add path in metadata section as we have \_index,  
> > \_type, \_id.  
> > I should try to do it with  
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/meta.html).
> > 
> > Could you open an issue for that?
> > 
> > ## Thanks for the feedback.
> > 
> > David 😉  
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > 
> > Le 16 mars 2013 à 01:44, Shadow [suryaveers...@gmail.com](mailto:suryaveers...@gmail.com) a écrit :
> > 
> > An update, I tried with Latest Elasticsearch and FS River plugin - 0.1.0,  
> > but still issue persists. The only difference was with 0.1.0, if I use  
> > {"filename\_as\_id": true} in the definition of "fs", then I do get the  
> > local file name (Not the complete URL of file/ relative url of the file  
> > from the url given in definition). I would definitely not use  
> > "filename\_as\_id" flag , because then chances of conflict are quite high, I  
> > would have expected with that flas, I get absolute/relative file path from  
> > the url given. can we get that?
> > 
> > Also, still looking for a solution where I see the content of the files  
> > completely in search result and also the full/relative file path where the  
> > search hit a match.
> > 
> > Thanks!
> > 
> > On Friday, March 15, 2013 4:22:52 PM UTC-7, Shadow wrote:
> > 
> > > FS River seems like a great plugin!  
> > > I tried to play around with some data. For one, it was quite fast (y).
> > > 
> > > But I ran into some issues/quirks, or maybe I didn't use it correctly.  
> > > I'm using version 0.0.3, with 0.20.5 of elasticsearch and using filesystem.
> > > 
> > > 1. When I create river with following setting:  
> > > curl -XDELETE 127.0.0.1:9200/\_river/foo  
> > > curl -XDELETE 127.0.0.1:9200/foo  
> > > curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
> > > "type": "fs",  
> > > "fs": {  
> > > "name": "Foo Data",  
> > > "url": "/Users/slodha/foo/content",  
> > > "update\_rate": 60000,  
> > > "includes": "\*.json",  
> > > "json\_support" : true  
> > > },  
> > > "index": {  
> > > "index": "foo",  
> > > "type": "foo",  
> > > "bulk\_size": 50  
> > > }  
> > > }'
> > > 
> > > and search on this with this query:  
> > > {  
> > > "query": {  
> > > "query\_string": {  
> > > "default\_field": "\_all",  
> > > "query": "slodha"  
> > > }  
> > > }  
> > > }
> > > 
> > > I get results like:
> > > 
> > > hits: [
> > > 
> > > - {
> > > - \_index: foo
> > > - \_type: foo
> > > - \_id: 18156b6b5a6b3a8e1ec5984f185e18
> > > - \_score: 6.9584246
> > > - \_source: {
> > > - sunnyVal: slodha  
> > > }  
> > > }
> > > 
> > > - {
> > > - \_index: foo
> > > - \_type: foo
> > > - \_id: d7b4df4222e0d075d74ffde8aaa04a56
> > > - \_score: 6.901722
> > > - \_source: {
> > > - fileNameTest: slodha  
> > > }  
> > > }
> > > 
> > > ]
> > > 
> > > -- I never get which file it belonged to - which I would definitely need  
> > > to be able to search in the filesystem eventually.
> > > 
> > > When I do this:
> > > 
> > > curl -XDELETE 127.0.0.1:9200/\_river/foo  
> > > curl -XDELETE 127.0.0.1:9200/foo  
> > > curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
> > > "type": "fs",  
> > > "fs": {  
> > > "name": "Foo Data",  
> > > "url": "/Users/slodha/foo/content",  
> > > "update\_rate": 60000  
> > > },  
> > > "index": {  
> > > "index": "city",  
> > > "type": "city",  
> > > "bulk\_size": 50  
> > > }  
> > > }'
> > > 
> > > Notice here I do not use any json restriction..  
> > > I get results like this:
> > > 
> > > hits: [
> > > 
> > > - {
> > > - \_index: foo
> > > - \_type: foo
> > > - \_id: 18156b6b5a6b3a8e1ec5984f185e18
> > > - \_score: 2.0015228
> > > - \_source: {
> > > - name: slodha\_1.json
> > > - postDate: 1363384941000
> > > - pathEncoded: 44d22b925f562f4e8d1d847253493336
> > > - rootpath: 948cd64d775db4119962b5a36dd530
> > > - virtualpath: t/sunnyTest
> > > - file: {
> > > - \_name: slodha\_1.json
> > > - content: ewoic3VubnlWYWwiIDogInNsb2RoYSIgCn0K  
> > > }  
> > > }  
> > > }
> > > 
> > > - {
> > > - \_index: foo
> > > - \_type: foo
> > > - \_id: d7b4df4222e0d075d74ffde8aaa04a56
> > > - \_score: 1.7533717
> > > - \_source: {
> > > - name: file1.json
> > > - postDate: 1363388628000
> > > - pathEncoded: 99d79f46f1ce275b6b9152a0de54d5
> > > - rootpath: 948cd64d775db4119962b5a36dd530
> > > - virtualpath: t/sunnyTest/sunnyTest2
> > > - file: {
> > > - \_name: file1.json
> > > - content: ewoiZmlsZU5hbWVUZXN0IiA6ICJzbG9kaGEiCn0K  
> > > }  
> > > }  
> > > }
> > > 
> > > ]
> > > 
> > > Now, here I get to know the exact file paths, but this time the content  
> > > is all a hash sum, and not readable..
> > > 
> > > I'm sure there should be a way for me to see both content and file paths  
> > > as human readable with just one river. Can you suggest what I'm doing wrong?
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 18, 2013, 6:16pm UTC](https://discuss.elastic.co/t/question-about-fsriver-0-0-3/11174/6 "2013-03-18T18:16:39Z")

</div>

No you can't have an id with `/` I think. Or it needs to be encoded.  
But, if I can add path in meta, I can add the full path or the relative one as needed.

Perhaps, you can modify your issue or add a comment?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 18 mars 2013 à 18:52, Shadow [suryaveersingh.lodha@gmail.com](mailto:suryaveersingh.lodha@gmail.com) a écrit :

> Also, when using JSON support and "filename as ID" is it possible to get the absolute/relative "filepath" as ID, and not just the filename? maybe another key "filepath\_as\_id" ? I personally feel that would be quite helpful as well.
> 
> On Friday, March 15, 2013 11:48:07 PM UTC-7, Shadow wrote:  
> Thanks, opened an issue - [Can we add feature to be able to to see both file content and file paths as human readable format (text) with just one river for text files · Issue #11 · dadoonet/fscrawler · GitHub](https://github.com/dadoonet/fsriver/issues/11)
> 
> I think the idea behind such a design would have been to be able to just search across files and see what files changed, after a given time or something like that, but I assume, if we want to do search using this on file system for text files - json, xml, any custom text format (instead of grepping around), we should be able show the content & location of that content along with the result of the search too.
> 
> Cheers  
> Sunny
> 
> On Friday, March 15, 2013 7:58:27 PM UTC-7, David Pilato wrote:  
> This is by design.
> 
> If you want to inject JSON files (json\_support: true), it's injected as is without any modification.  
> When you disable json\_support (default), files (whatever the format) are sent to the mapper attachment plugin and content is extracted with Tika and indexed.
> 
> What you see in file.content is the raw file base64 encoded.
> 
> A nice option would be to add path in metadata section as we have \_index, \_type, \_id.  
> I should try to do it with [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/meta.html).
> 
> Could you open an issue for that?
> 
> ## Thanks for the feedback.
> 
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 16 mars 2013 à 01:44, Shadow [suryaveers...@gmail.com](mailto:suryaveers...@gmail.com) a écrit :
> 
> An update, I tried with Latest Elasticsearch and FS River plugin - 0.1.0, but still issue persists. The only difference was with 0.1.0, if I use {"filename\_as\_id": true} in the definition of "fs", then I do get the local file name (Not the complete URL of file/ relative url of the file from the url given in definition). I would definitely not use "filename\_as\_id" flag , because then chances of conflict are quite high, I would have expected with that flas, I get absolute/relative file path from the url given. can we get that?
> 
> Also, still looking for a solution where I see the content of the files completely in search result and also the full/relative file path where the search hit a match.
> 
> Thanks!
> 
> On Friday, March 15, 2013 4:22:52 PM UTC-7, Shadow wrote:  
> FS River seems like a great plugin!  
> I tried to play around with some data. For one, it was quite fast (y).
> 
> But I ran into some issues/quirks, or maybe I didn't use it correctly. I'm using version 0.0.3, with 0.20.5 of elasticsearch and using filesystem.
> 
> 1. When I create river with following setting:  
> curl -XDELETE 127.0.0.1:9200/\_river/foo  
> curl -XDELETE 127.0.0.1:9200/foo  
> curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
> "type": "fs",  
> "fs": {  
> "name": "Foo Data",  
> "url": "/Users/slodha/foo/content",  
> "update\_rate": 60000,  
> "includes": "\*.json",  
> "json\_support" : true  
> },  
> "index": {  
> "index": "foo",  
> "type": "foo",  
> "bulk\_size": 50  
> }  
> }'
> 
> and search on this with this query:  
> {  
> "query": {  
> "query\_string": {  
> "default\_field": "\_all",  
> "query": "slodha"  
> }  
> }  
> }
> 
> I get results like:
> 
> hits: [  
> {  
> \_index: foo  
> \_type: foo  
> \_id: 18156b6b5a6b3a8e1ec5984f185e18  
> \_score: 6.9584246  
> \_source: {  
> sunnyVal: slodha  
> }  
> }  
> {  
> \_index: foo  
> \_type: foo  
> \_id: d7b4df4222e0d075d74ffde8aaa04a56  
> \_score: 6.901722  
> \_source: {  
> fileNameTest: slodha  
> }  
> }  
> ]
> 
> -- I never get which file it belonged to - which I would definitely need to be able to search in the filesystem eventually.
> 
> When I do this:
> 
> curl -XDELETE 127.0.0.1:9200/\_river/foo  
> curl -XDELETE 127.0.0.1:9200/foo  
> curl -XPUT 'localhost:9200/\_river/foo/\_meta' -d '{  
> "type": "fs",  
> "fs": {  
> "name": "Foo Data",  
> "url": "/Users/slodha/foo/content",  
> "update\_rate": 60000  
> },  
> "index": {  
> "index": "city",  
> "type": "city",  
> "bulk\_size": 50  
> }  
> }'
> 
> Notice here I do not use any json restriction..  
> I get results like this:
> 
> hits: [  
> {  
> \_index: foo  
> \_type: foo  
> \_id: 18156b6b5a6b3a8e1ec5984f185e18  
> \_score: 2.0015228  
> \_source: {  
> name: slodha\_1.json  
> postDate: 1363384941000  
> pathEncoded: 44d22b925f562f4e8d1d847253493336  
> rootpath: 948cd64d775db4119962b5a36dd530  
> virtualpath: t/sunnyTest  
> file: {  
> \_name: slodha\_1.json  
> content: ewoic3VubnlWYWwiIDogInNsb2RoYSIgCn0K  
> }  
> }  
> }  
> {  
> \_index: foo  
> \_type: foo  
> \_id: d7b4df4222e0d075d74ffde8aaa04a56  
> \_score: 1.7533717  
> \_source: {  
> name: file1.json  
> postDate: 1363388628000  
> pathEncoded: 99d79f46f1ce275b6b9152a0de54d5  
> rootpath: 948cd64d775db4119962b5a36dd530  
> virtualpath: t/sunnyTest/sunnyTest2  
> file: {  
> \_name: file1.json  
> content: ewoiZmlsZU5hbWVUZXN0IiA6ICJzbG9kaGEiCn0K  
> }  
> }  
> }  
> ]
> 
> Now, here I get to know the exact file paths, but this time the content is all a hash sum, and not readable..
> 
> I'm sure there should be a way for me to see both content and file paths as human readable with just one river. Can you suggest what I'm doing wrong?
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:46am UTC](https://discuss.elastic.co/t/question-about-fsriver-0-0-3/11174/7 "2017-07-06T02:46:02Z")

</div>