Fscrawler using fiddler from remote machine

Elastic Stack Elasticsearch
1

If i want to call fscrawler from a remote machine and pass the file which is on remote machine as well, how can i do it ?

Here is what I am trying to do. I've a web job running on some server which will pick the file from blob storage and then call the fscrawler rest API and pass the content to it to index. But I am not sure How can i call FSCrawler REST API for upload.

Thanks
Himal

2

Once you have the file let say in /temp you can upload it to FSCrawler:

curl -F "file=@/temp/test.txt" "http://127.0.0.1:8080/fscrawler/_upload"
3

Hi,

Thanks for your reply. But does that mean that i need to copy the files into the temp folder ? I am planning to call the REST API remotely and pass the base64 content to it. Something similar to what mentioned here.

Also, When I try to do the same, I get "HTTP/1.1 415 Unsupported Media Type" error. Here is my request from fiddler.

POST : http://elasticsearch:8080/fscrawler/_upload

User-Agent: Fiddler
Host: elasticsearch:8080
Content-Length: 105
Content-Type: application/json

{
"filename": "test.txt",
"binary_content": VGhpcyBpcyByYW5kb20gdGVzdCBmb3IgZWxhc3RpYyBzZWFyY2gu
}

Thanks,
Himal

4

It's not supported but this can be added as an option. But why not using ingest-attachment plugin instead?

5

Got it. Was confused between ingest-attachment and elastic search. but seems like if i pass base64 data to ingest-attachment, that will index the content. Let me try it out. Thanks for your help.