# Fscrawler pipeline feature
**URL:** https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041
**Category:** Elasticsearch
**Created:** [June 15, 2018, 7:28am UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041 "2018-06-15T07:28:47Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@pa26992](https://discuss.elastic.co/u/pa26992)
#### Post date: [June 15, 2018, 7:28am UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/1 "2018-06-15T07:28:47Z")
Hey I am unable to use the fscrawler pipeline feature. I wanted to use it to replace '\_' in path.real to '/'. I am using 6.X kibana and elasticsearch and 2.4 fscrawler.
Config is as follows.
```auto
{
"name" : "test1",
"fs" : {
"url" : "/tmp/es",
"update_rate" : "15m",
"excludes" : ["~*"],
"json_support" : false,
"filename_as_id" : false,
"add_filesize" : true,
"remove_deleted" : true,
"add_as_inner_object" : false,
"store_source" : false,
"index_content" : true,
"attributes_support" : false,
"raw_metadata" : false,
"xml_support" : false,
"index_folders" : true,
"lang_detect" : false,
"continue_on_error" : false,
"pdf_ocr" : true,
"ocr" : {
"language" : "eng"
}
},
"elasticsearch" : {
"nodes" : [ {
"host" : "127.0.0.1",
"port" : 9200,
"scheme" : "HTTP"
} ],
"pipeline": "pipeline1",
"bulk_size" : 100,
"flush_interval" : "5s"
},
"rest" : {
"scheme" : "HTTP",
"host" : "127.0.0.1",
"port" : 8080,
"endpoint" : "fscrawler"
}
}
```
Creating pipeline as follows:
```auto
PUT _ingest/pipeline/pipeline1
{
"description" : "testing pipeline",
"processors" : [
{
"gsub": {
"field": "path.real",
"pattern": "_",
"replacement": "/"
}
}
]
}
```
I am uploading pdfs using rest feature of fscrawler
Please help.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [June 15, 2018, 10:24am UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/2 "2018-06-15T10:24:49Z")
Please format your code, logs or configuration files using `` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) and not the citation button. It will make your post more readable.
Or use markdown style like:
````
```
CODE
```
````
This is the icon to use if you are not using markdown format:
There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.
---
### Author:  [@pa26992](https://discuss.elastic.co/u/pa26992)
#### Post date: [June 15, 2018, 11:50am UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/3 "2018-06-15T11:50:42Z")
Apologies, I have formatted the query now.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [June 15, 2018, 12:03pm UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/4 "2018-06-15T12:03:18Z")
Some few things to check.
First can you try the latest SNAPSHOT of FSCrawler?
Then, did you simulate your pipeline with the simulate endpoint to make sure your regex does what you expect?
If so, could you share the simulate call and the response?
---
### Author:  [@pa26992](https://discuss.elastic.co/u/pa26992)
#### Post date: [June 15, 2018, 3:12pm UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/5 "2018-06-15T15:12:41Z")
> [@dadoonet](#):
>
> First can you try the latest SNAPSHOT of FSCrawler?
Sorry wont be able to do that, dont have the required rights on the system.
> [@dadoonet](#):
>
> Then, did you simulate your pipeline with the simulate endpoint to make sure your regex does what you expect?
> If so, could you share the simulate call and the response?
I tried simulating even a small "set" but doesnt seem to be working on Elasticsearch. please help me if I am missing something, I am very new to elasticsearch and fscrawler. Following is the request and response
```auto
POST _ingest/pipeline/pipeline1/_simulate
{
"pipeline":{
"description" : "testing pipeline",
"processors" : [
{
"set": {
"field":"path",
"value": "bar123"
}
}
]
},
"docs": [
{
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"path": "test_123"
}
}
]
}
```
```auto
{
"docs": [
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"path": "test_123",
"content": "bar"
},
"_ingest": {
"timestamp": "2018-06-15T15:07:39.718Z"
}
}
}
]
}
```
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [June 15, 2018, 4:23pm UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/6 "2018-06-15T16:23:17Z")
This is working well:
```auto
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": "testing pipeline",
"processors": [
{
"gsub": {
"field": "path",
"pattern": "_",
"replacement": "/"
}
}
]
},
"docs": [
{
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"path": "test_123"
}
}
]
}
```
It gives:
```auto
{
"docs": [
{
"doc": {
"_index": "index",
"_type": "_doc",
"_id": "id",
"_source": {
"path": "test/123"
},
"_ingest": {
"timestamp": "2018-06-15T16:22:10.750350Z"
}
}
}
]
}
```
Can you do it now with a typical document sent by FSCrawler to Elasticsearch?
---
### Author:  [@pa26992](https://discuss.elastic.co/u/pa26992)
#### Post date: [June 26, 2018, 10:33am UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/7 "2018-06-26T10:33:18Z")
Hi,
Thanks for the reply. Yes, it did work with elastic search but am afraid not with fscrawler.
Following is how am i using this.
```auto
PUT _ingest/pipeline/pipeline1
{
"description": "testing pipeline",
"processors": [
{
"gsub": {
"field": "meta.raw.Application-Name",
"pattern": " ",
"replacement": "/"
}
}
]
}
}
```
FS Crawler config
```auto
{
"name" : "pipeline_testing",
"fs" : {
"url" : "/home/testfolder1",
"update_rate" : "1m",
"excludes" : ["~*"],
"json_support" : false,
"filename_as_id" : false,
"add_filesize" : true,
"remove_deleted" : true,
"add_as_inner_object" : false,
"store_source" : false,
"index_content" : true,
"attributes_support" : false,
"raw_metadata" : false,
"xml_support" : false,
"index_folders" : true,
"lang_detect" : false,
"continue_on_error" : false,
"pdf_ocr" : true,
"ocr" : {
"language" : "eng"
}
},
"elasticsearch" : {
"nodes" : [ {
"host" : "127.0.0.1",
"port" : 9200,
"scheme" : "HTTP"
} ],
"pipeline": "pipeline1",
"bulk_size" : 100,
"flush_interval" : "5s"
},
"rest" : {
"scheme" : "HTTP",
"host" : "127.0.0.1",
"port" : 8080,
"endpoint" : "fscrawler"
}
}
```
It does not indexes any document with this config. removing pipeline indexes but doesn't uses pipeline. Same worked well with simulate and when elasticsearch is used directly. Am I missing something here if you can suggest.
Thanks
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [June 27, 2018, 7:36am UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/8 "2018-06-27T07:36:35Z")
Can you share the output of:
```auto
GET pipeline_testing/_search
{
"size": 1
}
```
---
### Author:  [@pa26992](https://discuss.elastic.co/u/pa26992)
#### Post date: [June 28, 2018, 5:38am UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/9 "2018-06-28T05:38:43Z")
Following is the output of the above query when pipeline is set in fscrawler config
```auto
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
```
As soon as I remove line "pipeline":"pipeline1", from the config it returns the following
```auto
{
"took": 0,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 100,
"max_score": 1,
"hits": [
{
"_index": "pipeline_testing",
"_type": "doc",
"_id": "b5db833fe0ee409de6b926d67e63b1",
"_score": 1,
"_source": {
"content": """
.........
........
```
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [June 28, 2018, 5:50am UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/10 "2018-06-28T05:50:58Z")
Could you share the full output of the latest?
---
### Author:  [@pa26992](https://discuss.elastic.co/u/pa26992)
#### Post date: [June 28, 2018, 6:02am UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/11 "2018-06-28T06:02:17Z")
Apologies, issue was on my side. I tried searching for error on elastic search console and found the following error. Found that I changed the key earlier while simulating to check if its drills down in json and forgot to correct that in actual pipeline. I corrected that and its working well now.
```auto
[2018-06-28T11:05:19,468][DEBUG][o.e.a.b.TransportBulkAction] [C8-OFwE] failed to execute pipeline [pipeline1] for document [pipeline_testing/doc/bd1cd038f977f76ce54bc1cace51d4]
org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [raw] not present as part of path [meta.raw.Application-Name]
at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:156) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:107) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:58) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.ingest.PipelineExecutionService.innerExecute(PipelineExecutionService.java:169) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.ingest.PipelineExecutionService.access$000(PipelineExecutionService.java:42) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:94) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [raw] not present as part of path [meta.raw.Application-Name]
... 11 more
Caused by: java.lang.IllegalArgumentException: field [raw] not present as part of path [meta.raw.Application-Name]
```
Thanks for your time. It really helped me.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 26, 2018, 6:12am UTC](https://discuss.elastic.co/t/fscrawler-pipeline-feature/136041/12 "2018-07-26T06:12:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.