Hey I am unable to use the fscrawler pipeline feature. I wanted to use it to replace '_' in path.real to '/'. I am using 6.X kibana and elasticsearch and 2.4 fscrawler.
Config is as follows.
Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.
Or use markdown style like:
```
CODE
```
This is the icon to use if you are not using markdown format:
There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.
First can you try the latest SNAPSHOT of FSCrawler?
Then, did you simulate your pipeline with the simulate endpoint to make sure your regex does what you expect?
If so, could you share the simulate call and the response?
Sorry wont be able to do that, dont have the required rights on the system.
I tried simulating even a small "set" but doesnt seem to be working on Elasticsearch. please help me if I am missing something, I am very new to elasticsearch and fscrawler. Following is the request and response
It does not indexes any document with this config. removing pipeline indexes but doesn't uses pipeline. Same worked well with simulate and when elasticsearch is used directly. Am I missing something here if you can suggest.
Thanks
Apologies, issue was on my side. I tried searching for error on elastic search console and found the following error. Found that I changed the key earlier while simulating to check if its drills down in json and forgot to correct that in actual pipeline. I corrected that and its working well now.
[2018-06-28T11:05:19,468][DEBUG][o.e.a.b.TransportBulkAction] [C8-OFwE] failed to execute pipeline [pipeline1] for document [pipeline_testing/doc/bd1cd038f977f76ce54bc1cace51d4]
org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [raw] not present as part of path [meta.raw.Application-Name]
at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:156) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:107) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:58) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.ingest.PipelineExecutionService.innerExecute(PipelineExecutionService.java:169) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.ingest.PipelineExecutionService.access$000(PipelineExecutionService.java:42) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:94) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [raw] not present as part of path [meta.raw.Application-Name]
... 11 more
Caused by: java.lang.IllegalArgumentException: field [raw] not present as part of path [meta.raw.Application-Name]
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.