Parsing word and pdf file in elasticsearch using logstash

Elastic Stack Logstash
1

I m trying to read pdf file using logstash
i use following config file

input {
file {
path => "D:\ELKstack\elastic\page_ELK.pdf"
start_position => "beginning"
}
}

filter {
if [type] == ".pdf"{
multiline {
pattern => "^\n"
what => "next"
negate=>false
}
}
}
output {
elasticsearch{ hosts => ["localhost:9000"] index => "logstash-pdf" }
stdout {codec => "rubydebug" }
}

output is-

ԿD\xA6r\xBD_\xFF,Gh\xFE}ؙ\xB8g+[\u0006ۿ\xF2\xC5gp^\xE8\xDA\xE7Ym\u0083\xA7\u0013\xF8'X\xB5\x9E.\xBD\xC5\xFB\xFCR\x97\xBA\x8Fۘ\xB1\x80QH%wE1\u001F\xB5\x9B\x9B\xA2Et\x8B\u001Fd\u0005\x9AC\xA4\xDDKdq@\x96\xC4\xF1\u001AY\xA7L05\xBA\xF0Ӗץ\x99\xEAy\x8E\u0011\xA0\xDD\xDEٽAEӀ\x8A\u0011\"_\x9C\xC4\u001D\xC9avA%\xA3\xB4#\xD7\xE9؋\x838\x96\xC8'\xB8bL\xED\xB2l;p\xF6\x86\f\t\xB9\u0001O\u0011NGf\xE8tYW\xD6\xF9\xD6t\xADRӐl\xE5e\xD5xL\xA9\xC6y\xE1\u05F9\u0015\xE6_\xE1j/х,\u0001\x8D\u007F\xB2Fw>\u000Fa\xC6\u0019|\xF6\xAE\xC9|\xE3\xCC\u0012Q\u00152\xAD~"

it is not readable.what should i do?

2

Logstash cannot parse these sorts of files.

See https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest-attachment.html

3

thanks @warkolm

4

can i do it using kibana?

5

Nope.

6

ok... thanks

7

i m trying to load pdf in elasticsearch using ingest-attachment
i tried this-

PUT _ingest/pipeline/attachment
{
"description" : "Extract attachment information",
"processors" : [
{
"attachment" : {
"field" : "data"
}
}
]
}

PUT my_index/my_type/my_id?pipeline=attachment
{
"data": "JVBERi0xLjUNCiW1tbW1DQoxIDAgb2JqDQo8PC9UeXBlL0NhdGFsb2cvUGFnZXMgMiAwIFIvTGFu=\n"
}

output-

{
"error": {
"root_cause": [
{
"type": "exception",
"reason": "java.lang.IllegalArgumentException: ElasticsearchParseException[Error parsing document in field [data]]; nested: IllegalArgumentException[Input byte array has wrong 4-byte ending unit];",
"header": {
"processor_type": "attachment"
}
}
]
can you please tell me what is wrong in this?

8

how can i use apache tika with elasticsearch???

9

Hii @warkolm
I m trying to load pdf in elasticsearch using ingest-attachment plugin

PUT my_index/my_type/my_id?pipeline=attachment
{
"data": "JVBERi0xLjUNCiW1tbW1DQoxIDAgb2JqDQo8PC9UeXBlL0NhdGFsb2cvUGFnZXMgMiAwIFIvTGFu="
}

output-
{
"error": {
"root_cause": [
{
"type": "illegal_state_exception",
"reason": "There are no ingest nodes in this cluster, unable to forward request to an ingest node."
}
],
"type": "illegal_state_exception",
"reason": "There are no ingest nodes in this cluster, unable to forward request to an ingest node."
},
"status": 500
}

can you please tell me the correction?

10

You need to make sure you have a node that can run the ingest pipelines.

11

@warkolm
I checked in my elasticsearch.yml file. There is no ingest node.
Can I add it manually?

12

can u please tell me,
where do i set ingest node in elasticsearch.yml???

13

Have you read the documentation?
We are happy to help, but unfortunately we don't have the time to walk you through every step.

14

yes i read the documentation ,but i didn't get where to set that ingest.node in elasticsearch.yml.
I installed the ingest attachment plugin.
Thanks @warkolm I will try to resolve it.If any query there then i will get back to you.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.