Good evening everyone,
I've been trying to track down some errors and get my first job running and collecting data from twitter, I found some examples on github and slightly modified them to fit my needs. At this point in time I don't really understand what the error is telling me although I've read up on people having similar issues I have not been able to figure this one out.
.conf file
input {
twitter {
consumer_key => "XXXXXX"
consumer_secret => "XXXXXX"
oauth_token => "XXXXXX"
oauth_token_secret => "XXXXXX"
keywords => [ "thor", "spiderman", "wolverine", "ironman", "hulk"]
full_tweet => true
}
}
filter { }
output {
stdout {
codec => dots
}
elasticsearch {
user => "XXXXXXX"
password => "XXXXXX"
hosts => "localhost:9200"
index => "twitter_elastic_example"
document_type => "tweets"
template => "./twitter_template.json"
template_name => "twitter_elastic_example"
template_overwrite => true
}
}
here is the .json
{
"template": "twitter_elastic_example",
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"_default_": {
"_all": {
"enabled": true
},
"properties": {
"@timestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"text": {
"type": "text"
},
"user": {
"type": "object",
"properties": {
"description": {
"type": "text"
}
}
},
"coordinates": {
"type": "object",
"properties": {
"coordinates": {
"type": "geo_point"
}
}
},
"entities": {
"type": "object",
"properties": {
"hashtags": {
"type": "object",
"properties": {
"text": {
"type": "text",
"fielddata": true
}
}
}
}
},
"retweeted_status": {
"type": "object",
"properties": {
"text": {
"type": "text"
}
}
}
},
"dynamic_templates": [
{
"string_template": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
}
]
}
}
}
So from what I've gathered is there is a problem during the concatenation between the two files, I've tried to join them but was still unable to find the error it was indicating was taking place. So currently my machine consists for 3 nodes a master, minion, and client node which should be set up as a search load balance. They are currently running on aws, and while I have a yellow status for the cluster it seem that I can monitor and communicate through out the different machines. I've also tried to start the job on the Master Node and the Ingest node, both with no luck on getting the task started.
First let me thank you in advance, This has been a really cool project although I will admit there has been a bit of a learning curve to overcome.