Error expecting # line

Good evening everyone,

I've been trying to track down some errors and get my first job running and collecting data from twitter, I found some examples on github and slightly modified them to fit my needs. At this point in time I don't really understand what the error is telling me although I've read up on people having similar issues I have not been able to figure this one out.

.conf file

input {
  twitter {
    consumer_key       => "XXXXXX"
    consumer_secret    => "XXXXXX"
    oauth_token        => "XXXXXX"
    oauth_token_secret => "XXXXXX"
    keywords           => [ "thor", "spiderman", "wolverine", "ironman", "hulk"]
    full_tweet         => true
  }
}

filter { }

output {
  stdout {
    codec => dots
  }
  elasticsearch {
      user  => "XXXXXXX"
      password => "XXXXXX"
      hosts => "localhost:9200"
      index         => "twitter_elastic_example"
      document_type => "tweets"
      template      => "./twitter_template.json"
      template_name => "twitter_elastic_example"
      template_overwrite => true
  }
}

here is the .json

{
  "template": "twitter_elastic_example",
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true
      },
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "dateOptionalTime"
        },
        "text": {
          "type": "text"
        },
        "user": {
          "type": "object",
          "properties": {
            "description": {
              "type": "text"
            }
          }
        },
        "coordinates": {
          "type": "object",
          "properties": {
            "coordinates": {
              "type": "geo_point"
            }
          }
        },
        "entities": {
          "type": "object",
          "properties": {
            "hashtags": {
              "type": "object",
              "properties": {
                "text": {
                  "type": "text",
                  "fielddata": true
                }
              }
            }
          }
        },
        "retweeted_status": {
          "type": "object",
          "properties": {
            "text": {
              "type": "text"
            }
          }
        }
      },
      "dynamic_templates": [
        {
          "string_template": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword"
            }
          }
        }
      ]
    }
  }
}

So from what I've gathered is there is a problem during the concatenation between the two files, I've tried to join them but was still unable to find the error it was indicating was taking place. So currently my machine consists for 3 nodes a master, minion, and client node which should be set up as a search load balance. They are currently running on aws, and while I have a yellow status for the cluster it seem that I can monitor and communicate through out the different machines. I've also tried to start the job on the Master Node and the Ingest node, both with no luck on getting the task started.

First let me thank you in advance, This has been a really cool project although I will admit there has been a bit of a learning curve to overcome.

Can you post the actual error?

Sorry I probably should have done that the first post, but here it is

[2017-02-28T00:24:44,126][ERROR][logstash.agent ] fetched an invalid config {:config=>"input {\n twitter {\n consumer_key => \"xxxxxxx\"\n consumer_secret => \"xxxxxxxxx\"\n oauth_token => \"xxxxxxxxxxx\"\n oauth_token_secret => \"xxxxxxxxxxx\"\n keywords => [ \"thor\", \"spiderman\", \"wolverine\", \"ironman\", \"hulk\"]\n full_tweet => true\n }\n}\n\nfilter { }\n\noutput {\n stdout {\n codec => dots\n }\n elasticsearch {\n user => \"xxxxxxxxxx\"\n password => \"xxxxxxxxxxxxx\"\n hosts => \"localhost:9200\"\n index => \"twitter_elastic_example\"\n document_type => \"tweets\"\n template => \"./twitter_template.json\"\n template_name => \"twitter_elastic_example\"\n template_overwrite => true\n }\n}\n\n{ \n \"template\": \"twitter_elastic_example\",\n \"settings\": {\n \"number_of_shards\": 1,\n \"number_of_replicas\": 0\n },\n \"mappings\": {\n \"_default_\": {\n \"_all\": {\n \"enabled\": true\n },\n \"properties\": {\n \"@timestamp\": {\n \"type\": \"date\",\n \"format\": \"dateOptionalTime\"\n },\n \"text\": {\n \"type\": \"text\"\n },\n \"user\": {\n \"type\": \"object\",\n \"properties\": {\n \"description\": {\n \"type\": \"text\"\n }\n }\n },\n \"coordinates\": {\n \"type\": \"object\",\n \"properties\": {\n \"coordinates\": {\n \"type\": \"geo_point\"\n }\n }\n },\n \"entities\": {\n \"type\": \"object\",\n \"properties\": {\n \"hashtags\": {\n \"type\": \"object\",\n \"properties\": {\n \"text\": {\n \"type\": \"text\",\n \"fielddata\": true\n }\n }\n }\n }\n },\n \"retweeted_status\": {\n \"type\": \"object\",\n \"properties\": {\n \"text\": {\n \"type\": \"text\"\n }\n }\n }\n },\n \"dynamic_templates\": [\n {\n \"string_template\": {\n \"match\": \"*\",\n \"match_mapping_type\": \"string\",\n \"mapping\": {\n \"type\": \"keyword\"\n }\n }\n }\n ]\n }\n }\n}\n\n", :reason=>"Expected one of #, input, filter, output at line 30, column 1 (byte 804) after "}

I can't see a line 30 in your config?

So going off another post I was reading in the process of trying to get this to work it was explained that the files are joined using a command something along these lines

`cat /etc/logstash/conf.d/*  > ./test.conf` 

which basically vertically concatenation the .conf file to the .json, then it's sent off to the next stop. So line 30 actually falls in either the first of 2nd line of the json file when they are linked like that.

here is the link http://stackoverflow.com/questions/34164603/error-expected-one-of-input-filter-output-at-line-24-column-1-byte-528

What json file, the template? If so then no, it doesn't do that, it only cat's the LS config files. Not the template.

Well that isn't good then, Not sure why it's throwing an error on line 30. I guess I'll have to do some reading and playing around with the files.

Do you have the template JSON file in the same directory as the Logstash configuration files? Don't do that.

2 Likes

Ok awesome I'll relocate that document, I guess it came down to a problem with the logstash.yml, I was able to get it to run by using the following /usr/share/logstash/bin/logstash --path.settings=/etc/logstash/logstash.yml twitter.conf So I'll have to dig into that this morning. However I'll also get the json moved out of there and only keep my config files there.

If you had the template file in the same dir as the .conf file, that would explain the error.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.