Location Mapping

Yup

Line 4 has a space right..

start position => "beginning"

Should be

start_position => "beginning"

also see quotes on the next line where not needed on the left side

The conf file needs to be correct to execute... Just be careful I think you can take it from here.

Aadityas-MacBook:documents aadi2391$ logstash -f /Users/aadi2391/Desktop/Documents/logstash-7.2.0/config/logstash-Boston_Crimes.conf -t
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
[2019-08-04T15:00:48,548][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-04T15:00:49,305][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 18, column 11 (byte 541) after
[2019-08-04T15:00:49,322][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

what is the error now? syntax error has been eliminated but i cant understand this

input{
file {
path => " /Users/aadi2391/Desktop/Documents/CIS5900/Project/crime 2.csv "
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
csv {
seperator => ";"
columns => [ "OFFENSE_CODE" , "OFFENSE_CODE_GROUP" , "OFFENSE_DESCRIPTION" , "DISTRICT" ,
"REPORTING_AREA" , "SHOOTING" , "OCCURRED_ON_DATE" , "YEAR" , "MONTH" , "DAY_OF_WEEK" ,
"HOUR" , "UCR_PART" , "STREET" , "Lat", "Long" , "Location" ]
}
}

    mutate { convert => {"Latitude" => "float"} }
    mutate { convert => {"Longitude" => "float"} }
    mutate { rename => {"Latitude" => "[location][lat]"} }
    mutate { rename => {"Longitude" => "[location][lon]"} }
    
    }

output{

elasticsearch {
Hosts => ["localhost:9200"]
index => "nyc311calls"
document_type => "calls"
user => "elastic"
password => "changeme"}
stdout { codec => rubydebug { metadata => true } }
stdout { codec => dots }
}

can you go through this and let me know where exactly lies the mistake?

Remove the } before the mutate filters. That closes the filter section and the mutates need to be inside that. Also, add a } at the end of the file.

Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
[2019-08-04T15:33:17,231][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-04T15:33:18,434][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, => at line 32, column 12 (byte 971) after output {

elasticsearch {
Hosts => ["localhost:9200"]
index => "Boston_crimes"
document_type => "records"
user => "elastic"
password => "changeme"
stdout
[2019-08-04T15:33:18,451][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

i got this now?

input{
file {
path => " /Users/aadi2391/Desktop/Documents/CIS5900/Project/crime 2.csv "
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
csv {
seperator => ";"
columns => [ "OFFENSE_CODE" , "OFFENSE_CODE_GROUP" , "OFFENSE_DESCRIPTION" , "DISTRICT" ,
"REPORTING_AREA" , "SHOOTING" , "OCCURRED_ON_DATE" , "YEAR" , "MONTH" , "DAY_OF_WEEK" ,
"HOUR" , "UCR_PART" , "STREET" , "Lat", "Long" , "Location" ]
}

    mutate { convert => {"Latitude" => "float"} }
    mutate { convert => {"Longitude" => "float"} }
    mutate { rename => {"Latitude" => "[location][lat]"} }
    mutate { rename => {"Longitude" => "[location][lon]"} }
    
    }

output {

elasticsearch {
Hosts => ["localhost:9200"]
index => "Boston_crimes"
document_type => "records"
user => "elastic"
password => "changeme"
stdout { codec => rubydebug { metadata => true } }
stdout { codec => dots }
}

check this

Missing the last } at the end of the file as @Badger said

Aadityas-MacBook:documents aadi2391$ logstash -f /Users/aadi2391/Desktop/Documents/logstash-7.2.0/config/logstash-Boston_Crimes.conf -t
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
[2019-08-04T15:56:53,175][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-04T15:56:55,910][INFO ][org.reflections.Reflections] Reflections took 73 ms to scan 1 urls, producing 19 keys and 39 values
[2019-08-04T15:56:56,823][ERROR][logstash.filters.csv ] Unknown setting 'seperator' for csv
[2019-08-04T15:56:56,844][FATAL][logstash.runner ] The given configuration is invalid. Reason: Something is wrong with your configuration.
[2019-08-04T15:56:56,860][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

I added all those things what was needed but still this small error persist. Below is the conf file for ur kind reference:

input{
file {
path => " /Users/aadi2391/Desktop/Documents/CIS5900/Project/crime 2.csv "
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
csv {
seperator => ";"
columns => [ "OFFENSE_CODE" , "OFFENSE_CODE_GROUP" , "OFFENSE_DESCRIPTION" , "DISTRICT" ,
"REPORTING_AREA" , "SHOOTING" , "OCCURRED_ON_DATE" , "YEAR" , "MONTH" , "DAY_OF_WEEK" ,
"HOUR" , "UCR_PART" , "STREET" , "Lat", "Long" , "Location" ]
}

    mutate { convert => {"Latitude" => "float"} }
    mutate { convert => {"Longitude" => "float"} }
    mutate { rename => {"Latitude" => "[location][lat]"} }
    mutate { rename => {"Longitude" => "[location][lon]"} }
    
    }

output {

elasticsearch {
Hosts => ["localhost:9200"]
index => "Boston_crimes"
document_type => "records"

stdout { codec => rubydebug { metadata => true } }

stdout { codec => dots }

}
}

Kindly advice.

Hmmm see if you can figure out from the error and this...

https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-separator

This is basically a computer program ... Typos will always fail...

sir, I am still getting the error. Kindly please help

In the error message

[2019-08-04T15:56:56,823][ERROR][logstash.filters.csv ] Unknown setting 'seperator' for csv

Typo ...seperator vs separator

I also notice

path => " /Users/aadi2391/Desktop/Documents/CIS5900/Project/crime 2.csv

is there really supposed to be a space at the beginning before /User and in the file name crime 2.csv?

Perhaps pay close attention to the typos you have had many...

Good luck on the debugging ... You will figure it out!

Apologies when I say typos... they are among other items...

  • Incorrect Spelling
  • Incorrect capitalization or case (Upper or Lower)
  • Missing or added spaces or _ etc
  • Missing or additional {
  • Any other in correct entry.
    The conf must be typed in exactly.

Aadityas-MacBook:documents aadi2391$ logstash -f /Users/aadi2391/Desktop/Documents/logstash-7.2.0/config/logstash-Boston_Crimes.conf -t
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
[2019-08-04T19:39:24,275][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-04T19:39:28,112][INFO ][org.reflections.Reflections] Reflections took 215 ms to scan 1 urls, producing 19 keys and 39 values
[2019-08-04T19:39:29,077][ERROR][logstash.outputs.elasticsearch] Unknown setting 'Hosts' for elasticsearch
[2019-08-04T19:39:29,094][FATAL][logstash.runner ] The given configuration is invalid. Reason: Something is wrong with your configuration.
[2019-08-04T19:39:29,112][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

i am not able to understand besides typos

It seems you have used Hosts instead of hosts in your Elasticsearch output. Spelling and case matters.

1 Like

Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

how can u configure multiple instance?

How run this command (PUT) ? Because Im newbie.. Thank you for your advise

Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

how can u configure multiple instance?

Add --path.data to the command line used to start each instance and point the two instances at two different directories.

Alternatively, copy the contents of /etc/logstash to another directory, edit path.data in logstash.yml and then use --path.settings on the command line to point one instance at it.

Mr Badger,

Can you provide any example? I want to understand the syntax

/usr/share/logstash/bin/logstash --path.data /tmp/someDir --path.settings /etc/logstash

is how I start logstash.

{
$schema: https://vega.github.io/schema/vega/v3.0.json
config: {
kibana: {
type: "map"
latitude: 37.0902
longitude: 95.7129
zoom: 4
}
}
data: [
{
"name": "cities",
"values": [
{"city": "Boston", "lon": 42.308126191, "lat": -71.07692974, "pop": 4823991},
{"city": "Boston", "lon": 42.34432328, "lat": -71.15778368, "pop": 2270800},
{"city": "Boston", "lon": 42.37525782, "lat": -71.02466343, "pop": 4485211},
{"city": "Boston", "lon": 42.32324363, "lat": -71.10892316, "pop": 397397},
{"city": "Adelaide", "lon": 42.28076737, "lat": -71.04736497, "pop": 1295714}
]
transform: [
{
type: geopoint
projection: projection
fields: ["lon", "lat"]
}
]
}
]
marks: [
{
type: symbol
from: {data: "cities"}
encode: {
enter: {
size: {value: 100}
fill: {value: "black"}
stroke: {value: "orange"}
}
update: {
x: {field: "x"}
y: {field: "y"}
}
}
}
]
}

I want to plot the map of US with Boston as the main city.