Location Mapping

stephenb · August 4, 2019, 9:35pm

Yup

Line 4 has a space right..

start position => "beginning"

Should be

start_position => "beginning"

also see quotes on the next line where not needed on the left side

The conf file needs to be correct to execute... Just be careful I think you can take it from here.

aaditya_alekar · August 4, 2019, 10:02pm

Aadityas-MacBook:documents aadi2391$ logstash -f /Users/aadi2391/Desktop/Documents/logstash-7.2.0/config/logstash-Boston_Crimes.conf -t
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
[2019-08-04T15:00:48,548][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-04T15:00:49,305][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 18, column 11 (byte 541) after
[2019-08-04T15:00:49,322][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

what is the error now? syntax error has been eliminated but i cant understand this

aaditya_alekar · August 4, 2019, 10:12pm

input{
file {
path => " /Users/aadi2391/Desktop/Documents/CIS5900/Project/crime 2.csv "
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
csv {
seperator => ";"
columns => [ "OFFENSE_CODE" , "OFFENSE_CODE_GROUP" , "OFFENSE_DESCRIPTION" , "DISTRICT" ,
"REPORTING_AREA" , "SHOOTING" , "OCCURRED_ON_DATE" , "YEAR" , "MONTH" , "DAY_OF_WEEK" ,
"HOUR" , "UCR_PART" , "STREET" , "Lat", "Long" , "Location" ]
}
}

    mutate { convert => {"Latitude" => "float"} }
    mutate { convert => {"Longitude" => "float"} }
    mutate { rename => {"Latitude" => "[location][lat]"} }
    mutate { rename => {"Longitude" => "[location][lon]"} }
    
    }

output{

elasticsearch {
Hosts => ["localhost:9200"]
index => "nyc311calls"
document_type => "calls"
user => "elastic"
password => "changeme"}
stdout { codec => rubydebug { metadata => true } }
stdout { codec => dots }
}

can you go through this and let me know where exactly lies the mistake?

Badger · August 4, 2019, 10:26pm

Remove the } before the mutate filters. That closes the filter section and the mutates need to be inside that. Also, add a } at the end of the file.

aaditya_alekar · August 4, 2019, 10:34pm

Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
[2019-08-04T15:33:17,231][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-04T15:33:18,434][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, => at line 32, column 12 (byte 971) after output {

elasticsearch {
Hosts => ["localhost:9200"]
index => "Boston_crimes"
document_type => "records"
user => "elastic"
password => "changeme"
stdout
[2019-08-04T15:33:18,451][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

i got this now?

aaditya_alekar · August 4, 2019, 10:35pm

input{
file {
path => " /Users/aadi2391/Desktop/Documents/CIS5900/Project/crime 2.csv "
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
csv {
seperator => ";"
columns => [ "OFFENSE_CODE" , "OFFENSE_CODE_GROUP" , "OFFENSE_DESCRIPTION" , "DISTRICT" ,
"REPORTING_AREA" , "SHOOTING" , "OCCURRED_ON_DATE" , "YEAR" , "MONTH" , "DAY_OF_WEEK" ,
"HOUR" , "UCR_PART" , "STREET" , "Lat", "Long" , "Location" ]
}

    mutate { convert => {"Latitude" => "float"} }
    mutate { convert => {"Longitude" => "float"} }
    mutate { rename => {"Latitude" => "[location][lat]"} }
    mutate { rename => {"Longitude" => "[location][lon]"} }
    
    }

output {

elasticsearch {
Hosts => ["localhost:9200"]
index => "Boston_crimes"
document_type => "records"
user => "elastic"
password => "changeme"
stdout { codec => rubydebug { metadata => true } }
stdout { codec => dots }
}

check this

stephenb · August 4, 2019, 10:55pm

Missing the last } at the end of the file as @Badger said

aaditya_alekar · August 4, 2019, 10:58pm

Aadityas-MacBook:documents aadi2391$ logstash -f /Users/aadi2391/Desktop/Documents/logstash-7.2.0/config/logstash-Boston_Crimes.conf -t
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
[2019-08-04T15:56:53,175][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-04T15:56:55,910][INFO ][org.reflections.Reflections] Reflections took 73 ms to scan 1 urls, producing 19 keys and 39 values
[2019-08-04T15:56:56,823][ERROR][logstash.filters.csv ] Unknown setting 'seperator' for csv
[2019-08-04T15:56:56,844][FATAL][logstash.runner ] The given configuration is invalid. Reason: Something is wrong with your configuration.
[2019-08-04T15:56:56,860][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

I added all those things what was needed but still this small error persist. Below is the conf file for ur kind reference:

input{
file {
path => " /Users/aadi2391/Desktop/Documents/CIS5900/Project/crime 2.csv "
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
csv {
seperator => ";"
columns => [ "OFFENSE_CODE" , "OFFENSE_CODE_GROUP" , "OFFENSE_DESCRIPTION" , "DISTRICT" ,
"REPORTING_AREA" , "SHOOTING" , "OCCURRED_ON_DATE" , "YEAR" , "MONTH" , "DAY_OF_WEEK" ,
"HOUR" , "UCR_PART" , "STREET" , "Lat", "Long" , "Location" ]
}

    mutate { convert => {"Latitude" => "float"} }
    mutate { convert => {"Longitude" => "float"} }
    mutate { rename => {"Latitude" => "[location][lat]"} }
    mutate { rename => {"Longitude" => "[location][lon]"} }
    
    }

output {

elasticsearch {
Hosts => ["localhost:9200"]
index => "Boston_crimes"
document_type => "records"

stdout { codec => rubydebug { metadata => true } }

stdout { codec => dots }

}
}

Kindly advice.

stephenb · August 4, 2019, 11:08pm

Hmmm see if you can figure out from the error and this...

https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-separator

This is basically a computer program ... Typos will always fail...

aaditya_alekar · August 5, 2019, 12:50am

sir, I am still getting the error. Kindly please help

stephenb · August 5, 2019, 1:52am

In the error message

[2019-08-04T15:56:56,823][ERROR][logstash.filters.csv ] Unknown setting 'seperator' for csv

Typo ...seperator vs separator

I also notice

path => " /Users/aadi2391/Desktop/Documents/CIS5900/Project/crime 2.csv

is there really supposed to be a space at the beginning before /User and in the file name crime 2.csv?

Perhaps pay close attention to the typos you have had many...

Good luck on the debugging ... You will figure it out!

Apologies when I say typos... they are among other items...

Incorrect Spelling
Incorrect capitalization or case (Upper or Lower)
Missing or added spaces or _ etc
Missing or additional {
Any other in correct entry.
The conf must be typed in exactly.

aaditya_alekar · August 5, 2019, 2:53am

Aadityas-MacBook:documents aadi2391$ logstash -f /Users/aadi2391/Desktop/Documents/logstash-7.2.0/config/logstash-Boston_Crimes.conf -t
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
[2019-08-04T19:39:24,275][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-04T19:39:28,112][INFO ][org.reflections.Reflections] Reflections took 215 ms to scan 1 urls, producing 19 keys and 39 values
[2019-08-04T19:39:29,077][ERROR][logstash.outputs.elasticsearch] Unknown setting 'Hosts' for elasticsearch
[2019-08-04T19:39:29,094][FATAL][logstash.runner ] The given configuration is invalid. Reason: Something is wrong with your configuration.
[2019-08-04T19:39:29,112][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

i am not able to understand besides typos

Christian_Dahlqvist · August 5, 2019, 3:04am

It seems you have used Hosts instead of hosts in your Elasticsearch output. Spelling and case matters.

aaditya_alekar · August 5, 2019, 3:55am

Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

how can u configure multiple instance?

psanggabuana · August 5, 2019, 10:05am

How run this command (PUT) ? Because Im newbie.. Thank you for your advise

aaditya_alekar · August 5, 2019, 5:13pm

Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

how can u configure multiple instance?

Badger · August 5, 2019, 5:20pm

Add --path.data to the command line used to start each instance and point the two instances at two different directories.

Alternatively, copy the contents of /etc/logstash to another directory, edit path.data in logstash.yml and then use --path.settings on the command line to point one instance at it.

aaditya_alekar · August 5, 2019, 5:49pm

Mr Badger,

Can you provide any example? I want to understand the syntax

Badger · August 5, 2019, 6:06pm

/usr/share/logstash/bin/logstash --path.data /tmp/someDir --path.settings /etc/logstash

is how I start logstash.

aaditya_alekar · August 5, 2019, 7:59pm

{
$schema: https://vega.github.io/schema/vega/v3.0.json
config: {
kibana: {
type: "map"
latitude: 37.0902
longitude: 95.7129
zoom: 4
}
}
data: [
{
"name": "cities",
"values": [
{"city": "Boston", "lon": 42.308126191, "lat": -71.07692974, "pop": 4823991},
{"city": "Boston", "lon": 42.34432328, "lat": -71.15778368, "pop": 2270800},
{"city": "Boston", "lon": 42.37525782, "lat": -71.02466343, "pop": 4485211},
{"city": "Boston", "lon": 42.32324363, "lat": -71.10892316, "pop": 397397},
{"city": "Adelaide", "lon": 42.28076737, "lat": -71.04736497, "pop": 1295714}
]
transform: [
{
type: geopoint
projection: projection
fields: ["lon", "lat"]
}
]
}
]
marks: [
{
type: symbol
from: {data: "cities"}
encode: {
enter: {
size: {value: 100}
fill: {value: "black"}
stroke: {value: "orange"}
}
update: {
x: {field: "x"}
y: {field: "y"}
}
}
}
]
}

I want to plot the map of US with Boston as the main city.

Topic		Replies	Views
LogStash, GeoJSON and Kibana Logstash	5	701	July 30, 2021
Generating geo_point based on a couple float Logstash	1	319	December 29, 2019
Latitude and Longitude Field conversion to Geo Point Kibana	3	4665	February 22, 2017
Convert lat and long to geohash Kibana	6	3255	October 5, 2019
Converting Location field into lat and long Kibana	6	768	November 19, 2018

Location Mapping

stdout { codec => rubydebug { metadata => true } }

stdout { codec => dots }

Related topics