Help with Logstash Conf file

Elastic Stack Logstash
1

HI I need assistance in getting logstash config for reading json format file into logstash. I need to be able to extract
1460157360000 as timestamp and 0.6706035137176514 as metric value within the message. At this stage Im getting message: \r

Below is the sample of the file:

{"result":{"dataPoints":{"HOST-B95753DFBAAD5CFC":[[1460157240000,0.6706035137176514],[1460157300000,0.47489500045776367],[1460157360000,0.6633559465408325],[1460157420000,0.664337674776713],[1460157480000,0.48868878682454425],[1460157540000,0.5531626145044962],[1460157600000,0.5602931976318359],[1460157660000,0.46219921112060547],[1460157720000,0.6900641918182373],[1460157780000,0.7065688769022623],[1460157840000,0.48181156317392987],[1460157900000,0.6183704535166422],[1460157960000,0.7815582752227783],[1460158020000,0.462338924407959],[1460158080000,0.6249697605768839],[1460158140000,0.5535382429758707],[1460158200000,0.32572468121846515],[1460158260000,0.5986524820327759],[1460158320000,0.6185022989908854],[1460158380000,0.6317550341288248],[1460158440000,0.5991113185882568],[1460158500000,0.6576340595881144],[1460158560000,0.6448680957158407],[1460158620000,0.43584291140238446],[1460158680000,0.6118787129720052],[1460158740000,0.46228031317392987],[1460158800000,0.33152655760447186],[1460158860000,0.7297681172688802],[1460158920000,0.488211194674174],[1460158980000,0.5012081066767374],[1460159040000,0.6772387822469076],[1460159100000,0.4495040575663249],[1460159160000,0.56641685962677],[1460159220000,0.7032074928283691],[1460159280000,0.6242591540018717],[1460159340000,0.46868149439493817],[1460159400000,0.3904886643091838],[1460159460000,0.6906340916951498],[1460159520000,0.4947916666666667],[1460159580000,0.5795085430145264],[1460159640000,0.605602502822876],[1460159700000,0.42277057965596515],[1460159760000,0.7291666666666666],[1460159820000,0.6253341833750407],[1460159880000,0.5013046264648438],[1460159940000,0.6379194259643555],[1460160060000,0.37136884530385333],[1460160120000,0.6573647260665894],]]},"timeseriesId":"com.r.builtin:host.cpu.user","unit":"%","entities":{"HOST-E8631391336AA0AC":"AX.com","},"resolutionInMillisUTC":3610000,"aggregationType":"AVG"}}

2

What have you got so far in the way of config?

3

Hi Mark,
I have now setup a http poller successfully and I can see its sending data to elastic search but I'm unable to see any output in Kibana. The index ruxit2 is there in elastic search, is there any way I can troubleshoot.

Here is my current logstash conf:

input {
http_poller {

urls => { poller1 => "https://bxz58617.live.ruxit.com/api/v1/timeseries/?relativeTime=hour&Api-Token=ph3HD6pcSjqeOrGu&entity=HOST-AXAOSN-2.corp.axieo.com&aggregationType=AVG&timeseriesId=com.ruxit.builtin:host.cpu.user" }

request_timeout => 120
interval => 60
socket_timeout => 60
codec => "json"
metadata_target => "http_poller_metadata"
 }

}
#filter {

#if http_poller_metadata[poller1][01] {

mutate {

  #add_field => {
    #"timestamp" => "%{@timestamp}"
  #}
#}

#}
output {

#csv {
#type => 'csv'
#fields => ["resolutionInMillisUTC", "aggregationType"]
#path => "D:\ruxit\ruxit-%{@timestamp}.csv"

#}

elasticsearch {
hosts => "localhost:9200"
index => "ruxit2"
}
stdout {codec => json }

}

4

If there data is there in ES then you should be able to see it in KB.
Are you sure the timeframe is correct?

Also it appears that you are indexing time based data, you should use time based indices as well.

5

Hi Mark, Thanks for your response, I just observed that its seems not be indexed, as you can see below ruxit 2 the value 0. How do I troubleshoot this.

yellow open 4csv-4 5 1 609 0 573.2kb 573.2kb
yellow open ruxit 5 1 25 0 78.1kb 78.1kb
yellow open ruxit2 5 1 0 0 795b 795b
yellow open .kibana 1 1 4 0 17.2kb 17.2kb

6

Hi
I'm new to elastic search, how I go about creating an index for this.

7

Hi Mark,
May be you meant
index => -ruxit2-%{@timestamp}
Is that what I should use

8

ES debug log- Does this provide any hint

[2016-04-09 07:24:28,152][DEBUG][action.bulk ] [Grand Director] [ruxit2][3] failed to execute bulk item (index) index {[ruxit2][logs][AVP56Fx4OuDrhkiUfPSY], source[{"result":{"dataPoints":{"HOST-":[[1460183040000,0.5599784056345621],[1460183100000,0.5986287196477255],[1460183160000,0.5984710852305094],[1460183220000,0.5337768395741781],[1460183280000,0.7751784324645996],[1460183340000,0.5338440338770548],[1460183400000,0.48186250527699787],[1460183460000,0.7488510608673096],[1460183520000,0.4688095649083455],[1460183580000,0.4231974283854167],[1460183640000,0.6317665576934814],[1460183700000,0.5991417169570923],[1460183760000,0.5078125],[1460183820000,0.6744218667348226],[1460183880000,0.6445516347885132],[1460183940000,0.4361625909805298],[1460184000000,0.6579706271489462],[1460184060000,0.5799161195755005],[1460184120000,0.3384987513224284],[1460184180000,0.5396430492401123],[1460184240000,0.6968499819437662],[1460184300000,0.5535586277643839],[1460184360000,0.6117763121922811],[1460184420000,1.0353707472483318],[1460184480000,0.8013400236765543],[1460184540000,1.1201181411743164],[1460184600000,0.9637682437896729],[1460184660000,0.622464140256246],[1460184720000,0.48838313420613605],[1460184780000,0.4948383569717407],[1460184840000,0.5599367618560791],[1460184900000,0.5340606371561686],[1460184960000,0.6770838101704916],[1460185980000,0.34506479899088544],[1460186040000,0.5343042612075806],[1460186100000,0.6571340958277384],[1460186160000,0.5142620801925659],[1460186220000,0.6838487784067789],[1460186280000,0.5925066471099854],[1460186340000,0.5278586149215698],[1460186400000,0.6311729351679484],[1460186460000,0.6643789211908976],[1460186520000,0.47532153129577637],[1460186580000,null],[1460186640000,null]],"HOST-E8631391336AA0AC":[[1460183040000,0.2994597951571147],[1460183100000,1.3204102516174316],[1460183160000,0.22148660818735758],[1460183220000,1.015655755996704],[1460183280000,0.28002981344858807],[1460183340000,0.2344361941019694],[1460183400000,0.31911255915959674],[1460183460000,1.796875],[1460183520000,0.1628215511639913],[1460183580000,0.3186192711194356],],[1460184480000,1.7975783348083496],[1460184540000,4.218913396199544],[1460184600000,4.146780649820964],[1460184660000,1.9272055625915527],[1460184720000,1.7857200304667156],[1460184780000,0.20184367895126343],[1460184840000,0.1888225277264913],[1460184900000,0.20183336734771729],[1460184960000,0.23453799883524576],[1460185020000,10.066106796264648],[1460185080000,1.6753816604614258],[1460185140000,2.604492664337158],[1460185200000,1.5754502614339192],[1460185260000,1.0161751906077068],[1460185320000,4.635995229085286],[1460185380000,0.5142248074213663],[1460185440000,1.8144783973693848],[1460185500000,0.17577137549718222],[1460185560000,0.2541201909383138],[1460185620000,1.2503669261932373],[1460185680000,0.3123581608136495],[1460185740000,0.24103877941767374],[1460185800000,0.2212634285291036],[1460185860000,0.18868221839269003],[1460185920000,0.7090568542480469],[1460185980000,0.4039620558420817],[1460186040000,1.5799395243326824],[1460186100000,0.6125293572743734],[1460186160000,0.3516236146291097],[1460186220000,0.2213950753211975],[1460186280000,0.9766032695770264],[1460186340000,0.24704360961914062],[1460186400000,2.365400791168213],[1460186460000,0.1888835827509562],[1460186520000,0.16924055417378744],[1460186580000,0.27346805731455487],[1460186640000,null]]},"timeseriesId":"com.ruxit.builtin:host.cpu.user","unit":"%","entities":{"HOST-E8631391336AA0AC":"AXCLIN-1.corp.axieo.com","HOST-B95753DFBAAD5CFC":"AXAOSN-2.corp.axieo.com"},"resolutionInMillisUTC":3610000,"aggregationType":"AVG"},"@version":"1","@timestamp":"2016-04-09T07:24:26.134Z","http_poller_metadata":{"name":"poller1","host":"AXDTRACEP-1","request":{"method":"get","url":"https://bxz58617.live.ruxit.com/api/v1/timeseries/?relativeTime=hour&Api-Token=_&entity=HOST-AXAOSN-2.corp.axieo.com&aggregationType=AVG&timeseriesId=com.ruxit.builtin:host.cpu.user"},"runtime_seconds":1.484,"code":200,"response_headers":{"date":"Sat, 09 Apr 2016 07:24:25 GMT","content-type":"application/json;charset=utf-8","server":"ruxit server","x-ratelimit-limit":"10","x-ratelimit-remaining":"9","x-ratelimit-reset":"1460186725540000","content-length":"4508","strict-transport-security":"max-age=31536000;includeSubDomains","set-cookie":"SRV=server6; path=/; HttpOnly; Secure","cache-control":"private"},"response_message":"OK","times_retried":0}}]}
MapperParsingException[failed to parse]; nested: IllegalArgumentException[mapper [result.dataPoints.HOST-B95753DFBAAD5CFC] of different type, current_type [long], merged_type [double]];
at org.elasticsearch.index.mapper.DocumentParser.innerParse

9

You may want to edit your posts, you have included your API token in there.

1 Like
10

HI Mark,
Thanks for noticing but i edited it before uploading. I will go and take it out . Can you help here, I'm struggling ES to index the output or if I try to save it locally as CSV I get permission denied error.