Struggling here, grateful for any help

Elastic Stack Elasticsearch
1

Hi there,

So I'm probably confusing myself here but I am trying to import logs from our Cloudflare instance into Elasticsearch, via a file that has many json entries and looks like the following (this is just a single log)

{ "brandId":100,"flags":2,"hosterId":0,"ownerId":XXXXXX,"rayId":"XXXXXXXXX","securityLevel":"med","timestamp":1504731172284000000,"unstablePublic":null,"zoneId":XXXXXXX,"zoneName":"test.com","zonePlan":"enterprise","client":{"asNum":4648,"country":"XX","deviceType":"desktop","ip":"XXXXXXXX","ipClass":"noRecord","srcPort":63970,"sslCipher":"NONE","sslFlags":0,"sslProtocol":"none"},"clientRequest":{"accept":"text/html, application/xhtml+xml, /","body":null,"bodyBytes":0,"bytes":246,"cookies":null,"flags":1,"headers":,"httpHost":"www.test.com","httpMethod":"GET","httpProtocol":"HTTP/1.1","referer":"","uri":"/","userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"},"edge":{"bbResult":"0","cacheResponseTime":0,"colo":26,"enabledFlags":12,"endTimestamp":1504731172288000000,"flServerIp":"XXXXXXXX","flServerName":"26f27","flServerPort":80,"pathingOp":"wl","pathingSrc":"macro","pathingStatus":"nr","startTimestamp":1504731172284000000,"usedFlags":0,"rateLimit":{"ruleId":0,"mitigationId":null,"sourceId":"","processedRules":null},"dnsResponse":{"rcode":0,"error":"ok","cached":false,"duration":0,"errorMsg":"","overrideError":false}},"edgeResponse":{"bodyBytes":5,"bytes":285,"compressionRatio":0,"contentType":"","headers":null,"setCookies":null,"status":301}}

Originally I set up mappings as follows

curl -XPUT http://localhost:9200/cloudflare -d '
{
"mappings" : {
"default" : {
"properties" : {
"securityLevel" : {"type": "string" },
"zoneName" : {"type": "string" },
"cacheExternalIP" : {"type": "string" },
"cacheInternalIP" : {"type": "string" },
"cacheExternalPort" : { "type" : "integer" },
"ruleType" : { "type" : "integer" }
}
}
}
}
';

When I go to import the logs using curl -XPUT localhost:9200/cloudflare/_bulk --data-binary @cloudflare.json I get the following error message.

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_NUMBER]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_NUMBER]"},"status":400}

I did find this threat (_https://discuss.elastic.co/t/bulk-insert-file-having-many-json-entries-into-elasticsearch/46470_) and I understand the format is wrong when I am trying to insert using the Bulk format but I am not having much luck changing it.

Would anyone please be able to help me?

Thank you

2

This is what the format for a bulk post needs to look like - https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docs-bulk.html

3

Hey!

So basically take one entry from the cloudflare.json file, and try and insert it into bulk format. That way, you can ensure that your data is correct and bulk format is working. Then try working a small subset of the data into bulk format and inserting! If that works as well, then do the full set. If that doesn't work, you know you have hit an issue with bulk inserting from the ES end and not your data or formatting :slight_smile:

Try that and let us know :slight_smile:

Best of luck

4

Thanks, so this what I've done so far.

action_and_meta_data
{"brandId":100,"flags":2,"hosterId":0,"ownerId":4855861,"rayId":"XXXXXXXXXX","securityLevel":"med","timestamp":XXXXXXXXX,"unstablePublic":null,"zoneId":XXXXXX,"zoneName":"test.com","zonePlan":"enterprise","cache":{"bckType":"c3","cacheExternalIp":"","cacheExternalPort":0,"cacheFileKey":"XXXXXXXXX","cacheInternalIp":"XXXXXXX","cacheServerName":"12c161","cacheStatus":"hit","cacheTokens":0,"endTimestamp":1504669892158999808,"startTimestamp":1504669892158999808},"cacheResponse":{"bodyBytes":0,"bytes":2505,"contentType":"text/javascript","retriedStatus":0,"status":200},"client":{"asNum":15003,"country":"XX","deviceType":"desktop","ip":"104.238.45.55","ipClass":"noRecord","srcPort":50406,"sslCipher":"XXXXXXXXXXXX","sslFlags":1,"sslProtocol":"TLSv1.2"},"clientRequest":{"accept":"/","body":null,"bodyBytes":0,"bytes":1063,"cookies":null,"flags":1,"headers":,"httpHost":"www.test.com","httpMethod":"GET","httpProtocol":"HTTP/1.1","referer":"https://www.ehs.com/","uri":"/wp-content/themes/velocity/lib/js/resizer.js?ver=1.0.0","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"},"edge":{"bbResult":"0","cacheResponseTime":5000000,"colo":12,"enabledFlags":12,"endTimestamp":1504669892160000000,"flServerIp":"104.20.7.52","flServerName":"12f190","flServerPort":443,"pathingOp":"wl","pathingSrc":"macro","pathingStatus":"nr","startTimestamp":1504669892155000064,"usedFlags":0,"rateLimit":{"ruleId":0,"mitigationId":null,"sourceId":"","processedRules":null},"dnsResponse":{"rcode":0,"error":"ok","cached":true,"duration":0,"errorMsg":"","overrideError":false}},"edgeRequest":{"bodyBytes":0,"bytes":1833,"headers":null,"httpHost":"www.test.com","httpMethod":"GET","keepaliveStatus":"reuseAccepted","uri":"/wp-content/themes/velocity/lib/js/resizer.js?ver=1.0.0"},"edgeResponse":{"bodyBytes":599,"bytes":1017,"compressionRatio":2.47,"contentType":"text/javascript","headers":null,"setCookies":null,"status":200}}

Which then gives me this error.

{"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"parse_exception","reason":"Failed to derive xcontent"},"status":400}

Any thoughts? Thanks in advance!

5

That's still not a valid bulk request.
You need to look lower in that doc page for the actual structure, what you have there is simple logical representation of the structure and not valid json.

6

Thanks! I've reached out to Cloudflare for their assistance.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.