Issue indexing data from Python


(Matt Altman) #1

I am trying to index data but I am receiving two different error messages depending on how I try and index.

Here is my mapping

request_body = {
    "settings": {
        "number_of_shards": 5,
        "number_of_relicas": 0
    },
    "mappings": {
        "logs": {
            "properties": {
                "date": { "type": "date", "format": "YYYY-MM-DD" },
                "time": { "type": "string" },
                "client_ip": { "type": "ip" },
                "method": { "type": "string" },
                "request": { "type": "string" },
                "status_code": { "type": "integer" },
                "bytes": { "type": "integer" },
                "time_taken": { "type": "integer" },
                "referrer": { "type": "string" },
                "user_agent": { "type": "string" },
                "cookie": { "type": "string" }
            }
        }
    }
}

When I use this line

es.bulk(index = index_name, doc_type = 'logs', body = result.group(x))

I get "parse exception: failed to derive xcontent"

And when I use this line

es.index(index = index_name, doc_type = 'logs', body = result.group(x))

I get "parse exception: caused by compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"

Here is my regex pattern

pattern = '(\d{4}[-]\d{2}[-]\d{2})\s+(\d{2}[:]\d{2}[:]\d{2})\s+(\d{1,3}[.]\d{1,3}[.]\d{1,3}[.]\d{1,3})\s+([GET|PUT|POST]+)\s+([^\s]+)\s+(\d+)\s+(\d+)\s+(\d+)\s+([^\s]+)\s+(\".*?\")\s+(\".*?\")'

And finally, here is a line that I am trying to index

2016-06-14 18:20:07 1.1.1.1 GET /origin.com/products/PublishingImages/file.png 200 10915 1 "http://www.referrer.com/CategoryPage.aspx?" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0)" "ASP.NET_SessionId=yjumie0ar12dl; __SessionCookie=7FKbuqkC6pmz2ZKewLVMAjOtKTbLpb8zK8j0TCr+U; __CSALastPageName=XCOaCvdhusg==; __CSALastHierarchy=L7ATLEuRZmXj3ddzXRPF7BVVZFzs=; __CSALastReportSuiteName=0eWICfQzEtcnfMSQ==; __CSARedirectTags=ABOcOdYBWK/O5Rw==; pck=0aa46f231-abc9-cc8d12f4b2d9; __CSCookie=RVqW7dLcpb4QVYDUPoADt5vqtK0oo9QKniVXlY0O83Cet4n73Fah5UvdTzbfayAEgnHsTch/OrCOSTZaK4peWy/1kXq3fX2uHLPbrw4BgXhaZvjRUp4yRhaL+Ug2qcj6pQny6oyot+GPOBP4OgqtbzTZJv7NxykKZHKZRzjYBvBLtbOf9brrzjzeaTbDw7r7SEmVzxQOihmDBGDG0Rn16jC9T8MtbUl3VKz3GojZeLTmYTCXX67oBmnGZPPtV8joGqYrVcHrSj0XKxyeFBluQONFGvEGhw/7TKoshKQfd5oNo7A9eJ+UfEvG51iwT/3ePMd6MvzFMdj+eMuizFcjdas9v+bWH923AYt7rUnb99T8AX6+8cJy0/fj69AkdhG/FTvNQrV05Lpbr5gjL5uy8N4Ng2zsoKMdgrFYalARQNM/8h0i7HhxQyx+OY6PXR0VbTGI=; ETE1=fvClmAi4/MuJaZFB+0Cac7U1k4DgF5ylbBWLfPA5UuABTPG9Xp+KAVSMf3ES0cJm94ok4Zw/AiaX1j7lf6BQpcoI9UwGAe9PUFnpSnxiAlp9I5fSbCJyho3YDa0MoJ9xC7fwwPigfrd7oxdyi8nvCKs5//Imb3d015qESq0aThMWwhuKwT7C7xyk4Q1mdD0l3kFum4C1g2cbIaNQ5qMJ3M+DJvu7SbjVIAZanGwrt1aZLL/1Eo1MPVEuGmMOj/p6hHtjBc; icxid=1461781512418-308716; gigyaCookieNotification=acknowledged; __PublicSiteRecentSearchesCookie=UeUbHRveUtzmA==; __CSUserIbcIDCookie=dHEe9zrGnQX6OLzOhtqfft1DC571R+670C7m6; __CSUserIbcMonikerCookie=Oiz4bVglhbAzQ==; __qca=P0-455562-1461759; icxid=146512418-3087371016; s_vi=[CS]v1|2B90051D1FEE-4000196D[CE]; s_fid=0C6E0CDA143CF-181DECF8985; ddtLatC=549p52.5p46.5p37p41p55.5p35p105; dtCookie=B66FBB0F22CBB49B863198A2"


(system) #2