Btukfyl

I'm basically trying to index data from a dataframe coming from a csv file.

I created an indices successfully.

es.indices.create(index='hash_test', ignore=400)

And added a baseline indexes with the columns and sample data contained in my dataframe

       es.index(index="hash_test", doc_type="hash-test", id=rand_id, body={

         'FILENAME': '6.js', 

         'HASH': 'b4d44ed618112e41cb7e8f33bb19a414', 

         'DATE': '2018-11-15'})

Which ran fine.

Below is how I want to parse my dataframe into the proper format and iterate through the rows and index the data into Elasticsearch similar to the above.

def index_data(data_path, chunksize, index_name, doc_type):

    f = open(data_path)

    csvfile = pd.read_csv(f, iterator=True, chunksize=chunksize,sep="£",encoding="utf-8-sig",index_col=0,engine="python") 

    dictionary = {'Â':''}

    es = Elasticsearch('http://*.*.*.*:9200/')



    for i,df in enumerate(csvfile):

        rand_id = uuid.uuid4();

        df.replace(dictionary, regex=True, inplace=True)

        df.columns = df.columns.str.replace('Â', '')

        records=df.where(pd.notnull(df),None).T.to_dict()

        list_records=[records[it] for it in records]

        json_data = str(''.join(str(v) for v in list_records))

        try:

            es.index(index_name, doc_type, rand_id, json_data)

        except:

            print("error!")

            pass

I had to do some parsing of the dataframe as a weird character was in every row and column (Â).

When I print the values I want to index

print(index_name, doc_type, rand_id, json_data)

I get exactly what I want

hash_test hash-test 51eacee2-e2b1-4886-82f5-1373ec59c640 {'FILENAME': '6.js', 'HASH': 'b4d44ed618112e41cb7e8f33bb19a414', 'DATE': '2018-11-15'}

However I get the following error when I run it;

RequestError: RequestError(400, 'mapper_parsing_exception', 'failed to parse')

Which is attempting to PUT the following data:

{"_index":"hash_test","_type":"hash-test","_id":"{'FILENAME': '8.js', 'HASH': 'b4d44ed618112e41cb7e8f33bb19a414', 'DATE': '2018-11-15'}","found":false}

It completely ignores the rand_id parameter completely, and when I do the following:

es.index(index_name, doc_type, json_data, rand_id)

It ignores the json_data paramter.....

{"_index":"hash_test","_type":"hash-test","_id":"93eadd1b-6859-474b-9750-b618b800b4d5","found":false}

I don't understand the differences in the output I'm getting, and I'm stumped as to how the body is ending up in the _id field when I specified the id parameter.

Cheers in advance for any help.

So, not surprisingly, i was over complicating what I needed to do by creating a clean JSON string from my dataframe. Instead of using a dictionary and then using a list (which I imagine is the source of my errors), I learned it's much easier to just use the to_json function in pandas.

The below code clears this up and indexes my dataframe into my Elasticsearch instance.

def index_data(data_path, chunksize, index_name, doc_type):

    f = open(data_path)

    csvfile = pd.read_csv(f, iterator=True, chunksize=chunksize,sep="£",encoding="utf-8-sig",index_col=0,engine="python") 

    es = Elasticsearch('http://*.*.*.*:9200/')



    for i,df in enumerate(csvfile):

        rand_id = uuid.uuid4(); #create a random id

        data=df.to_json(orient='records', lines=True)

        try:

            es.index(index=index_name,doc_type=doc_type,id=rand_id,body=data)

        except TransportError as e:

            print(e.info)