Nested dictionaries - lists of dictionaries in each other - dictionary

Question

I originally had a jsonL file that I imported line by line and am now trying to get into the Pandas dataframe. One entry looks something like the following:
*[{u'country': u'denmark',
u'gender': u'F',
u'item_type': u'user',
u'location': u'Denmark',
u'name': u'Mona',
u'profile_text': u'',
u'reviews': [{u'company_id': u'stylepit.dk',
u'date': u'2013-10-06T18:54:49.000+00:00',
u'rating': u'4',
u'text': [u'Altid glad for at handle hos Smartkids - stort sortiment af mange m\xe6rker nemt og hurtigt'],
u'title': u'Som altid kommer varerne hurtigt - super fint'},
{u'company_id': u'www.coolshop.dk',
u'date': u'2012-10-28T19:00:56.000+00:00',
u'rating': u'5',
u'text': [u'F\xf8rste gang jeg har handlet hos Coolshop, det var super nemt og hurtigt og de har et fint udvalg, hjemmesiden har flotte fotos af varen s\xe5 jeg var ikke i tvivl om hvad jeg bestilte. Jeg k\xf8ber gerne igen hos Coolshop.',
u'med venlig hilsen',
u'Mona Pedersen Ulstrup'],
u'title': u'Super hurtig ekspedering'},
{u'company_id': u'www.yourkids.dk',
u'date': u'2010-09-26T19:47:51.000+00:00',
u'rating': u'5',
u'text': [u'Har k\xf8bt rigtig mange cars via yourkids, nok den eneste side i Danmark hvor du bare kan f\xe5 alle de sidste nye biler.',
u'Lige sagen for en cars samler. Der er altid rigtig meget service, min bedste anbefaling'],
u'title': u'Super super service'}]*
So, I have a bunch of users that may have one or more reviews as a list of dictionaries. I need to match the list of reviews with each user in pandas. I just did the following:
dataframe = pd.DataFrame(reviews)
sub_data = []
for i in dataframe['reviews']:
for j in i:
sub_data.append(j)
subdata_frame = pd.DataFrame(sub_data)
subdata_frame
Which gives me two individual dataframes which, when concatenated, do not, obviously match the user with a review, but rather does add the new columns to the mutual dataframe, and just adds the extra reviews at the bottom with no user information.
How do I associate the reviews data with my users?

Answer 1

I answered this before over here Creating pandas dataframe from list of dictionaries containing lists of data. However, I've seen similiar questions like this and the past and decided to post a more generalized solution
Typically for nested data types like this, I merge the inner data with the outer. In your case your inner data is reviews, which is something by itself can be represented nicely with a DataFrame.
def fixInnerData(df,innerDataCol,cols = False):
if cols: f = lambda x:pd.DataFrame(x.iloc[0],columns = cols)
else: f = lambda x:pd.DataFrame(x.iloc[0])
fix = df.groupby(level=0)[innerDataCol].apply(f)
fix = fix.reset_index(level=1,drop=True)
df = pd.merge(fix,df.drop([innerDataCol],1),how='inner',left_index=True,right_index=True)
return df
df = pd.DataFrame(data)
innerDataCol = 'reviews'
fixInnerData(df,innerDataCol)
This method is great because you can use it with a loop and pull out all levels of a dictionary