Unable to get graph api user groups using access token using MSALjs - asp.net

Question

I am using msaljs in angular with .net. I am able to get a v2 oauth token but in .net, the GetMyMemberOfGroupsAsyncmethod returns a 401 with invalid compact token error. I tried changing the scopes in the application but to no use.
Here's the angular config file
{
"msalConfig": {
"clientId": "CLIENT_ID",
"authority": "https://login.microsoftonline.com/TENANT_ID/",
"validateAuthority": true,
"redirectUri": "http://localhost:3000/",
"postLogoutRedirectUri": "http://localhost:3000/"
},
"angular": {
"popUp": false,
"consentScopes": [
"user.read.all",
"directory.Read.all"
]
}
}
Below is the token parsed via JWT.io
{
"aud": "AUD",
"iss": "https://login.microsoftonline.com/TENANT_ID/v2.0",
"iat": 1592242137,
"nbf": 1592242137,
"exp": 1592246037,
"aio": "",
"email": "",
"family_name": "",
"given_name": "",
"name": "",
"nonce": "",
"oid": "",
"preferred_username": "USERNAME",
"sub": "",
"tid": "",
"uti": "",
"ver": "2.0"
}
The exception occurs in the last line of the below code segment:
public async Task<List<Group>> GetMyMemberOfGroupsAsync(string accessToken)
{
List<Group> groups = new List<Group>();
PrepareAuthenticatedClient(accessToken);
// Get groups the current user is a direct member of.
IUserMemberOfCollectionWithReferencesPage memberOfGroups = await graphServiceClient.Me.MemberOf.Request().GetAsync();
I get the following exception on my .net backend
fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
An unhandled exception has occurred while executing the request.
Status Code: Unauthorized
Microsoft.Graph.ServiceException: Code: InvalidAuthenticationToken
Message: CompactToken parsing failed with error code: 80049217
Inner error:
AdditionalData:
date: 2020-06-16T12:44:50
request-id: ce0a83bf-3e33-4510-b4f6-9840dfb85a8a
ClientRequestId: ce0a83bf-3e33-4510-b4f6-9840dfb85a8a
The endpoints use v2.0. Also, i tried adding protected resource maps like the following in the config file but the token seems to be unaffected:
protectedResourceMap: [
['https://graph.microsoft.com/v1.0/me', ['user.read']]
];