Signal connecting to hub but can't invoke methods - signalr

Question

The problem
We are experiencing some weird problems with our signalR service. The signalR service sometimes works, and sometimes it doesn't. The service connects without issues, but when we try to invoke some of the methods i get the following error:
An unexpected error occurred invoking XXXX on the server.
We have tried enabling detailed logging, but the error message was the same. In the snippet below you can see the error we are getting, and you can furthermore see that the client keeps pinging the server without problems after the error.
SignalR ws image
What we have tried
We have 3 different environments set up with signalR. One of the environments is working but we are experiencing the above mentioned error in our 2 other environments. We tried checking the code on the working environment down on one of those that didn't work, and that didn't fix our signalR problem. What we tried so far:
Restarting the signalR service.
Renaming the hubs.
Cleaning and rebuilding the solution.
The signalR logs doesn't tell us much too. This is the message we are getting out of the logs on Azure:
"message":"Connection aborted. Reason: Connection ping timeout.","type":"ConnectivityLogs","collection":"Connection"
This is the method that fails:
public async Task AddToItemGroupTest(Guid itemId)
{
var securityLogger = await CreateSecurityLoggerForConnectionChanges(itemId);
await Groups.AddToGroupAsync(Context.ConnectionId, GetItemGroupIdentifier(itemId, _applicationSettings.EnvironmentShortName));
securityLogger.LogDataCreate($"connection to itemhub on item with id: {itemId}");
}
Client side:
NotificationService.getItemConnection().invoke("AddToItemGroup" + Constants.environmentShortName, itemId)
.then(() => {
this.addGroupToActiveGroups("item", "AddToItemGroup" + Constants.environmentShortName, itemId);
connectionChangedCallback(connectionStatus.connected);
callback();
})
.catch((error) => {
connectionChangedCallback(connectionStatus.disconnected);
console.log(`Failed adding client to group. Error: ${error}`);
throw Error(`Failed adding client to group. Error: ${error}`);
});
getItemConnection looks like this:
private static getItemConnection() {
if (!this.itemConnection) {
this.itemConnection = new SignalR.HubConnectionBuilder()
.withUrl(Constants.itemHubUrl, {
accessTokenFactory: () => {
return getUserAccesTokenOrForceLogin(this.authService);
}})
.withAutomaticReconnect()
.build();
}
return this.itemConnection;
}