I am trying the async examples from the GNOME project site. I get the follwoing warning which I don't under stand on how to fix.
async.vala:8.2-8.17: warning: delegates with scope="async" must be owned
Code
async double do_calc_in_bg(double val) throws ThreadError {
SourceFunc callback = do_calc_in_bg.callback;
double[] output = new double[1];
// Hold reference to closure to keep it from being freed whilst
// thread is active.
// WARNING HERE
ThreadFunc<bool> run = () => {
// Perform a dummy slow calculation.
// (Insert real-life time-consuming algorithm here.)
double result = 0;
for (int a = 0; a<100000000; a++)
result += val * a;
output[0] = result;
Idle.add((owned) callback);
return true;
};
new Thread<bool>("thread-example", run);
yield;
return output[0];
}
void main(string[] args) {
var loop = new MainLoop();
do_calc_in_bg.begin(0.001, (obj, res) => {
try {
double result = do_calc_in_bg.end(res);
stderr.printf(#"Result: $result\n");
} catch (ThreadError e) {
string msg = e.message;
stderr.printf(#"Thread error: $msg\n");
}
loop.quit();
});
loop.run();
}
The warning is pointing at the run variable inside the async function. Who or what needs to be owned? The reference to the closure?
The delegate needs to have a well defined owner all the time. The error message is a bit misleading.
To fix it you have to explicitly transfer the ownership from the delegate to the thread constructor:
new Thread<bool>("thread-example", (owned) run);
Instead of
new Thread<bool>("thread-example", run);
See also: https://wiki.gnome.org/Projects/Vala/Tutorial#Ownership
PS: The generated C code is fine in both cases. (at least with valac 0.46.6)
Related
I'm having trouble writing an async function in Vala.
I understand that the function's callback has to get registered somewhere in order for execution to continue after the yield statements, so I add it to the GLib mainloop to be called periodically using Idle.add. Here is my dummy code:
async void loop_func()
{
var id = Idle.add(loop_func.callback);
message("Added idle %u",id);
yield;
message("yielded successfully 1");
yield;
message("yielded successfully 2");
for (var i = 0; i < 20; i++)
{
message("%d",i);
yield;
}
message("finished");
}
int main()
{
var loop = new GLib.MainLoop();
loop_func.begin(() => {loop.quit();});
loop.run();
return 0;
}
Despite this, the code after the 2nd yield statement never gets executed.
This can be seen from the output:
$ ./async
** Message: 20:07:24.932: async.vala:4: Added idle 1
** Message: 20:07:24.932: async.vala:7: yielded successfully 1
And then it hangs.
What am I missing here?
Although GLib calls an idle function repeatedly until it tells it to stop (by returning false), the async function's .callback seems to quit repeating immediately after being called.
This can be overcome by adding the callback again every time before the yield statement:
async void loop_func()
{
for (var i = 0; i < 20; i++)
{
Idle.add(loop_func.callback);
yield;
}
}
An alternative (and possibly more effective) way to do this would be to ignore the value that the async function's callback returns:
async void loop_func()
{
var id = Idle.add(() => {loop_func.callback(); return Source.CONTINUE;});
for (var i = 0; i < 20; i++)
{
yield;
}
Source.remove(id);
}
Greeting,
I'm currently facing a problem that my coroutine can't start. This is the first time I facing this issues and I can't find a proper solution online. Much appreciated if anyone can point me to the right direction to solve this issue.
Here are the code.
path_reference.GetDownloadUrlAsync().ContinueWith((Task<Uri> task) => {
if (!task.IsFaulted && !task.IsCanceled)
{
Debug.Log("Download URL: " + task.Result);
StartCoroutine(DownloadStuff(task.Result));
}
else
{
Debug.Log(task.Exception.ToString());
}
});
}
IEnumerator DownloadStuff(Uri uri)
{
Debug.Log("Start Download");
using (var www = UnityWebRequestTexture.GetTexture(uri))
{
yield return www.SendWebRequest();
if (www.isNetworkError || www.isHttpError)
{
Debug.Log(www.error);
}
else
{
var texture = DownloadHandlerTexture.GetContent(www);
//Texture2D texture = new Texture2D(1, 1);
//if you need sprite for SpriteRenderer or Image
Sprite sprite = Sprite.Create(texture, new Rect(0.0f, 0.0f, texture.width,
texture.height), new Vector2(0.5f, 0.5f), 100.0f);
Debug.Log("Finished downloading!");
}
Debug.Log(www.downloadProgress);
}
}'
The task returned by Firebase probably finishes execution on a thread other than the main thread, and Unity coroutines can only run on the main thread.
Unity's support of multithreading and async is pretty spotty, including "eating" some errors if the continuations of those errors would execute on another thread other than the main thread.
To fix this, you need to change the function that starts your coroutine:
try {
// Important: ConfigureAwait(true) ensures the code after this will run in the
// same thread as the code before (which is the main thread!)
var url = await path_reference.GetDownloadUrlAsync().ConfigureAwait(true);
StartCoroutine(DownloadStuff(url));
} catch (Exception ex) {
// Tip: when logging errors, use LogException and pass the whole exception,
// that way you will get pretty links to the error line in the whole stack trace.
Debug.LogException(ex);
}
As an aside, I usually have a few extension methods on all my projects to deal with that while staying in async-world instead of coroutine-world (because at least with async I can catch errors and not just "halt and catch fire" like Unity's coroutines)
The main ones are:
public static Task ToTask(this YieldInstruction self, MonoBehaviour owner) {
var source = new TaskCompletionSource<object>();
IEnumerable Routine() {
yield return self;
source.SetResult(null);
}
return source.Task;
}
private static Task SendAsync(this UnityWebRequest self, MonoBehaviour owner) {
var source = new TaskCompletionSource<object>();
await self.SendWebRequest().ToTask(owner);
if (
self.isHttpError
|| self.isNetworkError
) {
source.SetException(new Exception(request.error));
yield break;
}
source.SetResult(null);
}
Which you can use like this, inside a MonoBehaviour:
await new WaitForSeconds(0.2f).ToTask(this);
UnityWebRequest request = /* etc */;
await request.SendAsync(this);
var texture = DownloadHandlerTexture.GetContent(request);
Note that these methods do not require ConfigureAwait, since their SetResult/SetException invocations are ran from Unity-provided coroutine continuations.
I am currently attemping to build a custom Editor tool for Unity that utilizes the Firebase Realtime Database. This tool would allow someone to right-click on a scene asset in the inspector and select to 'lock' or 'unlock' the scene. Within our Firebase Database, this locking is represented by a dictionary, with each scene name as a key and each value being either "locked" or "unlocked". This funcitonality will be expanded later, but for now, I'm just trying to get things set up so that I can actually connect to and use the Firebase Realtime Database.
I had looked at the Firebase Quickstart Unity project for the Realtime Database (the one that functions like a leaderboard) and saw that it worked fine. I could replace the database URL in the project with the URL for my app's database, and when I entered in values, they appeared inside my realtime database.
So, I based the code for my custom editor script on the code from the quickstart. In fact, I copy-pasted most of it. I will post the script itself, and then describe the errors I receive, as well as the lines that give the errors:
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEditor;
using Firebase;
using Firebase.Unity.Editor;
using Firebase.Database;
[CustomEditor(typeof(SceneAsset))]
[ExecuteInEditMode]
public class SceneLockingEditor : Editor
{
static string sceneName;
DependencyStatus dependencyStatus = DependencyStatus.UnavailableOther;
protected virtual void OnEnable()
{
Debug.Log("OnEnable Called");
sceneName = target.name;
FirebaseApp.CheckAndFixDependenciesAsync().ContinueWith(task => {
dependencyStatus = task.Result;
if (dependencyStatus == DependencyStatus.Available)
{
InitializeFirebase();
}
else
{
Debug.LogError(
"Could not resolve all Firebase dependencies: " + dependencyStatus);
}
});
}
// Initialize the Firebase database:
protected virtual void InitializeFirebase()
{
Debug.Log("Initializing Firebase");
FirebaseApp app = FirebaseApp.DefaultInstance;
app.SetEditorDatabaseUrl(CENSORING MY DATABASE SORRY);
if (app.Options.DatabaseUrl != null) app.SetEditorDatabaseUrl(app.Options.DatabaseUrl);
}
static TransactionResult SceneLockTransaction(MutableData mutableData)
{
List<object> sceneLocks = mutableData.Value as List<object>;
if (sceneLocks == null)
{
sceneLocks = new List<object>();
}
if(mutableData.ChildrenCount > 0)
{
//Look at every child in the scene locks directory.
foreach (var child in sceneLocks)
{
Debug.Log("Checking next child.");
if (!(child is Dictionary<string, object>))
continue;
//If we find the scene we're looking for...
Debug.Log("Checking if the scene has the name we want");
foreach(string key in ((Dictionary<string, object>)child).Keys)
{
Debug.Log("Key: " + key);
}
if(((Dictionary<string, object>)child).ContainsKey(sceneName))
{
string childLockStatus = (string)((Dictionary<string, object>)child)["lockStatus"];
//If the scene is already locked, just abort.
if (childLockStatus == "locked")
{
Debug.Log("Scene is already locked. Abort.");
return TransactionResult.Abort();
}
else
{
Debug.Log("Scene existed in the database and was not locked. Locking it.");
// If the scene existed in the database but was not locked, we will lock it.
((Dictionary<string, object>)child)[sceneName] = "locked";
// You must set the Value to indicate data at that location has changed.
mutableData.Value = sceneLocks;
return TransactionResult.Success(mutableData);
}
}
}
}
Debug.Log("Scene did not exist in the database. Adding it as locked.");
// If the scene didn't exist in the database before, we will add it as locked.
Dictionary<string, object> newSceneLock = new Dictionary<string, object>();
newSceneLock[sceneName] = "locked";
sceneLocks.Add(newSceneLock);
// You must set the Value to indicate data at that location has changed.
mutableData.Value = sceneLocks;
return TransactionResult.Success(mutableData);
}
static TransactionResult SceneUnlockTransaction(MutableData mutableData)
{
List<object> sceneLocks = mutableData.Value as List<object>;
if (sceneLocks == null)
{
sceneLocks = new List<object>();
}
if (mutableData.ChildrenCount > 0)
{
//Look at every child in the scene locks directory.
foreach (var child in sceneLocks)
{
Debug.Log("Checking next child.");
if (!(child is Dictionary<string, object>))
continue;
//If we find the scene we're looking for...
Debug.Log("Checking if the scene has the name we want");
foreach (string key in ((Dictionary<string, object>)child).Keys)
{
Debug.Log("Key: " + key);
}
if (((Dictionary<string, object>)child).ContainsKey(sceneName))
{
string childLockStatus = (string)((Dictionary<string, object>)child)["lockStatus"];
//If the scene is already locked, just abort.
if (childLockStatus == "unlocked")
{
Debug.Log("Scene is already unlocked. Abort.");
return TransactionResult.Abort();
}
else
{
Debug.Log("Scene existed in the database and was locked. Unlocking it.");
// If the scene existed in the database but was not locked, we will lock it.
((Dictionary<string, object>)child)[sceneName] = "unlocked";
// You must set the Value to indicate data at that location has changed.
mutableData.Value = sceneLocks;
return TransactionResult.Success(mutableData);
}
}
}
}
Debug.Log("Scene did not exist in the database. Adding it as unlocked.");
// If the scene didn't exist in the database before, we will add it as locked.
Dictionary<string, object> newSceneLock = new Dictionary<string, object>();
newSceneLock[sceneName] = "unlocked";
sceneLocks.Add(newSceneLock);
// You must set the Value to indicate data at that location has changed.
mutableData.Value = sceneLocks;
return TransactionResult.Success(mutableData);
}
static public void AddSceneLock()
{
Debug.Log("Attempting to add scene lock to database.");
DatabaseReference reference = FirebaseDatabase.DefaultInstance.GetReference("SceneLocks");
Debug.Log("Running Transaction...");
// Use a transaction to ensure that we do not encounter issues with
// simultaneous updates that otherwise might create more than MaxScores top scores.
reference.RunTransaction(SceneLockTransaction)
.ContinueWith(task => {
if (task.Exception != null)
{
Debug.Log(task.Exception.ToString());
}
else if (task.IsCompleted)
{
Debug.Log("Transaction complete.");
}
});
}
static public void RemoveSceneLock()
{
Debug.Log("Attempting to add scene lock to database.");
DatabaseReference reference = FirebaseDatabase.DefaultInstance.GetReference("SceneLocks");
Debug.Log("Running Transaction...");
// Use a transaction to ensure that we do not encounter issues with
// simultaneous updates that otherwise might create more than MaxScores top scores.
reference.RunTransaction(SceneUnlockTransaction)
.ContinueWith(task => {
if (task.Exception != null)
{
Debug.Log(task.Exception.ToString());
}
else if (task.IsCompleted)
{
Debug.Log("Transaction complete.");
}
});
}
[MenuItem("CONTEXT/SceneAsset/Lock Scene", false, 0)]
public static void LockScene()
{
Debug.Log("LockScene Called for scene " + sceneName + ".");
AddSceneLock();
}
[MenuItem("CONTEXT/SceneAsset/Unlock Scene", false, 0)]
public static void UnlockScene()
{
Debug.Log("UnlockScene Called for scene " + sceneName + ".");
RemoveSceneLock();
}
}
The errors always come from this line:
FirebaseDatabase.DefaultInstance.GetReference("SceneLocks");
Any line that has to do with "FirebaseDatabase.DefaultInstance" will throw one of the following two errors
Error 1:
InvalidOperationException: SyncContext not initialized.
Firebase.Unity.UnitySynchronizationContext.get_Instance ()
Firebase.Platform.PlatformInformation.get_SynchronizationContext ()
Firebase.FirebaseApp.get_ThreadSynchronizationContext ()
Firebase.Database.DotNet.DotNetPlatform+SynchronizationContextTarget..ctor ()
Firebase.Database.DotNet.DotNetPlatform.NewEventTarget (Firebase.Database.Internal.Core.Context c)
Firebase.Database.Internal.Core.Context.EnsureEventTarget ()
Firebase.Database.Internal.Core.Context.InitServices ()
Firebase.Database.Internal.Core.Context.Freeze ()
Firebase.Database.Internal.Core.RepoManager.CreateLocalRepo (Firebase.Database.Internal.Core.Context ctx, Firebase.Database.Internal.Core.RepoInfo info, Firebase.Database.FirebaseDatabase firebaseDatabase)
Firebase.Database.Internal.Core.RepoManager.CreateRepo (Firebase.Database.Internal.Core.Context ctx, Firebase.Database.Internal.Core.RepoInfo info, Firebase.Database.FirebaseDatabase firebaseDatabase)
Firebase.Database.FirebaseDatabase.EnsureRepo ()
Firebase.Database.FirebaseDatabase.get_RootReference ()
SceneLockingEditor.OnInspectorGUI () (at Assets/Bitloft/SCRIPTS/Editor/SceneLockingEditor.cs:37)
UnityEditor.InspectorWindow.DrawEditor (UnityEditor.Editor[] editors, Int32 editorIndex, Boolean rebuildOptimizedGUIBlock, System.Boolean& showImportedObjectBarNext, UnityEngine.Rect& importedObjectBarRect) (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:1242)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)
Error 2:
Exception: Custom Run loops are not supported!
Firebase.Database.Internal.Core.Context.GetExecutorService ()
Firebase.Database.Internal.Core.Context.GetConnectionContext ()
Firebase.Database.Internal.Core.Context.NewPersistentConnection (Firebase.Database.Internal.Connection.HostInfo info, IDelegate delegate_)
Firebase.Database.Internal.Core.Repo..ctor (Firebase.Database.Internal.Core.RepoInfo repoInfo, Firebase.Database.Internal.Core.Context ctx, Firebase.Database.FirebaseDatabase firebaseDatabase)
Firebase.Database.Internal.Core.RepoManager.CreateLocalRepo (Firebase.Database.Internal.Core.Context ctx, Firebase.Database.Internal.Core.RepoInfo info, Firebase.Database.FirebaseDatabase firebaseDatabase)
Firebase.Database.Internal.Core.RepoManager.CreateRepo (Firebase.Database.Internal.Core.Context ctx, Firebase.Database.Internal.Core.RepoInfo info, Firebase.Database.FirebaseDatabase firebaseDatabase)
Firebase.Database.FirebaseDatabase.EnsureRepo ()
Firebase.Database.FirebaseDatabase.get_RootReference ()
SceneLockingEditor.OnInspectorGUI () (at Assets/Bitloft/SCRIPTS/Editor/SceneLockingEditor.cs:37)
UnityEditor.InspectorWindow.DrawEditor (UnityEditor.Editor[] editors, Int32 editorIndex, Boolean rebuildOptimizedGUIBlock, System.Boolean& showImportedObjectBarNext, UnityEngine.Rect& importedObjectBarRect) (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:1242)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)
It's always one error or the other, and I can't determine what causes one error to appear rather than the other. Both errors stop whatever operation I'm trying to do on the database, which means I can't interact with my database at all.
I've taken a look at the quickstart projects and watched some videos of people setting up Firebase to work with their projects, and I can't seem to determine what I have messed up during the process. I have imported my google-services.json into the unity project. The quickstart projects worked just fine interacting with my database. It's just this particular script that won't work. I cannot find any mention of these two errors anywhere on Google. I even contacted the official Firebase support and they couldn't give me any advice on what the errors mean or what could be causing them.
I considered one problem might be in my initialization function. Instead of doing:
FirebaseApp app = FirebaseApp.DefaultInstance;
I figured that maybe I am supposed to use FirebaseApp.Create() with a custom name passed in, but that resulted in the same errors being thrown on the same line. I am at a loss for how to proceed with this problem. I don't know of anyone else who has had these particular errors, and I've done very much experimentation with different ways to access the database over the past several days. If anybody has an idea of what I am doing wrong here, or what causes these errors (and, how to fix them), I would really appreciate it.
At first you should initialize firebase with new instance of FirebaseApp with unique name. I do it like this:
FirebaseApp firebaseApp = FirebaseApp.Create(
FirebaseApp.DefaultInstance.Options,
"FIREBASE_EDITOR");
The second is setup references (DatabaseReference, StorageReference etc.) with this firebaseApp instance and use it only after FirebaseApp.CheckAndFixDependenciesAsync()
Overall code will look like this:
public static void Initialize(bool isEditor = false)
{
if (isEditor)
{
FirebaseApp firebaseApp = FirebaseApp.Create(
FirebaseApp.DefaultInstance.Options,
"FIREBASE_EDITOR");
firebaseApp.SetEditorDatabaseUrl("https://project.firebaseio.com/");
FirebaseApp.CheckAndFixDependenciesAsync().ContinueWith(task =>
{
if (task.Result == DependencyStatus.Available)
{
database = FirebaseDatabase.GetInstance(firebaseApp).RootReference;
storage = FirebaseStorage.GetInstance(firebaseApp).RootReference;
auth = FirebaseAuth.GetAuth(firebaseApp);
}
else
{
Debug.LogError(
"Could not resolve all Firebase dependencies: " + task.Result);
}
});
}
else
{
FirebaseApp.DefaultInstance.SetEditorDatabaseUrl("https://project.firebaseio.com/");
database = FirebaseDatabase.DefaultInstance.RootReference;
storage = FirebaseStorage.DefaultInstance.RootReference;
auth = FirebaseAuth.DefaultInstance;
}
IsInitialized = true;
}
I had the same errors. I spent several hours solving this and it worked for me
I need pass returnValue to a method as argument passed by reference and adjust original var value when function id done. So using ReferenceArgumentHelper class.
What's wrong in code bellow when returnValue is unintentionally deleted (when it is a node, i.e. string) and valgrind detects it. callMethod("onFunctionExit" calls an Qore script method and I can see there correct returnValue value. I suspect it's deleted when exiting onFunctionExit when ReferenceArgumentHelper is destroyed. rah.getArg() references reference variable, so it should not be deleted in callMethod.
DLLLOCAL ThreadDebugEnum callMethod(const char* name, const ThreadDebugEnum defaultResult, QoreProgram *pgm, int paramCount, AbstractQoreNode** params, ExceptionSink* xsink) {
int rv;
QoreListNode* l = new QoreListNode();
qore_program_to_object_map_t::iterator i = qore_program_to_object_map.find(pgm);
if (i == qore_program_to_object_map.end()) {
return defaultResult;
}
i->second->ref();
l->push(i->second);
for (int i=0; i<paramCount; i++) {
if (params[i])
params[i]->ref();
l->push(params[i]);
}
rv = qo->intEvalMethod(name, l, xsink);
l->deref(xsink);
return (ThreadDebugEnum) rv;
}
DLLLOCAL virtual ThreadDebugEnum onFunctionExit(QoreProgram *pgm, const StatementBlock *blockStatement, QoreValue& returnValue, ExceptionSink* xsink) {
AbstractQoreNode* params[2];
params[0] = getLocation(blockStatement);
ReferenceArgumentHelper rah(returnValue.takeNode(), xsink); // grab node from returnValue and pass to helper
params[1] = rah.getArg(); // caller owns ref
ThreadDebugEnum rv = callMethod("onFunctionExit", DBG_SB_RUN, pgm, 2, params, xsink);
AbstractQoreNode* rc = rah.getOutputValue(); // caller owns ref
returnValue.assign(rc); // takes reference
// returnValue.ref();
}
return rv;
}
When looking deeply I did not get why compiler is happy with code in /lib/ReferenceArgumentHelper.cpp:
struct lvih_intern {
LocalVar lv;
ExceptionSink* xsink;
ReferenceNode* ref;
DLLLOCAL lvih_intern(AbstractQoreNode* val, ExceptionSink* xs) : lv("ref_arg_helper", 0), xsink(xs) {
printd(5, "ReferenceArgumentHelper::ReferenceArgumentHelper() instantiating %p (val: %p type: '%s') \n", &lv, val, val ? val->getTypeName() : "n/a");
lv.instantiate(val); <--------------
VarRefNode* vr = new VarRefNode(strdup("ref_arg_helper"), VT_LOCAL);
vr->ref.id = &lv;
ref = new ReferenceNode(vr, 0, vr, 0);
}
class LocalVar {
....
DLLLOCAL void instantiate(QoreValue nval) const {
What is behind conversion AbstractQoreNode* to QoreValue in method call? I did not find an overloaded operator or so. I'm looking what exactly happens with references.
** EDIT **
To make a long story short, ReferenceArgumentHelper was buggy; it hadn't been used in years and was not up to date. The class has been fixed which should fix your issue I hope.
Thank you for pointing this out, and let me know if you have any further problems with this or the fix to the affected code.
I have code like this:
public bool Set(IEnumerable<WhiteForest.Common.Entities.Projections.RequestProjection> requests)
{
var documentSession = _documentStore.OpenSession();
//{
try
{
foreach (var request in requests)
{
documentSession.Store(request);
}
//requests.AsParallel().ForAll(x => documentSession.Store(x));
documentSession.SaveChanges();
documentSession.Dispose();
return true;
}
catch (Exception e)
{
_log.LogDebug("Exception in RavenRequstRepository - Set. Exception is [{0}]", e.ToString());
return false;
}
//}
}
This code gets called many times. After i get to around 50,000 documents that have passed through it i get an OutOfMemoryException.
Any idea why ? perhaps after a while i need to declare a new DocumentStore ?
thank you
**
UPDATE:
**
I ended up using the Batch/Patch API to perform the update I needed.
You can see the discussion here: https://groups.google.com/d/topic/ravendb/3wRT9c8Y-YE/discussion
Basically since i only needed to update 1 property on my objects, and after considering ayendes comments about re-serializing all the objects back to JSON, i did something like this:
internal void Patch()
{
List<string> docIds = new List<string>() { "596548a7-61ef-4465-95bc-b651079f4888", "cbbca8d5-be45-4e0d-91cf-f4129e13e65e" };
using (var session = _documentStore.OpenSession())
{
session.Advanced.DatabaseCommands.Batch(GenerateCommands(docIds));
}
}
private List<ICommandData> GenerateCommands(List<string> docIds )
{
List<ICommandData> retList = new List<ICommandData>();
foreach (var item in docIds)
{
retList.Add(new PatchCommandData()
{
Key = item,
Patches = new[] { new Raven.Abstractions.Data.PatchRequest () {
Name = "Processed",
Type = Raven.Abstractions.Data.PatchCommandType.Set,
Value = new RavenJValue(true)
}}});
}
return retList;
}
Hope this helps ...
Thanks alot.
I just did this for my current project. I chunked the data into pieces and saved each chunk in a new session. This may work for you, too.
Note, this example shows chunking by 1024 documents at a time, but needing at least 2000 before we decide it's worth chunking. So far, my inserts got the best performance with a chunk size of 4096. I think that's because my documents are relatively small.
internal static void WriteObjectList<T>(List<T> objectList)
{
int numberOfObjectsThatWarrantChunking = 2000; // Don't bother chunking unless we have at least this many objects.
if (objectList.Count < numberOfObjectsThatWarrantChunking)
{
// Just write them all at once.
using (IDocumentSession ravenSession = GetRavenSession())
{
objectList.ForEach(x => ravenSession.Store(x));
ravenSession.SaveChanges();
}
return;
}
int numberOfDocumentsPerSession = 1024; // Chunk size
List<List<T>> objectListInChunks = new List<List<T>>();
for (int i = 0; i < objectList.Count; i += numberOfDocumentsPerSession)
{
objectListInChunks.Add(objectList.Skip(i).Take(numberOfDocumentsPerSession).ToList());
}
Parallel.ForEach(objectListInChunks, listOfObjects =>
{
using (IDocumentSession ravenSession = GetRavenSession())
{
listOfObjects.ForEach(x => ravenSession.Store(x));
ravenSession.SaveChanges();
}
});
}
private static IDocumentSession GetRavenSession()
{
return _ravenDatabase.OpenSession();
}
Are you trying to save it all in one call?
The DocumentSession need to turn all of the objects that you pass it into a single request to the server. That means that it may allocate a lot of memory for the write to the server.
Usually we recommend on batches of about 1,024 items in you are doing bulks saves.
DocumentStore is a disposable class, so I worked around this problem by disposing the instance after each chunk. I highly doubt this is the most efficient way to run operations, but it will prevent significant memory overhead from happening.
I was running a sort of "delete all" operation like so. You can see the using blocks disposing both the DocumentStore and the IDocumentSession objects after each chunk.
static DocumentStore GetDataStore()
{
DocumentStore ds = new DocumentStore
{
DefaultDatabase = "test",
Url = "http://localhost:8080"
};
ds.Initialize();
return ds;
}
static IDocumentSession GetDbInstance(DocumentStore ds)
{
return ds.OpenSession();
}
static void Main(string[] args)
{
do
{
using (var ds = GetDataStore())
using (var db = GetDbInstance(ds))
{
//The `Take` operation will cap out at 1,024 by default, per Raven documentation
var list = db.Query<MyClass>().Skip(deleteSum).Take(5000).ToList();
deleteCount = list.Count;
deleteSum += deleteCount;
foreach (var item in list)
{
db.Delete(item);
}
db.SaveChanges();
list.Clear();
}
} while (deleteCount > 0);
}