I have json data that is being compressed using ServiceStacks's inbuilt ToOptimizedResult method. This has been working fine for a while now, recently though, when the data to be returned is high (50k+) rows etc. I get the Out Memory Error below:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Text.StringBuilder.ToString()
at ServiceStack.Text.JsonSerializer.SerializeToString(Object value, Type type)
at ServiceStack.Text.JsonSerializer.SerializeToString[T](T value)
at ServiceStack.ServiceModel.Serialization.JsonDataContractSerializer.SerializeToString[T](T obj)
at ServiceStack.Common.Web.HttpResponseFilter.SerializeToString(IRequestContext requestContext, Object response)
at ServiceStack.ServiceHost.RequestContextExtensions.ToOptimizedResult[T](IRequestContext requestContext, T dto)
at MyService.post(PostDDSReportDataSourceData Input) in MyService.vb:line 452
at lambda_method(Closure , Object , Object )
at ServiceStack.ServiceHost.ServiceRunner`1.Execute(IRequestContext requestContext, Object instance, TRequest request)
The weird part is that the exact same data is serialized ok on Dev, the error only appears in Production. The main solutions I've seen online recommend disabling buffering, however I've only seen examples that disable buffering entirely for the entire application. How can I disable buffering just for that one request?
Figured it out myself eventually. In the method simply turn off buffering like this:
CType(MyBase.Response.OriginalResponse, System.Web.HttpResponse).BufferOutput = False
As an addendum to the above solution, you will still receive this error if you store the json string in a variable. Only call the ToOptimized method when returning the result. Turning off buffering won't make a difference if you store the serialized string in a variable.
I eventually ended up serializing my list to a file, then sending the file down by returning an HttpResult below:
Dim PathToTheSerializedJsonFile as String = "C:\SomeFile.Json"
Using fl As StreamWriter = File.CreateText(PathToTheSerializedJsonFile)
Dim Serr As New Newtonsoft.Json.JsonSerializer()
Serr.Serialize(fl, Input)
End Using
Return New HttpResult(New FileInfo(PathToTheSerializedJsonFile),asAttachment:=True)
Related
1)When exactly we need to use HTTPContext (exclusively), please elaborate with an example if possible.
2)What does the following code do
internal string QueryString( string name )
{
if ( HttpContext.Current.Request.QueryString[name] == null )
return string.Empty;
else {
return HttpContext.Current.Request.QueryString[name].ToString();
}
3)Is there any other alternative way where we avoid using HttpContext
Q1:
HttpContext is an object which encapsulates all HTTP-specific information about an individual HTTP request. So, if you need to get any information about receiving HTTP request (for example in order to get the query string parameter values, referral urls, User's IP address etc), you need to use HttpContext object. Further more, this object contains information of current Request, Response, Server, Session, Cache, User and etc.
More information
Usage and examples
Get current session Id: HttpContext.Session.SessionID
Get Timestamp of current request: HttpContext.Timestamp.ToString()
Q2:
In your code sample, you are trying get the value of query string parameter 'name'.
Your dummy request Url looks like - http://www.yourdomain.com/something?name=queryStringValue
QueryString parameter = name
QueryString parameter value = queryStringValue
Hope this helps.
I might be confused about something, but when I store a custom object from the Java Riak client and then try to read that object using the Python Riak client, I end up with a raw json string instead of a dict.
However, if I store a the object in python, I am able to output a python dictionary when fetching that object.
I could simply use a json library on the python side to resolve this, but the very fact that I am experiencing this discrepancy makes me think that I am doing something wrong.
On the Java side, this is my object:
class DocObject
{
public String status; // FEEDING | PERSISTED | FAILED | DELETING
public List<String> messages = new ArrayList<String>();
}
class PdfObject extends DocObject
{
public String url;
public String base_url;
}
This is how I am storing that object in Riak:
public void feeding(IDocument doc) throws RiakRetryFailedException {
PdfObject pdfObject = new PdfObject();
pdfObject.url = doc.getElement("url").getValue().toString();
pdfObject.base_url = doc.getElement("base_url").getValue().toString();
pdfObject.status = "FEEDING";
String key = hash(pdfObject.url);
pdfBucket.store(key, pdfObject).execute();
}
And this is what I am doing in Python to fetch the data:
# Connect to Riak.
client = riak.RiakClient()
# Choose the bucket to store data in.
bucket = client.bucket('pdfBucket')
doc = bucket.get('7909aa2f84c9e0fded7d1c7bb2526f54')
doc_data = doc.get_data()
print type(doc_data)
The result of the above python is:
<type 'str'>
I am expecting that to be <type 'dict'>, just like how the example here works:
http://basho.github.com/riak-python-client/tutorial.html#getting-single-values-out
I am perplexed as to why when the object is stored from Java it is stored as a JSON string and not as an object.
I would appreciate if anybody could point out an issue with my approach that might be causing this discrepancy.
Thanks!
It would appear you've found a bug in our Python client with the HTTP protocol/transport.
Both the version you're using and the current one in master are not decoding JSON properly. Myself and another dev looked into this this morning and it appears to stem from an issue with charset parameter being returned from Riak with the content-type as Christian noted in his comment ("application/json; charset=UTF-8")
We've opened an issue on github (https://github.com/basho/riak-python-client/issues/227) and will get this corrected.
In the mean time the only suggestion I have is to decode the returned JSON string yourself, or using the 1.5.2 client (latest stable from pypy) and the Protocol Buffers transport:
client = riak.RiakClient(port=8087, transport_class=riak.RiakPbcTransport)
it will return the decoded JSON as a dict as you're expecting.
The code is trivial so rather than explain, here's the relevant portion:
protected void Page_Load(object sender, EventArgs e)
{
if ((Request.HttpMethod == "POST") && (Request.ContentType == "text/xml"))
{
string filename = string.Format("{0:yyyyMMddHHmmss}.xml", DateTime.UtcNow);
var path = Path.Combine(Request.MapPath("Chapters"), Request.Headers["X-MAC"]);
Directory.CreateDirectory(path);
string xml;
using (ChapterWriterClient client = new ChapterWriterClient())
using (StreamReader sr = new StreamReader(Request.InputStream))
{
xml = sr.ReadToEnd();
client.Write(xml);
System.Diagnostics.Trace.TraceInformation(xml);
}
Request.SaveAs(Path.Combine(path, filename), false);
}
There's more but the rest of it works as expected.
Here's the problem: the call client.Write(string xml) is erratic in whether it succeeds.
Some known facts:
No error is ever reported to the debugger or the event log
Sometimes the XML makes it into the message queue and sometimes it doesn't
Request.SaveAs(string filepath, bool includeHeaders) call always succeeds and the resultant file always contains the expected XML. This has several important implications:
The call to WCF is being executed.
Because the string xml had the expected value immediately after the failed WCF call, and assignment to this variable occurs only prior to the WCF call, it must have had the correct value at the time of the call.
There is some variation in message size so I'm checking the service config in that regard but past experience tells me to expect an exception thrown when the payload is too big.
Any clues would be appreciated.
Also, as you can probably see the whole point of the exercise is a somewhat naiive attempt to use an HTTP POST to put some XML into a message on MSMQ. I'd be perfectly happy with a "Don't do it like that, do it like this you ignorant savage" type answer.
Will this code throw an exception:
public class MyHttpHandlerFactory : IHttpHandlerFactory
{
public IHttpHandler GetHandler(HttpContext context, string requestType, string url, string pathTranslated)
{
if (...)
return null;
...
return new MyHttpHandler();
}
}
If you take a look at this article on MSDN, you'll see that in their example they return null from GetHandler. However, they only return null if the request is not a GET or a POST, which should never happen based on the way they have set up the factory in the web.config.
I set up a quick sample using the code from the article using ASP.NET 4.0/IIS 7.5/Integrated Pipeline and if you do return null from a call to GetHandler all that appears to happen is an empty 200/OK response is returned from the server (I checked using Fiddler). So it seems that ASP.NET 4.0 (at least) handles this condition gracefully. To answer the question as asked, no it does not appear that there is a runtime limitation to returning null. In practice, however, you may want to limit the requests that your HandlerFactory is receiving so that it never returns null, or at least consider how the other parts of your application will react to a request to this handler factory returning an empty 200/OK response.
I have a site which runs in ASP.NET 3.5, NHibernate 2.2 and Sprint .NET for Dependency Injection. On our test server a rather strange error occurrs, and also almost everytime there are multiple users online. After the problem has occurred, this error is displayed for every user and every request they make - until you do an IISRESET. Then it is all ok again.
Here's the exception:
'count' must be non-negative.
Parameter name: count
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.ArgumentOutOfRangeException: 'count' must be non-negative.
Parameter name: count
Source Error:
[No relevant source lines]
Source File: c:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\root\4bf9aa39\6dcf5fc6\App_Web_z9ifuy6t.6.cs Line: 0
Stack Trace:
[ArgumentOutOfRangeException: 'count' must be non-negative.
Parameter name: count]
System.String.CtorCharCount(Char c, Int32 count) +10082288
Spring.Objects.Factory.Support.AbstractObjectFactory.GetObjectInternal(String name, Type requiredType, Object[] arguments, Boolean suppressConfigure) +3612
Spring.Objects.Factory.Support.AbstractObjectFactory.GetObject(String name) +75
Spring.Objects.Factory.Support.DefaultListableObjectFactory.GetObjectsOfType(Type type, Boolean includePrototypes, Boolean includeFactoryObjects) +365
Spring.Context.Support.AbstractApplicationContext.GetObjectsOfType(Type type, Boolean includePrototypes, Boolean includeFactoryObjects) +136
Spring.Context.Support.AbstractApplicationContext.GetObjectsOfType(Type type) +66
[ActivationException: Activation error occured while trying to get instance of type InfoTextService, key ""]
Microsoft.Practices.ServiceLocation.ServiceLocatorImplBase.GetInstance(Type serviceType, String key) in c:\Home\Chris\Projects\CommonServiceLocator\main\Microsoft.Practices.ServiceLocation\ServiceLocatorImplBase.cs:57
Microsoft.Practices.ServiceLocation.ServiceLocatorImplBase.GetInstance() in c:\Home\Chris\Projects\CommonServiceLocator\main\Microsoft.Practices.ServiceLocation\ServiceLocatorImplBase.cs:90
OurProjectsNamespace.Infrastructure.ObjectLocator.LocateService() +86
This is indeed a very wierd error. When you look at the source of the AbstractObjectFactory.GetObjectInternal you will see the following structure:
[ThreadStatic]
private int nestingCount;
protected object GetObjectInternal(...)
{
const int INDENT = 3;
bool hasErrors = false;
try
{
nestingCount++;
if (log.IsDebugEnabled)
{
log.Debug("msg" +
new String(' ', nestingCount * INDENT));
}
// More code: Calls self recursively.
}
catch
{
nestingCount--;
hasErrors = true;
if (log.IsErrorEnabled)
{
log.Error("msg" +
new String(' ', nestingCount * INDENT));
}
}
finally
{
if (!hasErrors)
{
nestingCount--;
if (log.IsDebugEnabled)
{
log.Debug("msg" +
new String(' ', nestingCount * INDENT));
}
}
}
}
The exception you are seeing must be thrown by one of the three new String(' ', nestingCount * INDENT) calls. That particular string constructor call throws when the supplied value is negative. Because INDENT is a const, nestingCount must have a negative value in that case. nestingCount is a thread-static variable. Thread-static variables are always initialized with their default value (0 in this case) and can't be influenced by other threads. Further more, nestingCount is never used outside this method.
Because nestingCount is thread-static and only used in that method, it is hard to imagine a scenario were nestingCount can get negative. Perhaps in the case of an asynchronous (ThreadAbort) exception, but even this I find hard to imagine. Other option is that the thread-static variable is changed by someone else using reflection.
But the big question is: how to solve this?
Solution:
There's only one thing I can think of and that is reconfigure log4net in a way that debug information isn't logged. When disallowing debug information, the string(char, int) constructor will probably never get called again, which will hide the problem. Not very pretty, but possibly effective. This might work, because the AbstractObjectFactory logs using the log variable that is initialized as follows:
this.log = LogManager.GetLogger(this.GetType());
You can do this by globally disabling the writing of debug information in log4net, or –when you think this is overkill- by configuring log4net to disable debug info for the type Spring.Objects.Factory.Support.DefaultListableObjectFactory (the instance that is actually causes the exception).
Good luck.
I have seen this error occur when a database column is mapped to more than one property. A typical case is when a foreign key column is mapped to a property and to collection. A second or third pair of eyes on the config files helps spot these.
One twist with this is that it occurs for all users. Do you have a persistent object that is stored in application state?
In my case, this error occurred after some time during performance testing. It starts fine and after some time this error pops up.
Turns out it was caused by a totally unrelated [ThreadLocal] variable I used in my code. I replaced it with a method parameter and now it works fine.