ExecuteXmlReader Fails on Specific JSON from SQL Server - .net-core

I have an existing working process which receives generated JSON from store procedures in SQL Server using the for JSON directive. But upon receiving a specific text in the column data, there is a failure by ExecuteXmlReader in the Read operation.
Exception
XmlException: '=' is an unexpected token. The expected token is ';'.
Line 1, position 94
Which if I controlled the output, I would most likely put it in a CDATA section.
Data Returned from SQL Server
JSON_F52E2B61-18A1-11d1-B105-00805F49916B
{"photoId":1000000007,"photoType":"image\/gif","photoUrl":"https:\/\/slack-imgs.com\/?c=1&url=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2F3o84U9arAYRM73AIvu%2Fgiphy-downsized.gif" }
Ultimate JSON String Which Should be Returned by Read
{
"photoId": 1000000007,
"photoType": "image/gif",
"photoUrl": "https://slack-imgs.com/?c=1&url=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2F3o84U9arAYRM73AIvu%2Fgiphy-downsized.gif",
"isActive": true
}
URL saved to Table
https://slack-imgs.com/?c=1&url=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2F3o84U9arAYRM73AIvu%2Fgiphy-downsized.gif
Ultimately this is a SQL Server 2016 change but I will need a fix sooner than what can be provided from Microsoft. So, is there a work around to handle this either by SQL or C# .Net code?
Oddly enough when one clicks on the table column value JSON in SSMS, it gives the same error.

I'll go ahead and propose that you are using the wrong APIs to read the FOR JSON query results. Here's a little helper class that implements a SqlCommand.ExecuteJsonReader() extension method.
static class SqlJsonUtils
{
public static Newtonsoft.Json.JsonReader ExecuteJsonReader(this SqlCommand cmd)
{
var rdr = cmd.ExecuteReader();
var jr = new Newtonsoft.Json.JsonTextReader(new SqlJSONReader(rdr));
return jr;
}
class SqlJSONReader : System.IO.TextReader
{
SqlDataReader rdr;
string currentLine = "";
int currentPos = 0;
public SqlJSONReader(SqlDataReader rdr)
{
this.rdr = rdr;
}
public override int Peek()
{
return GetChar(false);
}
public override int Read()
{
return GetChar(true);
}
public int GetChar(bool Advance)
{
while (currentLine.Length == currentPos)
{
if (!rdr.Read())
{
return -1;
}
currentLine = rdr.GetString(0);
currentPos = 0;
}
int rv = (int)currentLine[currentPos];
if (Advance) currentPos += 1;
return rv;
}
public override void Close()
{
rdr.Close();
}
}
}

I have since made this a downloadable NUGET package. See SQLJSONReader with updates for async.
I took David's advice and marked his as the answer, but I needed the whole raw JSON.
So I added removed the call to to Newtonsoft.Json.JsonTextReader because it wouldn't return me just a string and modified David's extension class to return the whole JSON by calling ReadAll.
Note that it uses Newtonsoft's JsonConvert.DeserializeObject.
Code
public static SqlJSONReader ExecuteJsonReader(this SqlCommand cmd)
{
var rdr = cmd.ExecuteReader();
return new SqlJSONReader(rdr);
}
public class SqlJSONReader : System.IO.TextReader
{
private SqlDataReader SqlReader { get; set; }
private string CurrentLine { get; set; }
private int CurrentPostion { get; set; }
public SqlJSONReader(SqlDataReader rdr)
{
CurrentLine = "";
CurrentPostion = 0;
this.SqlReader = rdr;
}
public override int Peek()
{
return GetChar(false);
}
public override int Read()
{
return GetChar(true);
}
public int GetChar(bool Advance)
{
while (CurrentLine.Length == CurrentPostion)
{
if (!SqlReader.Read())
{
return -1;
}
CurrentLine = SqlReader.GetString(0);
CurrentPostion = 0;
}
var rv = CurrentLine[CurrentPostion];
if (Advance)
CurrentPostion += 1;
return rv;
}
public string ReadAll()
{
var sbResult = new StringBuilder();
if (SqlReader.HasRows)
{
while (SqlReader.Read())
sbResult.Append(SqlReader.GetString(0));
}
else
return string.Empty;
// Clean up any JSON escapes before returning
return JsonConvert.DeserializeObject(sbResult.ToString()).ToString();
}
public override void Close() { SqlReader.Close(); }
}
Usage
using (SqlConnection conn = new SqlConnection(connectionString))
using (SqlCommand cmd = conn.CreateCommand())
{
cmd.CommandText = "exec [dbo].[GetPhoto] #PhotoId=4";
conn.Open();
var rdr = cmd.ExecuteJsonReader();
string jsonResult = rdr.ReadAll();
conn.Close();
}

Related

How to implement asynchronous data streaming in .Net Core Service Bus triggered Azure Function processing huge data not to get OutOfMemoryException?

I have a service bus triggered Azure Function which listens for messages containing just blob URL strings of JSON data which each one of them is at least 10MB.
Message queue is near real-time(If I use the correct term) so producers keep putting messaging to the queue with a frequency so there is always data in the queue to be processed.
I have designed a solution but it gets OutOfMemoryException most of the time. The steps involved in the current solution sequentially are:
Consume a message
Download the file from the URL within the consumed message to a temporary folder
Read the whole file as a string
Deserialize it to an object
Partition into the chunks to supply Mongo bulk upsert limit
Bulk upsert to Mongo
I have tried to solve OutOfMemoryException and I thought that it's because my function/consumer don't have the same pace with the producer, so I think that at the time t1 when it gets the first message and process it and then while it's upserting to the mongo the function keeps getting the messages and they accumulate in the memory and waiting to be upserted.
Is my reasoning right?
Thus I think that If I could implement a streaming solution starting from #3, reading from file by chunking and putting it to a stream then I would prevent the memory keep growing and reduce time also. I have mostly Java background and I somehow know that with custom iterator/spliterator/iterable it is possible to do streaming and asynchronous processing.
How can I do asynchronous data streaming with .Net Core in an Azure Function?
Are there other approaches to solve this problem?
namespace x.y.Z
{
public class MyFunction
{
//...
[FunctionName("my-func")]
public async Task Run([ServiceBusTrigger("my-topic", "my-subscription", Connection = "AzureServiceBus")] string message, ILogger log, ExecutionContext context)
{
var data = new PredictionMessage();
try
{
data = myPredictionService.genericDeserialize(message);
await myPredictionService.ValidateAsync(data);
await myPredictionService.AddAsync(data);
}
catch (Exception ex)
{
//...
}
}
}
}
public class PredictionMessage
{
public string BlobURL { get; set; }
}
namespace x.y.z.Prediction
{
public abstract class BasePredictionService<T> : IBasePredictionService<T> where T : PredictionMessage, new()
{
protected readonly ILogger log;
private static JsonSerializer serializer;
public BasePredictionService(ILogger<BasePredictionService<T>> log)
{
this.log = log;
serializer = new JsonSerializer();
}
public async Task ValidateAsync(T message)
{
//...
}
public T genericDeserialize(string message)
{
return JsonConvert.DeserializeObject<T>(message);
}
public virtual Task AddAsync(T message)
{
throw new System.NotImplementedException();
}
public async Task<string> SerializePredictionResult(T message)
{
var result = string.Empty;
using (WebClient client = new WebClient())
{
var tempPath = Path.Combine(Path.GetTempPath(), DateTime.Now.Ticks + ".json");
Uri srcPath = new Uri(message.BlobURL);
await client.DownloadFileTaskAsync(srcPath, tempPath);
using (FileStream fs = File.Open(tempPath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
using (BufferedStream bs = new BufferedStream(fs))
using (StreamReader sr = new StreamReader(bs))
{
result = sr.ReadToEnd();
}
}
Task.Run(() =>
{
File.Delete(tempPath);
});
return result;
}
}
protected TType StreamDataDeserialize<TType>(string streamResult)
{
var body = default(TType);
using (MemoryStream stream = new MemoryStream(Encoding.Default.GetBytes(streamResult)))
{
using (StreamReader streamReader = new StreamReader(stream))
{
body = (TType)serializer.Deserialize(streamReader, typeof(TType));
}
}
return body;
}
protected List<List<TType>> Split<TType>(List<TType> list, int chunkSize = 1000)
{
List<List<TType>> retVal = new List<List<TType>>();
while (list.Count > 0)
{
int count = list.Count > chunkSize ? chunkSize : list.Count;
retVal.Add(list.GetRange(0, count));
list.RemoveRange(0, count);
}
return retVal;
}
}
}
namespace x.y.z.Prediction
{
public class MyPredictionService : BasePredictionService<PredictionMessage>, IMyPredictionService
{
private readonly IMongoDBRepository<MyPrediction> repository;
public MyPredictionService(IMongoDBRepoFactory mongoDBRepoFactory, ILogger<MyPredictionService> log) : base(log)
{
repository = mongoDBRepoFactory.GetRepo<MyPrediction>();
}
public override async Task AddAsync(PredictionMessage message)
{
string streamResult = await base.SerializePredictionResult(message);
var body = base.StreamDataDeserialize<List<MyPrediction>>(streamResult);
if (body != null && body.Count > 0)
{
var chunkList = base.Split(body);
await BulkUpsertProcess(chunkList);
}
}
private async Task BulkUpsertProcess(List<List<MyPrediction>> chunkList)
{
foreach (var perChunk in chunkList)
{
var filterContainers = new List<IDictionary<string, object>>();
var updateContainer = new List<IDictionary<string, object>>();
foreach (var item in perChunk)
{
var filter = new Dictionary<string, object>();
var update = new Dictionary<string, object>();
filter.Add(/*...*/);
filterContainers.Add(filter);
update.Add(/*...*/);
updateContainer.Add(update);
}
await Task.Run(async () =>
{
await repository.BulkUpsertAsync(filterContainers, updateContainer);
});
}
}
}
}

Issue with Custom Pipeline component

The Custom Pipeline component developed reads the incoming stream to a folder and pass only some meta data through the MessageBox.I am using the one already availaible in Code Project
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.BizTalk.Component.Interop;
using System.IO;
namespace SendLargeFilesDecoder
{
[ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
[ComponentCategory(CategoryTypes.CATID_Decoder)]
[System.Runtime.InteropServices.Guid("53fd04d5-8337-42c2-99eb-32ac96d1105a")]
public class SendLargeFileDecoder : IBaseComponent,
IComponentUI,
IComponent,
IPersistPropertyBag
{
#region IBaseComponent
private const string _description = "Pipeline component used to save large files to disk";
private const string _name = "SendLargeFileDecoded";
private const string _version = "1.0.0.0";
public string Description
{
get { return _description; }
}
public string Name
{
get { return _name; }
}
public string Version
{
get { return _version; }
}
#endregion
#region IComponentUI
private IntPtr _icon = new IntPtr();
public IntPtr Icon
{
get { return _icon; }
}
public System.Collections.IEnumerator Validate(object projectSystem)
{
return null;
}
#endregion
#region IComponent
public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
{
if (_largeFileLocation == null || _largeFileLocation.Length == 0)
_largeFileLocation = Path.GetTempPath();
if (_thresholdSize == null || _thresholdSize == 0)
_thresholdSize = 4096;
if (pInMsg.BodyPart.GetOriginalDataStream().Length > _thresholdSize)
{
Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();
string srcFileName = pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();
string largeFilePath = _largeFileLocation + System.IO.Path.GetFileName(srcFileName);
FileStream fs = new FileStream(largeFilePath, FileMode.Create);
// Write message to disk
byte[] buffer = new byte[1];
int bytesRead = originalStream.Read(buffer, 0, buffer.Length);
while (bytesRead != 0)
{
fs.Flush();
fs.Write(buffer, 0, buffer.Length);
bytesRead = originalStream.Read(buffer, 0, buffer.Length);
}
fs.Flush();
fs.Close();
// Create a small xml file
string xmlInfo = "<MsgInfo xmlns='http://SendLargeFiles'><LargeFilePath>" + largeFilePath + "</LargeFilePath></MsgInfo>";
byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(xmlInfo);
MemoryStream ms = new MemoryStream(byteArray);
pInMsg.BodyPart.Data = ms;
}
return pInMsg;
}
#endregion
#region IPersistPropertyBag
private string _largeFileLocation;
private int _thresholdSize;
public string LargeFileLocation
{
get { return _largeFileLocation; }
set { _largeFileLocation = value; }
}
public int ThresholdSize
{
get { return _thresholdSize; }
set { _thresholdSize = value; }
}
public void GetClassID(out Guid classID)
{
classID = new Guid("CA47347C-010C-4B21-BFCB-22F153FA141F");
}
public void InitNew()
{
}
public void Load(IPropertyBag propertyBag, int errorLog)
{
object val1 = null;
object val2 = null;
try
{
propertyBag.Read("LargeFileLocation", out val1, 0);
propertyBag.Read("ThresholdSize", out val2, 0);
}
catch (ArgumentException)
{
}
catch (Exception ex)
{
throw new ApplicationException("Error reading PropertyBag: " + ex.Message);
}
if (val1 != null)
_largeFileLocation = (string)val1;
if (val2 != null)
_thresholdSize = (int)val2;
}
public void Save(IPropertyBag propertyBag, bool clearDirty, bool saveAllProperties)
{
object val1 = (object)_largeFileLocation;
propertyBag.Write("LargeFileLocation", ref val1);
object val2 = (object)_thresholdSize;
propertyBag.Write("ThresholdSize", ref val2);
}
#endregion
}
}
The issue here is the LargeFileLocation is configurable in the receive pipeline. If I give a location for the first time for example E:\ABC\ the files are sent to the location.
But if I change the location to E:\DEF\ the files are still being sent to the previous location E:\ABC. I tried to create a new biztalk application deleting the old one but still I get the files dropped in to the old location E:\ABC\ not sure why.
Most likely the issue is with respect to Property definition of LargeFileLocation and its implementation and usage in IPersistPropertyBag interfaces. You can try following things:
Check if you have added E:\ABC path in Pipeline at design time. If
yes remove it from there and set in Admin console for first time
also and see how it behaves, my feeling is it will take temp path
location.
Change the Properties and IPersistPropertyBag implementation to use property with declaration such as public string LargeFileName {get;set;} i.e. no local variables _largeFileName.
Have you deleted the dll in %BizTalkFolder%\Pipeline Components\ ?
To refresh the pipeline component, you need delete the old dll file/remove the item in VS toolbox. then restart the VS, deploy it again.
and for this LargeFileLocation , I suggest you make it as a property so you can config it.

ServiceStack REST API path variables from root throwing exception

I am trying to write a REST web service using ServiceStack that accepts variable paths off of route. For example:
[Route("/{group}"]
public class Entity : IReturn<SomeType> {}
This throws a NotSupported Exception "RestPath '/{collection}' on type Entity is not supported". However, if I change the path as follows (along with the associated path in AppHost configuration) to:
[Route("/rest/{group}"]
It works just fine. In order to integrate with the system that I am working with, I need to use /{group}.
ServiceStack now allows you to add a fallback route from the / root path to handle any un-matched requests, that's not handled by a catch-all handler or refers to an existing static file. So in v3.9.56 you can now do:
[FallbackRoute("/{group}"]
public class Entity : IReturn<SomeType> {}
An alternative option is to register a IAppHost.CatchAllHandlers to handle un-matched routes, but you would need to return your own IHttpHandler to handle the request or alternatively return a RedirectHttpHandler to redirect to a different route that is managed by ServiceStack.
My current work in progress, a plugin to serve the default page to all 'not found' routes without changing the url in the browser, includes most of what you'll need to handle a global wildcard route. Use it to get you started.
To understand what this code is doing, it helps to understand ServiceStack's routing priority, and how CatchAllHandlers fit into the process. ServiceStack calls ServiceStackHttpHandlerFactory.GetHandler to get the handler for the current route.
ServiceStackHttpHandlerFactory.GetHandler returns:
A matching RawHttpHandler, if any.
If the domain root, the handler returned by GetCatchAllHandlerIfAny(...), if any.
If the route matches a metadata uri (I'm skipping over the exact logic here, as it's not important for your question), the relevant handler, if any.
The handler returned by ServiceStackHttpHandlerFactory.GetHandlerForPathInfo if any.
NotFoundHandler.
ServiceStackHttpHandlerFactory.GetHandlerForPathInfo returns:
If the url matches a valid REST route, a new RestHandler.
If the url matches an existing file or directory, it returns
the handler returned by GetCatchAllHandlerIfAny(...), if any.
If it's a supported filetype, a StaticFileHandler,
If it's not a supported filetype, the ForbiddenHttpHandler.
The handler returned by GetCatchAllHandlerIfAny(...), if any.
null.
The CatchAllHandlers array contains functions that evaluate the url and either return a handler, or null. The functions in the array are called in sequence and the first one that doesn't return null handles the route. Let me highlight some key elements:
First, the plugin adds a CatchAllHandler to the appHost.CatchAllHandlers array when registered.
public void Register(IAppHost appHost)
{
appHost.CatchAllHandlers.Add((string method, string pathInfo, string filepath) =>
Factory(method, pathInfo, filepath));
}
Second, the CatchAllHandler. As described above, the function may be called for the domain root, an existing file or directory, or any other unmatched route. Your method should return a handler, if your criteria are met, or return null.
private static Html5ModeFeature Factory(String method, String pathInfo, String filepath)
{
var Html5ModeHandler = Html5ModeFeature.Instance;
List<string> WebHostRootFileNames = RootFiles();
// handle domain root
if (string.IsNullOrEmpty(pathInfo) || pathInfo == "/")
{
return Html5ModeHandler;
}
// don't handle 'mode' urls
var mode = EndpointHost.Config.ServiceStackHandlerFactoryPath;
if (mode != null && pathInfo.EndsWith(mode))
{
return null;
}
var pathParts = pathInfo.TrimStart('/').Split('/');
var existingFile = pathParts[0].ToLower();
var catchAllHandler = new Object();
if (WebHostRootFileNames.Contains(existingFile))
{
var fileExt = Path.GetExtension(filepath);
var isFileRequest = !string.IsNullOrEmpty(fileExt);
// don't handle directories or files that have another handler
catchAllHandler = GetCatchAllHandlerIfAny(method, pathInfo, filepath);
if (catchAllHandler != null) return null;
// don't handle existing files under any event
return isFileRequest ? null : Html5ModeHandler;
}
// don't handle non-physical urls that have another handler
catchAllHandler = GetCatchAllHandlerIfAny(method, pathInfo, filepath);
if (catchAllHandler != null) return null;
// handle anything else
return Html5ModeHandler;
}
In the case of the wildcard at the root domain, you may not want to hijack routes that can be handled by another CatchAllHandler. If so, to avoid infinite recursion, you'll need a custom GetCatchAllHandlerIfAny method.
//
// local copy of ServiceStackHttpHandlerFactory.GetCatchAllHandlerIfAny, prevents infinite recursion
//
private static IHttpHandler GetCatchAllHandlerIfAny(string httpMethod, string pathInfo, string filePath)
{
if (EndpointHost.CatchAllHandlers != null)
{
foreach (var httpHandlerResolver in EndpointHost.CatchAllHandlers)
{
if (httpHandlerResolver == Html5ModeFeature.Factory) continue; // avoid infinite recursion
var httpHandler = httpHandlerResolver(httpMethod, pathInfo, filePath);
if (httpHandler != null)
return httpHandler;
}
}
return null;
}
Here's the complete, and completely untested, plugin. It compiles. It carries no warranty of fitness for any specific purpose.
using ServiceStack;
using ServiceStack.Common.Web;
using ServiceStack.Razor;
using ServiceStack.ServiceHost;
using ServiceStack.Text;
using ServiceStack.WebHost.Endpoints;
using ServiceStack.WebHost.Endpoints.Formats;
using ServiceStack.WebHost.Endpoints.Support;
using ServiceStack.WebHost.Endpoints.Support.Markdown;
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Web;
namespace MyProject.Support
{
public enum DefaultFileFormat
{
Markdown,
Razor,
Static
}
public class Html5ModeFeature : EndpointHandlerBase, IPlugin
{
private FileInfo fi { get; set; }
private DefaultFileFormat FileFormat { get; set; }
private DateTime FileModified { get; set; }
private byte[] FileContents { get; set; }
public MarkdownHandler Markdown { get; set; }
public RazorHandler Razor { get; set; }
public object Model { get; set; }
private static Dictionary<string, string> allDirs;
public string PathInfo { get; set; }
public void Register(IAppHost appHost)
{
appHost.CatchAllHandlers.Add((string method, string pathInfo, string filepath) =>
Factory(method, pathInfo, filepath));
}
private Html5ModeFeature()
{
foreach (var defaultDoc in EndpointHost.Config.DefaultDocuments)
{
if (PathInfo == null)
{
var defaultFileName = Path.Combine(Directory.GetCurrentDirectory(), defaultDoc);
if (!File.Exists(defaultFileName)) continue;
PathInfo = (String)defaultDoc; // use first default document found.
}
}
SetFile();
}
private static Html5ModeFeature instance;
public static Html5ModeFeature Instance
{
get { return instance ?? (instance = new Html5ModeFeature()); }
}
public void SetFile()
{
if (PathInfo.EndsWith(MarkdownFormat.MarkdownExt) || PathInfo.EndsWith(MarkdownFormat.TemplateExt))
{
Markdown = new MarkdownHandler(PathInfo);
FileFormat = DefaultFileFormat.Markdown;
return;
}
if (PathInfo.EndsWith(Razor.RazorFormat.RazorFileExtension)) {
Razor = new RazorHandler(PathInfo);
FileFormat = DefaultFileFormat.Razor;
return;
}
FileContents = File.ReadAllBytes(PathInfo);
FileModified = File.GetLastWriteTime(PathInfo);
FileFormat = DefaultFileFormat.Static;
}
//
// ignore request.PathInfo, return default page, extracted from StaticFileHandler.ProcessResponse
//
public void ProcessStaticPage(IHttpRequest request, IHttpResponse response, string operationName)
{
response.EndHttpHandlerRequest(skipClose: true, afterBody: r =>
{
TimeSpan maxAge;
if (r.ContentType != null && EndpointHost.Config.AddMaxAgeForStaticMimeTypes.TryGetValue(r.ContentType, out maxAge))
{
r.AddHeader(HttpHeaders.CacheControl, "max-age=" + maxAge.TotalSeconds);
}
if (request.HasNotModifiedSince(fi.LastWriteTime))
{
r.ContentType = MimeTypes.GetMimeType(PathInfo);
r.StatusCode = 304;
return;
}
try
{
r.AddHeaderLastModified(fi.LastWriteTime);
r.ContentType = MimeTypes.GetMimeType(PathInfo);
if (fi.LastWriteTime > this.FileModified)
SetFile(); //reload
r.OutputStream.Write(this.FileContents, 0, this.FileContents.Length);
r.Close();
return;
}
catch (Exception ex)
{
throw new HttpException(403, "Forbidden.");
}
});
}
private void ProcessServerError(IHttpRequest httpReq, IHttpResponse httpRes, string operationName)
{
var sb = new StringBuilder();
sb.AppendLine("{");
sb.AppendLine("\"ResponseStatus\":{");
sb.AppendFormat(" \"ErrorCode\":{0},\n", 500);
sb.AppendFormat(" \"Message\": HTML5ModeHandler could not serve file {0}.\n", PathInfo.EncodeJson());
sb.AppendLine("}");
sb.AppendLine("}");
httpRes.EndHttpHandlerRequest(skipClose: true, afterBody: r =>
{
r.StatusCode = 500;
r.ContentType = ContentType.Json;
var sbBytes = sb.ToString().ToUtf8Bytes();
r.OutputStream.Write(sbBytes, 0, sbBytes.Length);
r.Close();
});
return;
}
private static List<string> RootFiles()
{
var WebHostPhysicalPath = EndpointHost.Config.WebHostPhysicalPath;
List<string> WebHostRootFileNames = new List<string>();
foreach (var filePath in Directory.GetFiles(WebHostPhysicalPath))
{
var fileNameLower = Path.GetFileName(filePath).ToLower();
WebHostRootFileNames.Add(Path.GetFileName(fileNameLower));
}
foreach (var dirName in Directory.GetDirectories(WebHostPhysicalPath))
{
var dirNameLower = Path.GetFileName(dirName).ToLower();
WebHostRootFileNames.Add(Path.GetFileName(dirNameLower));
}
return WebHostRootFileNames;
}
private static Html5ModeFeature Factory(String method, String pathInfo, String filepath)
{
var Html5ModeHandler = Html5ModeFeature.Instance;
List<string> WebHostRootFileNames = RootFiles();
// handle domain root
if (string.IsNullOrEmpty(pathInfo) || pathInfo == "/")
{
return Html5ModeHandler;
}
// don't handle 'mode' urls
var mode = EndpointHost.Config.ServiceStackHandlerFactoryPath;
if (mode != null && pathInfo.EndsWith(mode))
{
return null;
}
var pathParts = pathInfo.TrimStart('/').Split('/');
var existingFile = pathParts[0].ToLower();
var catchAllHandler = new Object();
if (WebHostRootFileNames.Contains(existingFile))
{
var fileExt = Path.GetExtension(filepath);
var isFileRequest = !string.IsNullOrEmpty(fileExt);
// don't handle directories or files that have another handler
catchAllHandler = GetCatchAllHandlerIfAny(method, pathInfo, filepath);
if (catchAllHandler != null) return null;
// don't handle existing files under any event
return isFileRequest ? null : Html5ModeHandler;
}
// don't handle non-physical urls that have another handler
catchAllHandler = GetCatchAllHandlerIfAny(method, pathInfo, filepath);
if (catchAllHandler != null) return null;
// handle anything else
return Html5ModeHandler;
}
//
// Local copy of private StaticFileHandler.DirectoryExists
//
public static bool DirectoryExists(string dirPath, string appFilePath)
{
if (dirPath == null) return false;
try
{
if (!ServiceStack.Text.Env.IsMono)
return Directory.Exists(dirPath);
}
catch
{
return false;
}
if (allDirs == null)
allDirs = CreateDirIndex(appFilePath);
var foundDir = allDirs.ContainsKey(dirPath.ToLower());
//log.DebugFormat("Found dirPath {0} in Mono: ", dirPath, foundDir);
return foundDir;
}
//
// Local copy of private StaticFileHandler.CreateDirIndex
//
static Dictionary<string, string> CreateDirIndex(string appFilePath)
{
var indexDirs = new Dictionary<string, string>();
foreach (var dir in GetDirs(appFilePath))
{
indexDirs[dir.ToLower()] = dir;
}
return indexDirs;
}
//
// Local copy of private StaticFileHandler.GetDirs
//
static List<string> GetDirs(string path)
{
var queue = new Queue<string>();
queue.Enqueue(path);
var results = new List<string>();
while (queue.Count > 0)
{
path = queue.Dequeue();
try
{
foreach (string subDir in Directory.GetDirectories(path))
{
queue.Enqueue(subDir);
results.Add(subDir);
}
}
catch (Exception ex)
{
Console.Error.WriteLine(ex);
}
}
return results;
}
//
// local copy of ServiceStackHttpHandlerFactory.GetCatchAllHandlerIfAny, prevents infinite recursion
//
private static IHttpHandler GetCatchAllHandlerIfAny(string httpMethod, string pathInfo, string filePath)
{
if (EndpointHost.CatchAllHandlers != null)
{
foreach (var httpHandlerResolver in EndpointHost.CatchAllHandlers)
{
if (httpHandlerResolver == Html5ModeFeature.Factory) continue; // avoid infinite recursion
var httpHandler = httpHandlerResolver(httpMethod, pathInfo, filePath);
if (httpHandler != null)
return httpHandler;
}
}
return null;
}
public override void ProcessRequest(IHttpRequest httpReq, IHttpResponse httpRes, string operationName)
{
switch (FileFormat)
{
case DefaultFileFormat.Markdown:
{
Markdown.ProcessRequest(httpReq, httpRes, operationName);
break;
}
case DefaultFileFormat.Razor:
{
Razor.ProcessRequest(httpReq, httpRes, operationName);
break;
}
case DefaultFileFormat.Static:
{
fi.Refresh();
if (fi.Exists) ProcessStaticPage(httpReq, httpRes, operationName); else ProcessServerError(httpReq, httpRes, operationName);
break;
}
default:
{
ProcessServerError(httpReq, httpRes, operationName);
break;
}
}
}
public override object CreateRequest(IHttpRequest request, string operationName)
{
return null;
}
public override object GetResponse(IHttpRequest httpReq, IHttpResponse httpRes, object request)
{
return null;
}
}
}

nice overload structure

I have created a library function and want to add an overload that does a very similar thing with an additional parameter. The existing code looks like this:
public class MealsAllocation
{
public int mealId;
public List<CrewSummary> crew;
private MealsAllocation() { }
public MealsAllocation(int MealId) {
mealId = MealId;
string connStr = ConfigurationManager.ConnectionStrings["LocalSqlServer"].ConnectionString;
SqlConnection conn = new SqlConnection(connStr);
//first fill an ienumerable redemption object for the meal
List<MealRedemption> mealRedemptions = new List<MealRedemption>();
SqlCommand cmdRed = new SqlCommand("tegsGetMealsRedemption", conn);
cmdRed.CommandType = CommandType.StoredProcedure;
cmdRed.Parameters.Add(new SqlParameter("#mealId", MealId));
conn.Open();
SqlDataReader drRed = cmdRed.ExecuteReader();
while (drRed.Read())
{
MealRedemption mr = new MealRedemption(Convert.ToInt32(drRed["crewId"]), Convert.ToDateTime(drRed["creation"]), Convert.ToInt32(drRed["redeemed"]));
mealRedemptions.Add(mr);
}
conn.Close();
//then fill the crew list
crew = new List<CrewSummary>();
SqlCommand cmdCrew = new SqlCommand("tegsGetMealsAllocation", conn);
cmdCrew.CommandType = CommandType.StoredProcedure;
cmdCrew.Parameters.Add(new SqlParameter("#mealId", MealId));
conn.Open();
SqlDataReader drCrew = cmdCrew.ExecuteReader();
while (drCrew.Read())
{
int drCid = Convert.ToInt32(drCrew["id"]);
List<MealRedemption> drMr = mealRedemptions.FindAll(red => red.crewId == drCid) ;
CrewSummary cs = new CrewSummary(drCid, Convert.ToInt32(drCrew["allocation"]), drMr );
crew.Add(cs);
}
conn.Close();
}
So then now I wish to add a new overload that will look a bit like this:
public MealsAllocation(int MealId, int crewId)
{
}
and essentially this will do much the same but slightly different from the above.
What would be a good strategy to avoid "copy and paste inheritance" ?
ie a nice way to refactor the above so that it lends itself more easily to the overload?
How about moving your logic to an internal function so it's only accessible in this assembly, and to this class and use optional parameters... something like this:
public class MealsAllocation
{
public int mealId;
public List<CrewSummary> crew;
private MealsAllocation()
{
}
public MealsAllocation(int MealId)
{
DoWork(MealId);
}
public MealsAllocation(int MealId, int crewId)
{
DoWork(MealId, crewId);
}
internal void DoWork(int MealId, int crewId = -1)
{
// have your logic here based on your parameter list
// valid crewId passed, then add new param for DB proc
if (crewId > -1)
{
cmdCrew.Parameters.Add(new SqlParameter("#crewId", crewId));
}
}
}
You can user object initializer
var mealRedemption = new
{
MealId = yourvlue,
Crew = crew
};
Link : http://msdn.microsoft.com/en-us/library/bb384062.aspx
Since you want to overload your constructor, you could also try such an approach:
public MealsAllocation(int MealId) : this (MealId, null)
{
}
public MealsAllocation(int MealId, int? crewId)
{
// Initialize your instance as needed
if (crewId.HasValue)
{
// Do some more stuff
}
}
Although I don't recommend doing all that inside the constructor, you can simply add an optional param to the end:
public class MealsAllocation
{
public int MealId { get; set; }
public int CrewId { get; set; }
public List<CrewSummary> Crew { get; set; };
public MealsAllocation(int mealId, int crewId = 0)
{
this.MealId = mealId;
this.CrewId = crewId;
if(this.CrewId = 0) // etc...
}
Side note: You need to add the using statement around your SqlConnection, SqlCommand and SqlDataReader object or you could run into connection and/or memory leaks. Personally, I'd create a Data Access layer and put all of the data related methods there to make them reusable across your entire business layer.
Also, I think this might be a good candidate for the Lazy<T> object: http://msdn.microsoft.com/en-us/library/dd642331.aspx
The first thing that comes to mind is to split that big chunk of code in two different methods
giving at each method a specialized functionality
public MealsAllocation(int MealId)
{
List<MealRedemption> mealRedemptions = LoadMealRedemptions(MealID);
LoadCrewSummaryByMeal(mealRedemptions, MealID);
}
while the other constructor could be
public MealsAllocation(int MealId, int crewId)
{
List<MealRedemption> mealRedemptions = LoadMealRedemptions(MealID);
LoadCrewSummaryByCrew(mealRedemptions, MealID, crewID);
}
In the first constructor you call the private method where you load the MealRedemptions list, get its output and pass to a specialized method that load the CrewSummary list using only the MealID and the list obtained from the first method.
In the second constructor you could use the same method used in the first one and then use a different one for the loading of the CrewSummary. The requirements of your second constructor are not clear and could change the design of this second method (I mean, how do you use the crewID parameter to change the inner workings to build the CrewSummary list?)

Storage for DataMappers in ASP.NET WebApplication

In Martin Fowler's "Patterns of Enterprise Application Architecture"
is described approach for organizing DAL like a set of mappers for entities. Each has it's own IdentityMap storing specific entity.
for example in my ASP.NET WebApplication:
//AbstractMapper - superclass for all mappers in DAL
public abstract class AbstractMapper
{
private readonly string _connectionString;
protected string ConnectionString
{
get { return _connectionString; }
}
private readonly DbProviderFactory _dbFactory;
protected DbProviderFactory DBFactory
{
get { return _dbFactory; }
}
#region LoadedObjects (IdentityMap)
protected Hashtable LoadedObjects = new Hashtable();
public void RegisterObject(long id, DomainObject obj)
{
LoadedObjects[id] = obj;
}
public void UnregisterObject(long id)
{
LoadedObjects.Remove(id);
}
#endregion
public AbstractMapper(string connectionString, DbProviderFactory dbFactory)
{
_connectionString = connectionString;
_dbFactory = dbFactory;
}
protected virtual string DBTable
{
get
{
throw new NotImplementedException("database table is not defined in class " + this.GetType());
}
}
protected virtual T Find<T>(long id, IDbTransaction tr = null) where T : DomainObject
{
if (id == 0)
return null;
T result = (T)LoadedObjects[id];
if (result != null)
return result;
IDbConnection cn = GetConnection(tr);
IDbCommand cmd = CreateCommand(GetFindStatement(id), cn, tr);
IDataReader rs = null;
try
{
OpenConnection(cn, tr);
rs = cmd.ExecuteReader(CommandBehavior.SingleRow);
result = (rs.Read()) ? Load<T>(rs) : null;
}
catch (DbException ex)
{
throw new DALException("Error while loading an object by id in class " + this.GetType(), ex);
}
finally
{
CleanUpDBResources(cmd, cn, tr, rs);
}
return result;
}
protected virtual T Load<T>(IDataReader rs) where T : DomainObject
{
long id = GetReaderLong(rs["ID"]);
T result = (T)LoadedObjects[id];
if (result != null)
return result;
result = (T)DoLoad(id, rs);
RegisterObject(id, result);
return result;
}
// another CRUD here ...
}
// Specific Mapper for entity Account
public class AccountMapper : AbstractMapper
{
internal override string DBTable
{
get { return "Account"; }
}
public AccountMapper(string connectionString, DbProviderFactory dbFactory) : base(connectionString, dbFactory) { }
public Account Find(long id)
{
return Find<Account>(id);
}
public override DomainObject DoLoad(long id, IDataReader rs)
{
Account account = new Account(id);
account.Name = GetReaderString(rs["Name"]);
account.Value = GetReaderDecimal(rs["Value"]);
account.CurrencyID = GetReaderLong(rs["CurrencyID"]);
return account;
}
// ...
}
The question is: where to store these mappers? How system services (entities) should call mappers?
I decided to create MapperRegistry containing all mappers. So services can call mappers like:
public class AccountService : DomainService
{
public static Account FindAccount(long accountID)
{
if (accountID > 0)
return MapperRegistry.AccountMapper.Find(accountID);
return null;
}
...
}
But where can I store MapperRegistry instance? I see following variants, but don't like any of them:
MapperRegistry is global for application (Singleton)
Not applicable because of necessity of synchronization in multi-thread ASP.NET application (at least Martin says that only mad can choose this variant)
MapperRegistry per Session
Seems not so good too. All ORMs (NHibernate, LINQ to SQL, EntityFramework) masters advise to use DataContext (NHibernateSession, ObjectContext) per Request and not to store context in Session.
Also in my WebApp almost all requests are AJAX-requests to EntityController.asmx (with attribute ScriptService) returning JSON. And session is not allowed.
MapperRegistry per Request
There are a lot of separate AJAX calls. In this case life cycle of MapperRegistry will be too small. So the data almost always will be retrieved from database, as a result - low performance.
Dear Experts, please help me with architectural solution.

Resources