Related
With web.config going away, what is the preferred way to store sensitive info (passwords, tokens) in the configurations of a web app built using ASP.NET Core?
Is there a way to automatically get encrypted configuration sections in appsettings.json?
User secrets looks like a good solution for storing passwords, and, generally, application secrets, at least during development.
Check the official Microsoft documentation. You can also review this other SO question.
This is just a way to "hide" your secrets during development process and to avoid disclosing them into the source tree; the Secret Manager tool does not encrypt the stored secrets and should not be treated as a trusted store.
If you want to bring an encrypted appsettings.json to production, you can do so by building a custom configuration provider.
For example:
public class CustomConfigProvider : ConfigurationProvider, IConfigurationSource
{
public CustomConfigProvider()
{
}
public override void Load()
{
Data = UnencryptMyConfiguration();
}
private IDictionary<string, string> UnencryptMyConfiguration()
{
// do whatever you need to do here, for example load the file and unencrypt key by key
//Like:
var configValues = new Dictionary<string, string>
{
{"key1", "unencryptedValue1"},
{"key2", "unencryptedValue2"}
};
return configValues;
}
private IDictionary<string, string> CreateAndSaveDefaultValues(IDictionary<string, string> defaultDictionary)
{
var configValues = new Dictionary<string, string>
{
{"key1", "encryptedValue1"},
{"key2", "encryptedValue2"}
};
return configValues;
}
public IConfigurationProvider Build(IConfigurationBuilder builder)
{
return new CustomConfigProvider();
}
}
Define a static class for your extension method:
public static class CustomConfigProviderExtensions
{
public static IConfigurationBuilder AddEncryptedProvider(this IConfigurationBuilder builder)
{
return builder.Add(new CustomConfigProvider());
}
}
And then you can activate it:
// Set up configuration sources.
var builder = new ConfigurationBuilder()
.AddJsonFile("appsettings.json")
.AddEncryptedProvider()
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true);
I agree with #CoderSteve that writing a whole new provider is too much work. It also doesn't build on the existing standard JSON architecture. Here is a solution that I come up with the builds on top of the standard JSON architecture, uses the preferred .Net Core encryption libraries, and is very DI friendly.
public static class IServiceCollectionExtensions
{
public static IServiceCollection AddProtectedConfiguration(this IServiceCollection services)
{
services
.AddDataProtection()
.PersistKeysToFileSystem(new DirectoryInfo(#"c:\keys"))
.ProtectKeysWithDpapi();
return services;
}
public static IServiceCollection ConfigureProtected<TOptions>(this IServiceCollection services, IConfigurationSection section) where TOptions: class, new()
{
return services.AddSingleton(provider =>
{
var dataProtectionProvider = provider.GetRequiredService<IDataProtectionProvider>();
section = new ProtectedConfigurationSection(dataProtectionProvider, section);
var options = section.Get<TOptions>();
return Options.Create(options);
});
}
private class ProtectedConfigurationSection : IConfigurationSection
{
private readonly IDataProtectionProvider _dataProtectionProvider;
private readonly IConfigurationSection _section;
private readonly Lazy<IDataProtector> _protector;
public ProtectedConfigurationSection(
IDataProtectionProvider dataProtectionProvider,
IConfigurationSection section)
{
_dataProtectionProvider = dataProtectionProvider;
_section = section;
_protector = new Lazy<IDataProtector>(() => dataProtectionProvider.CreateProtector(section.Path));
}
public IConfigurationSection GetSection(string key)
{
return new ProtectedConfigurationSection(_dataProtectionProvider, _section.GetSection(key));
}
public IEnumerable<IConfigurationSection> GetChildren()
{
return _section.GetChildren()
.Select(x => new ProtectedConfigurationSection(_dataProtectionProvider, x));
}
public IChangeToken GetReloadToken()
{
return _section.GetReloadToken();
}
public string this[string key]
{
get => GetProtectedValue(_section[key]);
set => _section[key] = _protector.Value.Protect(value);
}
public string Key => _section.Key;
public string Path => _section.Path;
public string Value
{
get => GetProtectedValue(_section.Value);
set => _section.Value = _protector.Value.Protect(value);
}
private string GetProtectedValue(string value)
{
if (value == null)
return null;
return _protector.Value.Unprotect(value);
}
}
}
Wire up your protected config sections like this:
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc();
// Configure normal config settings
services.Configure<MySettings>(Configuration.GetSection("MySettings"));
// Configure protected config settings
services.AddProtectedConfiguration();
services.ConfigureProtected<MyProtectedSettings>(Configuration.GetSection("MyProtectedSettings"));
}
You can easily create encrypted values for your config files using a controller like this:
[Route("encrypt"), HttpGet, HttpPost]
public string Encrypt(string section, string value)
{
var protector = _dataProtectionProvider.CreateProtector(section);
return protector.Protect(value);
}
Usage:
http://localhost/cryptography/encrypt?section=SectionName:KeyName&value=PlainTextValue
I didn't want to write a custom provider – way too much work. I just wanted to tap into JsonConfigurationProvider, so I figured out a way that works for me, hope it helps someone.
public class JsonConfigurationProvider2 : JsonConfigurationProvider
{
public JsonConfigurationProvider2(JsonConfigurationSource2 source) : base(source)
{
}
public override void Load(Stream stream)
{
// Let the base class do the heavy lifting.
base.Load(stream);
// Do decryption here, you can tap into the Data property like so:
Data["abc:password"] = MyEncryptionLibrary.Decrypt(Data["abc:password"]);
// But you have to make your own MyEncryptionLibrary, not included here
}
}
public class JsonConfigurationSource2 : JsonConfigurationSource
{
public override IConfigurationProvider Build(IConfigurationBuilder builder)
{
EnsureDefaults(builder);
return new JsonConfigurationProvider2(this);
}
}
public static class JsonConfigurationExtensions2
{
public static IConfigurationBuilder AddJsonFile2(this IConfigurationBuilder builder, string path, bool optional,
bool reloadOnChange)
{
if (builder == null)
{
throw new ArgumentNullException(nameof(builder));
}
if (string.IsNullOrEmpty(path))
{
throw new ArgumentException("File path must be a non-empty string.");
}
var source = new JsonConfigurationSource2
{
FileProvider = null,
Path = path,
Optional = optional,
ReloadOnChange = reloadOnChange
};
source.ResolveFileProvider();
builder.Add(source);
return builder;
}
}
I managed to create a custom JSON configuration provider which uses DPAPI to encrypt and decrypt secrets. It basically uses simple regular expressions that you can define to specify what parts of the JSON needs to be encrypted.
The following steps are performed:
Json file is loaded
Determine whether the JSON parts that match the given regular expressions are already encrypted (or not). This is done by base-64 decoding of the JSON part and verify whether it starts with the expected prefix !ENC!)
If not encrypted, then encrypt the JSON part by first using DPAPI and secondly add the prefix !ENC! and encode to base-64
Overwrite the unencrypted JSON parts with the encrypted (base-64) values in the Json file
Note that the base-64 does not bring better security, but only hides the prefix !ENC! for cosmetic reasons. This is just a matter of taste of course ;)
This solution consists of the following classes:
ProtectedJsonConfigurationProvider class (= custom JsonConfigurationProvider)
ProtectedJsonConfigurationSource class (= custom JsonConfigurationSource)
AddProtectedJsonFile() extension method on the IConfigurationBuilder in order to simple add the protected configuration
Assuming the following initial authentication.json file:
{
"authentication": {
"credentials": [
{
user: "john",
password: "just a password"
},
{
user: "jane",
password: "just a password"
}
]
}
}
Which becomes (sort of) the following after loading
{
"authentication": {
"credentials": [
{
"user": "john",
"password": "IUVOQyEBAAAA0Iyd3wEV0R=="
},
{
"user": "jane",
"password": "IUVOQyEBAAAA0Iyd3wEV0R=="
}
]
}
}
And assuming the following configuration class based on the json format
public class AuthenticationConfiguration
{
[JsonProperty("credentials")]
public Collection<CredentialConfiguration> Credentials { get; set; }
}
public class CredentialConfiguration
{
[JsonProperty("user")]
public string User { get; set; }
[JsonProperty("password")]
public string Password { get; set; }
}
Below the sample code:
//Note that the regular expression will cause the authentication.credentials.password path to be encrypted.
//Also note that the byte[] contains the entropy to increase security
var configurationBuilder = new ConfigurationBuilder()
.AddProtectedJsonFile("authentication.json", true, new byte[] { 9, 4, 5, 6, 2, 8, 1 },
new Regex("authentication:credentials:[0-9]*:password"));
var configuration = configurationBuilder.Build();
var authenticationConfiguration = configuration.GetSection("authentication").Get<AuthenticationConfiguration>();
//Get the decrypted password from the encrypted JSON file.
//Note that the ProtectedJsonConfigurationProvider.TryGet() method is called (I didn't expect that :D!)
var password = authenticationConfiguration.Credentials.First().Password
Install the Microsoft.Extensions.Configuration.Binder package in order to get the configuration.GetSection("authentication").Get<T>() implementation
And finally the classes in which the magic happens :)
/// <summary>Represents a <see cref="ProtectedJsonConfigurationProvider"/> source</summary>
public class ProtectedJsonConfigurationSource : JsonConfigurationSource
{
/// <summary>Gets the byte array to increse protection</summary>
internal byte[] Entropy { get; private set; }
/// <summary>Represents a <see cref="ProtectedJsonConfigurationProvider"/> source</summary>
/// <param name="entropy">Byte array to increase protection</param>
/// <exception cref="ArgumentNullException"/>
public ProtectedJsonConfigurationSource(byte[] entropy)
{
this.Entropy = entropy ?? throw new ArgumentNullException(Localization.EntropyNotSpecifiedError);
}
/// <summary>Builds the configuration provider</summary>
/// <param name="builder">Builder to build in</param>
/// <returns>Returns the configuration provider</returns>
public override IConfigurationProvider Build(IConfigurationBuilder builder)
{
EnsureDefaults(builder);
return new ProtectedJsonConfigurationProvider(this);
}
/// <summary>Gets or sets the protection scope of the configuration provider. Default value is <see cref="DataProtectionScope.CurrentUser"/></summary>
public DataProtectionScope Scope { get; set; }
/// <summary>Gets or sets the regular expressions that must match the keys to encrypt</summary>
public IEnumerable<Regex> EncryptedKeyExpressions { get; set; }
}
/// <summary>Represents a provider that protects a JSON configuration file</summary>
public partial class ProtectedJsonConfigurationProvider : JsonConfigurationProvider
{
private readonly ProtectedJsonConfigurationSource protectedSource;
private readonly HashSet<string> encryptedKeys = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
private static readonly byte[] encryptedPrefixBytes = Encoding.UTF8.GetBytes("!ENC!");
/// <summary>Checks whether the given text is encrypted</summary>
/// <param name="text">Text to check</param>
/// <returns>Returns true in case the text is encrypted</returns>
private bool isEncrypted(string text)
{
if (text == null) { return false; }
//Decode the data in order to verify whether the decoded data starts with the expected prefix
byte[] decodedBytes;
try { decodedBytes = Convert.FromBase64String(text); }
catch (FormatException) { return false; }
return decodedBytes.Length >= encryptedPrefixBytes.Length
&& decodedBytes.AsSpan(0, encryptedPrefixBytes.Length).SequenceEqual(encryptedPrefixBytes);
}
/// <summary>Converts the given key to the JSON token path equivalent</summary>
/// <param name="key">Key to convert</param>
/// <returns>Returns the JSON token path equivalent</returns>
private string convertToTokenPath(string key)
{
var jsonStringBuilder = new StringBuilder();
//Split the key by ':'
var keyParts = key.Split(':');
for (var keyPartIndex = 0; keyPartIndex < keyParts.Length; keyPartIndex++)
{
var keyPart = keyParts[keyPartIndex];
if (keyPart.All(char.IsDigit)) { jsonStringBuilder.Append('[').Append(keyPart).Append(']'); }
else if (keyPartIndex > 0) { jsonStringBuilder.Append('.').Append(keyPart); }
else { jsonStringBuilder.Append(keyPart); }
}
return jsonStringBuilder.ToString();
}
/// <summary>Writes the given encrypted key/values to the JSON oconfiguration file</summary>
/// <param name="encryptedKeyValues">Encrypted key/values to write</param>
private void writeValues(IDictionary<string, string> encryptedKeyValues)
{
try
{
if (encryptedKeyValues == null || encryptedKeyValues.Count == 0) { return; }
using (var stream = new FileStream(this.protectedSource.Path, FileMode.Open, FileAccess.ReadWrite))
{
JObject json;
using (var streamReader = new StreamReader(stream, Encoding.UTF8, true, 4096, true))
{
using (var jsonTextReader = new JsonTextReader(streamReader))
{
json = JObject.Load(jsonTextReader);
foreach (var encryptedKeyValue in encryptedKeyValues)
{
var tokenPath = this.convertToTokenPath(encryptedKeyValue.Key);
var value = json.SelectToken(tokenPath) as JValue;
if (value.Value != null) { value.Value = encryptedKeyValue.Value; }
}
}
}
stream.Seek(0, SeekOrigin.Begin);
using (var streamWriter = new StreamWriter(stream))
{
using (var jsonTextWriter = new JsonTextWriter(streamWriter) { Formatting = Formatting.Indented })
{
json.WriteTo(jsonTextWriter);
}
}
}
}
catch (Exception exception)
{
throw new Exception(string.Format(Localization.ProtectedJsonConfigurationWriteEncryptedValues, this.protectedSource.Path), exception);
}
}
/// <summary>Represents a provider that protects a JSON configuration file</summary>
/// <param name="source">Settings of the source</param>
/// <see cref="ArgumentNullException"/>
public ProtectedJsonConfigurationProvider(ProtectedJsonConfigurationSource source) : base(source)
{
this.protectedSource = source as ProtectedJsonConfigurationSource;
}
/// <summary>Loads the JSON data from the given <see cref="Stream"/></summary>
/// <param name="stream"><see cref="Stream"/> to load</param>
public override void Load(Stream stream)
{
//Call the base method first to ensure the data to be available
base.Load(stream);
var expressions = protectedSource.EncryptedKeyExpressions;
if (expressions != null)
{
//Dictionary that contains the keys (and their encrypted value) that must be written to the JSON file
var encryptedKeyValuesToWrite = new Dictionary<string, string>();
//Iterate through the data in order to verify whether the keys that require to be encrypted, as indeed encrypted.
//Copy the keys to a new string array in order to avoid a collection modified exception
var keys = new string[this.Data.Keys.Count];
this.Data.Keys.CopyTo(keys, 0);
foreach (var key in keys)
{
//Iterate through each expression in order to check whether the current key must be encrypted and is encrypted.
//If not then encrypt the value and overwrite the key
var value = this.Data[key];
if (!string.IsNullOrEmpty(value) && expressions.Any(e => e.IsMatch(key)))
{
this.encryptedKeys.Add(key);
//Verify whether the value is encrypted
if (!this.isEncrypted(value))
{
var protectedValue = ProtectedData.Protect(Encoding.UTF8.GetBytes(value), protectedSource.Entropy, protectedSource.Scope);
var protectedValueWithPrefix = new List<byte>(encryptedPrefixBytes);
protectedValueWithPrefix.AddRange(protectedValue);
//Convert the protected value to a base-64 string in order to mask the prefix (for cosmetic purposes)
//and overwrite the key with the encrypted value
var protectedBase64Value = Convert.ToBase64String(protectedValueWithPrefix.ToArray());
encryptedKeyValuesToWrite.Add(key, protectedBase64Value);
this.Data[key] = protectedBase64Value;
}
}
}
//Write the encrypted key/values to the JSON configuration file
this.writeValues(encryptedKeyValuesToWrite);
}
}
/// <summary>Attempts to get the value of the given key</summary>
/// <param name="key">Key to get</param>
/// <param name="value">Value of the key</param>
/// <returns>Returns true in case the key has been found</returns>
public override bool TryGet(string key, out string value)
{
if (!base.TryGet(key, out value)) { return false; }
else if (!this.encryptedKeys.Contains(key)) { return true; }
//Key is encrypted and must therefore be decrypted in order to return.
//Note that the decoded base-64 bytes contains the encrypted prefix which must be excluded when unprotection
var protectedValueWithPrefix = Convert.FromBase64String(value);
var protectedValue = new byte[protectedValueWithPrefix.Length - encryptedPrefixBytes.Length];
Buffer.BlockCopy(protectedValueWithPrefix, encryptedPrefixBytes.Length, protectedValue, 0, protectedValue.Length);
var unprotectedValue = ProtectedData.Unprotect(protectedValue, this.protectedSource.Entropy, this.protectedSource.Scope);
value = Encoding.UTF8.GetString(unprotectedValue);
return true;
}
/// <summary>Provides extensions concerning <see cref="ProtectedJsonConfigurationProvider"/></summary>
public static class ProtectedJsonConfigurationProviderExtensions
{
/// <summary>Adds a protected JSON file</summary>
/// <param name="configurationBuilder"><see cref="IConfigurationBuilder"/> in which to apply the JSON file</param>
/// <param name="path">Path to the JSON file</param>
/// <param name="optional">Specifies whether the JSON file is optional</param>
/// <param name="entropy">Byte array to increase protection</param>
/// <returns>Returns the <see cref="IConfigurationBuilder"/></returns>
/// <exception cref="ArgumentNullException"/>
public static IConfigurationBuilder AddProtectedJsonFile(this IConfigurationBuilder configurationBuilder, string path, bool optional, byte[] entropy, params Regex[] encryptedKeyExpressions)
{
var source = new ProtectedJsonConfigurationSource(entropy)
{
Path = path,
Optional = optional,
EncryptedKeyExpressions = encryptedKeyExpressions
};
return configurationBuilder.Add(source);
}
}
public static IServiceCollection ConfigureProtected<TOptions>(this IServiceCollection services, IConfigurationSection section) where TOptions: class, new()
{
return services.AddSingleton(provider =>
{
var dataProtectionProvider = provider.GetRequiredService<IDataProtectionProvider>();
var protectedSection = new ProtectedConfigurationSection(dataProtectionProvider, section);
var options = protectedSection.Get<TOptions>();
return Options.Create(options);
});
}
This method is correct
Just a few clarifications to help avoid problems. When you encrypt a value, it's using the section as 'Purpose' (https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/consumer-apis/purpose-strings?view=aspnetcore-2.2) When you get a 'Payload not valid' or something similar, it's likely that the purpose you used to encrypt it, differs from the purpose use to decrypt it. So, let's say I have a first level section in my appsettings.json named 'SecureSettings' and within it a connection string:
{
"SecureSettings":
{
"ConnectionString":"MyClearTextConnectionString"
}
}
To encrypt the value, I'd call: http://localhost/cryptography/encrypt?section=SecureSettings:ConnectionString&value=MyClearTextConnectionString
You may not want to keep an Encrypt controller in the app itself btw.
I can save a picture in iOS and Android but I can't find a way to save an image in UWP. Any ideas?
Thank you in advance.
I can't find a way to save an image in UWP
No, we can't extract the image data from an Xamarin ImageSource object.
There is StreamImagesourceHandler class implementation in UWP, see here
public sealed class StreamImageSourceHandler : IImageSourceHandler
{
public async Task<Windows.UI.Xaml.Media.ImageSource> LoadImageAsync(ImageSource imagesource, CancellationToken cancellationToken = new CancellationToken())
{
BitmapImage bitmapimage = null;
//Omitted
return bitmapimage;
}
}
So we need to extract data from BitmapImage.
Actually the BitmapImage class is inherited from ImageSource class, while we can't extract the image data from ImageSource, for the reason, please see these two questions:
Convert ImageSource to WriteableBitmap in Metro Windows 8
How save BitmapImage WinRT
The solution here is to use different way for Windows Runtime(W/WP8.1 & UWP) app, extracting image data from System.IO.Stream class is supported in UWP.
We can use DependencyService to access native platform features, firstly, create an interface in PCL:
public interface ISaveImage
{
void SavePictureToDisk(ImageSource imgSrc, string Id, bool OverwriteIfExist = false);
void SavePictureToDiskWINRT(System.IO.Stream imgStream, string Id, bool OverwriteIfExist = false);
}
In the code behind of Xamarin page:
var memoryStream = new MemoryStream(Convert.FromBase64String("iVBOxxxxxxxxxxMVEX/uQOAuwPzUxxxxxxxxxxxx="));
ImageSource imgsource = ImageSource.FromStream(() => memoryStream);
if (Device.OS == TargetPlatform.Windows|| Device.OS == TargetPlatform.WinPhone)
DependencyService.Get<ISaveImage>().SavePictureToDiskWINRT(memoryStream, "1");
else
DependencyService.Get<ISaveImage>().SavePictureToDisk(imgsource, "1");
Implement the interface in UWP platform:
using Xamarin.Forms;
using WorkingWithImages.WinUniversal;
using System.IO;
using System;
using Windows.Storage.Streams;
[assembly: Xamarin.Forms.Dependency(typeof(SaveImageImplementation))]
namespace WorkingWithImages.WinUniversal
{
public class SaveImageImplementation : ISaveImage
{
public SaveImageImplementation() { }
public void SavePictureToDisk(ImageSource imgSrc, string Id, bool OverwriteIfExist = false)
{
throw new NotImplementedException();
}
public async void SavePictureToDiskWINRT(Stream imgStream, string Id, bool OverwriteIfExist = false)
{
var inStream = imgStream.AsRandomAccessStream();
var fileBytes = new byte[inStream.Size];
using (DataReader reader = new DataReader(inStream))
{
await reader.LoadAsync((uint)inStream.Size);
reader.ReadBytes(fileBytes);
}
var file = await Windows.Storage.ApplicationData.Current.LocalFolder.CreateFileAsync(Id+".jpg", Windows.Storage.CreationCollisionOption.ReplaceExisting);
using (var fs = await file.OpenAsync(Windows.Storage.FileAccessMode.ReadWrite))
{
var outStream = fs.GetOutputStreamAt(0);
var dataWriter = new DataWriter(outStream);
dataWriter.WriteBytes(fileBytes);
await dataWriter.StoreAsync();
dataWriter.DetachStream();
await outStream.FlushAsync();
outStream.Dispose();
fs.Dispose();
}
}
}
}
Please check my completed demo in here
About UWP File storage guidance, please see Create, write, and read a file
maybe someone needs a solution for iOS and Android (below). Meanwhile I'm waiting an idea for UWP.
iOS
/// <summary>
/// Saves the picture to disk.
/// </summary>
/// <returns>The picture to disk.</returns>
/// <param name="imgSrc">Image source.</param>
/// <param name="id">Identifier.</param>
/// <param name="overwriteIfExist">if set to <c>true</c> overwrite if exist.</param>
/// <returns>The picture to disk.</returns>
public async void SaveImage(ImageSource imgSrc, string id, bool overwriteIfExist = false)
{
var renderer = new StreamImagesourceHandler();
var photo = await renderer.LoadImageAsync(imgSrc);
string jpgFilename = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Personal), id + ".jpg");
if (File.Exists(jpgFilename))
{
File.Delete(jpgFilename);
}
NSData imgData = photo.AsJPEG();
NSError err;
if (imgData.Save(jpgFilename, false, out err))
{
Console.WriteLine("saved as " + jpgFilename);
}
else
{
Console.WriteLine("NOT saved as " + jpgFilename
+ " because" + err.LocalizedDescription);
}
}
Good to know, when iOS saves an image as jpg, the image header says png.
Android
/// <summary>
/// Saves the picture to disk.
/// </summary>
/// <param name="imgSrc">Image source.</param>
/// <param name="id">The image identifier.</param>
/// <param name="overwriteIfExist">if set to <c>true</c> overwrite if exist.</param>
/// <returns>The picture to disk.</returns>
public async void SaveImage(ImageSource imgSrc, string id,
bool overwriteIfExist = false)
{
var renderer = new StreamImagesourceHandler();
var photo = await renderer.LoadImageAsync(imgSrc, Forms.Context);
string jpgFilename = System.IO.Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Personal), id + ".jpg");
if (File.Exists(jpgFilename))
{
File.Delete(jpgFilename);
}
using (FileStream fs = new FileStream(jpgFilename, FileMode.OpenOrCreate))
{
photo.Compress(Bitmap.CompressFormat.Jpeg, 100, fs);
}
}
want to update a C# Winforms application to use await.
The application calls the MYOB Accountright API via an SDK.
I am using Dot Net Framework 4.5.1
The old code is like this
public void GetItems( CompanyFile companyFile )
{
var itemSvc = new ItemService(MyConfiguration, null, MyOAuthKeyService);
string pageFilter = string.Format("$top={0}&$skip={1}&$orderby=Date desc", PageSize,
PageSize * (_currentPage - 1));
itemSvc.GetRange(MyCompanyFile, pageFilter, MyCredentials, OnComplete, OnError);
}
/// <summary>
/// Method called on Async complete
/// </summary>
/// <param name="statusCode"></param>
/// <param name="items"></param>
/// <remarks></remarks>
private void OnComplete(System.Net.HttpStatusCode statusCode,
PagedCollection<Item> items)
{
myItems = items;
}
/// <summary>
/// Callback if there is an error
/// </summary>
/// <param name="uri"></param>
/// <param name="ex"></param>
/// <remarks></remarks>
private void OnError(Uri uri, Exception ex)
{
Trace.WriteLine("In OnError");
MessageBox.Show(ex.Message);
}
I want code something like this
private async Task FetchItemsAsync()
{
var itemSvc = new ItemService(MyConfiguration, null, MyOAuthKeyService);
string pageFilter = string.Format("$top={0}&$skip={1}&$orderby=Date desc", PageSize,
PageSize * (_currentPage - 1));
itemSvc.GetRange(MyCompanyFile, pageFilter, MyCredentials, OnComplete, OnError);
var totalPages = (int)Math.Ceiling((double)(myItems.Count / PageSize));
while (_currentPage < totalPages)
{
await LoadMore(); // how do I write this?
}
}
How do I do that?
[Update5]
I tried
private const double PageSize = 400;
protected CancellationTokenSource MyCancellationTokenSource;
protected CompanyFile MyCompanyFile;
protected IApiConfiguration MyConfiguration;
protected ICompanyFileCredentials MyCredentials;
protected ItemService MyItemService;
protected IOAuthKeyService MyOAuthKeyService;
private int _currentPage = 1;
private int _totalPages;
public void FetchItems(CompanyFile companyFile, IApiConfiguration configuration, ICompanyFileCredentials credentials)
{
MyCompanyFile = companyFile;
MyConfiguration = configuration;
MyCredentials = credentials;
MyCancellationTokenSource = new CancellationTokenSource();
MyItemService = new ItemService(MyConfiguration, null, MyOAuthKeyService);
FetchAllItemsAsync();
}
private async void FetchAllItemsAsync()
{
try
{
var items = new List<Item>();
int totalPages = 0;
do
{
string pageFilter = string.Format("$top={0}&$skip={1}&$orderby=Date desc", PageSize, PageSize * (_currentPage - 1));
CancellationToken ct = MyCancellationTokenSource.Token;
Log("About to Await GetRange");
Task<PagedCollection<Item>> tpc = MyItemService.GetRangeAsync(MyCompanyFile, pageFilter, MyCredentials, ct, null);
Log("About to Await GetRange B");
PagedCollection<Item> newItems = await tpc; // fails here
Log("Page {0} retrieved {1} items", _currentPage, newItems.Count);
if (totalPages == 0)
{
totalPages = (int)Math.Ceiling((items.Count / PageSize));
}
items.AddRange(newItems.Items.ToArray());
_currentPage++;
}
while (_currentPage < totalPages);
MessageBox.Show(string.Format("Fetched {0} items", items.Count));
}
catch (ApiCommunicationException ex)
{
Log(ex.ToString());
throw;
}
catch (Exception exception)
{
Log(exception.ToString());
throw;
}
}
However I get a ValidationException
{"Encountered a validation error (http://localhost:8080/AccountRight/ab5c1f96-7663-4052-8360-81004cfe8598/Inventory/Item/?$top=400&$skip=0&$orderby=Date desc)"}
[MYOB.AccountRight.SDK.ApiValidationException]: {"Encountered a validation error (http://localhost:8080/AccountRight/ab5c1f96-7663-4052-8360-81004cfe8598/Inventory/Item/?$top=400&$skip=0&$orderby=Date desc)"}
base: {"Encountered a validation error (http://localhost:8080/AccountRight/ab5c1f96-7663-4052-8360-81004cfe8598/Inventory/Item/?$top=400&$skip=0&$orderby=Date desc)"}
ErrorInformation: "Warning, error messages have not been finalised in this release and may change"
Errors: Count = 1
RequestId: "e573dfed-ec68-4aff-ac5e-3ffde1c2f943"
StatusCode: BadRequest
URI: {http://localhost:8080/AccountRight/ab5c1f96-7663-4052-8360-81004cfe8598/Inventory/Item/?$top=400&$skip=0&$orderby=Date desc}
I have cross posted this problem to
MYOB Support
The latest version of the MYOB.AccountRight.API.SDK you are referencing already has overloads for supporting async/await on .NET4, .NET45 and PCL.
The Sample code was created as an example for someone using .NET 3.5 (hence no async/await). Another sample (windows phone) shows async/await in action using the SDK
[Update]
You are probably getting an OData related exception as the Item entity does not have a Date field for which you can filter by (see docs).
When you catch an ApiCommunicationException (of which ApiValidationException is a subclass) there is an Errors property that provides more detail.
There is also a RequestId (and some other properties) which are very useful should you need to talk to the support guys if you have issues talking to the cloud hosted API.
You can use a TaskCompletionSource object that you can resolve with the result or error callback. I'm not sure what the signature of the error callback is so that part probably wont work.
private Task<PagedCollection<Item>> FetchItemsAsync()
{
var taskSource = new TaskCompletionSource<PagedCollection<Item>>();
var itemSvc = new ItemService(MyConfiguration, null, MyOAuthKeyService);
string pageFilter = string.Format("$top={0}&$skip={1}&$orderby=Date desc", PageSize,
PageSize * (_currentPage - 1));
itemSvc.GetRange(
MyCompanyFile,
pageFilter,
MyCredentials,
(statusCode, items) => taskSource.TrySetResult(items),
(error) => taskSource => taskSource.TrySetException(error) // Not sure if this is correct signature
);
return taskSource.Task;
}
You can then return the Task object that it creates which you can use for async things. I'm not really sure about what logic you are trying to implement because your question is not very detailed but you can use the method with the await command because it returns a Task object like the following.
private async void FetchAllItemsAsync()
{
int totalPages;
do
{
items = await FetchItemsAsync()
totalPages = (int)Math.Ceiling((double)(items.Count / PageSize));
_currentPage++;
} while (_currentPage < totalPages)
}
Using the default Visual Studio 2013 Web API project template with individual user accounts, and posting to the /token endpoint with an Accept header of application/xml, the server still returns the response in JSON:
{"access_token":"...","token_type":"bearer","expires_in":1209599}
Is there a way to get the token back as XML?
According to RFC6749 the response format should be JSON and Microsoft implemented it accordingly. I found out that JSON formatting is implemented in Microsoft.Owin.Security.OAuth.OAuthAuthorizationServerHandler internal class with no means of extension.
I also encountered the need to have token response in XML.
The best solution I came up with was to implement HttpModule converting JSON to XML when stated in Accept header.
public class OAuthTokenXmlResponseHttpModule : IHttpModule
{
private static readonly string FilterKey = typeof(OAuthTokenXmlResponseHttpModule).Name + typeof(MemoryStreamFilter).Name;
public void Init(HttpApplication application)
{
application.BeginRequest += ApplicationOnBeginRequest;
application.EndRequest += ApplicationOnEndRequest;
}
private static void ApplicationOnBeginRequest(object sender, EventArgs eventArgs)
{
var application = (HttpApplication)sender;
if (ShouldConvertToXml(application.Context.Request) == false) return;
var filter = new MemoryStreamFilter(application.Response.Filter);
application.Response.Filter = filter;
application.Context.Items[FilterKey] = filter;
}
private static bool ShouldConvertToXml(HttpRequest request)
{
var isTokenPath = string.Equals("/token", request.Path, StringComparison.InvariantCultureIgnoreCase);
var header = request.Headers["Accept"];
return isTokenPath && (header == "text/xml" || header == "application/xml");
}
private static void ApplicationOnEndRequest(object sender, EventArgs eventArgs)
{
var context = ((HttpApplication) sender).Context;
var filter = context.Items[FilterKey] as MemoryStreamFilter;
if (filter == null) return;
var jsonResponse = filter.ToString();
var xDocument = JsonConvert.DeserializeXNode(jsonResponse, "oauth");
var xmlResponse = xDocument.ToString(SaveOptions.DisableFormatting);
WriteResponse(context.Response, xmlResponse);
}
private static void WriteResponse(HttpResponse response, string xmlResponse)
{
response.Clear();
response.ContentType = "application/xml;charset=UTF-8";
response.Write(xmlResponse);
}
public void Dispose()
{
}
}
public class MemoryStreamFilter : Stream
{
private readonly Stream _stream;
private readonly MemoryStream _memoryStream = new MemoryStream();
public MemoryStreamFilter(Stream stream)
{
_stream = stream;
}
public override void Flush()
{
_stream.Flush();
}
public override int Read(byte[] buffer, int offset, int count)
{
return _stream.Read(buffer, offset, count);
}
public override void Write(byte[] buffer, int offset, int count)
{
_memoryStream.Write(buffer, offset, count);
_stream.Write(buffer, offset, count);
}
public override string ToString()
{
return Encoding.UTF8.GetString(_memoryStream.ToArray());
}
#region Rest of the overrides
public override bool CanRead
{
get { throw new NotImplementedException(); }
}
public override bool CanSeek
{
get { throw new NotImplementedException(); }
}
public override bool CanWrite
{
get { throw new NotImplementedException(); }
}
public override long Seek(long offset, SeekOrigin origin)
{
throw new NotImplementedException();
}
public override void SetLength(long value)
{
throw new NotImplementedException();
}
public override long Length
{
get { throw new NotImplementedException(); }
}
public override long Position
{
get
{
throw new NotImplementedException();
}
set
{
throw new NotImplementedException();
}
}
#endregion
}
Ok I had such a fun time trying to figure this out using OWIN I thought I would share my solution with the community, I borrowed some insight from other posts https://stackoverflow.com/a/26216511/1148288 and https://stackoverflow.com/a/29105880/1148288 along with the concepts Alexei describs in his post. Nothing fancy doing with implementation but I had a requirement for my STS to return an XML formatted response, I wanted to keep with the paradigm of honoring the Accept header, so my end point would examine that to determine if it needed to run the XML swap or not. This is what I am current using:
private void ConfigureXMLResponseSwap(IAppBuilder app)
{
app.Use(async (context, next) =>
{
if (context.Request != null &&
context.Request.Headers != null &&
context.Request.Headers.ContainsKey("Accept") &&
context.Request.Headers.Get("Accept").Contains("xml"))
{
//Set a reference to the original body stream
using (var stream = context.Response.Body)
{
//New up and set the response body as a memory stream which implements the ability to read and set length
using (var buffer = new MemoryStream())
{
context.Response.Body = buffer;
//Allow other middlewares to process
await next.Invoke();
//On the way out, reset the buffer and read the response body into a string
buffer.Seek(0, SeekOrigin.Begin);
using (var reader = new StreamReader(buffer))
{
string responsebody = await reader.ReadToEndAsync();
//Using our responsebody string, parse out the XML and add a declaration
var xmlVersion = JsonConvert.DeserializeXNode(responsebody, "oauth");
xmlVersion.Declaration = new XDeclaration("1.0", "UTF-8", "yes");
//Convert the XML to a byte array
var bytes = Encoding.UTF8.GetBytes(xmlVersion.Declaration + xmlVersion.ToString());
//Clear the buffer bits and write out our new byte array
buffer.SetLength(0);
buffer.Write(bytes, 0, bytes.Length);
buffer.Seek(0, SeekOrigin.Begin);
//Set the content length to the new buffer length and the type to an xml type
context.Response.ContentLength = buffer.Length;
context.Response.ContentType = "application/xml;charset=UTF-8";
//Copy our memory stream buffer to the output stream for the client application
await buffer.CopyToAsync(stream);
}
}
}
}
else
await next.Invoke();
});
}
Of course you would then wire this up during startup config like so:
public void Configuration(IAppBuilder app)
{
HttpConfiguration httpConfig = new HttpConfiguration();
//Highly recommend this is first...
ConfigureXMLResponseSwap(app);
...more config stuff...
}
Hope that helps any other lost souls that find there way to the this post seeking to do something like this!
take a look here i hope it can help how to set a Web API REST service to always return XML not JSON
Could you retry by doing the following steps:
In the WebApiConfig.Register(), specify
config.Formatters.XmlFormatter.UseXmlSerializer = true;
var supportedMediaTypes = config.Formatters.XmlFormatter.SupportedMediaTypes;
if (supportedMediaTypes.Any(it => it.MediaType.IndexOf("application/xml", StringComparison.InvariantCultureIgnoreCase) >= 0) ==false)
{
supportedMediaTypes.Insert(0,new MediaTypeHeaderValue("application/xml"));
}
I normally just remove the XmlFormatter altogether.
// Remove the XML formatter
config.Formatters.Remove(config.Formatters.XmlFormatter);
Add the line above in your WebApiConfig class...
public static class WebApiConfig
{
public static void Register(HttpConfiguration config)
{
// Web API configuration and services
// Configure Web API to use only bearer token authentication.
config.SuppressDefaultHostAuthentication();
config.Filters.Add(new HostAuthenticationFilter(OAuthDefaults.AuthenticationType));
// Web API routes
config.MapHttpAttributeRoutes();
// Remove the XML formatter
config.Formatters.Remove(config.Formatters.XmlFormatter);
config.Routes.MapHttpRoute(
name: "DefaultApi",
routeTemplate: "api/{controller}/{id}",
defaults: new { id = RouteParameter.Optional }
);
}
}
I successfully used System.ServiceModel.Syndication.SyndicationFeed to add some Atom10 output from my ASP.NET 3.5 web site. It was my first production use of ServiceStack, and it all work fine.
My first attempt resulted in UTF-16 instead of UTF-8, which was ok for all browsers except IE. So I had to create XmlWriterResult class to solve this. My solution works, but how should I have done?
public class AsStringService : IService<AsString>
{
public object Execute(AsString request)
{
SyndicationFeed feed = new SyndicationFeed("Test Feed " + request.Name, "This is a test feed", new Uri("http://Contoso/testfeed"), "TestFeedID", DateTime.Now);
SyndicationItem item = new SyndicationItem("Test Item", "This is the content for Test Item", new Uri("http://localhost/ItemOne"), "TestItemID", DateTime.Now);
List<SyndicationItem> items = new List<SyndicationItem>();
items.Add(item);
feed.Items = items;
Atom10FeedFormatter atomFormatter = new Atom10FeedFormatter(feed);
return new XmlWriterResult(xmlWriter => atomFormatter.WriteTo(xmlWriter));
}
}
XmlWriterResult is:
public delegate void XmlWriterDelegate(XmlWriter xmlWriter);
/// <summary>
/// From https://groups.google.com/forum/?fromgroups=#!topic/servicestack/1U02g7kViRs
/// </summary>
public class XmlWriterResult : IDisposable, IStreamWriter, IHasOptions
{
private readonly XmlWriterDelegate _writesToXmlWriter;
public XmlWriterResult(XmlWriterDelegate writesToXmlWriter)
{
_writesToXmlWriter = writesToXmlWriter;
this.Options = new Dictionary<string, string> {
{ HttpHeaders.ContentType, "text/xml" }
};
}
public void Dispose()
{
}
public void WriteTo(Stream responseStream)
{
using (XmlWriter xmlWriter = XmlWriter.Create(responseStream))
{
_writesToXmlWriter(xmlWriter);
}
}
public IDictionary<string, string> Options { get; set; }
}
(Yes, I like delegates, I also do a lot of F#)
As this isn't a question with any clear answer I'd just tell you how I'd do it.
Assuming SyndicationFeed is a clean DTO / POCO you should just return that in your service:
public class AsStringService : IService
{
public object Any(AsString request)
{
SyndicationFeed feed = new SyndicationFeed("Test Feed " + request.Name,
"This is a test feed", new Uri("http://Contoso/testfeed"),
"TestFeedID", DateTime.Now);
SyndicationItem item = new SyndicationItem("Test Item",
"This is the content for Test Item",
new Uri("http://localhost/ItemOne"),
"TestItemID", DateTime.Now);
List<SyndicationItem> items = new List<SyndicationItem>();
items.Add(item);
feed.Items = items;
return feed;
}
}
This example uses ServiceStack's New API which is much nicer, you should try using it for future services.
This will allow you to get Content Negotiation in all of ServiceStack's registered Content-Types.
Registering a Custom Media Type
You could then register a Custom Media Type as seen in ServiceStack's Northwind v-card example:
private const string AtomContentType = "application/rss+xml";
public static void Register(IAppHost appHost)
{
appHost.ContentTypeFilters.Register(AtomContentType, SerializeToStream,
DeserializeFromStream);
}
public static void SerializeToStream(IRequestContext requestContext,
object response, Stream stream)
{
var syndicationFeed = response as SyndicationFeed;
if (SyndicationFeed == null) return;
using (XmlWriter xmlWriter = XmlWriter.Create(stream))
{
Atom10FeedFormatter atomFormatter = new Atom10FeedFormatter(feed);
atomFormatter.WriteTo(xmlWriter);
}
}
public static object DeserializeFromStream(Type type, Stream stream)
{
throw new NotImplementedException();
}
Now the rss+xml format should appear in the /metadata pages and ServiceStack will let you request the format in all the supported Content-Negotation modes, e.g:
Accept: application/rss+xml HTTP Header
Using the predefined routes, e.g: /rss+xml/syncreply/AsString
Or appending /route?format=rss+xml to the QueryString
Note: you may need to url encode the '+' character