I am trying to insert a .NET serialized JSON document to CouchBase but get gibberish with non-English characters.
I've tried to put:
name = " الفورية مترجم نصوص مجاني إلى "
But I get:
"name": " ״§„ˆ״±״© …״×״±״¬… †״µˆ״µ …״¬״§† ״¥„‰ ",
When viewing it in CouchBase administration page.
Any solution?
I am using ASP.NET 4.5 and latest CouchBase API with CouchBase 2.0 beta.
Also noted on http://www.couchbase.com/forums/thread/gibrish-non-english-characters-using-c-client-and-coucbase-2-0-beta, but copied here for reference:
Hi Idanm
The .NET client stores strings using UTF-8. Are you seeing the data coming out of the client correctly, or are the mixed up strings in the admin console?
Also, you can try the new extension methods that use Newtonsoft.JSON for Json serialization. In the .NET Couchbase Client Beta, you'll find Couchbase.Extensions with the following class:
public static class CouchbaseClientExtensions
{
public static bool StoreJson(this CouchbaseClient client, StoreMode storeMode, string key, object value)
{
var json = JsonConvert.SerializeObject(value);
return client.Store(storeMode, key, json);
}
public static T GetJson<T>(this CouchbaseClient client, string key) where T : class
{
var json = client.Get<string>(key);
return json == null ? null : JsonConvert.DeserializeObject<T>(json);
}
}
It seems possible that the Encoding.Default.GetBytes in the extensions you're using might be at fault here...
Although I'm not familiar with the Couchbase storage system, you're definitely running into a text encoding issue. I'd check that your JSON serializer is serializing to UTF-8 encoding, as well as ensuring that you're specifying UTF-8 encoding on the Couchbase side. Check their APIs and storage types to ensure that they are UTF-8 based.
Related
I am trying to set up a Web API to be used for capturing the Clocking In and Clocking Out times of Employees after scanning a QR code.
I have set up this SQL Server database to test out the logic for Web API and IONIC 3 project.
My SQL Server code are as follows:
-- Creating table 1 'FarmWorker'
CREATE TABLE [dbo].[FarmWorker]
(
[FarmWorkerNum] int primary key identity(1,1),
[FarmWorkerFName] varchar (15) not null,
[FarmWorkerLName] varchar (15) not null,
[FarmWorkerIDNum] char (13) not null
);
GO
-- Creating table 2 'AttendenceSheet'
CREATE TABLE [dbo].[AttendenceSheet]
(
[AttendenceSheetID] int primary key identity(1,1),
[ClockInTime] datetime not null,
[ClockOutTime] datetime not null,
[FarmWorkerNum] int not null
FOREIGN KEY REFERENCES [FarmWorker](FarmWorkerNum)
);
GO
My Web API has been set up using Visual Studio 2015 and ASP.Net Web API 2.
My controller code for the Web API are as follows:
namespace AttendanceService.Controllers
{
public class FarmWorkerController : ApiController
{
public IEnumerable<FarmWorker> Get()
{
using (AttDBEntities entities = new AttDBEntities())
{
return entities.FarmWorkers.ToList();
}
}
public FarmWorker Get(int id)
{
using (AttDBEntities entities = new AttDBEntities())
{
return entities.FarmWorkers.FirstOrDefault(e =>Convert.ToInt32(e.FarmWorkerIDNum) == id);
}
}
}
}
When I run my Web API in the browser and attempt to run the first Get (to return a list of all FarmWorkers), I get the following error:
An error has occurred
System.InvalidOperationException: the 'ObjectContent`1' type failed to serialize the response body for content type 'application/xml; charset=utf-8'.
Type 'System.Data.Entity.DynamicProxies.FarmWorker_DA4B06B144222E86714953DE48C1952FFB 59C5CE216BAE8D8D506D8AEE9CEEBB' with data contract name 'FarmWorker_DA4B06B144222E86714953DE48C1952FFB59C5CE216BAE8D8D506D8AEE9CEEBB:htt p://schemas.datacontract.org/2004/07/System.Data.Entity.DynamicProxies' is not expected. Consider using a DataContractResolver if you are using DataContractSerializer or add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to the serializer.
Does anyone know how to resolve this error? Any help will greatly be appreciated.
Thank you in advance.
[EDIT / UPDATE 1] - 2 hour after posting
I have managed to resolve the error message by adding in this code into the WebApiConfig.cs file:
config.Formatters.Remove(config.Formatters.XmlFormatter);
config.Formatters.Add(config.Formatters.JsonFormatter);
config.Formatters.JsonFormatter.SerializerSettings.Formatting = Newtonsoft.Json.Formatting.Indented;
QUESTION: Why do I receive an error when it is using XML format over JSON format? If possible, I would like to use the XML format, is there any way to make this possible ?
Thanks in advance again.
My guess is that whatever you're using to call the endpoint (e.g. your browser, Fiddler, Postman), is specifying a Content-Type of application/json in the request. Try inspecting the request to see if that is the value of the header being sent.
If you really do wish to use XML, add the XML formatter again and make a request using a Content-Type of application/xml, which should return the content in XML format for you.
Solving the issue of 'ObjectContent1', I had to insert code into the WebApiConfig.cs
config.Formatters.Remove(config.Formatters.XmlFormatter);
config.Formatters.Add(config.Formatters.JsonFormatter);
config.Formatters.JsonFormatter.SerializerSettings.Formatting = Newtonsoft.Json.Formatting.Indented;
I have the following endpoint
#RequestMapping("missedcall")
fun missedCall(#RequestParam("v") encryptedValue : String, model:
ModelMap): String {
//decrypt encryptedValue here
}
When I execute this endpoint with "http://myurl.com/missedcall?v=this+is+my+encrypted+string", encryptedValue initialized as "this is my encrypted string". I actually want the pluses as they are part of the encryption and I can't decrypt the string without them.
The work around would be URL encode the string back to restore pluses and other special characters, BUT is there a cleaner way? Maybe disabling URL decoding for this particular endpoint?
Note: I can't pass this param in body, it has to be part of the query string. Also this is written in Kotlin, but I am 100% sure Java has similar issues, so don't feel discouraged by Kotlin :).
Any web framework will decrypt the query path for you, as that's expected behavior. If that's not what you want, you will have to define a method argument of type HttpServletRequest and parse the query yourself using HttpServletRequest.getQueryString().
Thanks to #SeanPatrickFloyd, implemented it as
#RequestMapping("missedcall")
fun missedCall(request: HttpServletRequest, model: ModelMap): String {
val encodedValue = request.queryString.split("=")[1]
//decrypt encryptedValue here
}
I have a web API in ASP.NET Core 2. I am using FormatFilter as defined in https://andrewlock.net/formatting-response-data-as-xml-or-json-based-on-the-url-in-asp-net-core/
I have a function defined like this:
[HttpGet("/api/Values/{userKey}/{variableKey}/GetValue")]
[HttpGet("/api/Values/{userKey}/{variableKey}/GetValue.{format}"), FormatFilter]
public async Task<string> GetValue(string userKey, string variableKey)
In Startup I have:
services.AddMvc(options =>
{
options.FormatterMappings.SetMediaTypeMappingForFormat("xml", "application/xml");
options.FormatterMappings.SetMediaTypeMappingForFormat("js", "application/json");
})
.AddXmlSerializerFormatters();
It works fine except that I'd like the default format to be XML and not json when I call /GetValue.
I still wish to continue getting json when I call /GetValue.js and XML when I call /GetValue.xml
I cannot find any doc on how to make XML the default format.
How can I achieve this?
We can pass a default value to a placeholder, so I changed a little the format of the URL and made it like this:
[HttpGet("/api/Values/{userKey}/{variableKey}/GetValue/{format=xml}"), FormatFilter]
Then /GetValue returns xml formatted
/GetValue/xml returns xml formatted
/GetValue/js returns json formatted
Has anyone tried (and succeeded) to get subscriptions out of RavenDB with the Raven DB .Net client?
Having some Json serialisation issues that when the following runs, it throws with a
"Error converting value "Subscriber.Messages.Events.MyEvent, Version=1.0.0.0" to type 'NServiceBus.Unicast.Subscriptions.MessageType'. message
The code is simple :
var documentStore = new DocumentStore
{
Url = "http://localhost:8080/",
DefaultDatabase = "publisher",
};
documentStore.Initialize();
using (var session = documentStore.OpenSession())
{
return session.Query<NServiceBus.Unicast.Subscriptions.Raven.Subscription>("Raven/DocumentsByEntityName").ToArray();
}
It's definitely a serialisation issue as the retrieval works. As it does using the alternative below:
session.Advanced.LuceneQuery<Subscription>("Raven/DocumentsByEntityName").QueryResult.Results[0]
In the RaveDB studio I can see the following document in the publisher database.
{
"MessageType": "Subscriber.Messages.Events.MyEvent, Version=1.0.0.0",
"Clients": [
{
"Queue": "samplesubscriber",
"Machine": "myDesktopHere"
}
]
}
Error converting value "Subscriber.Messages.Events.MyEvent, Version=1.0.0.0" to type 'NServiceBus.Unicast.Subscriptions.MessageType'.
Anyone have a clue why the serialization fails?
I'm using NServiveBus.Host 4.2, Raven-DB client 1.0.616, and Newtonsoft.json 4.0.5.
Incidentally I've pulled up the types using dotpeek and created local versions. I created my own subcription, MessageType, MessageTypeConvertor from the NSB dll's. I then managed to deserialise the strings without issue. Any thoughts?
EDIT
As per suggestions the advance Lucene query does a great job of retrieving the results. But then deserialization fails. For example, search results are returned in the first line, but fail to deserlize in the return statement. I've pulled up a local version of the Subscription type from the NSB dll's, and implemented the type converter, again pulled up from the NSB libraries, and using those in place of NServiceBus.Unicast.Subscriptions.Raven.Subscription works fine. Inevitably, that's not a stable choice.
var searchResults = session.Advanced.LuceneQuery<NServiceBus.Unicast.Subscriptions.Raven.Subscription>("Raven/DocumentsByEntityName").WhereEquals("Tag", "Subscription").QueryResult.Results;
return searchResults.Select(subscriptionJsonObject => JsonConvert.DeserializeObject<NServiceBus.Unicast.Subscriptions.Raven.Subscription>(subscriptionJsonObject.ToString())).ToList();
Any further thoughts?
When you query with this form:
session.Query<Entity>("IndexName")
You are asking for all items of that index to be returned from the query, and then telling the RavenDB client to deserialize them all as the type you specified.
Normally, that works out quite well because a specific index would be built for the types you're working with. For example, you would usually build your own index of subscriptions for a specific purpose, and query that:
session.Query<Subscription>("Subscriptions/ByWhatever")
or if you built the index from C#, you might like this syntax instead to avoid the string:
session.Query<Subscription, Subscriptions_ByWhatever>()
You could also just let Raven build an index for you automatically:
session.Query<Subscription>()
Since you used the built-in Raven/DocumentsByEntityName index, you are returning all documents in the database, not just those that are subscriptions. Since only some can be deserialized as a Subscription type, the others are failing with serialization errors.
If you'd like to continue to use that index, you would need to filter it to just those of that type:
session.Advanced.LuceneQuery<Subscription>("Raven/DocumentsByEntityName")
.WhereEquals("Tag", "Subscriptions")
The LuceneQuery form is easier here, since you are filtering a field by its string name.
You could use a regular LINQ query, but you'd have to create a type that contains the Tag field, like this:
class RDBENIndexEntry
{
public string Tag { get; set; }
}
...
session.Query<RDBENIndexEntry>("Raven/DocumentsByEntityName")
.Where(x => x.Tag == "Subscriptions").OfType<Subscription>()
I might be confused about something, but when I store a custom object from the Java Riak client and then try to read that object using the Python Riak client, I end up with a raw json string instead of a dict.
However, if I store a the object in python, I am able to output a python dictionary when fetching that object.
I could simply use a json library on the python side to resolve this, but the very fact that I am experiencing this discrepancy makes me think that I am doing something wrong.
On the Java side, this is my object:
class DocObject
{
public String status; // FEEDING | PERSISTED | FAILED | DELETING
public List<String> messages = new ArrayList<String>();
}
class PdfObject extends DocObject
{
public String url;
public String base_url;
}
This is how I am storing that object in Riak:
public void feeding(IDocument doc) throws RiakRetryFailedException {
PdfObject pdfObject = new PdfObject();
pdfObject.url = doc.getElement("url").getValue().toString();
pdfObject.base_url = doc.getElement("base_url").getValue().toString();
pdfObject.status = "FEEDING";
String key = hash(pdfObject.url);
pdfBucket.store(key, pdfObject).execute();
}
And this is what I am doing in Python to fetch the data:
# Connect to Riak.
client = riak.RiakClient()
# Choose the bucket to store data in.
bucket = client.bucket('pdfBucket')
doc = bucket.get('7909aa2f84c9e0fded7d1c7bb2526f54')
doc_data = doc.get_data()
print type(doc_data)
The result of the above python is:
<type 'str'>
I am expecting that to be <type 'dict'>, just like how the example here works:
http://basho.github.com/riak-python-client/tutorial.html#getting-single-values-out
I am perplexed as to why when the object is stored from Java it is stored as a JSON string and not as an object.
I would appreciate if anybody could point out an issue with my approach that might be causing this discrepancy.
Thanks!
It would appear you've found a bug in our Python client with the HTTP protocol/transport.
Both the version you're using and the current one in master are not decoding JSON properly. Myself and another dev looked into this this morning and it appears to stem from an issue with charset parameter being returned from Riak with the content-type as Christian noted in his comment ("application/json; charset=UTF-8")
We've opened an issue on github (https://github.com/basho/riak-python-client/issues/227) and will get this corrected.
In the mean time the only suggestion I have is to decode the returned JSON string yourself, or using the 1.5.2 client (latest stable from pypy) and the Protocol Buffers transport:
client = riak.RiakClient(port=8087, transport_class=riak.RiakPbcTransport)
it will return the decoded JSON as a dict as you're expecting.