I have a web application that places the user's search term in the query string, in a similar way to Google. E.g. the address might be www.example.com/mysearchpage.aspx?q=searchTerm.
Usually this works fine, but if there is a special character in the search term such as â, the action attribute on the form is encoded to percent encoding and the character is replaced with %u00e2.
If I search for chât I will end up with the URL www.example.com/mysearchpage.aspx?q=châtin the browser's address bar but the action attribute on the form that comes back from the server would be www.example.com/mysearchpage.aspx?q=ch%u00e2t which means that a subsequent form submission fails because the URL is incorrectly formatted.
I have ensured that in IIS the encoding is set to be UTF-8 for Requests, Response Headers and Responses. I have also inspected the page being delivered from IIS in Fiddler and that already includes the incorrectly encoded action.
The encoded format appears to be in a non-standard format as explained in this wikipedia article.
Is there a way to prevent IIS from encoding the form's action in this way?
The solution was to add targetFramework=4.5.2 into the httpRuntime tag in the web.config file.
Previously this was not specified but was specified in the compilation tag, however specifying targetFramework=4.5.1 still caused the problem.
Attempting to make my first ASP.NET page. Got IIS 5.1 on XP, configured to run .NET 4. Created a new virtual directory and added an .aspx file. When I browse the file, non-ASCII characters are corrupted. For instance, an ü (U+00FC) is transformed to ü (U+00C3 U+00BC), which is the I-don't-get-this-is-UTF-8 equivalent.
I have tried various ways of availing this:
I made sure the .aspx file is indeed encoded as UTF-8.
I set the meta tag:
<meta charset="UTF-8">
I set the virtual directory to handle .aspx as text/html;charset=utf-8 under HTTP Headers > File Type in IIS.
I added ResponseEncoding="utf-8" to <%# Page ... %>.
I inserted the string in HttpUtility.HtmlEncoded(). Now the ü was transformed to ü (U+00C3 U+00BC).
Finally, I found 2 ways that worked:
Replacing non-ASCII characters with character references, such as ü This was okay in the 90's, not today.
Adding a web.config file to the virtual directory, with this content:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.web>
<globalization fileEncoding="utf-8"/>
</system.web>
</configuration>
Without fileEncoding setting, the ASP.NET parser will read the .aspx and corrupt every non-ASCII character without attempting to infer the file encoding. Is this just something you pros have learned to live with, or am I missing something? Is a web.config file with globalization settings the way to handle "international" characters on .aspx pages? I don't remember having similar problems with PHP, so I'm puzzled why this crops up with ASP.NET.
To use non-ASCII characters you need to have two things. Save the files using UTF-8, by choosing this encoding for the files and be sure that you have these settings on your web.config
<globalization requestEncoding="utf-8" responseEncoding="utf-8" fileEncoding="utf-8" />
Note that there is always a web.config on ASP.NET. There is the global one that also has these settings and lives in the asp.net directory {drive:}\WINDOWS\Microsoft.NET\Framework\{version}\CONFIG\, and then the web.config on your project. Sometimes the global one sets the encoding from the current country. In this case you need to set it back to UTF-8 in your project.
You have found all that already, I just point out the 3 settings:
Save your files with unicode.
Set the requestEncoding="utf-8"
Set the responseEncoding="utf-8"
You have three options.
Option 1 - either entity-encode all characters that don't fit into ASCII or replace them with similarly looking ASCII equivalents. This is error-prone and hard to maintain. The next time you have to incorporate a large piece of text you may forget to check the included piece and it "looks garbage" again.
Option 2 - save the .aspx as "UTF-8 with BOM". Such files are properly handled automatically - that's documented in description of fileEncoding property of system.web/globalization section of web.config. This is also hard to maintain - the next time you get the file resaved as "UTF-8" (without BOM) it "looks garbage" again and it may go unnoticed. When you add new .aspx files you'll have to check they are saved as "UTF-8 with BOM" too. This approach is error prone - for example, some file comparison tools don't show adding/removing BOM (at least with default settings).
Option 3 - ensure the file is saved as either "UTF-8" or "UTF-8 with BOM" and at the same time set fileEncoding property of system.web/globalization section of web.config to utf-8. The default value of this property is "single byte character encoding" so files with non-ASCII character saved as UTF-8 are handled improperly and result "looks garbage". This is the most maintainable approach - it's easy to see and easy to verify and don't randomly break when a file is resaved. fileEncoding is the only one of the three ???Encoding properties which defaults to "single byte character encoding" - responseEncoding and requestEncoding default to utf-8 so in most cases there's no need to change (or set) them, setting fileEncoding is usually enough.
On my ASP.NET MVC application, I am trying to implement a URL like below :
/product/tags/for+families
When I try to run my application with default configurations, I am getting this message with 404.11 Response Code :
HTTP Error 404.11 - Not Found
The request filtering module is configured to deny a request that
contains a double escape sequence.
I can get around with this error by implementing the below code inside my web.config :
<system.webServer>
<security>
<requestFiltering allowDoubleEscaping="true" />
</security>
</system.webServer>
So, now I am not getting any 404.11.
What I am wondering is that what kind of security holes I am opening with this implementation.
BTW, my application is under .Net Framework 4.0 and running under IIS 7.5.
The security holes that you might open up have to do with code injection - HTML injection, JavaScript injection or SQL injection.
The default settings protect you from attacks semi-efficiently by not allowing common injection strategies to work. The more default security you remove, the more you have to think about what you do with the input provided through URLs, GET request querystrings, POST request data, HTTP headers and so on...
For instance, if you are building dynamic SQL queries based on the id parameter of your action method, like this:
public ActionResult Tags(string id)
{
var sql = "SELECT * FROM Tags Where tagName = '" + id + "'";
// DO STUFF...
}
(...which is NOT a good idea), the default protection, put in place by the .NET framework, might stop some of the more dangerous scenarios, like the user requesting this URL:
/product/tags/1%27;drop%20table%20Tags;%20--
The whole idea is to treat every part of urls and other inputs to action methods as possible threats. The default security setting does provide some of that protection for you. Each default security setting you change opens up for a little more potential badness that you need to handle manually.
I assume that you are not building SQL queries this way. But the more sneaky stuff comes when you store user input in your database, then later displaying them. The malevolent user could store JavaScript or HTML in your database that go out unencoded, which would in turn threaten other users of your system.
Security Risk
The setting allowDoubleEscaping only applies to the path (cs-uri-stem) and is best explained by OWASP Double Encoding. The technique is used to get around security controls by URL Encoding the request twice. Using your URL as an example:
/product/tags/for+families --> /product/tags/for%2Bfamilies --> /product/tags/for%252Bfamilies
Suppose there are security controls specifically for /product/tags/for+families. A request arrives for /product/tags/for%252Bfamilies which is the same resource though is unchecked by the aforementioned security controls. I used the generalized term of security controls because they could be anything such as requiring an authenticated user, checking for SQLi, etc.
Why Does IIS Block?
The plus sign (+) is a reserved character per RFC2396:
Many URI include components consisting of or delimited by, certain
special characters. These characters are called "reserved", since
their usage within the URI component is limited to their reserved
purpose. If the data for a URI component would conflict with the
reserved purpose, then the conflicting data must be escaped before
forming the URI.
reserved = ";" | "/" | "?" | ":" | "#" | "&" | "=" | "+" |
"$" | ","
Wade Hilmo has an excellent post titled How IIS blocks characters in URLs. There's lots of information and background provided. The part specifically for the plus sign is as follows:
So allowDoubleEscaping/VerifyNormalization seems pretty straightforward. Why did I say that it causes confusion? The issue is when a ‘+’ character appears in a URL. The ‘+’ character doesn’t appear to be escaped, since it does not involve a ‘%’. Also, RFC 2396 notes it as a reserved character that can be included in a URL when it’s in escaped form (%2b). But with allowDoubleEscaping set to its default value of false, we will block it even in escaped form. The reason for this is historical: Way back in the early days of HTTP, a ‘+’ character was considered shorthand for a space character. Some canonicalizers, when given a URL that contains a ‘+’ will convert it to a space. For this reason, we consider a ‘+’ to be non-canonical in a URL. I was not able to find any reference to a RFC that calls out this ‘+’ treatment, but there are many references on the web that talk about it as a historical behavior.
From my own experience I know that when IIS logs a request spaces are substituted with a plus sign. Having a plus sign in the name may cause confusion when parsing logs.
Solution
There are three ways to fix this and two ways to still use the plus sign.
allowDoubleEscaping=true - This will allow double escaping for your entire website/application. Depending on the content, this could be undesirable to say the least. The following command will set allowDoubleEscaping=true.
appcmd.exe set config "Default Web Site" -section:system.webServer/security/requestFiltering /allowDoubleEscaping:True
alwaysAllowedUrls - Request Filtering offers a whitelist approach. By adding that URL path to alwaysAllowedUrls, the request will not be checked by any other Request Filtering settings and continue on in the IIS request pipeline. The concern here is that Request Filtering will not check the request for:
Request Limits: maxContentLength, maxUrl, maxQueryString
Verbs
Query - query string parameters will not be checked
Double Escaping
High Bit Characters
Request Filtering Rules
Request Header Limits
The following command will add /product/tags/for+families to alwaysAllowedUrls on the Default Web Site.
appcmd.exe set config "Default Web Site" -section:system.webServer/security/requestFiltering /+"alwaysAllowedUrls.[url='/product/tags/for+families']"
Rename - yes, just rename the file/folder/controller/etc. if possible. This is the easiest solution.
I have made a work arround for this .
so when you want to put the encripted string inside the url for(IIS)
you have to clean it from dirty :{ ";", "/", "?", ":", "#", "&", "=", "+", "$", "," };
and when you want to decript it and use it again , you have to make it dirty again before decript it (to get the wanted result) .
Here is my code , i hope it helps somebody :
public static string cleanUpEncription(string encriptedstring)
{
string[] dirtyCharacters = { ";", "/", "?", ":", "#", "&", "=", "+", "$", "," };
string[] cleanCharacters = { "p2n3t4G5l6m","s1l2a3s4h","q1e2st3i4o5n" ,"T22p14nt2s", "a9t" , "a2n3nd","e1q2ua88l","p22l33u1ws","d0l1ar5","c0m8a1a"};
foreach (string dirtyCharacter in dirtyCharacters)
{
encriptedstring=encriptedstring.Replace(dirtyCharacter, cleanCharacters[Array.IndexOf(dirtyCharacters, dirtyCharacter)]);
}
return encriptedstring;
}
public static string MakeItDirtyAgain(string encriptedString)
{
string[] dirtyCharacters = { ";", "/", "?", ":", "#", "&", "=", "+", "$", "," };
string[] cleanCharacters = { "p2n3t4G5l6m", "s1l2a3s4h", "q1e2st3i4o5n", "T22p14nt2s", "a9t", "a2n3nd", "e1q2ua88l", "p22l33u1ws", "d0l1ar5", "c0m8a1a" };
foreach (string symbol in cleanCharacters)
{
encriptedString = encriptedString.Replace(symbol, dirtyCharacters[Array.IndexOf(cleanCharacters,symbol)]);
}
return encriptedString;
}
Encode the encrypted string separated:
return HttpServerUtility.UrlTokenEncode(Encoding.UTF8.GetBytes("string"));
Decode the encrypted string separated:
string x = Encoding.UTF8.GetString(HttpServerUtility.UrlTokenDecode(id));
So I ran into this when I was calling an API from an MVC app. Instead of opening the security hole, I modified my path.
First off, I recommend NOT disabling this setting. It is more appropriate to modify the design of the application/resource (e.g. encode the path, pass the data in a header or in the body).
Although this is an older post, I thought I would share how you could resolve this error if you are receiving this from a call to an API by using HttpUtility.UrlPathEncode method in System.Web.
I use RestSharp for making calls out, so my example is using the RestRequest:
var tags = new[] { "for", "family" };
var apiRequest = new RestRequest($"product/tags/{HttpUtility.UrlPathEncode(string.Join("+", tags))}");
This produces a path equal to:
/product/tags/for%2Bfamilies
On another note, do NOT build a dynamic query based on a user's inputs. You SHOULD always use a SqlParameter. Also, it is extremely important from a security perspective to return the values with the appropriate encoding to prevent injection attacks.
I have a TextBox in an Asp.net form. there is a simple javascript witch separates each three digits in TextBox. it works fine when you enter data in TextBox. I used coma , for separating digits and used dot . as floating point character.
as I said every thing works fine when I am entering data in TextBox. but when the post-back occurs and saved data returns to client, every .(s) has been removed (for example 2.3 saved as 23 and digits in TextBox are separated by . instead of ,.
this problem occurs just in a specific server (windows server 2003 sp1) and works fine in other windows server 2003 (SP1)! I am experiencing this problem for first time!
But I think the problem is because of specific Regional & Language Options in the server. This server is joined to a domain controller. when I change the regional and language options to this set:
Decimal Symbol -> .
Digit Grouping Symbol -> ,
nothing changes.
when I check the following item after customizing settings :
Apply All Settings to the current user account and to the default user profile -> checked
when I restart the Server, It jumps out from domain and need to be re-joined to domain controller! and of-course nothing changes again!
Do you had this problem? any solution please!
I can not post code here, because the code is too complex and I am sure problem is not because of code because it is working every where unless the specified server.
EDIT
Also setting regional and language options for network service user may help to solve the problem. any body knows how can I do this ?
Have you tried using the globalization tag in your web.config? This prevents you from running into trouble when multiple servers are configured differently (ie. different languagepacks).
<configuration>
<system.web>
<globalization
culture="en-US"
uiCulture="en-US" />
</system.web>
</configuration>
After goofing around with a similar problem for WAY to long I did the following with the help of a number of clues (also found on StackOverFlow, StackOverFlow rocks by the way...)
The first thing I did was dump out what the server was actually thinking (Page_Load):
var dtInfo = System.Globalization.DateTimeFormatInfo.CurrentInfo;
DisplayDebugInfo(String.Format(
"Culture({0}/{1}), DateFormat(SD:{2},DS:{3})",
System.Globalization.CultureInfo.CurrentCulture.Name,
System.Globalization.CultureInfo.CurrentUICulture.Name,
dtInfo.ShortDatePattern, dtInfo.DateSeparator));
Also on Windows 2003, I tried fixing the regional setting via the regular control panel but with no success.
I also tried setting the globalization settings in the web.config as mentioned in the other solution but with little effect.
It seems that once you start messing with the regional setting you can quickly get to the point where things are messed up. I decided to avoid messing with the registry and go for a code solution because then I would not have to worry when my code was released to production.
I added the following code to the base class for my page so that it would fix it everywhere. You could also place it in the Page_Load.
using System.Globalization;
using System.Threading;
// Fix the cultural settings...
CultureInfo culture = (CultureInfo)CultureInfo.CurrentCulture.Clone();
culture.DateTimeFormat.ShortDatePattern = "MM/dd/yyyy";
culture.DateTimeFormat.DateSeparator = "/";
Thread.CurrentThread.CurrentCulture = culture;
Problem solved. For me anyway.
When i visit http://localhost:17357/u/a%2fa/m/ssd-10 and look at HttpContext.Current.Request.Url in Application_BeginRequest i see http://localhost:17357/u/a/a/m/ssd-10 huh? shouldnt i get http://localhost:17357/u/a%2fa/m/ssd-10? i thought the point of escaping urls is so ?, &, / and other special symbols not be confused with their special meaning in urls. Maybe theres a config i need to tweak?
I created 4 usernames, there are
a?#!&ee
a?#!/&ee
as d
ｸｲﾝ
with the links as
a?#!&ee<br>
a?#!/&ee<br>
as d<br>
ｸｲﾝ
The last two work, but the first two i get the exceptio
A first chance exception of type 'System.ArgumentException' occurred in mscorlib.dll
Additional information: Illegal characters in path.
then
A first chance exception of type 'System.Web.HttpException' occurred in System.Web.dll
Additional information: '/u/a?#!&ee' is not a valid virtual path.
and my page says Bad Request. How can i allow these usernames to work. If its impossible how can i write a workaround?
You need to escape it again. Use %252f instead of %2f. To clarify, the URL is unencoded when the server receives it. URL encoding allows you to pass in a / that the server processes as a character instead of the special function that a reserved character would normally trigger. See the Wikipedia page for more info.
Concerning your error with the a?#!&ee username, it seems almost certain that you're running into a problem that ASP.NET has with special characters (even urlencoded properly) that are not in the query string (that is, after the ? part of the URL). Joshua Flanagan talks about it in a blog post, and identifies %, &, *, and : as the problematic characters.
He points to a Dirk.Net blog post that offers a couple of fixes. First, you can edit the registry to allow restricted characters (adding a DWORD key AllowRestrictedChars to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters and setting its boolean value to true). Or, you can ensure that you have the .NET framework 1.1 SP1 and edit the registry to set ASP.NET VErification Compatibility to true (DWORD HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET VerificationCompatibility = 1). Third, you can try setting ValidateRequest to false on the ASPX page. Finally, as Joshua decided to do, you can pass the information using the query string, i.e. after the ? as ASP.Net originally (pre MVC) expected.
I wrote my own solution. Its nice to have a username with / but not consider as / when getting a GET request.