Libcurl offers CURLOPT_HEADERFUNCTION and CURLOPT_WRITEFUNCTION callbacks. That's great until you use pipelining and multistack. How do you correlate the header with the body? Let's say tons of requests and bunch of easy handles cause libcurl to establish multiple connections to the server. Let's assume first response header arrives, and there is a delay in receiving the body. In the mean time, second header shows up along with the body. Does libcurl ensure that the second header is not delivered to the application until first response is complete?
This is important because header needs to be associated with the body. I am in the same predicament even when I don't use HEADERFUNCTION. Even if I use just the WRITEFUNCTION, it could receive the replies out of order in a mixed fashion. So the question is: Does libcurl ensure that the responses are delivered as a whole? If it's a single connection, we can be sure that the response order will follow request order. But I see libcurl making multiple connections when I use pipeling and multistack. Let's say 5 connections are made to same server because we are talking about Pipelining here. Response header for Conn1 arrives, Before we get the body from Conn1, we get the ResponseHeader from Conn2. Does LibCurl ensure that the Conn2ResponseHeader is not delivered to the application before BodyFromConn1? Otherwise following code will break.
class CEasyHandle
{
CURL* m_pCurl;
bool m_bInUse;
};
class CMultiStack
{
public:
CURLM* m_pCurlMulti;
deque<CEasyHandle*>& m_listEasyHandles;
static CEasyHandle* gpCurrentlyReceivingEasyHandle;
CEasyHandle* GetAvailableEasyHandle()
{
// Iterate through m_listEasyHandles and find one that is currently not added to multistack (m_bInUse)
// if none free, return NULL
}
bool MakeRequest(const char* pUrl)
{
CEasyHandle* pEasyHandle = GetAvailableEasyHandle();
if(!pEasyHandle) pEasyHandle = CreateNewEasyHandleAndAddToList();
curl_easy_setopt(pEasyHandle->m_pCurl, CURLOPT_HEADERFUNCTION, header_callback);
curl_easy_setopt(pEasyHandle->m_pCurl, CURLOPT_HEADERDATA, pEasyHandle); // header gets the EasyHandle
curl_easy_setopt(pEasyHandle->m_pCurl, CURLOPT_WRITEFUNCTION, write_callback);
curl_easy_setopt(pEasyHandle->m_pCurl, CURLOPT_WRITEDATA, this); // body gets MultiStack
// set options, add to multistack, pEasyHandle->m_bInUse = true;
}
static size_t header_callback(char *buffer, size_t size, size_t nmemb, void *userdata)
{
gpCurrentlyReceivingEasyHandle = (CEasyHandle*)userdata;
// if no data expected, of course set gpCurrentlyReceivingEasyHandle->m_bInUse = false;
}
static size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata)
{
CMultiStack* pThisObj = (CMultiStack*)userdata;
pThisObj->PerformSomeWork();
// once complete, gpCurrentlyReceivingEasyHandle->m_bInUse = false;
}
};
Why don't use case CEasyHandle as the user data to the WRITEFUNCTION? And store a back-pointer to the CMultiStack in each CEasyHandle when you create them? Then you can always find any of the pieces you need in the write_callback, and you don't need to worry about the order.
Related
I'm working with multi-program UDP MPEG-2 TS streams that, -unfortunately- dynamically re-map their elementary stream PIDs at random intervals. The stream is being demuxed using Microsoft's MPEG-2 demultiplexer filter.
I'm using the PSI-Parser filter (an example filter included in the DirectShow base classes) in order to react to the PAT/PMT changes.
The code is properly reacting to the change, yet I am experiencing some odd crashes (heap memory corruption) right after I remap the Demuxer pins to their new ID's. (The re-mapping is performed inside the thread that is processing graph events, while the EC_PROGRAMCHANGED message is being processed).
The crash could be due to faulty code in my part, yet I have not found any reference that tells me if changing the pin PID mapping is safe while the graph is running.
Can anyone provide some info if this is operation is safe, and if it is not, what could I do to minimize capture disruption?
I managed to find the source code for a Windows CE version of the demuxer filter. Inspecting it, indeed, it seems that it is safe to remap a pin while the filter is running.
I also managed to find the source of my problems with the PSI-Parser filter.
When a new transport stream is detected, or the PAT version changes, the PAT is flushed, (all programs are removed, the table is re-parsed and repopulated).
There is a subtle bug within the CPATProcessor::flush() method.
//
// flush
//
// flush an array of struct: m_mpeg2_program[];
// and unmap all PMT_PIDs pids, except one: PAT
BOOL CPATProcessor::flush()
{
BOOL bResult = TRUE;
bResult = m_pPrograms->free_programs(); // CPrograms::free_programs() call
if(bResult == FALSE)
return bResult;
bResult = UnmapPmtPid();
return bResult;
}// flush
Here's the CPrograms::free_programs() implementation.
_inline BOOL free_programs()
{
for(int i= 0; i<m_ProgramCount; i++){
if(!HeapFree(GetProcessHeap(), 0, (LPVOID) m_programs[i] ))
return FALSE;
}
return TRUE;
}
The problem here is that the m_ProgramCount member is never cleared. So, -apart from reporting the wrong number of programs in the table after a flush (since it is updated incrementally for each program found in the table)-, the next time the table is flushed, it will try to release memory that was already released.
Here's my updated version that fixes the heap corruption errors:
_inline BOOL free_programs()
{
for(int i= 0; i<m_ProgramCount; i++){
if(!HeapFree(GetProcessHeap(), 0, (LPVOID) m_programs[i] ))
return FALSE;
}
m_ProgramCount = 0; // This was missing, next call will try to free memory twice
return TRUE;
}
Although Nginx is a really interesting piece of software, the lack of documentation is making me crazy.
Goal: capture the whole response body, which would be logged on the server.
Problem: I have always a single buffer which size is ZERO.
Approach
I would expect to be able to accomplish this requirement with a body filter, which would "wait" for last_buf before iterate the full buffers chain.
/**
* #param ngx_http_request_t *r HTTP request
* #param ngx_chain_t *in Buffer chain
*/
static ngx_int_t
create_response_snapshot(ngx_http_request_t *r, ngx_chain_t *in)
{
ngx_chain_t *chain = NULL;
int chain_contains_last_buffer = 0;
size_t buffer_size = 0;
// check if body is complete
chain = in;
for ( ; ; )
{
if (chain->buf->last_buf)
{
chain_contains_last_buffer = 1;
}
if (NULL == chain->next)
break;
chain = chain->next;
}
if (0 == chain_contains_last_buffer)
{
// response is not complete
return ngx_http_next_body_filter(r, in);
}
// Response Content-Length
ngx_log_error(NGX_LOG_ALERT,r->connection->log,0,"Content-Length: %d",
r->headers_out.content_length_n);
// lets iterate buffers chain
for (chain = in; NULL != chain; chain = chain->next)
{
buffer_size = ngx_buf_size(chain->buf);
ngx_log_error(NGX_LOG_ALERT,r->connection->log,0,"buffer_size#%d",buffer_size);
}
return ngx_http_next_body_filter(r, in);
}
My comment got too big to be a comment, but I don't feel like it's a proper answer - oh well.
To re-iterate, the problem with the code you've posted is that your module's body filter function won't be called on the whole chain at once. It gets called on the first piece, then the second piece, until the nth piece. Finally it gets called on a completely empty chain, for whatever reason the buf with last_buf = 1 is always by itself and empty.
So I think what you want to do is "dam" the flow of buffers by accumulating them in your module without releasing any to the next filter until you have all of them at once.
Check out the substitution filter module: http://lxr.nginx.org/source//src/http/modules/ngx_http_sub_filter_module.c
It uses a "busy" chain which is what I was referring to. From what I've been able to tell it uses it to keep track of which buffers have actually been sent (when this happens the size gets set to zero) and adds those to the module context's free list for re-use. See ngx_http_sub_output on line 438 for this behavior.
My suggestion was to do something like what that module does, except without calling the next filter until you have the entire page. You can't call next_filter if you want to process the entire page as a whole, since doing that will result in data getting sent to the client. Again this runs counter to Nginx's design, so I think you should find an alternative that doesn't require the whole response body at once if you can.
What tools or techniques can I use to protect my ASP.NET web application from Denial Of Service attacks
For sure a hardware solution is the best option to prevent DOS attacks, but considering a situation in which you have no access to hardware config or IIS settings, this is definitely why a developer must have something handy to block or at least decrease dos attack effect.
The core concept of logic relies on a FIFO (First In First Out) collection such as Queue, but as it has some limitations I decided to create my own collection.
Without discussing more details this is the complete code I use:
public class AntiDosAttack
{
readonly static List<IpObject> items = new List<IpObject>();
public static void Monitor(int Capacity, int Seconds2Keep, int AllowedCount)
{
string ip = HttpContext.Current.Request.UserHostAddress;
if (ip == "")
return;
// This part to exclude some useful requesters
if(HttpContext.Current.Request.UserAgent != null && HttpContext.Current.Request.UserAgent == "Some good bots")
return;
// to remove old requests from collection
int index = -1;
for (int i = 0; i < items.Count; i++)
{
if ((DateTime.Now - items[i].Date).TotalSeconds > Seconds2Keep)
{
index = i;
break;
}
}
if (index > -1)
{
items.RemoveRange(index, items.Count - index);
}
// Add new IP
items.Insert(0, new IpObject(ip));
// Trim collection capacity to original size, I could not find a better reliable way
if (items.Count > Capacity)
{
items.RemoveAt(items.Count - 1);
}
// Count of currect IP in collection
int count = items.Count(t => t.IP == ip);
// Decide on block or bypass
if (count > AllowedCount)
{
// alert webmaster by email (optional)
ErrorReport.Report.ToWebmaster(new Exception("Blocked probable ongoing ddos attack"), "EvrinHost 24 / 7 Support - DDOS Block", "");
// create a response code 429 or whatever needed and end response
HttpContext.Current.Response.StatusCode = 429;
HttpContext.Current.Response.StatusDescription = "Too Many Requests, Slow down Cowboy!";
HttpContext.Current.Response.Write("Too Many Requests");
HttpContext.Current.Response.Flush(); // Sends all currently buffered output to the client.
HttpContext.Current.Response.SuppressContent = true; // Gets or sets a value indicating whether to send HTTP content to the client.
HttpContext.Current.ApplicationInstance.CompleteRequest(); // Causes ASP.NET to bypass all events and filtering in the HTTP pipeline chain of execution and directly execute the EndRequest event.
}
}
internal class IpObject
{
public IpObject(string ip)
{
IP = ip;
Date = DateTime.Now;
}
public string IP { get; set; }
public DateTime Date { get; set; }
}
}
The internal class is designed to keep the date of request.
Naturally DOS Attack requests create new sessions on each request while human requests on a website contain multiple requests packed in one session, so the method can be called in Session_Start.
usage:
protected void Session_Start(object sender, EventArgs e)
{
// numbers can be tuned for different purposes, this one is for a website with low requests
// this means: prevent a request if exceeds 10 out of total 30 in 2 seconds
AntiDosAttack.Monitor(30, 2, 10);
}
for a heavy request website you may change seconds to milliseconds but consider the extra load caused by this code.
I am not aware if there is a better solution to block intentional attacks on website, so I appreciate any comment and suggestion to improve the code. By then I consider this as a best practice to prevent DOS attacks on ASP.NET websites programmatically.
Try the Dynamic IP Restriction extension http://www.iis.net/download/dynamiciprestrictions
Not a perfect solution, but helps raise the bar =)
It's a broad area, so if you can be more specific about your application, or the level of threat you're trying to protect against, I'm sure more people can help you.
However, off the bat, you can go for a combination of a caching solution such as Squid: http://www.blyon.com/using-squid-proxy-to-fight-ddos/, Dynamic IP Restriction (as explained by Jim) and if you have the infrastructure, an active-passive failover setup, where your passive machine serves placeholder content which doesnt hit your database / any other machines. This is last-defence, so that you minimise the time a DDOS might bring your entire site offline for.
In order to support a legacy application that's in the field, I need my ASP.NET MVC app to return an empty response that also has a Content-Type. One of IIS, ASP.NET, or ASP.NET MVC is removing my Content-Type when I send back a null response. Is there any way around this?
(While not requiring an empty response with a set Content-Type would obviously be the ideal solution, the clients are already out there, and many of them cannot be upgraded.)
EDIT: Since there was a request for code: I'm proxying the request from the new web application to the one that older clients rely on. To do this, I have a subclass of ActionResult, called LegacyResult, that you can simply return for those methods that need to be handled by the old software. This is the relevant part of its code:
public override void ExecuteResult(ControllerContext context)
{
using (var legacyResponse = GetLegacyResponse(context))
{
var clientResponse = context.HttpContext.Response;
clientResponse.Buffer = false;
clientResponse.ContentType = legacyResponse.ContentType; /* Yes, I checked that legacyResponse.ContentType is never string.IsNullOrEmpty */
if (legacyResponse.ContentLength >= 0) clientResponse.AddHeader("Content-Length", legacyResponse.ContentLength.ToString());
var legacyInput = legacyResponse.GetResponseStream();
using (var clientOutput = clientResponse.OutputStream)
{
var rgb = new byte[32768];
int cb;
while ((cb = legacyInput.Read(rgb, 0, rgb.Length)) > 0)
{
clientOutput.Write(rgb, 0, cb);
}
clientOutput.Flush();
}
}
}
If legacyInput has data, then Content-Type is set appropriately. Otherwise, it's not. I can actually kluge the old backend to send an empty v. non-empty response for exactly the same request, and observe the difference in Fiddler.
EDIT 2: Poking around with Reflector reveals that, if headers have not been written at the time that HttpResponse.Flush is called, then Flush writes out the headers itself. The problem is that it only writes out a tiny subset of the headers. One of the missing ones is Content-Type. So it seems that, if I can force headers out to the stream, I can avoid this problem.
You have to trick the response into writing the headers, by falsely telling it there's content, then suppressing it:
/// [inside the writing block]
var didWrite = false;
while ((cb = legacyInput.Read(rgb, 0, rgb.Length)) > 0)
{
didWrite = true;
clientOutput.Write(rgb, 0, cb);
}
if (!didWrite)
{
// The stream needs a non-zero content length to write the correct headers, but...
clientResponse.AddHeader("Content-Length", "1");
// ...this actually writes a "Content-Length: 0" header with the other headers.
clientResponse.SuppressContent = true;
}
I need to load an external web (not local) page into my site (some link), but only a part of it.
What are the options for doing so?
That depends on whether or not the external page is local, or on a different domain. If it's local, you can use $.load() in the jQuery library. This has an optional parameter to specify which element in the remote-dom to load it:
$("#links").load("/Main_Page #jq-p-Getting-Started li");
If the page is on another domain, you'll need a proxy script. You can do this with PHP and the phpQuery (php port of jQuery) library. You'll just use file_get_contents() to get the actual remote-dom, and then pull out the elements you want based on jQuery-like selectors.
$f = fopen('http://www.quran.az/2/255', 'r');
and so on...
Once you get the whole page as Michael Todd outlined, you will likely need to either use substring methods for a static means to slice up the content or you can use regex's for a more dynamic way to grab the content. An intro article on Regex's in ASP.Net can be found here. Good luck!
To load a web page in .Net, use the HttpWebRequest class.
Example taken from MSDN, here:
private string StringGetWebPage(String uri)
{
const int bufSizeMax = 65536; // max read buffer size conserves memory
const int bufSizeMin = 8192; // min size prevents numerous small reads
StringBuilder sb;
// A WebException is thrown if HTTP request fails
try
{
// Create an HttpWebRequest using WebRequest.Create (see .NET docs)!
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
// Execute the request and obtain the response stream
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
// Content-Length header is not trustable, but makes a good hint.
// Responses longer than int size will throw an exception here!
int length = (int)response.ContentLength;
// Use Content-Length if between bufSizeMax and bufSizeMin
int bufSize = bufSizeMin;
if (length > bufSize)
bufSize = length > bufSizeMax ? bufSizeMax : length;
// Allocate buffer and StringBuilder for reading response
byte[] buf = new byte[bufSize];
sb = new StringBuilder(bufSize);
// Read response stream until end
while ((length = responseStream.Read(buf, 0, buf.Length)) != 0)
sb.Append(Encoding.UTF8.GetString(buf, 0, length));
}
catch (Exception ex)
{
sb = new StringBuilder(ex.Message);
}
return sb.ToString();
}
Note that this will return the entire page and not just a portion of it. You'll then need to sift through the page to find the information you're looking for.