I'm working with multi-program UDP MPEG-2 TS streams that, -unfortunately- dynamically re-map their elementary stream PIDs at random intervals. The stream is being demuxed using Microsoft's MPEG-2 demultiplexer filter.
I'm using the PSI-Parser filter (an example filter included in the DirectShow base classes) in order to react to the PAT/PMT changes.
The code is properly reacting to the change, yet I am experiencing some odd crashes (heap memory corruption) right after I remap the Demuxer pins to their new ID's. (The re-mapping is performed inside the thread that is processing graph events, while the EC_PROGRAMCHANGED message is being processed).
The crash could be due to faulty code in my part, yet I have not found any reference that tells me if changing the pin PID mapping is safe while the graph is running.
Can anyone provide some info if this is operation is safe, and if it is not, what could I do to minimize capture disruption?
I managed to find the source code for a Windows CE version of the demuxer filter. Inspecting it, indeed, it seems that it is safe to remap a pin while the filter is running.
I also managed to find the source of my problems with the PSI-Parser filter.
When a new transport stream is detected, or the PAT version changes, the PAT is flushed, (all programs are removed, the table is re-parsed and repopulated).
There is a subtle bug within the CPATProcessor::flush() method.
//
// flush
//
// flush an array of struct: m_mpeg2_program[];
// and unmap all PMT_PIDs pids, except one: PAT
BOOL CPATProcessor::flush()
{
BOOL bResult = TRUE;
bResult = m_pPrograms->free_programs(); // CPrograms::free_programs() call
if(bResult == FALSE)
return bResult;
bResult = UnmapPmtPid();
return bResult;
}// flush
Here's the CPrograms::free_programs() implementation.
_inline BOOL free_programs()
{
for(int i= 0; i<m_ProgramCount; i++){
if(!HeapFree(GetProcessHeap(), 0, (LPVOID) m_programs[i] ))
return FALSE;
}
return TRUE;
}
The problem here is that the m_ProgramCount member is never cleared. So, -apart from reporting the wrong number of programs in the table after a flush (since it is updated incrementally for each program found in the table)-, the next time the table is flushed, it will try to release memory that was already released.
Here's my updated version that fixes the heap corruption errors:
_inline BOOL free_programs()
{
for(int i= 0; i<m_ProgramCount; i++){
if(!HeapFree(GetProcessHeap(), 0, (LPVOID) m_programs[i] ))
return FALSE;
}
m_ProgramCount = 0; // This was missing, next call will try to free memory twice
return TRUE;
}
Related
I am trying to consume a max of 1000 messages from kafka at a time. (I am doing this because i need to batch insert into MSSQL.) I was under the impression that kafka keeps an internal queue which fetches messages from the brokers and when i use the consumer.consume() method it just checks if there are any messages in the internal queue and returns if it finds something. otherwise it just blocks until the internal queue is updated or until timeout.
I tried to use the solution suggested here: https://github.com/confluentinc/confluent-kafka-dotnet/issues/1164#issuecomment-610308425
but when i specify TimeSpan.Zero (or any other timespan up to 1000ms) the consumer never consumes any messages. but if i remove the timeout it does consume messages but then i am unable to exit the loop if there are no more messages left to be read.
I also saw an other question on stackoverflow which suggested to read the offset of the last message sent to kafka and then read messages until i reach that offset and then break from the loop. but currently i only have one consumer and 6 partitions for a topic. I haven't tried it yet but i think managing offsets for each of the partition might make the code messy.
Can someone please tell me what to do?
static List<RealTime> getBatch()
{
var config = new ConsumerConfig
{
BootstrapServers = ConfigurationManager.AppSettings["BootstrapServers"],
GroupId = ConfigurationManager.AppSettings["ConsumerGroupID"],
AutoOffsetReset = AutoOffsetReset.Earliest,
};
List<RealTime> results = new List<RealTime>();
List<string> malformedJson = new List<string>();
using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
{
consumer.Subscribe("RealTimeTopic");
int count = 0;
while (count < batchSize)
{
var consumerResult = consumer.Consume(1000);
if (consumerResult?.Message is null)
{
break;
}
Console.WriteLine("read");
try
{
RealTime item = JsonSerializer.Deserialize<RealTime>(consumerResult.Message.Value);
results.Add(item);
count += 1;
}
catch(Exception e)
{
Console.WriteLine("malformed");
malformedJson.Add(consumerResult.Message.Value);
}
}
consumer.Close();
};
Console.WriteLine(malformedJson.Count);
return results;
}
I found a workaround.
For some reason the consumer first needs to be called without a timeout. That means it will wait for a message until it gets at least one. after that using consume with timeout zero fetches all the rest of the messages one by one from the internal queue. this seems to work out for the best.
I had a similar problem, updating the Confluent.Kafka and lidrdkafka libraries from version 1.8.2 to 2.0.2 helped
Although Nginx is a really interesting piece of software, the lack of documentation is making me crazy.
Goal: capture the whole response body, which would be logged on the server.
Problem: I have always a single buffer which size is ZERO.
Approach
I would expect to be able to accomplish this requirement with a body filter, which would "wait" for last_buf before iterate the full buffers chain.
/**
* #param ngx_http_request_t *r HTTP request
* #param ngx_chain_t *in Buffer chain
*/
static ngx_int_t
create_response_snapshot(ngx_http_request_t *r, ngx_chain_t *in)
{
ngx_chain_t *chain = NULL;
int chain_contains_last_buffer = 0;
size_t buffer_size = 0;
// check if body is complete
chain = in;
for ( ; ; )
{
if (chain->buf->last_buf)
{
chain_contains_last_buffer = 1;
}
if (NULL == chain->next)
break;
chain = chain->next;
}
if (0 == chain_contains_last_buffer)
{
// response is not complete
return ngx_http_next_body_filter(r, in);
}
// Response Content-Length
ngx_log_error(NGX_LOG_ALERT,r->connection->log,0,"Content-Length: %d",
r->headers_out.content_length_n);
// lets iterate buffers chain
for (chain = in; NULL != chain; chain = chain->next)
{
buffer_size = ngx_buf_size(chain->buf);
ngx_log_error(NGX_LOG_ALERT,r->connection->log,0,"buffer_size#%d",buffer_size);
}
return ngx_http_next_body_filter(r, in);
}
My comment got too big to be a comment, but I don't feel like it's a proper answer - oh well.
To re-iterate, the problem with the code you've posted is that your module's body filter function won't be called on the whole chain at once. It gets called on the first piece, then the second piece, until the nth piece. Finally it gets called on a completely empty chain, for whatever reason the buf with last_buf = 1 is always by itself and empty.
So I think what you want to do is "dam" the flow of buffers by accumulating them in your module without releasing any to the next filter until you have all of them at once.
Check out the substitution filter module: http://lxr.nginx.org/source//src/http/modules/ngx_http_sub_filter_module.c
It uses a "busy" chain which is what I was referring to. From what I've been able to tell it uses it to keep track of which buffers have actually been sent (when this happens the size gets set to zero) and adds those to the module context's free list for re-use. See ngx_http_sub_output on line 438 for this behavior.
My suggestion was to do something like what that module does, except without calling the next filter until you have the entire page. You can't call next_filter if you want to process the entire page as a whole, since doing that will result in data getting sent to the client. Again this runs counter to Nginx's design, so I think you should find an alternative that doesn't require the whole response body at once if you can.
I have an application which plays video using DirectShow.Net and also records video. When I try to record to a filename that has just been used for playback by DirectShow it fails as the file is still in use. Sometimes it will work but take anywhere from 5 - 60 seconds until the file is unlocked. Before the recording is attempted the playback graph has definitely been destroyed. The code for creating and destroying the graph is below. If I stop and start my application following playback I can record to the same filename and there are no file locking issues.
Can anyone advise how I can correctly release the source filter so the file is not locked?
Creating the graph
try
{
graphBuilder = (IGraphBuilder)new FilterGraph();
#if DEBUG
// "Connect to remote graph" in GraphEdit
rotEntry = new DsROTEntry(graphBuilder);
#endif
hr = graphBuilder.AddSourceFilter(filename, filename, out baseFilter);
DsError.ThrowExceptionForHR(hr);
vmr9 = (IBaseFilter)new VideoMixingRenderer9();
ConfigureVMR9InWindowlessMode();
hr = graphBuilder.AddFilter(vmr9, "Video Mixing Renderer 9");
DsError.ThrowExceptionForHR(hr);
FilterGraphTools.ConnectFilters(graphBuilder, baseFilter, "Output", vmr9, "VMR Input0", true);
}
Destroying the graph
if (vmr9 != null)
{
Marshal.ReleaseComObject(vmr9);
vmr9 = null;
windowlessCtrl = null;
}
if (graphBuilder != null)
{
// Remove and release all filters
FilterGraphTools.RemoveAllFilters(graphBuilder);
Marshal.ReleaseComObject(graphBuilder);
graphBuilder = null;
baseFilter = null;
}
#if DEBUG
if (rotEntry != null)
{
rotEntry.Dispose();
rotEntry = null;
}
#endif
Eventually the graph is a set of connected COM objects, and successful graph termination depends on correct release, without any leaked references. References that you might have left unreleased are leading to objects kept alive and possibly locking certain resources.
The best you can do is explicit termination/removal of individual objects:
Stop the graph
Remove all fitlers explicitly using IFilterGraph2.RemoveFilter
Use filter dependent methods calls to possibly terminate individual filters, such as by putting empty path to file source/sink filters
If even leak takes place, the graph should no longer reference resources. Note that you sometimes can also reuse filters if you are re-creating the graph.
My asp.net(c#) method looks as follows:
static public bool GetVideoLength(string fileName, out double length)
{
DirectShowLib.FilterGraph graphFilter = new DirectShowLib.FilterGraph();
DirectShowLib.IGraphBuilder graphBuilder;
DirectShowLib.IMediaPosition mediaPos;
length = 0.0;
try
{
graphBuilder = (DirectShowLib.IGraphBuilder)graphFilter;
graphBuilder.RenderFile(fileName, null);
mediaPos = (DirectShowLib.IMediaPosition)graphBuilder;
mediaPos.get_Duration(out length);
return true;
}
catch
{
return false;
}
finally
{
mediaPos = null;
graphBuilder = null;
graphFilter = null;
}
}
I got the duration with the above method. But my problem is i can't delete the physical file
after my operation. I used
File.Delete(FilePath);
While performing this action i got an exception as follows:
"The process cannot access the file because it is being used by another process."
My Os is windows 7(IIS 7)
Any one please help me to sort this out?
I've got no experience in coding directshow apps in C#, but plenty of experience in C++.
DirectShow is based on a technology called COM - which uses reference counting to tell it when an object is in use.
It would use a COM object to represent the IGraphBuilder for example.
In C++, we would have to deconstruct the graph, by removing all its filters, then release the graph.
I understand that C# has its own garbage collection etc., but unless you explicitly release the objects you use, they'll remain in memory.
It seems from the code you've quoted, that the graph is still opened, even though playback may have finished. In that case, it'll hold a reference to the file which you've played back, which would explain why you can't delete it - e.g. there's a read lock on the file.
Hope this points you in the right direction!
I recently started this question in another thread (to which Reed Copsey
graciously responded) but I don't feel I framed the question well.
At the core of my question, I would like an illustration of how to gain
access to data AS it is being get/set.
I have Page.aspx.cs and, in the codebehind, I have a loop:
List<ServerVariable> files = new List<ServerVariable>();
for (i = 0; i <= Request.Files.Count - 1; i++)
{
m_objFile = Request.Files[i];
m_strFileName = m_objFile.FileName;
m_strFileName = Path.GetFileName(m_strFileName);
files.Add(new ServerVariable(i.ToString(),
this.m_strFileName, "0"));
}
//CODE TO COPY A FILE FOR UPLOAD TO THE
//WEB SERVER
//WHEN THE UPLOAD IS DONE, SET THE ITEM TO
//COMPLETED
int index = files.FindIndex(p => p.Completed == "0");
files[index] = new ServerVariable(i.ToString(),
this.m_strFileName, "1");
The "ServerVariable" type gets and sets ID, File, and Completed.
Now, I need to show the user the file upload "progress" (in effect,
the time between when the loop adds the ServerVariable item to the
list to when the Completed status changes from 0 to 1.
Now, I have a web service method "GetStatus()" that I would like to
use to return the files list (created above) as a JSON string (via
JQuery). Files with a completed status of 0 are still in progress,
files with a 1 are done.
MY QUESTION IS - what does the code inside GetStatus() look like? How
do I query List **as* it is being populated and
return the results real-time? I have been advised that I need to lock
the working process (setting the ServerVariable data) while I query
the values returned in GetStatus() and then unlock that same process?
If I have explained myself well, I'd appreciate a code illustration of
the logic in GetStatus().
Thanks for reading.
Have a look at this link about multi threading locks.
You need to lock the object in both read and write.