I know that stacks are per thread, but registers are limited,like on IA32,only 8 registers.
So how are registers shared among threads?
Registers are used by the CPU when it's currently running a particular thread. When the OS decides to switch from one thread to another, the OS saves the current values of all the register into a private memory area specific to the first thread. Before the second thread starts running, the OS loads the values of all the registers from its saved area. This is called a context switch.
Related
I am teaching myself OS by going through the lecture notes of the course at IIT Bombay (https://www.cse.iitb.ac.in/~mythili/os/). One of the questions in the Process worksheet asks which of the following doesn't always happen in the situation described at the title. The answer is C.
A. The process moves to kernel mode.
B. The program counter of the CPU shifts to the kernel part of the address space.
C. The process is context-switched out and a separate kernel process starts execution.
D. The OS code that deals with handling TCP/IP packets is invoked
I'm a bit confused though. I thought when an interrupt routine occurs the process is context-switched out so other processes can run and the CPU is not idle during that time. The kernel, then, will take care of the packet sending. Why would C not be correct then?
You are right in saying that "when an interrupt routine occurs the process is context-switched out so other processes can run and the CPU is not idle during that time", but the words "generally or mostly" need to be added to it.
In most cases, there is another process waiting for CPU time and that can be scheduled. However it is not the case 100% of the time. The question is about the word "always" and while other options always occur in the given situation, option C is a choice that OS makes at run time. If OS determines that switching out this process can be sub optimal than performing the system call and resuming the same process, then it may not perform the context switching.
There is a cost associated with context switching and if other processes are also blocked on some I/O then it may be optimal for OS to NOT switch the context or there might be other reasons to not switch the context such as what if only 1 process is running, there is no other process to switch the context to!
Suppose we invoke a system call for asynchronous IO. At the time of invoking system call, the mode changes from user mode to kernel mode . After invocation, the mode should immediately change back to user mode so that user application can proceed further(as it is non blocking).
Now if the mode is changed to user mode then how will kernel proceed with IO as mode is changed from kernel to user mode ? Will kernel perform asynchronous IO in user mode ?
IO means two different things (at two different levels of abstractions):
from an application point of view, from a process running in user-mode, calling any system call (listed in syscalls(2) for Linux) related to input or output, e.g. read(2), .... Notice that aio_read(3) is not listed as a system call (it is some library function using other system calls, see aio(7)).
on the raw hardware, any physical input or output operation sending data (or orders) to actual IO devices (e.g. SATA disks, USB ports, etc...)
Asynchronous or synchronous IO for a process means just calling some suitable subset of system calls, since system calls are the only way a process can interact with the kernel, and since in user-mode no physical IO is directly possible.
Read Operating Systems: Three Easy Pieces (freely downloadable) to get a better view of OSes.
Will kernel perform asynchronous IO in user mode ?
This shows some confusion. In practice, inside the kernel, physical IO is generally (and probably always) initiated by interrupt handlers (which might configure some DMA etc...). A hardware interrupt switches the processor to "kernel-mode" (actually supervisor mode of the ISA).
A blocking system call (e.g. read(2) when physical IO is needed since the data is not in the page cache) don't block the entire computer: it is just the calling process which becomes "blocked" so is rescheduled. The kernel will schedule some other runnable process. Much later, after having the kernel handle many interrupts, the blocked process will become runnable and could be rescheduled to run.
Processes are themselves (with files) one of the major abstractions (provided by the kernel) to application code.
In other words, at the conceptual level, the kernel scheduler is coded in some continuation-passing style.
See also kernelnewbies and OSDEV.
The asynchronous IO will be performed on behalf of the process, the kernel will handle it almost as usual while the process continues to run. In blocking mode, the process is just suspended.
Kernel has access to every process space, so he can fill/read data from process user space whatever a process is currently doing.
In a uniprocessor (UP) system, there's only one CPU core, so only one thread of execution can be happening at once. This thread of execution is synchronous (it gets a list of instructions in a queue and run them one by one). When we write code, it compiles to set of CPU instructions.
How can we have asynchronous behavior in software on a UP machine? Isn't everything just run in some fixed order chosen by the OS?
Even an out-of-order execution CPU gives the illusion of running instructions in program order. (This is separate from memory reordering observed by other cores or devices in the system. In a UP system, runtime memory reordering is only relevant for device drivers.)
An interrupt handler is a piece of code that runs asynchronously to the rest of the code, and can happen in response to an interrupt from a device outside the CPU. In user-space, a signal handler has equivalent semantics.
(Or a hardware interrupt can cause a context switch to another software thread. This is asynchronous as far as the software thread is concerned.)
Events like interrupts from network packets arriving or disk I/O completing happen asynchronously with respect to whatever the CPU was doing before the interrupt.
Asynchronous doesn't mean simultaneous, just that it can run between any two machine instructions of the rest of the code. A signal handler in a user-space program can run between any two machine instructions, so the code in the main program must work in a way that doesn't break if this happens.
e.g. A program with a signal-handler can't make any assumptions about data on the stack below the current stack pointer (i.e. in the un-reserved part of the stack). The red-zone in the x86-64 SysV ABI is a modification to this rule for user-space only, since the kernel can respect it when transferring control to a signal handler. The kernel itself can't use a red-zone, because hardware interrupts write to the stack outside of software control, before running the interrupt handler.
In an OS where I/O completion can result in the delivery of a POSIX signal (i.e. with POSIX async I/O), the timing of a signal can easily be determined by the timing of a hardware interrupts, so user-space code runs asynchronously with timing determined by things external to the computer. It's not just an issue for the kernel.
On a multicore system, there are obviously far more ways for things to happen in different orders more of the time.
Many processors are capable of multithreading, and many operating systems can simulate multithreading on single-threaded processors by swapping tasks in and out of the processor.
I'm trying to understand the difference between SGX threads enabled by TCS and untrusted threading provided by SDK.
If I understand correctly, TCS enables multiple logical processors to enter the same enclave. Each logical processor will have its own TCS and hence its own entry point (the OENTRY field in TCS). Each thread runs until an AEX happens or reaches the end of the thread. However, these threads enabled by TCS have no way to synchronize with each other yet. At least, there is no SGX instruction for synchronize.
Then, on the other hand, the SGX SDK offers a set of Thread Synchronization Primitives, mainly mutex and condition variable. These primitives are not trusted since they're eventually served by OS.
My question is, are these Thread Synchronization Primitives meant to be used by TCS threads? If so, wouldn't this deteriorate the security? The OS is able to play with scheduling as it wishes.
First, let us deal with your somewhat unclear terminology of
SGX threads enabled by TCS and untrusted threading provided by SDK.
Inside an enclave, only "trusted" threads can execute. There is no "untrusted" threading inside an enclave. Possibly, the following sentence in the SDK Guide [1] misled you:
Creating threads inside the enclave is not supported. Threads that run inside the enclave are created within the (untrusted) application.
The untrusted application has to set up the TCS pages (for more background on TCS see [2]). So how can the TCS set up by the untrusted application be the foundation for trusted threads inside the enclave? [2] gives the answer:
EENTER is only guaranteed to perform controlled jumps inside an enclave’s code if the contents of all the TCS pages are measured.
By measuring the TCS pages, the integrity of the threads (the TCS defines the allowed entry points) can be verified through enclave attestation. So only known-good execution paths can be executed within the enclave.
Second, let us look at the synchronization primitives.
The SDK does offer synchronization primitives, which you say are not to be trusted because they are eventually served by the OS. Lets look at the description of these primitives in [1]:
sgx_spin_lock() and unlock operate solely within the enclave (using atomic operations), with no need for OS interaction (no OCALL). Using a spinlock, you could yourself implement higher-level primitives.
sgx_thread_mutex_init() also does not make an OCALL. The mutex data structure is initialized within the enclave.
sgx_thread_mutex_lock() and unlock potentially perform OCALLS. However, since the mutex data is within the enclave, they can always enforce correctness of locking within the secure enclave.
Looking at the descriptions of the mutex functions, my guess is that the OCALLs serve to implement non-busy waiting outside the enclave. This is indeed handled by the OS, and susceptible to attacks. The OS may choose not to wake a thread waiting outside the enclave. But it can also choose to interrupt a thread running inside an enclave. SGX does not protect against DoS attacks (Denial of Service) from the (potentially compromised) OS.
To summarize, spin-locks (and by extension any higher-level synchronization) can be implemented securely inside an enclave. However, SGX does not protect against DoS attacks, and therefor you cannot assume that a thread will run. This also applies to locking primitives: a thread waiting on a mutex might not be awakened when the mutex is freed. Realizing this inherent limitation, the SDK designers chose to use (untrusted) OCALLs to efficiently implement some synchronization primitives (i.e. non-busy waiting).
[1] SGX SDK Guide
[2] SGX Explained
qweruiop, regarding your question in the comment (my answer is too long for a comment):
I would still count that as a DoS attack: the OS, which manages the resources of enclaves, denies T access to the resource CPU processing time.
But I agree, you do have to design the other threads running in that enclave with the awareness that T might never run. The semantics are different from running threads on a platform you control. If you want to be absolutely sure that the condition variable is checked, you have to do so on a platform you control.
The sgx_status_t returned by each proxy function (e.g. when making an ECALL into an enclave) can return SGX_ERROR_OUT_OF_TCS. So the SDK should handle all threading for you - just make ECALLs from two different ("untrusted") threads A and B outside the enclave, and the execution flow should continue in two ("trusted") threads inside the enclave, each bound to a separate TCS (assuming 2 unused TCS are available).
I'm trying to get around the concept of cooperative multitasking system and exactly how it works in a single threaded application.
My understanding is that this is a "form of multitasking in which multiple tasks execute by voluntarily ceding control to other tasks at programmer-defined points within each task."
So if you have a list of tasks and one task is executing, how do you determine to pass execution to another task? And when you give execution back to a previous task, how do resume from where you were previously?
I find this a bit confusing because I don't understand how this can be achieve without a multithreaded application.
Any advice would be very helpeful :)
Thanks
In your specific scenario where a single process (or thread of execution) uses cooperative multitasking, you can use something like Windows' fibers or POSIX setcontext family of functions. I will use the term fiber here.
Basically when one fiber is finished executing a chunk of work and wants to voluntarily allow other fibers to run (hence the "cooperative" term), it either manually switches to the other fiber's context or more typically it performs some kind of yield() or scheduler() call that jumps into the scheduler's context, then the scheduler finds a new fiber to run and switches to that fiber's context.
What do we mean by context here? Basically the stack and registers. There is nothing magic about the stack, it's just a block of memory the stack pointer happens to point to. There is also nothing magic about the program counter, it just points to the next instruction to execute. Switching contexts simply saves the current registers somewhere, changes the stack pointer to a different chunk of memory, updates the program counter to a different stream of instructions, copies that context's saved registers into the CPU, then does a jump. Bam, you're now executing different instructions with a different stack. Often the context switch code is written in assembly that is invoked in a way that doesn't modify the current stack or it backs out the changes, in either case it leaves no traces on the stack or in registers so when code resumes execution it has no idea anything happened. (Again, the theme: we assume that method calls fiddle with registers, push arguments to the stack, move the stack pointer, etc but that is just the C calling convention. Nothing requires you to maintain a stack at all or to have any particular method call leave any traces of itself on the stack).
Since each stack is separate, you don't have some continuous chain of seemingly random method calls eventually overflowing the stack (which might be the result if you naively tried to implement this scheme using standard C methods that continuously called each other). You could implement this manually with a state machine where each fiber kept a state machine of where it was in its work, periodically returning to the calling dispatcher's method, but why bother when actual fiber/co-routine support is widely available?
Also remember that cooperative multitasking is orthogonal to processes, protected memory, address spaces, etc. Witness Mac OS 9 or Windows 3.x. They supported the idea of separate processes. But when you yielded, the context was changed to the OS context, allowing the OS scheduler to run, which then potentially selected another process to switch to. In theory you could have a full protected virtual memory OS that still used cooperative multitasking. In those systems, if a errant process never yielded, the OS scheduler never ran, so all other processes in the system were frozen. **
The next natural question is what makes something pre-emptive... The answer is that the OS schedules an interrupt timer with the CPU to stop the currently executing task and switch back to the OS scheduler's context regardless of whether the current task cares to release the CPU or not, thus "pre-empting" it.
If the OS uses CPU privilege levels, the (kernel configured) timer is not cancelable by lower level (user mode) code, though in theory if the OS didn't use such protections an errant task could mask off or cancel the interrupt timer and hijack the CPU. There are some other scenarios like IO calls where the scheduler can be invoked outside the timer, and the scheduler may decide no other process has higher priority and return control to the same process without a switch... And in reality most OSes don't do a real context switch here because that's expensive, the scheduler code runs inside the context of whatever process was executing, so it has to be very careful not to step on the stack, to save register states, etc.
** You might ask why not just fire a timer if yield isn't called within a certain period of time. The answer lies in multi-threaded synchronization. In a cooperative system, you don't have to bother taking locks, worry about re-entrance, etc because you only yield when things are in a known good state. If this mythical timer fires, you have now potentially corrupted the state of the program that was interrupted. If programs have to be written to handle this, congrats... You now have a half-assed pre-emptive multitasking system. Might as well just do it right! And if you are changing things anyway, may as well add threads, protected memory, etc. That's pretty much the history of the major OSes right there.
The basic idea behind cooperative multitasking is trust - that each subtask will relinquish control, of its own accord, in a timely fashion, to avoid starving other tasks of processor time. This is why tasks in a cooperative multitasking system need to be tested extremely thoroughly, and in some cases certified for use.
I don't claim to be an expert, but I imagine cooperative tasks could be implemented as state machines, where passing control to the task would cause it to run for the absolute minimal amount of time it needs to make any kind of progress. For example, a file reader might read the next few bytes of a file, a parser might parse the next line of a document, or a sensor controller might take a single reading, before returning control back to a cooperative scheduler, which would check for task completion.
Each task would have to keep its internal state on the heap (at object level), rather than on the stack frame (at function level) like a conventional blocking function or thread.
And unlike conventional multitasking, which relies on a hardware timer to trigger a context switch, cooperative multitasking relies on the code to be written in such a way that each step of each long-running task is guaranteed to finish in an acceptably small amount of time.
The tasks will execute an explicit wait or pause or yield operation which makes the call to the dispatcher. There may be different operations for waiting on IO to complete or explicitly yielding in a heavy computation. In an application task's main loop, it could have a *wait_for_event* call instead of busy polling. This would suspend the task until it has input to process.
There may also be a time-out mechanism for catching runaway tasks, but it is not the primary means of switching (or else it wouldn't be cooperative).
One way to think of cooperative multitasking is to split a task into steps (or states). Each task keeps track of the next step it needs to execute. When it's the task's turn, it executes only that one step and returns. That way, in the main loop of your program you are simply calling each task in order, and because each task only takes up a small amount of time to complete a single step, we end up with a system which allows all of the tasks to share cpu time (ie. cooperate).