Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
The powers-that-be have decided to bestow memory upgrades on our developer team. We're all now in control of Mac Pro's with 32GB of RAM. I would have preferred an SSD instead of half of that RAM and I'm struggling to think of ways to make the most of it all. To date I have installed the x64 version of Windows 7 and also set up a 4GB RAM drive for temp files, browser cache etc. as well as codefiles for the various apps I'm working on.
Despite this, even in the middle of a heavy-duty debug session with a massively multi-project solution I always seem to have what to me as obscene amounts of free memory left and I was wondering if there was anything else I could do to make the most of the available RAM. The only other thing I could think of was to run a virtual Windows server on my workstation for 'proper' (i.e. in a mirror of our production environment) local deployment/testing and so on, but any tools or tricks that could put the 4-6GB to good use in any developer or user-friendly ways would be very welcome.
I work with ASP.Net and SQL Server and use VS2010/12 so any 'tricks' specific for this set-up are especially welcome. I was saddened to see that all that RAM has not made VS2010 any less prone to fits of unresponsiveness.
Some ideas:
use RAMdisk, and put your dev. environment on that... This will do wonders! Way quicker than the quickest SSD... But be careful, this is volatile! You could have 16GB, or even 24GB RAMdisk, and still have enough room to play with. Project switching has never been quicker, not to mention all disk based activities.
you can run multiple virtual machines. Like if you use a DB for development, you could have a local copy, not having to rely on shared resources. This can have a lot of benefits, though it has some drawbacks too (replication of changes by other developers, etc.)
combine the above! Get a RAMdisk to run your VMs and the VS from that! This involves a lot of copying when starting work, but that is once/day... I think a coffee break, and reading through the emails would be enough. Benefits: quick... Quicker than anything - once it started.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I am a Computational Fluid Dynamist (CFD), but I dont know mpi very well.
Since heavy CFD jobs requires Infiniband support, and people say that mvapich is usually much better than other mpi implementations. I want to know is this true? Any one has any real experience and references that I can look at? And how come that mvapich is better than the openmpi, etc.? Is it written by Infiniband company or what?
Thanks a lot!
So the answer is "probably not, and it doesn't matter anyway".
You write your code using the MPI API, and you can always install multiple MPI libraries and test against each, as you might with several LAPACK implementations. If one's consistently faster for your application, use it. But the MPI community is very concerned with performance, and the free competitors are all open source and publish their methods in papers, and all publish lots of benchmarks. The friendly rivalary, combined with the openness, tends to mean that no implementation has significant performance advantages for long.
On our big x86 cluster we've done "real world" and micro benchmarks with MPICH2, OpenMPI, MVAPICH2, and IntelMPI. Amongst the three open-source versions, there was no clear winner; on some cases one would win by 10-20%, on others it would lose by the same amount. On those few occassions where we were interested enough to dig into the details to find out why, it was often just a matter of defaults for things like eager limits or crossover between different collective algorithms, and by setting a couple of environment variables we got performance within noise between them. In other cases, a performance advantage persisted but was either not large enough or not consistent enough for us to investigate further.
(IntelMPI, which costs significant amounts of money, was noticibly and mostly-consistently faster, although what we consider the big win there was substantially improved startup times for very large jobs.)
MVAPICH was one of the first MPI implementations to really go after Infiniband performance, after having lots of experience with Myrinet, and they did have a significant advantage there for quite some time, and there are probably benchmarks in which they still win; but ultimately there was no consistent and important performance win and we went with OpenMPI for our main Open Source MPI option.
I would agree with Jonathan regarding the answer and add a few points from a cluster administration perspective.
As a person who at times dips into cluster administration I would add that tuning InfiniBand on a large cluster is not an easy task. You have to make sure that the OFED stack sits well upon your kernel. That the hardware is not faulty and the switches are working as expected without performance issues in a sustained mode and the application maps correctly onto the InfiniBand topology and lots more.
OpenMPI stack is considerably different from MPICH/MVAPICH. I find that OpenMPI component architecture makes it easier to find and debug issues than the architecture of MPICH/MVAPICH which I find more monolithic.
Speaking of vendors recall that MPICH comes from the MCS department of Argonne.
Update: Since version 3.1 MPICH supports OFED InfiniBand via the ib network module. Since 3.2 MPICH will support also the Mellanox MXM interface.
MVAPICH is built on top of MPICH sources by the people from the department of CS&E at Ohio State.
Many hardware vendors build either on top of MPICH or MVAPICH to provide InfiniBand support for their respective HW. One example is Intel MPI. Another is Voltaire MPI.
OpenMPI is developed by several teams supported by InfiniBand switch vendors like Cisco.
HP MPI used to be another very good MPI implementation for generic clusters that is currently available from Platfrom.
CFL codes don't scale well.
I can't speak directly to MVAPICH2, but I would recommend using whatever MPI is native to your cluster. So if you are using a Cray machine, you would go with Cray's MPI. It works like magic. Using your vendors recommended mpi makes a significant difference.
To directly answer your question, if your message size falls in the short range MVAPICH2 has a sweet spot where it beats OpenMPI. I think your CFL codes may fall in this range. On large clusters I have found that something goes wrong with MVAPICH2 with latency values when operating on over 2k PEs - but people don't run CFL on 2k PEs.
Ultimately, there is sufficient motivation to test this theory. Which code are you running OpenFOAM, Fluent?
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
How should an experienced .NET & SQL developer go about becoming a BizTalk expert for a project starting in 1 month? How should I spend my limited time to gain some practical skill & knowledge in BizTalk so I can "walk the talk"?
I am self employed, and would not be willing to spend more than USD300. I have the book "Professional BizTalk Server 2006" by Wrox, but have not found it to be a particularly good learning resource (very dry, needs more real world examples).
The BizTalk Virtual Labs in MSDN are a pretty good place to start with. Pluralsight also has several good BizTalk courses, and their online subscription isn't too expensive; would likely be a good option.
I agree with everything written this far. All solid info.
I have a few addons, coming from a fellow freelancer working with BizTalk since 2002:
Unit testing.
It's not easy to do, but check out BizUnit. A Codeplex based toolset written and maintained by Kevin Smith. One of the early BizTalk heroes :-) http://bizunit.codeplex.com/
Deployment / getting things into production
But also keep in mind that none of the day to day development stuff will prepare you for the part of the project where you have to deploy the app and make sure that it is "manageable" by operations. This can be quite complex, and is a topic in it's own right.
Check out Apress Pro BizTalk 2009, it's got a decent (IMO) chapter on this.
The entire development process around BizTalk.
The first two chapters of the same book will give you a good impression on what a BizTalk project is about. Where to use it, and where to not use it, how to organize projects, and name your stuff. Really a good collection of info that you would only get by reading 5-6 years of blogs back in time :-)
And one last thing. Depending on the roles on the project, you might be asked to optimize and tune BizTalk. And if they don't ask you. Make sure that you ask if others have done that, because you have to do it. BizTalk should always be tuned towards what it is supposed to do. Low latency vs high throughput, tuned according to hardware, correct setup and config of network around the SQL boxes, etc etc etc. This can be hairy stuff, and you should be careful not to jump into it before reading up on it all. But it's a subject we as freelancers are often expected to be able to deal with ... so thought I might bring it up.
Example ... BizTalk x64 processes on an x64 box runs really bad out of the box, actually worse than on the x86 processes. The 64 bit processes need to be tuned to really use all the MEM that are availble to them.
Anyways ... a bag of mixed tips and I hope you can use some of them! And good luck! It can be a tough start, but if used right, BizTalk can be a great product/toolset.
And remember .... if it is ugly, or hard, or both. You are doing it wrong. And don't be afraid to dive into .net code, and bolt it onto the BizTalk box. We all do it ... some just won't admit to it :-D
Start with the advice of tomasr.
Then, try and build something as real as possible. Biztalk is the kind of product where everything seems fine when you read the book and follow the examples, then you sit down to do something and you are thinking "what do I do now".
As per Thomas and Shiraz - set up an environment and get your hands dirty. If you haven't done so already, download and install BizTalk Server 2010 Developer Edition
But just to temper your expectation, IMHO expertise in BizTalk (or any other EAI / BPM / ESB product) can take years to accumulate.
It isn't clear whether you are developing for a client with an established BizTalk installation, or if this is the client's first BizTalk deployment. If so, one thing not to be underestimated is that the operational considerations of running a production BizTalk environment (performance, redundancy, reliability, auditing, tracking, monitoring with SCOM etc) are as complex as the development and testing - but understanding of this will be important to 'walk the talk'.
W.r.t. dev, start with some a simple EAI type mapping project, and then work your way through the SDK samples progress to some common messaging patterns (e.g. batching with aggregator), and then move into the BPM type orchestrations. You can probably leave BAM and the BRE for later.
Good luck!
+1 to tomasr for mentioning the virtual labs. Getting hands-on is definitely the way to go, as Shiraz Bhaiji also mentions. Hopefully you're not starting with BizTalk 2006, and can go with the latest: 2010. If that's the case, you can get the Developer Ed. of BizTalk 2010 for free now (see link from nonnb).
I'd also recommend Richard Seroter's book: 'SOA Patterns with BizTalk Server 2009' (available on Amazon.com). There are many ways to do the "wrong" things with BizTalk, and this book does an excellent job of walking through both the how and the why of building BizTalk solutions (with the code samples available from the publisher's site). And yes, it pretty much takes a whole book to go through it all. It's a good (more readable) companion to the Pro BizTalk 20xx series (which is generally better for very specific questions/tasks).
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I'm an ASP.NET / C# developer. I use VS2010 all the time. I am thinking of enabling BitLocker on my laptop to protect the contents, but I am concerned about performance degradation. Developers who use IDEs like Visual Studio are working on lots and lots of files at once. More than the usual office worker, I would think.
So I was curious if there are other developers out there who develop with BitLocker enabled. How has the performance been? Is it noticeable? If so, is it bad?
My laptop is a 2.53GHz Core 2 Duo with 4GB RAM and an Intel X25-M G2 SSD. It's pretty snappy but I want it to stay that way. If I hear some bad stories about BitLocker, I'll keep doing what I am doing now, which is keeping stuff RAR'ed with a password when I am not actively working on it, and then SDeleting it when I am done (but it's such a pain).
2015 Update: I've been using Visual Studio 2015 on my Surface Pro 3 when I travel, which has BitLocker enabled by default. It feels pretty much like my desktop, which is an i7-2600k # 4.6 GHz. I think on modern hardware with a good SSD, you won't notice!
2021 Update: I have been enabling bitlocker on all my computers and it flies now. No worries. Get an NVMe SSD and don't look back.
With my T7300 2.0GHz and Kingston V100 64gb SSD the results are
Bitlocker off → on
Sequential read 243 MB/s → 140 MB/s
Sequential write 74.5 MB/s → 51 MB/s
Random read 176 MB/s → 100 MB/s
Random write, and the 4KB speeds are almost identical.
Clearly the processor is the bottleneck in this case. In real life usage however boot time is about the same, cold launch of Opera 11.5 with 79 tabs remained the same 4 seconds all tabs loaded from cache.
A small build in VS2010 took 2 seconds in both situations. Larger build took 2 seconds vs 5 from before. These are ballpark because I'm looking at my watch hand.
I guess it all depends on the combination of processor, ram, and ssd vs hdd. In my case the processor has no hardware AES so compilation is worst case scenario, needing cycles for both assembly and crypto.
A newer system with Sandy Bridge would probably make better use of a Bitlocker enabled SDD in a development environment.
Personally I'm keeping Bitlocker enabled despite the performance hit because I travel often. It took less than an hour to toggle Bitlocker on/off so maybe you could just turn it on when you are traveling then disable it afterwards.
Thinkpad X61, Windows 7 SP1
Some practical tests...
Dell Latitude E7440
Intel Core i7-4600U
16.0 GB
Windows 8.1 Professional
LiteOn IT LMT-256M6M MSATA 256GB
This test is using a system partition. Results for a non-system partition are a bit better.
Score decrease:
Read: 5%
Write: 16%
Without BitLocker:
With BitLocker:
So you can see that with a very strong configuration and a modern SSD disk you can see a small performance degradation with tests. I don't know what about a typical work, especially with the Visual Studio.
Having used a laptop with BitLocker enabled for almost 2 years now with more or less similar specs (although without the SSD unfortunately), I can say that it really isn't that bad, or even noticable. Although I have not used this particular machine without BitLocker enabled, it really does not feel sluggish at all when compared to my desktop machine (dual core, 16 GB, dual Raptor disks, no BitLocker). Building large projects might take a bit longer, but not enough to notice.
To back this up with more non-scientifical "proof": many of my co-workers used their machines intensively without BitLocker before I joined the company (it became mandatory to use it around the time I joined, even though I am pretty sure the two events are totally unrelated), and they have not experienced noticable performance degradation either.
For me personally, having an "always on" solution like BitLocker beats manual steps for encryption, hands-down. Bitlocker-to-go (new on Windows 7) for USB devices on the other hand is simply too annoying to work with, since you cannot easily exchange information with non-W7 machines. Therefore I use TrueCrypt for removable media.
I am talking here from a theoretical point of view; I have not tried BitLocker.
BitLocker uses AES encryption with a 128-bit key. On a Core2 machine, clocked at 2.53 GHz, encryption speed should be about 110 MB/s, using one core. The two cores could process about 220 MB/s, assuming perfect data transfer and core synchronization with no overhead, and that nothing requires the CPU in the same time (that one hell of an assumption, actually). The X25-M G2 is announced at 250 MB/s read bandwidth (that's what the specs say), so, in "ideal" conditions, BitLocker necessarily involves a bit of a slowdown.
However read bandwidth is not that important. It matters when you copy huge files, which is not something that you do very often. In everyday work, access time is much more important: as a developer, you create, write, read and delete many files, but they are all small (most of them are much smaller than one megabyte). This is what makes SSD "snappy". Encryption does not impact access time. So my guess is that any performance degradation will be negligible(*).
(*) Here I assume that Microsoft's developers did their job properly.
The difference is substantial for many applications. If you are currently constrained by storage throughput, particularly when reading data, BitLocker will slow you down.
It would be useful to compare with other software based whole disk or whole partition encryption like TrueCrypt (which has the advantage if you dual boot with Linux since it works for both Windows and Linux).
A much better option is to use hardware encryption, which is available in many SSDs as well as in Hitachi 7200 RPM HDD. The performance of encrypted v. not is undetectable, and the encryption is invisible to operating systems. If you have a decent laptop, you can use the built-in security functions to generate and store the key, which your password unlocks from the encrypted key storage of the laptop.
I used to use the PGP disk encryption product on a laptop (and ran NTFS compressed on top of that!). It didn't seem to have much effect if the amount of disk to be read was small; and most software sources aren't huge by disk standards.
You have lots of RAM and pretty fast processors. I spent most of my time thinking,
typing or debugging.
I wouldn't worry very much about it.
My current work machine came with bitlocker, and being an upgrade from the prior model. It only seemed faster to me. What I have found, however, is that bitlocker is more bullet proof than truecrypt, when it comes to accurately laying down the data. I do a lot of work in SAS which constantly writes backup copies to disk as it moves along and shoots a variety of output types to disk at the end. SAS works fine writing output from multithreaded processes back to bitlocker and doesn't seem to know it's there. This has not been the case for me with truecrypt. I'm not sure what happens or how, but I found that processes got out of synch when working with source/output data in a truecrypt container, which is what I installed on my second work computer since it had no bitlocker. The constant backups were shooting to an SSD while the truecrypt results were on a regular HD. Maybe that speed difference helped trip it up. Whatever the cause, I had to quit using truecrypt on that second computer because it made my SAS results out of synch with respect to processing order and it screwed up some of my processes and data. Scary stuff in my world.
I work with people who have successfully used Truecrypt on the exact same computer, but they weren't using a disk intensive app. like SAS.
Bitlocker to Go, the encryption which bitlocker applies to thumb-drives, does slow things down quite a bit when it comes to read/write times. It's not too hard to use as long as you remember your password on the thumbdrive, and are willing to wait for it to format/initialize the drive, but in my experience it made access to the flash drive about 4 times as slow. Don't know why it would slow down a thumb drive and not a disk but that's how it was for me and my coworker.
Based on my success with bitlocker at work, I bought Windows Pro for my home computer to get bitlocker and plan to encrypt some directories with it for things like financials.
I'm building a PC with the new Intel I7 quad core processor. With hyperthreading turned on it will report 8 cores in Task Manager.
Some of my colleagues are saying that hyperthreading will make the system unreliable and suggest turning it off.
Can any of you good people enlighten me and the rest of the stockoverflow users.
Follow on: I've been using hyperthreading constantly, and its been spot on. No instability whatsoever. I'm using:
Microsoft Server 2008 64 bit
Microsoft SQL Server 2008 64 bit
Microsoft Visual Studio 2008
Diskeeper Server
Lots of controls (Telerik, Dundas, Rebex, Resharper)
Stability isn't likely to be affected, since the abstraction is very low level and the OS just sees it as another CPU to provide work to. However, performance is another matter.
In all honesty I can't say if this is still the case, but at least when the HT-enabled CPUs first came out, there were known problems with at least some applications. For example, MySQL, and multi-threaded apps like the Java application I support for my day job were known to have decreased performance when HT was enabled. We always recommended it be removed, at least for our particular use case of a server-side enterprise application.
It's possible that this is no longer an issue, and in a desktop environment this is less likely to be a problem for most use cases. The ability to split work on the CPU generally would lead to more responsive applications when the CPU is heavily utilized. However, the context switching and overhead could be a detrement when the app is already heavily threaded and CPU-intensive such as in the case of a database server.
Off the top of my head I can think of a few reasons your colleagues might say this.
Several articles about SQL performance suffering under hyperthreading. I believe it winds up doing too much context switchings or cache thrashing. can't remember exactly.
Early on going from single proc to multi-proc or more likely for most people hyperthreaded procs, brought many threading issues into the open. Race conditions, deadlocks, etc, that they never saw before. Even though its a code problem some people blamed the procs.
Are they making the same claims about multi-core/multi-proc or just about hyperthreaded?
As for me, I've been developing on a hyperthreaded box for 4 years now, only problem has been a UI deadlock issue of my own making.
Hyperthreading will mainly make a difference in the scheduler behaviour/performance when dispatching threads to the same CPU as opposed to different CPU...
It will show in a badly coded application that does not handle race conditions between threads...
So it is usually bad design/code.... that suddendly find a failure mode condition
Unreliable? I doubt so. The only disadvantage of hyperthreading that I can think of is the fact that if the OS is not aware of it, it might schedule two threads on one physical processor when other physical processors are idle which will degrade performance.
There was a problem with SQL server and hyperthreading for some queries because SQL server has its own scheduler, maxdop 1 would solve that
To whatever degree Windows is unstable, it's highly unlikely that hyperthreading contributes significantly (or it would have made big news by now.)
I've had a hyperthreading PC for a couple years now. Not that many cores, but it's worked fine for me.
Wish I had test data to prove your colleagues wrong, but it sounds like it's just my opinion versus theirs at this point. ;)
The threads in a hyperthreaded CPU share the same cache, and as such don't suffer from the cache consistency problems that a multiple cpu architecture can. Though, if the developer of a piece of software is programming with multiple cpus in mind, they will (or should) be writing with read semantics (iirc, that's the term). i.e. all writes are flushed from the cache immediately.
As far as I know, from the OS's point of view, it doesn't see hyperthreading as any different from having actual multiple cores. From the OS's point of view, there is no difference - it's isolated.
So, aside from the fact that hyperthreading's "extra cores" aren't "real" (in the strictly technical sense) and don't have the full performance of "real" CPU cores, I can't see that it'd be any less reliable. Slower, perhaps, in some rare instances, but not less reliable.
Of course, it depends on what you're running - I suppose some applications might get "down & dirty" with the CPU and hyperthreading might confuse them, but that's probably pretty rare.
I myself have been running a PC with hyperthreading for several years now, and I have seen no stability problems.
Sorry I don't have more concrete data!
I own an i7 system, and I haven't had any issues.
If it works w/ multiple cores, it works with hyperthreading.
The short answer: yes.
The long answer, as with almost every question, is "it depends". Depends on the OS, the software, the CPU revision, etc. I have personally had to disable hyperthreading on two occasions to get software working properly (one, with the Synergy application, and two, with the Windows NT 4.0 installer), but your mileage may vary.
As long as you get windows installed detecting multiple HT cores from the beginning (it loads some relevant drivers and such), you can always disable (and re-enable) HT "after the fact". If you have bizarre stability issues with specific software that you can't resolve, it's not hard to disable HT to see if it has any impact.
I wouldn't disable it to start with because, frankly, it will probably work fine in 99.99% of your daily use. But be aware that yes, it can occasionally cause bizarre behaviors, so don't rule it out if you happen to be troubleshooting something very odd down the road.
Personally, I've found that hyperthreading, while not causing any problems, doesn't actually help all that much either. It might be like having an extra .1 of a processor. On my HT machine at work, I only very seldomly see my CPU go above 50%. I don't know if HT has gotten any better with newer processors like the i7, but I'm not optimistic.
Other than hearing a few reports about SQL Server, all I can report is positive. I get about 25% better performance on heavy multi-threaded apps with HT on. Have never run into a problem with it, and I'm using a first generation HT processor...
Late to the party, but for future referrence;
I'm currently having an issue with this with SQLServer. Basically, my understanding is Hyperthreading on the same processor shares the same L1 & L2 cache, which can cause issues between the two. Citrix also appears to have this problem from what I'm reading.
Slava Ok wrote a good blog post on it.
I'm here very late but found this page via Google. I may have discovered a very subtle problem. I have a i7 950 running 2003 Server and it's great. Initially I left hyperthreading on in the BIOS, but during some testing and pushing things hard, I ran a program called "crashme" by Carrette. This program tries to crash an OS by spawning a process and feeding it garbage to try and run. My dual Opteron setup ran it forever without a problem, but the 950 crashed within the hour. It didn't crash for anything else unless I did something stupid, so it was very surprising. On a whim I turned off HT and ran the program again. It runs all night, even multiple instances of it. One anecdote doesn't mean much, but try it and see what happens. Also, it seems that the processor is slightly cooler at any given load if HT is turned off. YMMV.