Which of them are preferred in which circumstances?
I'd like to see the list of evaluation crtieria for the various modes, and maybe a discussion of the applicability of each criterion.
For example, I think one of the criteria is "size of the code" for encryption and decryption, which is important for micro-code embedded systems, like 802.11 network adapters. IF the code required to implement CBC is much smaller than that required for CTR (I don't know this is true, it's just an example), then I could understand why the mode with the smaller code would be preferred. But if I am writing an app that runs on a server, and the AES library I am using implements both CBC and CTR anyway, then this criterion is irrelevant.
See what I mean by "list of evaluation criteria and applicability of each criterion" ??
This isn't really programming related but it is algorithm related.encryptionaes
Have you start by reading the information on this on Wikipedia - Block cipher modes of operation? Then follow the reference link on Wikipedia to NIST: Recommendation for Block Cipher Modes of Operation.
I know one aspect: Although CBC gives better security by changing the IV for each block, it's not applicable to randomly accessed encrypted content (like an encrypted hard disk).
So, use CBC (and the other sequential modes) for sequential streams and ECB for random access.
ECB should not be used if encrypting more than one block of data with the same key.
CBC, OFB and CFB are similar, however OFB/CFB is better because you only need encryption and not decryption, which can save code space.
CTR is used if you want good parallelization (ie. speed), instead of CBC/OFB/CFB.
XTS mode is the most common if you are encoding a random accessible data (like a hard disk or RAM).
OCB is by far the best mode, as it allows encryption and authentication in a single pass. However there are patents on it in USA.
The only thing you really have to know is that ECB is not to be used unless you are only encrypting 1 block. XTS should be used if you are encrypting randomly accessed data and not a stream.
You might want to chose based on what is widely available. I had the same question and here are the results of my limited research.
STM32L (low energy ARM cores) from ST Micro support ECB, CBC,CTR GCM CC2541 (Bluetooth Low Energy) from TI supports ECB, CBC, CFB, OFB, CTR, and CBC-MAC
Open source limitations
Original rijndael-api source - ECB, CBC, CFB1 OpenSSL - command line CBC, CFB, CFB1, CFB8, ECB, OFB OpenSSL - C/C++ API CBC, CFB, CFB1, CFB8, ECB, OFB and CTR EFAES lib  - ECB, CBC, PCBC, OFB, CFB, CRT ([sic] CTR mispelled) OpenAES  - ECB, CBC
A word of introduction: This answer was partly a response to a lot of questions I've seen under the
[encryption] tag that showed people deploying utterly insecure code. Addressing these programmers I wrote the following opening sentence with the intend to shake them up enough to rethink their approach to cryptography, before their application gets attacked. If you are here in the process of learning, that's great! We need more programmers with background knowledge in cryptography. Keep on asking and add a silent "yet!" to my opening:
I know this sounds harsh, so let me illustrate my point: Imagine you are building a web application and you need to store some session data. You could assign each user a session ID and store the session data on the server in hash map mapping session ID to session data. But then you have to deal with this pesky state on the server and if at some point you need more than one server things will get messy. So instead you have the idea to store the session data in a cookie on the client side. You will encrypt it of course so the user cannot read and manipulate the data. So what mode should you use? Coming here you read the top answer (sorry for singling you out myforwik). The first one covered - ECB - is not for you, you want to encrypt more than one block, the next one - CBC - sounds good and you don't need the parallelism of CTR, you don't need random access, so no XTS and patents are a PITA, so no OCB. Using your crypto library you realize that you need some padding because you can only encrypt multiples of the block size. You choose PKCS7 because it was defined in some serious cryptography standards. After reading somewhere that CBC is provably secure if used with a random IV and a secure block cipher, you rest at ease even though you are storing your sensitive data on the client side.
Years later after your service has indeed grown to significant size, an IT security specialist contacts you in a responsible disclosure. She's telling you that she can decrypt all your cookies using a padding oracle attack, because your code produces an error page if the padding is somehow broken.
This is not a hypothetical scenario: Microsoft had this exact flaw in ASP.NET until a few years ago.
The problem is there are a lot of pitfalls regarding cryptography and it is extremely easy to build a system that looks secure for the layman but is trivial to break for a knowledgeable attacker.
For live connections use TLS (be sure to check the hostname of the certificate and the issuer chain). If you can't use TLS, look for the highest level API your system has to offer for your task and be sure you understand the guarantees it offers and more important what it does not guarantee. For the example above a framework like Play offers client side storage facilities, it does not invalidate the stored data after some time, though, and if you changed the client side state, an attacker can restore a previous state without you noticing.
If there is no high level abstraction available use a high level crypto library. A prominent example is NaCl and a portable implementation with many language bindings is Sodium. Using such a library you do not have to care about encryption modes etc. but you have to be even more careful about the usage details than with a higher level abstraction, like never using a nonce twice.
If for some reason you cannot use a high level crypto library, for example because you need to interact with existing system in a specific way, there is no way around educating yourself thoroughly. I recommend reading Cryptography Engineering by Ferguson, Kohno and Schneier. Please don't fool yourself into believing you can build a secure system without the necessary background. Cryptography is extremely subtle and it's nigh impossible to test the security of a system.
To prevent padding oracle attacks and changes to the ciphertext, one can compute a message authentication code (MAC) on the ciphertext and only decrypt it if it has not been tampered with. This is called encrypt-then-mac and should be preferred to any other order. Except for very few use cases authenticity is as important as confidentiality (the latter of which is the aim of encryption). Authenticated encryption schemes (with associated data (AEAD)) combine the two part process of encryption and authentication into one block cipher mode that also produces an authentication tag in the process. In most cases this results in speed improvement.
Considering the importance of authentication I would recommend the following two block cipher modes for most use cases (except for disk encryption purposes): If the data is authenticated by an asymmetric signature use CBC, otherwise use GCM.
A formal analysis has been done by Phil Rogaway in 2011, here. Section 1.6 gives a summary that I transcribe here, adding my own emphasis in bold (if you are impatient, then his recommendation is use CTR mode, but I suggest that you read my paragraphs about message integrity versus encryption below).
Note that most of these require the IV to be random, which means non-predictable and therefore should be generated with cryptographic security. However, some require only a "nonce", which does not demand that property but instead only requires that it is not re-used. Therefore designs that rely on a nonce are less error prone than designs that do not (and believe me, I have seen many cases where CBC is not implemented with proper IV selection). So you will see that I have added bold when Rogaway says something like "confidentiality is not achieved when the IV is a nonce", it means that if you choose your IV cryptographically secure (unpredictable), then no problem. But if you do not, then you are losing the good security properties. Never re-use an IV for any of these modes.
Also, it is important to understand the difference between message integrity and encryption. Encryption hides data, but an attacker might be able to modify the encrypted data, and the results can potentially be accepted by your software if you do not check message integrity. While the developer will say "but the modified data will come back as garbage after decryption", a good security engineer will find the probability that the garbage causes adverse behaviour in the software, and then he will turn that analysis into a real attack. I have seen many cases where encryption was used but message integrity was really needed more than the encryption. Understand what you need.
I should say that although GCM has both encryption and message integrity, it is a very fragile design: if you re-use an IV, you are screwed -- the attacker can recover your key. Other designs are less fragile, so I personally am afraid to recommend GCM based upon the amount of poor encryption code that I have seen in practice.
If you need both, message integrity and encryption, you can combine two algorithms: usually we see CBC with HMAC, but no reason to tie yourself to CBC. The important thing to know is encrypt first, then MAC the encrypted content, not the other way around. Also, the IV needs to be part of the MAC calculation.
I am not aware of IP issues.
Now to the good stuff from Professor Rogaway:
ECB: A blockcipher, the mode enciphers messages that are a multiple of n bits by separately enciphering each n-bit piece. The security properties are weak, the method leaking equality of blocks across both block positions and time. Of considerable legacy value, and of value as a building block for other schemes, but the mode does not achieve any generally desirable security goal in its own right and must be used with considerable caution; ECB should not be regarded as a “general-purpose” confidentiality mode.
CBC: An IV-based encryption scheme, the mode is secure as a probabilistic encryption scheme, achieving indistinguishability from random bits, assuming a random IV. Confidentiality is not achieved if the IV is merely a nonce, nor if it is a nonce enciphered under the same key used by the scheme, as the standard incorrectly suggests to do. Ciphertexts are highly malleable. No chosen ciphertext attack (CCA) security. Confidentiality is forfeit in the presence of a correct-padding oracle for many padding methods. Encryption inefficient from being inherently serial. Widely used, the mode’s privacy-only security properties result in frequent misuse. Can be used as a building block for CBC-MAC algorithms. I can identify no important advantages over CTR mode.
CFB: An IV-based encryption scheme, the mode is secure as a probabilistic encryption scheme, achieving indistinguishability from random bits, assuming a random IV. Confidentiality is not achieved if the IV is predictable, nor if it is made by a nonce enciphered under the same key used by the scheme, as the standard incorrectly suggests to do. Ciphertexts are malleable. No CCA-security. Encryption inefficient from being inherently serial. Scheme depends on a parameter s, 1 ≤ s ≤ n, typically s = 1 or s = 8. Inefficient for needing one blockcipher call to process only s bits . The mode achieves an interesting “self-synchronization” property; insertion or deletion of any number of s-bit characters into the ciphertext only temporarily disrupts correct decryption.
OFB: An IV-based encryption scheme, the mode is secure as a probabilistic encryption scheme, achieving indistinguishability from random bits, assuming a random IV. Confidentiality is not achieved if the IV is a nonce, although a fixed sequence of IVs (eg, a counter) does work fine. Ciphertexts are highly malleable. No CCA security. Encryption and decryption inefficient from being inherently serial. Natively encrypts strings of any bit length (no padding needed). I can identify no important advantages over CTR mode.
CTR: An IV-based encryption scheme, the mode achieves indistinguishability from random bits assuming a nonce IV. As a secure nonce-based scheme, the mode can also be used as a probabilistic encryption scheme, with a random IV. Complete failure of privacy if a nonce gets reused on encryption or decryption. The parallelizability of the mode often makes it faster, in some settings much faster, than other confidentiality modes. An important building block for authenticated-encryption schemes. Overall, usually the best and most modern way to achieve privacy-only encryption.
XTS: An IV-based encryption scheme, the mode works by applying a tweakable blockcipher (secure as a strong-PRP) to each n-bit chunk. For messages with lengths not divisible by n, the last two blocks are treated specially. The only allowed use of the mode is for encrypting data on a block-structured storage device. The narrow width of the underlying PRP and the poor treatment of fractional final blocks are problems. More efficient but less desirable than a (wide-block) PRP-secure blockcipher would be.
ALG1–6: A collection of MACs, all of them based on the CBC-MAC. Too many schemes. Some are provably secure as VIL PRFs, some as FIL PRFs, and some have no provable security. Some of the schemes admit damaging attacks. Some of the modes are dated. Key-separation is inadequately attended to for the modes that have it. Should not be adopted en masse, but selectively choosing the “best” schemes is possible. It would also be fine to adopt none of these modes, in favor of CMAC. Some of the ISO 9797-1 MACs are widely standardized and used, especially in banking. A revised version of the standard (ISO/IEC FDIS 9797-1:2010) will soon be released .
CMAC: A MAC based on the CBC-MAC, the mode is provably secure (up to the birthday bound) as a (VIL) PRF (assuming the underlying blockcipher is a good PRP). Essentially minimal overhead for a CBCMAC-based scheme. Inherently serial nature a problem in some application domains, and use with a 64-bit blockcipher would necessitate occasional re-keying. Cleaner than the ISO 9797-1 collection of MACs.
HMAC: A MAC based on a cryptographic hash function rather than a blockcipher (although most cryptographic hash functions are themselves based on blockciphers). Mechanism enjoys strong provable-security bounds, albeit not from preferred assumptions. Multiple closely-related variants in the literature complicate gaining an understanding of what is known. No damaging attacks have ever been suggested. Widely standardized and used.
GMAC: A nonce-based MAC that is a special case of GCM. Inherits many of the good and bad characteristics of GCM. But nonce-requirement is unnecessary for a MAC, and here it buys little benefit. Practical attacks if tags are truncated to ≤ 64 bits and extent of decryption is not monitored and curtailed. Complete failure on nonce-reuse. Use is implicit anyway if GCM is adopted. Not recommended for separate standardization.
CCM: A nonce-based AEAD scheme that combines CTR mode encryption and the raw CBC-MAC. Inherently serial, limiting speed in some contexts. Provably secure, with good bounds, assuming the underlying blockcipher is a good PRP. Ungainly construction that demonstrably does the job. Simpler to implement than GCM. Can be used as a nonce-based MAC. Widely standardized and used.
GCM: A nonce-based AEAD scheme that combines CTR mode encryption and a GF(2128)-based universal hash function. Good efficiency characteristics for some implementation environments. Good provably-secure results assuming minimal tag truncation. Attacks and poor provable-security bounds in the presence of substantial tag truncation. Can be used as a nonce-based MAC, which is then called GMAC. Questionable choice to allow nonces other than 96-bits. Recommend restricting nonces to 96-bits and tags to at least 96 bits. Widely standardized and used.