GAS (Oxy-Acetylene)
On the plus side, this is a very versaile system in that you can fusion weld either with or without a rod, braze, siversolder, cut or just use the torch for heating of all sorts of things. Gas welding is great for small jobs or thin material such as sheet metal or thin walled tubing. The two main issues with gas are (1) they possibly represent the greatest danger in terms of the gas bottles - though fortunately accidents are not that common if appropriate precautions are taken, and (2) I'm not sure how they do things in the USA, but often the gas bottles are only rented, not bought. That can make gas welding expensive if you only use it very seldom.
ARC (with rod / stick)
This uses electric current to melt the welding rod at the point of the weld. The rod is coated in flux which melts at a suitable rate. The job of the flux is, as said above, to "keep the metal clean" by creating a small gas cloud that excludes oxygen from the surface of the molten metal. Arc welding sets are reasonably affordable and you can weld a variety of thicknesses with them. It can be quite difficult to use arc welding on thin material, but that seems to depend on the welding machine used since my father has been having much better success since he bought a new machine.
MIG
As CR1 said, a different form of arc welding. Instead of having to stop the weld to replace the rods / sticks periodically, wire is fed from a spool on a continuous basis. The other difference, as said before, is that the wire is not coated with a flux so an inert gas is used to achieve the same effect. This does mean that MIG sets have a gas bottle that needs to be maintained. If, like oxy-acetylene, the gas bottle is rented rather than bought, it may have the same disadvantage. While I have yet to use a MIG welder, I have heard it said that MIG welding is easier than standard arc welding. However Dennis, above, says it is the other way around for him, so perhaps it is a personal thing.
TIG
I'll defer to CR1 on this as I have no experience of TIG welding - and probably have less chance of gaining such experience as MIG welding equipment seems more common in the workshops where I have seen welding equipment available. This probably says something about TIG welding in itself, but whether it is cost, difficulty or both (or even something quite different) I do not know.