The thing I see with oil is that it is just rock solid predictable; we're (the world) gonna use somewhere around 80 million barrels a day, day after day, month after month, so, it's pretty bullet proof there. And your comment about gold goes for oil too; it can be pure fantasy. If enough people get involved, it's a simple matter of buying at 40, selling for a few dollars more over what, all sorts of options, then buy at the new price, sell at the newer price, they MAKE the market regardless of what anyone does, supply or demand, because nothing dramatic is very likely from either end.
Take that argument apart. Seriously, I'd enjoy your thoughts.
You might change your mind about that.
I think there are two separate issues here. One about trading in oil, and how much speculation can artificially inflate the price. And, the other about the differences (or similarities) between how well gold prices are tied to fundamental factors and how well oil prices are tied to fundamental factors. I'll tackle the first issue first.
First, let me say that there is no doubt that speculation plays a role, sometimes a large role, in the pricing of oil futures contracts. This is true with any trade-able instrument - truth be told, it is the nature of a trade-able instrument. In the strictest since, all trading of futures is speculation, but some of this speculation is predicated on an actual need to acquire or sell the physical commodity, and some of it is not predicated on those actual needs. The former activity is technically hedging, and usually its primary goal is to stabilize the price that an entity pays/receives for a commodity. The later activity is what we generally refer to as speculation, and usually its primary goal is to make money. From the market's perspective, speculation's primary function is to provide the liquidity that the market needs to operate efficiently for all of the hedgers. Other dynamics aside, however much is needed should organic flow in and lubricate the spaces that need it at any given time. I won't get lost in that dynamic, because I suspect you can understand it.
Here's the the simple, yet poignant, reality of trading contracts - there always has to be a party on both sides of a trade. Let's discuss this in a cartoon-ish way for a minute, so that we can establish a common foundation on which to view the matter (just bear with me, I realize that you are not stupid
).
A sold contract represents an obligation to deliver a certain quantity of oil at some date in the future. A bought contract represents an obligation to take delivery of a certain quantity of oil at some date in the future. Whenever a contract exists, both sides of it exist. When a buyer and a seller decide to make a contract, they effectively create an open contract by juxtaposing these opposing obligations against each other. Two exactly opposite halves are created out of nothingness, but their sum is exactly equal to nothing, so their creation need not have a tangible effect on the real world. The buyer and seller are mated by the market, because the buyer offers the highest unmatched price and the seller offers the lowest unmatched price. Once those halves are created, either party can trade their half and realize a financial gain or loss by selling it at a price different than the one they originally agreed to. But, the contract still exists, because the two halves haven't been put back together yet. We call that an 'open contract' - the parts of the contract haven't been put back together yet - when they are, they will become nothing again. All of the open contracts added together represent the 'open interest'.
Now, you don't actually have to have a need for oil, or oil you need to sell, in order to be one of the parties creating a contract. You can just pretend to - you create a contract with the opposing side, even though you never plan to deliver/take delivery of the oil. Both parties to a contract can just be pretending, or just one can, or neither can. Those pretending parties are the speculators. So, in theory, you could have an unlimited number of contracts created, even though there will ultimately only be a certain amount of oil delivered - there can be more 'open interest' than there is actual oil or need for oil.
But, here's the thing - all of that open interest must eventually be closed. If you actually have a need for the oil, you can close a buy interest by buying the oil (that's not really how it actually happens - but this is a cartoon-ish model). If you actually have oil to sell, you can close a sell interest by selling the oil. Any sell side of a contract can mate with any buy side of a contract - they don't have to be the halves that were created at the same time.
Now, if you don't actually have oil to sell, or don't actually want to buy oil, then you have to close your position by acquiring the other half of whatever open interest you have. You don't get a choice about this - if you sold a contract, you eventually have to buy a contract, so that you can put them together and make them go *poof*. (In order to trade on the exchange, you have to keep a certain amount of cash on account with them to cover a situation where you don't properly close your contracts - the rate changes based on the current pricing volatility, but it might be something like 20% of the price of the contract.)
Here is the essence of the effect of speculation. The only way that speculation can 'artificially' inflate the price of oil, is by buying a contract. This creates extra demand. A speculative buyer has to mate with a real seller, which creates extra buy interest with no extra sell interest to offset it. The real buyers still need to buy, but there are less real sellers relative to the them, so they have to pay more in order to entice more sellers into the game. Those new sellers might be real sellers, or they might be speculative sellers who got enticed in by the price being high enough. They might be some of the same people who were just speculative buyers. But here's the key, eventually the number of speculative buyers has to become equal to the number of speculative sellers. The open interest on both sides has to end up equal, and the actual deliverers of oil will always equal the actual receivers. The speculative interest can still have the net effect of increasing the price though, because for brief periods of time there can be more speculative buying than there is selling - and that can create an upward momentum that lingers, even when the number of speculative sellers catches up.
However - the higher the price rises, the less oil gets used (I'll get back to this in a minute), so the less real buyers there are. Also, higher prices bring out more real sellers (in practicality, the difference often affects when a potential real seller sells, not if). What eventually happens is that the 'artificial' effect of a surplus of speculative buyers, creates a small surplus of real sellers, relative to real buyers. So long as all the real sellers are on the same page, this dynamic can continue, and prices can continue to rise - but, but ... if some real sellers get scared that they can't sell their contract, because there is starting to be more real supply than real demand - and the price has gotten high enough that there aren't enough speculative buyers willing to take the chance and make up the difference - then some real sellers may be willing to take less, just to make sure they get their contract sold. The prices get so attractive to real sellers that they are happy to sell as much as they can, and now the speculative buyers have a problem. The whole process reverses itself, and those speculative buyers have to be speculative sellers in order to close their positions - and there is already a surplus of sellers, so the prices are going to be dropping.
These kinds of dynamics play out on multiple levels, and in multiple, varied time-frames - all overlapping and commingling with each other. It's a market - and very subtle forces are very intricately interwoven. But the overriding general dynamic is this - for every speculative buy which puts upward pressure on the oil price, eventually there has to be a speculative sell, which will put downward pressure on the oil price.
Speculation can move prices - but it can only do it by the amount, and for the period of time, that the real parties allow it to do so. At any time, they can call speculation's bluff, and seize on what they think is a good deal (whether they be a buyer thinking the price is low enough, or a seller thinking it is high enough). And, remember the collective desire of the real parties is neutral with regard to price. There are just as many buyers who want it to be lower, as there are sellers who want it to be higher. The only appreciable net effect of the real parties, is to want stability - that is what allows their businesses to operate most efficiently.
In their desire to assert control over the market, the real players have the fundamentals on their side. When it comes to calling the bluff of the speculators, they can find their confidence in the power of the reality of the supply or demand fundamentals. If there is way, way more oil on the market than is needed, and it is clear that demand won't grow fast enough to absorb it all, then a buyer can hang their hat on that and force the sellers to take less. The thing is, when the supply demand issues get kinda close, or it looks like they might get close in the future (because of increasing demand or something), then the real players might not have as much confidence to call the speculators bluff. The speculators can bully them into accepting their terms - because speculative money is, by its nature, bolder.
...