Horses are as a rule, pretty lazy, and most don't want to hurt their riders. This behavior doesn't sound like he's being difficult, he's trying to tell you something and you have to figure out what.
Does he get jiggy as soon as you put the saddle on? Does he also swish his tail or pin his ears the whole time you're riding? Does he snap or look back at one side or the other while under saddle? Elam at Loveville Leather is good at evaluating saddle fit if you want someone else to look at it. FYI - padding up a poor fitting saddle only creates more pressure in the areas that don't fit. If after a ride there are dry spots anywhere the tree sits along the back it is a good indication that it doesn't fit.
You can check his back for soreness by running your finger and thumb down his spine from wither to behind where the saddle sits with enough pressure to push into his flesh and don't tickle him, really press down firmly. If he flinches and sags his back down and away from that pressure, his back is sore.
Pain at or immediately behind the wither indicates a tree that is too narrow. There will be dry spots on either side of the withers.
Mid to lower back pain on both sides can indicate a saddle that bridges. Bridging refers to where the saddle makes contact only at the withers and lower back with no contact through the middle portion - there will be dry spots on either side of the spine. (Poor fit in this area might result in moving sideways away from the pain.)
Another saddle problem is a broken or poorly made saddle which has a twisted or uneven tree, which can create pressure to either side of the back. Usually this shows up as pressure sensitivity on one side behind the withers and on the other side in the lower back region.
Reaction only one side of the back can show rider body weight imbalance. An easy way to check rider balance is to use a pair of bathroom scales placed side by side. Stand with one foot on each scale and take your balance. Have someone check the weight on each scale. Perfect balance would be identical numbers. A difference of more than 10 pounds between the two scales is an area of concern for rider balance. Do you constantly have to push down into one stirrup or the other to reposition the saddle? This also causes pain.
If his back is sore on only one side, check the length of your stirrup leathers. Leather can stretch unevenly, especially with imbalances in leg/body weight. Stirrup leathers really should be switched out each month exchanging either side to the other you can compare the length of each to see any extreme difference in rider balance.
Hope these things help you figure out what's going on with your boy. To quote Clinton Anderson..."It's never the horse."
