The indication of state of charge on the solar panel controller is based on the voltage at the controller. When the battery is charging the voltage will be higher, when the battery is discharging it will be lower, so it is quite normal for it to go up and own. Really the only accurate state of charge indication will be when the battery is at rest -- not charging or discharging. This will probably be just before the sun comes up and just after it goes down, assuming there is no load on the battery.
The charging rate depends on the angle and intensity of light hitting the solar panel. Any shading from a tree or the Maxx fan will cause the charging rate to go down.
Do you know if you were running the fridge off the battery when you were driving? This might account for the state of charge starting off low as the fridge may have been drawing more power than the towing vehicle could supply. We run our fridge on propane except when we are hooked up to external power.
I installed a Trimetric battery monitor to give a more accurate indication of state of charge. It measures power going in and out of the battery to keep a running estimate of the real state of charge. (see separate thread on this topic) Determining State of Charge on Solar Systems