I really doubt this has anything to do with css caching in the browser. If the css was cached incorrectly it would be wrong the first time as well, but as you state - it's not. This means some action after the first load is changing a condition effecting the appearance. You've got something else at play here.
Let's see an example of this in action.