Because of how language develops, it’s nearly impossible to be completely literal.
For example, the literal meaning of drift is not “to move as if driven or borne along by current” (OED again). The “literal” (by the letter) meaning is the letters d-r-i-f-t. It is only when one is metaphorical about the meaning of “literal” that “drift” can mean “to move downstream”. Only by further metaphorical extension can it mean to simply wander, as in drifting thoughts.