I've never really thought about leaving IS turned on, but a recent test I did proved to me that it is worth thinking about. When IS is first started up, the lens elements may move quickly to begin compensating for your camera's motion. This can actually move faster than the camera is moving! What does this mean? It means that if you take your camera from 'off' to taking a picture, the shot may be more blurred than without IS turned on, even with a fast shutter speed. I suggest turning IS off as a rule when shooting outdoors, unless you need it. Or pay attention and give the system the half second it needs to get working.
IS is great, but know when to turn it off, too. I find that with kids around you often want to take a shot 'right away'. With IS on you'll need to delay the shutter press just a bit to get optimal results... When I first got an IS lens it seemed like it didn't help at all until I learned this tip!