Mine came loose last summer on a Volvo 5.0 Gi. At first, they caused intermittent power loss, and later, became wedged somewhere that they caused a significant power loss. I didn't remove them, but the mechanic at the dealership said he knocked them loose with a long breaker bar.
I doubt running with them out of place would cause engine damage, especially if they weren't obstructing exhaust flow enough to obviously reduce power. Volvo doesn't even sell replacement parts for them. They discontinued the part a number of years ago according to my parts guy.
_________________________
2006 Cobalt 263 Mercruiser 8.1
1999 Larson 206 SEi Volvo 5.0Gi (for sale)