You can drop things at any time (including during movement) in order to make room for stealing. The general rule (in most scenarios) is that you can only carry 2 collectibles at a time.