I added Sentry's cron monitoring into my code (ie. using sidekiq), and the check ins created a cron monitor in Sentry's Web UI. I have now removed the code, but the monitor remains in the web UI, and is reporting monitor failures.

This is expected behavior. The normal workflow is to create/delete cron monitors through the Sentry Web UI, but there are a few integrations like sidekiq which we are able to automatically create a monitor for once we receive the first check in. However, you will still have to delete or mute these monitors manually.