I want to know why a search on a custom tag returns no results even though the tag data exists on my events.

If the tag name conflicts with a reserved search keyword (for example project_id), a normal key:value query does not search your tag. Use explicit tag syntax:
​
​tags[project_id]:your_value

Avoid reserved keywords and Sentry’s automatic tags as custom tag names when you can. If you overwrite one, you must use tags[name]:value to query it. See explicit tag syntax and searchable properties.

If the data lives in context or extra instead of tags, it is not searchable.