Skip to main content

Sentry does not filter the Spider web crawler correctly, why?

Updated over a month ago

Issue

We have "Filter out known web crawlers" turned on, yet we're seeing issues which Sentry has correctly tagged as a "Spider" (device -> family). The device "Brand" & "Family" are also "Spider".

Applies To

  • All customers

Resolution

We don't have the web crawlers posted in the docs, but you can see the current list here on our GitHub repo.
​
They include the most common web crawlers as determined by our event ingestion engineering team.
​
The capital "S" in "Spider" is not matching based on how we identify crawlers. Since the web crawlers filters use the web-agent as it is, without parsing it, you need to use another option to filter the events, such as the ignoreErrors or other options in the SDK.

Here you can read more about this in our docs.

Did this answer your question?