Skip to main content

Scraping engine

The scraping engine is how you bring the accounts you track up to date. When a handle on your watchlist is looking stale, you start a scrape and MBER goes out and collects its recent posts, so your timeline reflects what is happening now instead of whenever it was last collected.

What it does for you​

  • You start the scrape. Collection runs when you ask for it. Pick a handle on your watchlist, kick off a scrape, and MBER pulls in its recent posts. Nothing is collected behind your back.
  • Collection is global. What one scrape pulls in is saved for everyone on MBER, not just you. If another user has already refreshed an account you follow, it is already current on your timeline.
  • The more people, the less you scrape. Popular handles get collected often because many watchlists point at them. When the accounts you track are already kept up to date by others, collecting them costs you no usage.

Account status​

Every handle carries a status indicator that shows how recent its data is. You can tell at a glance whether a timeline reflects the latest activity or is due for a refresh. As data ages, the indicator winds down; a new collection brings it back up.