Tractable near-optimal policies for crawling [Computer Sciences]

The problem of maintaining a local cache of n constantly changing pages arises in multiple mechanisms such as web crawlers and proxy servers. In these, the resources for polling pages for possible updates are typically limited. The goal is to devise a polling and fetching policy that maximizes the utility...

Source: Proceedings of the National Academy of Sciences - August 7, 2018 Category: Science Authors: Yossi Azar, Eric Horvitz, Eyal Lubetzky, Yuval Peres, Dafna Shahaf Tags: Physical Sciences Source Type: research