Web scraping
Using APIs
Many websites offer APIs for downloading information in a standardized format, making it easier for users to pull data from their pages. For companies like Analyze that collect this information, the advantages are:
- Speed: APIs are usually faster at producing the data, and there is no need to download an entire web page.
- Structure: the information an API returns is structured, which makes it easier to read.
However, although in theory APIs should be the ideal option, in practice, in our experience, we often prefer to skip the API and use web scraping techniques on the web pages themselves.
- Access: many APIs require you to register to obtain a key, and some limit the number of requests per day. We have run into cases where, without prior notice, the provider changed the conditions for accessing the API, leaving us with periods in which we could not download the information at all.
- Robustness: keeping the API running is rarely a priority. In many cases the API is the output of a project the company or organization ran at some point, but its maintenance is not a priority afterwards. If the API service goes down for a while, it does not create the same urgency to fix the problem as the website itself going down would. Likewise, new data added to the website (for example, a retailer adding shipping costs to its product pages) usually does not show up in the API until a new API project (which can take months) is launched to change that.
- Completeness: many APIs do not carry the same amount of information as the website, and in some cases the data even differs. APIs also generally do not let you download the images on the pages (for example, product images).
Our recommendation is that companies make the effort to have functional, well-maintained APIs. Some companies believe it is better not to have an API, so that others cannot access their data easily. That, however, is a solution that backfires. With no API available, interested parties will simply scrape the web pages, which ends up costing the website owners more. Web scraping consumes more processing resources on the website's servers than an API does. Serving a web page usually also involves pagination, loading images, generating the HTML, loading JS files and several AJAX calls. There are certainly techniques a web scraping developer can use to reduce this load, but novice scrapers (or off-the-shelf web scraping software available on the market) do not necessarily apply them.
For example, almost every website shows at most 20 or 50 items per page, with no option to show everything in a single load. This means the page has to be loaded over and over, 20 items at a time, putting extra load on the web service. From the point of view of the web scraping programmer, the only change is one extra line of code with a while() expression.
Web scraping also distorts the website's own metrics. For example, it artificially inflates the number of visits to each page, or the owner ends up paying for web advertising on pages that are only ever seen by web scraping "robots".
In conclusion
Keeping APIs up to date and complete pays off: those who want the information will use the API instead of overloading the web service (and will not generate false statistics). Trying to put up barriers against web scraping may stop amateurs from scraping, but it is no barrier for experienced companies. The main effect will be an overloaded web service.