Web scraping typically refers to an automated process of collecting data from websites.<\/em> On a high level, you’re essentially making a bot that visits a website, detects the data you’re interested in, and then stores it into some appropriate data structure, so you can easily analyze and access it later.<\/p>\n However, if you’re concerned about your anonymity on the Internet, you should probably take a little more care when scraping the web. Since your IP address is public, a website owner could track it down and, potentially, block it.<\/p>\n So, if you want to stay as anonymous as possible, and prevent being blocked from visiting a certain website, you should consider using proxies when scraping a web.<\/p>\n Proxies, also referred to as proxy servers, are specialized servers that enable you not to directly access the websites you’re scraping. Rather, you’ll be routing your scraping requests via a proxy server<\/em>.<\/p>\n That way, your IP address gets «hidden» behind the IP address of the proxy server you’re using. This can help you both stay as anonymous as possible, as well as not being blocked, so you can keep scraping as long as you want.<\/p>\n In this comprehensive guide, you’ll get a grasp of the basics of web scraping and proxies, you’ll see the actual, working example of scraping a website using proxies in Node.js<\/a>. Afterward, we’ll discuss why you might consider using existing scraping solutions (like ScraperAPI<\/a>) over writing your own web scraper. At the end, we’ll give you some tips on how to overcome some of the most common issues you might face when scraping the web.<\/p>\n Web scraping is the process of extracting data from websites. It automates what would otherwise be a manual process of gathering information, making the process less time-consuming and prone to errors.<\/p>\n That way you can collect a large amount of data quickly and efficiently. Later, you can analyze, store, and use it.<\/p>\n The primary reason you might scrape a website is to obtain data that is either unavailable through an existing API or too vast to collect manually.<\/p>\n It’s particularly useful when you need to extract information from multiple pages or when the data is spread across different websites.<\/p>\n There are many real-world applications that utilize the power of web scraping in their business model. The majority of apps helping you track product prices and discounts, find cheapest flights and hotels, or even find a job, use the technique of web scraping to gather the data that provides you the value.<\/p>\n Imagine you’re sending a request to a website. Usually, your request is sent from your machine (with your IP address) to the server that hosts a website you’re trying to access. That means that the server «knows» your IP address and it can block you based on your geo-location, the amount of traffic you’re sending to the website, and many more factors.<\/p>\n But when you send a request through a proxy, it routes the request through another server, hiding your original IP address behind the IP address of the proxy server. This not only helps in maintaining anonymity but also plays a crucial role in avoiding IP blocking, which is a common issue in web scraping.<\/p>\n By rotating through different IP addresses, proxies allow you to distribute your requests, making them appear as if they’re coming from various users. This reduces the likelihood of getting blocked and increases the chances of successfully scraping the desired data.<\/p>\n Typically, there are four main types of proxy servers — datacenter, residential, rotating, and mobile.<\/em><\/p>\n Each of them has its pros and cons, and based on that, you’ll use them for different purposes and at different costs.<\/p>\n Datacenter proxies<\/strong> are the most common and cost-effective<\/em> proxies, provided by third-party data centers. They offer high speed and reliability<\/em> but are more easily detectable and can be blocked by websites more frequently<\/em>.<\/p>\n Residential proxies<\/strong> route your requests through real residential IP addresses. Since they appear as ordinary user connections, they are less likely to be blocked<\/em> but are typically more expensive<\/em>.<\/p>\n Rotating proxies<\/strong> automatically change the IP address after each request or after a set period. This is particularly useful for large-scale scraping projects<\/em>, as it significantly reduces the chances of being detected and blocked<\/em>.<\/p>\n Mobile proxies<\/strong> use IP addresses associated with mobile devices. They are highly effective for scraping mobile-optimized websites or apps<\/em> and are less likely to be blocked<\/em>, but they typically come at a premium cost<\/em>.<\/p>\n Let’s walk through a practical example of a web scraping project, and demonstrate how to set up a basic scraper, integrate proxies, and use a scraping service like ScraperAPI<\/a>.<\/p>\n Before you dive into the actual scraping process, it’s essential to set up your development environment.<\/p>\n For this example, we’ll be using Node.js<\/a> since it’s well-suited for web scraping due to its asynchronous capabilities. We’ll use Axios<\/a> for making HTTP requests, and Cheerio<\/a> to parse and manipulate HTML (that’s contained in the response of the HTTP request).<\/p>\n First, ensure you have Node.js installed<\/em> on your system. If you don’t have it, download and install it from nodejs.org<\/a>.<\/p>\n Then, create a new directory for your project and initialize it:<\/p>\n $ mkdir my-web-scraping-project Finally, install Axios and Cheerio since they are necessary for you to implement your web scraping logic:<\/p>\n $ npm install axios cheerio<\/p>\n Now that your environment is set up, let’s create a simple web scraping script. We’ll scrape a sample website<\/a> to gather famous quotes and their authors.<\/p>\n So, create a JavaScript file named sample-scraper.js and write all the code inside of it. Import the packages you’ll need to send HTTP requests and manipulate the HTML:<\/p>\n const<\/span> axios = require<\/span>(‘axios’<\/span>); Next, create a wrapper function that will contain all the logic you need to scrape data from a web page. It accepts the URL of a website you want to scrape as an argument and returns all the quotes found on the page:<\/p>\n \/\/ Function to scrape data from a webpage<\/span> \/\/ Load the HTML into cheerio<\/span> \/\/ Extract all elements with the class ‘quote’<\/span> \/\/ Output the quotes<\/span> } catch<\/span> (error) { Note:<\/strong> All the quotes are stored in a separate div element with a class of quote. Each quote has its text and author<\/em> — text is stored under the span element with the class of text, and the author is within the small element with the class of author.<\/p>\n<\/div>\n<\/div>\n<\/div>\n Finally, specify the URL of the website you want to scrape — in this case, https:\/\/quotes.toscrape.com, and call the scrapeWebsite() function:<\/p>\n \/\/ URL of the website you want to scrape<\/span> \/\/ Call the function to scrape the website<\/span> All that’s left for you to do is to run the script from the terminal:<\/p>\n $ node sample-scraper.js<\/p>\n To use a proxy with axios, you specify the proxy settings in the request configuration. The axios.get() method can include the proxy configuration, allowing the request to route through the specified proxy server. The proxy object contains the host, port, and optional authentication details<\/em> for the proxy:<\/p>\n \/\/ Send a GET request to the webpage with proxy configuration<\/span> Note:<\/strong> You need to replace these placeholders with your actual proxy details.<\/p>\n<\/div>\n<\/div>\n<\/div>\n Other than this change, the entire script remains the same:<\/p>\n \/\/ Function to scrape data from a webpage<\/span> \/\/ Load the HTML into cheerio<\/span> \/\/ Extract all elements with the class ‘quote’<\/span> \/\/ Output the quotes<\/span> } catch<\/span> (error) { Using a scraping service like ScraperAPI<\/a> offers several advantages over manual web scraping since it’s designed to tackle all of the major problems you might face when scraping websites:<\/p>\n Automatically handles common web scraping obstacles<\/strong> such as CAPTCHAs, JavaScript rendering, and IP blocks. To implement the ScraperAPI proxy<\/strong> into the scraping script you’ve created so far, there are just a few tweaks you need to make in the axios configuration.<\/p>\n First of all, ensure you have created a free ScraperAPI account<\/a>. That way, you’ll have access to your API key, which will be necessary in the following steps.<\/p>\n Once you get the API key, use it as a password in the axios proxy configuration from the previous section:<\/p>\n \/\/ Send a GET request to the webpage with ScraperAPI proxy configuration<\/span> And, that’s it, all of your requests will be routed through the ScraperAPI proxy servers<\/em>.<\/p>\n But to use the full potential of a scraping service you’ll have to configure it using the service’s dashboard — ScraperAPI is no different here.<\/p>\n It has a user-friendly dashboard<\/strong> where you can set up the web scraping process to best fit your needs<\/em>. You can enable proxy or async mode, JavaScript rendering, set a region from where the requests will be sent, set your own HTTP headers, timeouts, and much more.<\/p>\n And the best thing is that ScraperAPI automatically generates a script<\/strong> containing all of the scraper settings, so you can easily integrate the scraper into your codebase<\/em>.<\/p>\n Not every proxy provider and its configuration are the same. So, it’s important to know what proxy service to choose and how to configure it properly.<\/p>\n Let’s take a look at some tips and tricks to help you with that!<\/p>\n Implement a proxy rotation strategy that changes the IP address after a certain number of requests or at regular intervals. This approach can mimic human browsing behavior, making it less likely for websites to flag your activities as suspicious.<\/p>\n Many websites enforce rate limits to prevent excessive scraping. To avoid hitting these limits, you can:<\/p>\n Introduce Delays<\/strong>: Add random delays between requests to simulate human behavior. Choosing high-quality proxies is crucial for successful web scraping. Quality proxies, especially residential ones, are less likely to be detected and banned<\/strong> by target websites. Using a mix of high-quality proxies can significantly enhance your chances of successful scraping without interruptions.<\/p>\n Quality proxy services often provide a wide range of IP addresses<\/strong> from different regions, enabling you to bypass geo-restrictions and access localized content.<\/p>\n Reliable proxy services can offer faster response times and higher uptime<\/strong>, which is essential when scraping large amounts of data.<\/p>\n As your scraping needs grow, having access to a robust proxy service allows you to scale your operations without the hassle of managing your own infrastructure<\/strong>.<\/p>\n Using a reputable proxy service often comes with customer support and maintenance<\/strong>, which can save you time and effort in troubleshooting issues related to proxies.<\/p>\n CAPTCHAs and anti-bot mechanisms are some of the most common obstacles you’ll encounter while scraping a web.<\/p>\n Websites use CAPTCHAs<\/strong> to prevent automated access by trying to differentiate real humans and automated bots. They’re achieving that by prompting the users to solve various kinds of puzzles, identify distorted objects, and so on. That can make it really difficult for you to automatically scrape data.<\/p>\n Even though there are many both manual and automated CAPTCHA solvers available online, the best strategy for handling CAPTCHAs is to avoid triggering them in the first place. Typically, they are triggered when non-human behavior is detected. For example, a large amount of traffic, sent from a single IP address, using the same HTTP configuration is definitely a red flag!<\/p>\n So, when scraping a website, try mimicking human behavior as much as possible:<\/p>\n Add delays between requests and spread them out as much as you can. Beyond CAPTCHAs, websites often use sophisticated anti-bot measures<\/strong> to detect and block scraping.<\/p>\n Some websites use JavaScript to detect bots<\/em>. Tools like Puppeteer<\/a> can simulate a real browser environment, allowing your scraper to execute JavaScript and bypass these challenges.<\/p>\n Websites sometimes add hidden form fields or links that only bots will interact with<\/em>. So, try avoiding clicking on hidden elements or filling out forms with invisible fields.<\/p>\n Advanced anti-bot systems go as far as tracking user behavior, such as mouse movements or time spent on a page<\/em>. Mimicking these behaviors using browser automation tools can help bypass these checks.<\/p>\nWeb Scraping<\/h2>\n
Web Proxies<\/h2>\n
Types of Proxies<\/h3>\n
Example Web Scraping Project<\/h2>\n
Setting up<\/h3>\n
\n$ cd my-web-scraping-project
\n$ npm init -y<\/p>\nSimple Web Scraping Script<\/h3>\n
\nconst<\/span> cheerio = require<\/span>(‘cheerio’<\/span>);<\/p>\n
\nasync<\/span> function<\/span> scrapeWebsite<\/span>(url<\/span>) <\/span>{
\n try<\/span> {
\n \/\/ Send a GET request to the webpage<\/span>
\n const<\/span> response = await<\/span> axios.get(url);<\/p>\n
\n const<\/span> $ = cheerio.load(response.data);<\/p>\n
\n const<\/span> quotes = [];
\n $(‘div.quote’<\/span>).each((index, element<\/span>) =><\/span> {
\n \/\/ Extracting text from span with class ‘text’<\/span>
\n const<\/span> quoteText = $(element).find(‘span.text’<\/span>).text().trim();
\n \/\/ Assuming there’s a small tag for the author<\/span>
\n const<\/span> author = $(element).find(‘small.author’<\/span>).text().trim();
\n quotes.push({ quote<\/span>: quoteText, author<\/span>: author });
\n });<\/p>\n
\n console<\/span>.log(«Quotes found on the webpage:»<\/span>);
\n quotes.forEach((quote, index<\/span>) =><\/span> {
\n console<\/span>.log(`${index + 1<\/span>}<\/span>: «${quote.quote}<\/span>» — ${quote.author}<\/span>`<\/span>);
\n });<\/p>\n
\n console<\/span>.error(`An error occurred: ${error.message}<\/span>`<\/span>);
\n }
\n}<\/p>\n
\nconst<\/span> url = ‘https:\/\/quotes.toscrape.com’<\/span>;<\/p>\n
\nscrapeWebsite(url);<\/p>\nIntegrating Proxies<\/h3>\n
\nconst<\/span> response = await<\/span> axios.get(url, {
\n proxy<\/span>: {
\n host<\/span>: proxy.host,
\n port<\/span>: proxy.port,
\n auth<\/span>: {
\n username<\/span>: proxy.username, \/\/ Optional: Include if your proxy requires authentication<\/span>
\n password<\/span>: proxy.password, \/\/ Optional: Include if your proxy requires authentication<\/span>
\n },
\n },
\n});<\/p>\n
\nasync<\/span> function<\/span> scrapeWebsite<\/span>(url<\/span>) <\/span>{
\n try<\/span> {
\n \/\/ Send a GET request to the webpage with proxy configuration<\/span>
\n const<\/span> response = await<\/span> axios.get(url, {
\n proxy<\/span>: {
\n host<\/span>: proxy.host,
\n port<\/span>: proxy.port,
\n auth<\/span>: {
\n username<\/span>: proxy.username, \/\/ Optional: Include if your proxy requires authentication<\/span>
\n password<\/span>: proxy.password, \/\/ Optional: Include if your proxy requires authentication<\/span>
\n },
\n },
\n });<\/p>\n
\n const<\/span> $ = cheerio.load(response.data);<\/p>\n
\n const<\/span> quotes = [];
\n $(‘div.quote’<\/span>).each((index, element<\/span>) =><\/span> {
\n \/\/ Extracting text from span with class ‘text’<\/span>
\n const<\/span> quoteText = $(element).find(‘span.text’<\/span>).text().trim();
\n \/\/ Assuming there’s a small tag for the author<\/span>
\n const<\/span> author = $(element).find(‘small.author’<\/span>).text().trim();
\n quotes.push({ quote<\/span>: quoteText, author<\/span>: author });
\n });<\/p>\n
\n console<\/span>.log(«Quotes found on the webpage:»<\/span>);
\n quotes.forEach((quote, index<\/span>) =><\/span> {
\n console<\/span>.log(`${index + 1<\/span>}<\/span>: «${quote.quote}<\/span>» — ${quote.author}<\/span>`<\/span>);
\n });<\/p>\n
\n console<\/span>.error(`An error occurred: ${error.message}<\/span>`<\/span>);
\n }
\n}<\/p>\nIntegrating a Scraping Service<\/h3>\n
\nAutomatically handles proxies<\/strong> — proxy configuration, rotation, and much more.
\nInstead of building your own scraping infrastructure, you can leverage ScraperAPI’s pre-built solutions<\/em>. This saves significant development time and resources<\/strong> that can be better spent on analyzing the scraped data.
\nScraperAPI offers various customization options such as geo-location targeting, custom headers, and asynchronous scraping<\/strong>. You can personalize the service to suit your specific scraping needs.
\nUsing a scraping API like ScraperAPI is often more cost-effective<\/strong> than building and maintaining your own scraping infrastructure. The pricing is based on usage, allowing you to scale up or down as needed.
\nScraperAPI allows you to scale your scraping efforts<\/strong> by handling millions of requests concurrently.<\/p>\n
\naxios.get(url, {
\n method<\/span>: ‘GET’<\/span>,
\n proxy<\/span>: {
\n host<\/span>: ‘proxy-server.scraperapi.com’<\/span>,
\n port<\/span>: 8001<\/span>,
\n auth<\/span>: {
\n username<\/span>: ‘scraperapi’<\/span>,
\n password<\/span>: ‘YOUR_API_KEY’<\/span> \/\/ Paste your API key here<\/span>
\n },
\n protocol<\/span>: ‘http’<\/span>
\n }
\n});<\/p>\nBest Practices for Using Proxies in Web Scraping<\/h2>\n
Rotate Proxies Regularly<\/h3>\n
Handle Rate Limits<\/h3>\n
\nMonitor Response Codes<\/strong>: Track HTTP response codes to detect when you are being rate-limited. If you receive a 429 (Too Many Requests) response, pause your scraping for a while before trying again.<\/p>\nUse Quality Proxies<\/h3>\n
Handling CAPTCHAs and Other Challenges<\/h2>\n
\nRegularly rotate between multiple IP addresses using a proxy service.
\nRandomize HTTP headers and user agents.<\/p>\n