💻
System Design
  • System Design
  • System Design Overview
  • Basic Concepts
    • Memory Levels
    • Functional vs Non-Functional Requirements
  • Key Concepts and Definations
    • Scalability
    • Performance
    • Availability
    • Reliability
    • Consistency
    • Stateless vs Stateful Servers
    • CAP Theorem
    • Data Storage and Retrieval
    • Consistent Hashing
    • Rate Limit
    • Networking and Communication
    • Security and Privacy
  • Building Blocks
    • REST APIs
      • HTTP Request
    • Authentication and Authorisation
    • Caching
      • Caching with Redis + Python
      • LRU Caching Implementation
      • Caching: Case Study
    • Database
      • DBMS
      • SQL
      • NoSQL
      • Properties
      • Database
        • Elasticsearch
        • Neo4j
        • Redis
        • OrientDB
      • Use Cases
    • Message Queue
      • Kafka
      • Pulsar
      • Celery
      • Rabbimq
      • Redis
    • Deployment
      • Docker
        • Distroless
      • Kubernetes
      • CI - CD Pipeline
    • CDN
    • Nginx
      • Reverse Proxy
      • Load Balancer
      • Caching
    • Web Scrapper and Automation
    • Parallel Processing
    • Scheduling
      • Airflow
      • Azure Logic App
    • Networking Fundamentals
    • Cloud Services
      • AWS
        • Dynamic Website on AWS
      • Google Cloud
        • Dynamic Website on GCP
      • Azure
  • Miscellaneous
  • Git
    • Interesting Repos
      • Quantitative Finance
  • Visualization
    • Power BI
    • Kibana
    • Redash
  • rocksdb and Python
  • Python Framework
    • Python Concepts
      • miscellaneous Concepts
      • OOP
      • Built-in Class Attributes
      • Built-in Functions
      • Python Clean Code
      • Async io
      • Concurrency vs Parallelism
      • Client-Server Model
    • Flask
    • FastAPI
    • Tornado
    • Django
  • Java Script
    • Node js
    • Express js
  • Operating System
  • Big Data
    • Spark
      • Pyspark Working
    • EDW
      • EDW Concepts
    • GraphQL
    • Map Reduce
  • Search
  • UI UX
    • Figma
  • Logging in Python
  • Communication in Python
  • Hashing
  • Sample Application Structure
  • Data Transfer over Network
    • jwt Authentication
  • Rough
    • Authentication and Authorization
    • Pyspark Kubernetes Operator
    • Pulsar vs Kafka
    • Storage vs Compute Server
    • Bigdata Storage Server
    • Bigdata Storage + Compute Servers
    • Stream Data Storage
    • Columnary vs Row based
    • Big data Processing
    • Stream data Processing
    • Metadata and usage
    • Data Cataloging and Usage
    • Iceberg vs Hive
    • Page 1
  • SQL - A Complete Guide
    • SQL Queries
  • Data Engineer Interview
  • Elasticsearch
  • Lambda Architecture
  • Technology Stack
  • Machine Learning
    • Learning rate warm-up
  • System Designs
    • Thinking Mental Model
    • Design Fundamental Systems
      • Design Rate Limiter
      • Design Key-Value Store
      • Design a Web Crawler
      • Design Logging and Monitoring System
    • Design Spotify
    • Design ChatGPT
    • Design Tinder
    • Design Twitter
    • Design Hotel Booking System
    • Design Calender Management System
    • Design Reddit
    • Design Whatsapp
    • Design Youtube
    • Design Instagram
    • To Read
Powered by GitBook
On this page

Was this helpful?

  1. Building Blocks

Web Scrapper and Automation

PreviousCachingNextParallel Processing

Last updated 3 years ago

Was this helpful?

Scrapping static data from the website

For scrapping particular information from loaded web-page, we will use python library Beautifulsoup

Scrapping dynamic data from the website (after logging in, into the site)

Scrapping data from the website with dynamic data and iframe, which also require to enter input

How to avoid getting blocked or detected?

  • Use multiple IP (proxies) and keep rotating them (to make less requests from a single IP)

Third party service (example Luminati / Bright data) for re-routing the IP (the IP from where requests are being made)

# when testing proxies, get location of the proxy
print(browser.get('http://lumtest.com/myip.json'))

  • Wait on every page to let the data load on the browser

  • Once done with browsing, close the browser

  • Impersonate human like behaviour while surfing the website

    • Scrolling over the page

    • Use random, proper pause on the website

  • Captcha solving service

    1. If you are scraping pages faster than a human possibly can, you will fall into a category called “bots”.

    2. Following the same pattern while scraping. Like for example, you are going through every page of that target domain for just collecting images or links.

    3. If you are scraping using the same IP for a certain period of time.

    4. User Agent missing. Maybe you are using a headerless browser like

If login and password is required, then it is difficult to scrap a lot of data from a single account without being getting noticed.

Example: Linkedin disables a lot of data for a robot if number of monthly requests exceeds a particular number (different for premium and other accounts)

Scrapping is not legal!!

How to speed up the scrapping?

  1. To speed up the processing, multiprocessing using Celery

  2. Avoid loading images, ads and other data which is not required, into the browser

Flower UI for monitoring of scrapping and task completion

To deploy on browser-less server, selenium-grid docker image, which provides support for running multiple sessions and multiple browsers

Reference for Further Reading

Selenium is a headless browser, so we can not send header with the request to further lower down being detected as robot scrapping the website. ...

Tor Browser
read more
LogoHow To Avoid Being Blocked With Web Scraping Best PracticesJust Understanding Data
LogoHow to avoid getting blocked while web scraping
Logo10 Tips to avoid getting Blocked while Scraping Websites | CodementorCodementor
LogoWeb Scraping without getting blocked | ScrapingBeescrapingbee