Web scraping & data pipeline development: Playwright, proxies, monitoring

Get the data that lives behind a login, a portal or a hundred web pages.

I build production scrapers and data pipelines that log in to portals, collect data from websites with no API, validate what they find, and deliver clean data on a schedule. They come with monitoring, so you know the moment something changes.

Web Scraping & Data Pipelines, part of Data & Scraping

Three competitors' sofa prices, in your sheet before you open

Sound familiar?

Your team copies data from supplier portals, government sites or competitor pages by hand every week.

A freelancer's scraper worked for a month, then silently broke when the site changed.

Sites block you with captchas, rate limits or bot detection as soon as volume increases.

You have the data, but it's messy, duplicated and never quite in the format your team needs.

Who it's built for

Operations teams pulling data from supplier, utility, carrier or government portals

E-commerce and pricing teams tracking catalogues, prices and stock across sites

Startups that need a data feed for their product or AI model

Analysts and researchers who need structured data from public sources

Scrapers are easy. Reliable data pipelines are not.

Anyone can write a script that pulls a page and extracts a few fields. The hard part is everything after the first successful run:

the site changes its layout
the portal logs you out
the anti-bot system notices you
a field silently comes back empty, and nobody notices for three weeks

That’s the difference between a script and a data pipeline, and it’s what I build.

I built this kind of system in production for a property manager collecting utility bills for 735+ units across 13 providers (read the case study):

Plugin architecture. Each provider is its own module, so a change on one portal never breaks the others.
Browser automation that doesn’t get flagged. Playwright runs through rotating residential proxies, with sticky sessions per provider so logins look consistent.
Resilience built in. Circuit breakers pause a failing provider instead of hammering it. Every task has a timeout, and an hourly cleanup recovers stuck runs.
Validation and reconciliation. Extracted values are checked against a second source, and mismatches are flagged for review instead of passing through silently.
An operations dashboard. The team can see every run, failure and result without reading logs.

What I can collect

Portal data. Invoices, bills, statements, shipment status and reports from supplier, utility, carrier, bank or government portals you have access to.
Catalogue and pricing data. Products, prices, stock and reviews from e-commerce sites and marketplaces.
Listings and directories. Property listings, business directories, job posts and tenders.
Documents. PDFs and files linked from websites, passed on to document AI extraction to get structured fields out.

Doing it responsibly

Good scraping is polite scraping. I respect rate limits, cache aggressively, and fetch only what’s needed. I check terms of use and avoid collecting personal data unless there’s a clear lawful basis. Where a source offers an official API or data export, I’ll recommend using it. It’s usually more reliable and always less risky.

From raw data to decisions

Collecting the data is only step one. The pipeline can also:

normalise currencies and units
de-duplicate across sources
enrich records
push results into the tools your team uses: Google Sheets, a Postgres database, a BI dashboard, your CRM or an AI step that summarises and classifies what changed

You get a feed you can trust, not a folder of CSVs nobody opens.

What you get

Production scrapers with retries, timeouts and rate limiting

A pipeline that validates, de-duplicates and normalises the data

Scheduled runs on your infrastructure or managed hosting

Delivery to your database, warehouse, Google Sheets, API or app

Monitoring and alerts for failures and unexpected changes in the data

How the build runs

Source review

I check each source's structure, volume, login and anti-bot measures, and its terms of use, then propose the most reliable and appropriate approach, using an official API where one exists.

Build & validate

Scrapers are built with Playwright or HTTP clients as appropriate. Every record is validated against a schema, so bad data is caught instead of delivered.

Schedule & deliver

Runs are scheduled and queued, and results land wherever your team works, from Sheets to Postgres to your own API.

Monitor & maintain

Alerts fire on failures, empty results or suspicious changes. Optional maintenance plans keep scrapers working when sources change.

Built with tools like Python, Playwright, httpx, BeautifulSoup, Celery, Redis, PostgreSQL, FastAPI.

Questions owners ask

Is web scraping legal?

It depends on what you collect, how, and where you are. Collecting publicly available, non-personal data is generally lower risk. Bypassing logins you don't have rights to, ignoring terms you agreed to, or collecting personal data raises legal issues. I review each source's terms and the type of data with you, prefer official APIs where they exist, and won't build scrapers for clearly abusive use. For anything borderline, check with a lawyer.

What happens when the website changes?

Scrapers break when sites change. That's normal. Mine are built to fail loudly, not silently. Validation catches missing or malformed fields, alerts tell you immediately, and selectors are kept in one place so fixes are quick. Maintenance plans cover ongoing changes.

Can you scrape sites that require a login?

Yes, for accounts you're entitled to use, such as your own supplier, utility or carrier portals. Credentials are stored securely, and sessions are managed carefully so the portal isn't hammered.

How do you deal with blocking and captchas?

First by being polite. That means reasonable rate limits, caching and only fetching what's needed. Where volume requires it, rotating or residential proxies with sticky sessions help. I don't build tools designed to overwhelm or attack sites.

Where does the data end up?

Wherever it's useful. Google Sheets, Excel, CSV exports, PostgreSQL or a data warehouse, a webhook into your app, or straight into an AI pipeline for summarisation and classification.