The Scenario
You do competitive intelligence for a D2C brand. You've tried scraping 30 competitor URLs before. Half of them are behind Cloudflare. Standard HTTP requests come back with a CAPTCHA page. The last time you tried building a scraper for these, it worked for three days and then stopped.
The 30 URLs are sitting in column A of your Excel workbook. You need product names and prices from each one. The sites have bot protection that specifically blocks the tools you'd normally reach for.
The bad version:
- Try a standard HTTP request library. Get Cloudflare challenge pages for 22 of 30 URLs.
- Switch to a Selenium script. Get blocked again after a few requests when the site fingerprints the browser.
- Spend three hours configuring rotating proxies. Get partial results. Document nothing. Repeat next month.
The issue isn't that web scraping is hard in general. It's that anti-bot-protected sites require infrastructure — stealth proxies, browser fingerprint rotation, IP diversity — that takes longer to configure than the data you're trying to get is worth. Unless you already have that infrastructure standing by.
The Easy Way: One Prompt in SheetXAI
SheetXAI is an AI agent that lives inside your Excel workbook. It reads the workbook and through its built-in ScrapingBee integration it can route requests through stealth proxy mode — the same setup that bypasses Cloudflare and equivalent bot-detection systems — writing extracted fields directly back into your columns.
For each URL in column A that is protected by Cloudflare or anti-bot measures, use ScrapingBee stealth proxy mode to scrape the page and extract product name and price into columns B and C.
What You Get
- Column B: product name as extracted from the page — title tag, H1, or product name element depending on the site's structure.
- Column C: price as listed on the page, including currency symbol.
- Rows where stealth proxy still couldn't access the page flagged in column D with "BLOCKED" — these are the sites worth reviewing manually or escalating.
- All 30 URLs attempted in one pass with stealth mode active, no proxy configuration required on your end.
What If the Data Is Not Quite Ready
Some pages require JavaScript rendering in addition to stealth proxy — they load prices dynamically after the initial HTML
For each URL in column A, use ScrapingBee with both stealth proxy and JavaScript rendering enabled to extract product name and price into columns B and C — if the price element isn't present in the initial HTML, wait for dynamic content before extracting.
The product names come back with trailing descriptors or variant info that makes them hard to match against your catalog
After extracting product names into column B using ScrapingBee stealth mode, clean the values in column D — strip parenthetical variant descriptions, remove trailing size or color suffixes, and standardize the names so they can be matched against our catalog in the Product Master worksheet.
You need to flag only the competitor products that overlap with your own SKU list
After scraping product names and prices into columns B and C using ScrapingBee stealth proxy, check each product name in column B against the SKU names in column A of the Our Catalog worksheet — write "OVERLAP" into column D if there's a match, "NEW" if there isn't, so I can filter to competitive overlaps only.
Kill chain: stealth scrape all 30 URLs, extract name and price, clean the names, match against catalog, and flag price undercuts
For each URL in column A, use ScrapingBee stealth proxy with JavaScript rendering to extract product name and price — clean the product names to remove variant descriptors, match against the SKU list in Our Catalog, write the match status into column D, and flag any matched row where the competitor price is lower than our price with "UNDERCUT" in column E.
Scraping, cleaning, and competitive analysis in a single prompt is the point — not three separate operations stitched together manually.
Try It
Get the 7-day free trial of SheetXAI and open any Excel workbook with anti-bot-protected competitor URLs in column A, then ask it to use ScrapingBee stealth proxy to extract product data. See also: Compile E-Commerce Search Results Into a Product Table in an Excel workbook and the full ScrapingBee integration overview.