The Scenario
Third time this sprint, your SEO lead has asked for a URL inventory of a competitor's site. The request keeps changing shape — first it was just the blog section, then the docs, now it's everything. You have 10 competitor domains in column A of a Google Sheet. You need a flat list of every discovered URL for each one, tagged by source domain, so the crawl analysis team can work with it.
The bad version:
- Download a desktop crawler like Screaming Frog, point it at the first domain, wait for it to finish, export the URLs as a CSV.
- Import that CSV into a new tab, clean up the columns, tag the domain, copy the rows into the master sheet.
- Repeat for each of the 10 domains, across 10 separate crawl jobs.
By the time you're done with domain three, domain one's structure has probably been updated.
The Easy Way: One Prompt in SheetXAI
SheetXAI is an AI agent inside your Google Sheet. It reads the domain list in column A, calls Tavily's site mapping capability for each one, and appends discovered URLs as new rows — tagged by source domain.
For each domain in column A (rows 2 through 11), use Tavily to map the site and write every discovered URL as a new row below the current data in this sheet. Tag each row with the source domain in column B and the discovered URL in column A.
What You Get
- New rows appended below your domain list, one per discovered URL.
- Column A contains each discovered URL; column B contains the source domain it came from.
- All 10 domains processed in a single pass — no separate crawl jobs per site.
- If Tavily finds no URLs for a domain (blocked, invalid, or empty), a single row is written with "NO PAGES FOUND" in column A and the domain in column B.
What If the Data Is Not Quite Ready
Some domains in column A include protocol prefixes or trailing slashes that need normalizing
Normalize all domains in column A — remove http://, https://, and trailing slashes — then use Tavily to map each cleaned domain and append discovered URLs as new rows, tagging each with the source domain in column B.
You only want URLs from the /blog or /docs path of each site
For each domain in column A, use Tavily to map only pages under the /blog and /docs paths and append the discovered URLs as new rows, tagging each with the source domain in column B and the path section (/blog or /docs) in column C.
You need a count of discovered URLs per domain written into the original domain rows
Map each domain in column A using Tavily, append discovered URLs as new rows tagged by domain in column B, and write the total count of discovered URLs back into column C of the original domain's row.
Normalize, map, filter, and count in one pass
Clean all domains in column A (strip protocols and trailing slashes), then use Tavily to map each one. Append only URLs under /blog, /docs, or /pricing as new rows tagged with the source domain in column B. Write the count of discovered URLs per domain back into column C of the original domain row. Flag domains that returned zero results with "NO PAGES FOUND" in column C.
Everything — normalization, filtering, mapping, and count rollup — in one prompt.
Try It
Get the 7-day free trial of SheetXAI and open any Google Sheet with a column of competitor domains, then ask it to build a flat URL inventory for each one using Tavily. Or explore crawling a specific site section and extracting clean page content.