The Scenario
Three days ago your team ran a Scrapfly crawl on a 300-page production site ahead of a migration. The crawl finished. The results are sitting in the Scrapfly dashboard. Now your engineering lead is asking for a full URL inventory with HTTP status codes in a workbook — specifically, they need to see all 404s and redirect chains before the migration window opens on Friday.
You do not have API access. The person who ran the crawl is traveling.
The bad version:
- Find the Scrapfly dashboard, locate the right crawl job, look for an export option, download a CSV with 12 columns you do not need, open it in Excel, delete the irrelevant columns, copy and paste the useful columns into the actual workbook you are working in, realize the formats do not match, reformat
- Discover that the export has 312 rows but your destination workbook already has some URL entries from a manual audit, figure out which rows overlap, deduplicate by hand
- Mark the 404s with red fill manually, find the redirect chains in a separate column, cross-reference them with another worksheet
The migration window is Friday. This needs to be done Thursday morning so engineering has time to act on it.
The Easy Way: One Prompt in SheetXAI
SheetXAI is an AI agent that lives inside your Excel workbook. Put the Scrapfly crawler ID in cell A1, and SheetXAI retrieves all the crawled URLs and status codes from the API and writes them into your workbook — with flagging logic built in — in a single pass.
Retrieve all crawled URLs and their status codes from the Scrapfly crawler with ID in cell A1 and write each URL and status code as a row in columns A and B, flagging any non-200 status in column C
What You Get
- Column A contains every URL discovered during the crawl, one per row
- Column B contains the HTTP status code returned for each URL (200, 301, 404, 500, etc.)
- Column C contains a flag for any non-200 status — "Redirect" for 3xx, "Not Found" for 404, "Server Error" for 5xx
- All rows are written in a single operation — no batching, no partial results
What If the Data Is Not Quite Ready
The crawl returned duplicate URLs from different internal linking paths
Large sites often link to the same page from multiple locations, which can produce duplicate entries in the crawl results. Ask SheetXAI to deduplicate before writing:
Retrieve all URLs and status codes from the Scrapfly crawler in cell A1, deduplicate by URL, and write the first occurrence of each URL with its status code into columns A and B, flagging non-200 statuses in column C
You need redirect chains, not just the final status
If a URL returns 301 but you need to know what it redirects to for the migration mapping:
Retrieve all URLs and status codes from the Scrapfly crawler with ID in cell A1 — for any 3xx status, write the redirect destination URL into column D and note the chain length in column E
You want to cross-reference the crawl results against your planned URL mapping
Your migration URL map is in a worksheet called New URLs with old paths in column A and new paths in column B. After pulling the crawl results, match them:
Pull all crawled URLs and status codes from the Scrapfly crawler in cell A1 into columns A and B of this worksheet, then in column D look up each URL against column A of the New URLs worksheet and write the planned new URL where a match exists
You want the full migration readiness report built in one shot
Retrieve all URLs and status codes from the Scrapfly crawler in cell A1, deduplicate, write into columns A and B, flag non-200 statuses in column C, look up each URL in the New URLs worksheet and write the redirect target in column D, then in column E mark any 404 that has no corresponding redirect mapping as Migration Risk
Engineering gets a complete readiness report. You did not open the Scrapfly dashboard once.
Try It
Get the 7-day free trial of SheetXAI and put your Scrapfly crawler ID in cell A1, then ask it to retrieve all crawled URLs with status codes and flag the broken links and redirects. If you also need to crawl a new site from scratch and export the results, see the spoke on crawling a competitor site for an SEO audit.