| Tool | Best for |
|---|
| Firecrawl | General scraping, JS-heavy sites |
| Apify | Complex scenarios, pre-built scrapers |
| HTTP requests | Simple APIs, static pages |
Basic Scraping
Use @Firecrawl to scrape https://example.com/products
Use @Firecrawl to scrape https://example.com/products and extract:
- Product name
- Price
- Description
- Image URL
Crawl Multiple Pages
Use @Firecrawl to crawl all product pages on example.com
Start from /products
Limit to 100 pages
Example: Price Monitoring
Step 1 — Define targets:
Create a list of competitor product URLs to monitor
Use @Firecrawl to scrape each URL and extract the current price
Compare to our prices in @Google Sheets
If any competitor is 10%+ cheaper, alert @Slack
Common Challenges
JavaScript-rendered content
Firecrawl handles this automatically. For other methods:
Wait for the page to fully load before extracting
Follow the "Next" button until no more pages
Rate limiting
Add 2-second delay between requests
Authentication
Log in first, then scrape the dashboard
Best Practices
Always check if scraping is permitted before proceeding.
- Respect robots.txt — Check if scraping is allowed
- Rate limit — Don’t overwhelm servers
- Handle errors — Sites change, pages disappear
- Check ToS — Ensure scraping is permitted
- Mind privacy laws — GDPR, CCPA for personal data
Next
