Using the scraper¶
Add user agent strings that you would like to crawl with by running the management command:
python manage.py useragents --add "Firefox / 15.0" --desktop python manage.py useragents --add "Fennec / 15.0" --primary python manage.py useragents --add "Android / WebKit"
Detecting UA-sniffing issues requires at least three user-agents to be added: a desktop user-agent to be used as baseline, a “primary” mobile user agent (the one we want to make sure sites are sniffing, if they sniff mobile UAs at all), and at least one other mobile UA to check against. A “UA sniffing issue” will be reported for a URL if that URL returns markedly different content for any non-primary mobile UA (compared to the desktop UA content), but returns the desktop content for the primary mobile UA.
Call the scrape command, giving it a text file of URLs to parse:
python manage.py scrape [newline delimited text file of URLS]