[citation][nom]precariousgray[/nom]If Google's web crawler is somehow bypassing authentication and retrieving pay-for content, then it sounds like someone needs to work on their site's security.(Hint: the someone is not Google.)[/citation]
Google is not bypassing authentication. News Corp specifically allows Google searches in their robot.txt file.
Google is not bypassing authentication. News Corp specifically allows Google searches in their robot.txt file.
User-agent: *
Disallow: /printer_friendly_story
Disallow: /projects/livestream
#
User-agent: gsa-crawler
Allow: /printer_friendly_story
Allow: /google_search_index.xml
Allow: /google_news_index.xml
Allow: /*.xml.gz
#