二十一、数据采集实战
第2关：网站爬取策略
任务描述
本关任务：编写一个爬虫实现深度优先爬虫。
相关知识
主要介绍两种爬虫爬取策略：1. 深度优先爬虫； 2. 广度优先爬虫。
深度优先爬虫（一路到底）
在一个网页中，当一个超链被选择后，被链接的网页将执行深度优先搜索，即在搜索其余的超链结果之前必须先完整地搜索单独的一条链。深度优先搜索沿着网页上的超链走到不能再深入为止，然后返回到某一个网页，再继续选择该网页中的其他超链。当不再有其他超链可选择时，说明搜索已经结束。
示例：
爬取顺序为：1->2->4->8->5->3->6->7
广度优先爬虫（逐层爬取）
广度优先爬虫过程就是从一系列的种子节点开始，把这些网页中的“子节点” 提取出来，放入队列中依次进行抓取，被处理过的链接需要放入一张表中。每次新处理一个链接之前，需要查看这个链接是否已经存在于表中。如果存在，证明链接已经处理过， 跳过，不做处理，否则进行下一步处理。
示例：
爬取顺序为：1->2->3->4->5->6->7->8
编程要求
请仔细阅读右侧代码，结合相关知识，在 Begin-End 区域内进行代码补充，编写一个爬虫实现深度优先爬虫，爬取的网站为 www.baidu.com。
测试说明
平台会对你编写的代码进行测试：
下方预期输出随着网页的更新，中间的数据会有一定的变化。
预期输出：
Add the seeds url ['http://www.baidu.com'] to the unvisited url list
Pop out one url "http://www.baidu.com" from unvisited url list
Get 10 new links
Visited url count: 1
Visited deepth: 1
10 unvisited links:
Pop out one url "http://news.baidu.com" from unvisited url list
Get 52 new links
Visited url count: 2
Visited deepth: 2
Pop out one url "http://www.hao123.com" from unvisited url list
Get 311 new links
Visited url count: 3
Visited deepth: 2
Pop out one url "http://map.baidu.com" from unvisited url list
Get 0 new links
Visited url count: 4
Visited deepth: 2
Pop out one url "http://v.baidu.com" from unvisited url list
Get 566 new links
Visited url count: 5
Visited deepth: 2
Pop out one url "http://tieba.baidu.com" from unvisited url list
Get 21 new links
Visited url count: 6
Visited deepth: 2
Pop out one url "http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1" from unvisited url list
Get 1 new links
Visited url count: 7
Visited deepth: 2
Pop out one url "http://home.baidu.com" from unvisited url list
Get 27 new links
Visited url count: 8
Visited deepth: 2
Pop out one url "http://ir.baidu.com" from unvisited url list
Get 22 new links
Visited url count: 9
Visited deepth: 2
Pop out one url "http://www.baidu.com/duty/" from unvisited url list
Get 1 new links
Visited url count: 10
Visited deepth: 2
Pop out one url "http://jianyi.baidu.com/" from unvisited url list
Get 4 new links
Visited url count: 11
Visited deepth: 2
2 unvisited links:
Pop out one url "http://baozhang.baidu.com/guarantee/" from unvisited url list
Get 0 new links
Visited url count: 12
Visited deepth: 3
Pop out one url "http://ir.baidu.com/phoenix.zhtml?c=188488&p=irol-irhome" from unvisited url list
Get 22 new links
Visited url count: 13
Visited deepth: 3
22 unvisited links:
注意：右侧预期输出为部分输出。
开始你的任务吧，祝你成功！
from bs4 import BeautifulSoup
import requests
import re
#自定义队列类
class linkQuence:
def __init__(self):
# 已访问的url集合
self.visted = []
# 待访问的url集合
self.unVisited = []
# 获取访问过的url队列
def getVisitedUrl(self):
return self.visted
# 获取未访问的url队列
def getUnvisitedUrl(self):
return self.unVisited
# 添加到访问过得url队列中
def addVisitedUrl(self, url):
self.visted.append(url)
# 移除访问过得url
def removeVisitedUrl(self, url):
self.visted.remove(url)
# 未访问过得url出队列 深度优先 pop() 后进先出，栈
def unVisitedUrlDeQuence(self):
try:
return self.unVisited.pop()
except:
return None
# 保证每个url只被访问一次
def addUnvisitedUrl(self, url):
if url != "" and url not in self.visted and url not in self.unVisited:
self.unVisited.insert(0, url)
# 获得已访问的url数目
def getVisitedUrlCount(self):
return len(self.visted)
# 获得未访问的url数目
def getUnvistedUrlCount(self):
return len(self.unVisited)
# 判断未访问的url队列是否为空
def unVisitedUrlsEnmpy(self):
return len(self.unVisited) == 0
class MyCrawler:
def __init__(self, seeds):
# 初始化当前抓取的深度
self.current_deepth = 1
# 使用种子初始化url队列
self.linkQuence = linkQuence()
if isinstance(seeds, str):
self.linkQuence.addUnvisitedUrl(seeds)
if isinstance(seeds, list):
for i in seeds:
self.linkQuence.addUnvisitedUrl(i)
print("Add the seeds url %s to the unvisited url list" % str(self.linkQuence.unVisited))
# 抓取过程主函数
def crawling(self, seeds, crawl_deepth):
# ********** Begin **********#
# 循环条件：抓取深度不超过crawl_deepth
while self.current_deepth <= crawl_deepth:
# 循环条件：待抓取的链接不空
while not self.linkQuence.unVisitedUrlsEnmpy():
# 队头url出队列 深度优先 pop
url = self.linkQuence.unVisitedUrlDeQuence()
print('Pop out one url "%s" from unvisited url list' % url)
# 获取超链接
new_links = self.getHyperLinks(url)
print(f"Get {len(new_links)} new links")
# 将url放入已访问的url中
self.linkQuence.addVisitedUrl(url)
print(f"Visited url count: {self.linkQuence.getVisitedUrlCount()}")
print(f"Visited deepth: {self.current_deepth}")
if self.linkQuence.getUnvistedUrlCount() > 0:
print(f"{self.linkQuence.getUnvistedUrlCount()} unvisited links:")
# 未访问的url入列
for link in new_links:
self.linkQuence.addUnvisitedUrl(link)
self.current_deepth += 1
# ********** End **********#
# 获取源码中得超链接
def getHyperLinks(self, url):
# ********** Begin **********#
links = []
source = self.getPageSource(url)
if source is None:
return links
soup = BeautifulSoup(source, "html.parser")
a_list = soup.find_all("a", href=True)
for a in a_list:
href = a['href']
# 简单处理http开头链接
if href.startswith("http"):
links.append(href)
return links
# ********** End **********#
# 获取网页源码
def getPageSource(self, url):
# ********** Begin **********#
headers = {
"User‑Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/100.0.0.0 Safari/537.36"
}
try:
resp = requests.get(url, headers=headers, timeout=3)
resp.encoding = resp.apparent_encoding
return resp.text
except Exception:
return None
# ********** End **********#
def main(seeds, crawl_deepth):
craw = MyCrawler(seeds)
craw.crawling(seeds, crawl_deepth)
if __name__ == '__main__':
main("http://www.baidu.com", 3)
有任何问题都可以随时关注私信！
