用 Python 爬取起點(diǎn)小說(shuō)網(wǎng)

目標(biāo)

爬取一本仙俠類的小說(shuō)下載并保存為txt文件到本地。本例為“大周仙吏”。

項(xiàng)目準(zhǔn)備

軟件：Pycharm

第三方庫(kù)：requests,fake_useragent,lxml

網(wǎng)站地址：https://book.qidian.com

網(wǎng)站分析

打開(kāi)網(wǎng)址：

網(wǎng)址變?yōu)椋?/span>https://book.qidian.com/info/1020580616#Catalog

判斷是否為靜態(tài)加載網(wǎng)頁(yè)，Ctrl+U打開(kāi)源代碼，Ctrl+F打開(kāi)搜索框，輸入：第一章。

在這里是可以找到的，判定為靜態(tài)加載。

反爬分析

同一個(gè)ip地址去多次訪問(wèn)會(huì)面臨被封掉的風(fēng)險(xiǎn)，這里采用fake_useragent，產(chǎn)生隨機(jī)的User-Agent請(qǐng)求頭進(jìn)行訪問(wèn)。

代碼實(shí)現(xiàn)

1.導(dǎo)入相對(duì)應(yīng)的第三方庫(kù)，定義一個(gè)class類繼承object，定義init方法繼承self，主函數(shù)main繼承self。

import??requests
from?fake_useragent?import?UserAgent
from?lxml?import?etree
class?photo_spider(object):
????def?__init__(self):
????????self.url?=?'https://book.qidian.com/info/1020580616#Catalog'
????????ua?=?UserAgent(verify_ssl=False)
????????#隨機(jī)產(chǎn)生user-agent
????????for?i?in?range(1,?100):
????????????self.headers?=?{
????????????????'User-Agent':?ua.random
????????????}
????def?mian(self):
?????pass
if?__name__?==?'__main__':
????spider?=?qidian()
????spider.main()

2.發(fā)送請(qǐng)求,獲取網(wǎng)頁(yè)。

????def?get_html(self,url):
????????response=requests.get(url,headers=self.headers)
????????html=response.content.decode('utf-8')
????????return?html

3.獲取圖片的鏈接地址。

import?requests
from?lxml?import?etree
from?fake_useragent?import?UserAgent
class?qidian(object):
????def?__init__(self):
????????self.url?=?'https://book.qidian.com/info/1020580616#Catalog'
????????ua?=?UserAgent(verify_ssl=False)
????????for?i?in?range(1,?100):
????????????self.headers?=?{
????????????????'User-Agent':?ua.random
????????????}
????def?get_html(self,url):
????????response=requests.get(url,headers=self.headers)
????????html=response.content.decode('utf-8')
????????return?html
????def?parse_html(self,html):
????????target=etree.HTML(html)
????????links=target.xpath('//ul[@class="cf"]/li/a/@href')#獲取鏈接
????????names=target.xpath('//ul[@class="cf"]/li/a/text()')#獲取每一章的名字
????????for?link,name?in?zip(links,names):
????????????print(name+'\t'+'https:'+link)
????def?main(self):
????????url=self.url
????????html=self.get_html(url)
????????self.parse_html(html)
if?__name__?==?'__main__':
????spider=qidian()
????spider.main()

打印結(jié)果：

4.解析鏈接，獲取每一章內(nèi)容。

????def?parse_html(self,html):
????????target=etree.HTML(html)
????????links=target.xpath('//ul[@class="cf"]/li/a/@href')
????????for?link?in?links:
????????????host='https:'+link
????????????#解析鏈接地址
????????????res=requests.get(host,headers=self.headers)
????????????c=res.content.decode('utf-8')
????????????target=etree.HTML(c)
????????????names=target.xpath('//span[@class="content-wrap"]/text()')
????????????results=target.xpath('//div[@class="read-content?j_readContent"]/p/text()')
????????????for?name?in?names:
????????????????print(name)
????????????for?result?in?results:
????????????????print(result)

打印結(jié)果：（下面內(nèi)容過(guò)多，只貼出一部分。）

5.保存為txt文件到本地。

?with?open('F:/pycharm文件/document/'?+?name?+?'.txt',?'a')?as?f:
??????for?result?in?results:
??????????#print(result)
??????????f.write(result+'\n')

效果顯示：

打開(kāi)文件目錄：

完整代碼

import?requests
from?lxml?import?etree
from?fake_useragent?import?UserAgent
class?qidian(object):
????def?__init__(self):
????????self.url?=?'https://book.qidian.com/info/1020580616#Catalog'
????????ua?=?UserAgent(verify_ssl=False)
????????for?i?in?range(1,?100):
????????????self.headers?=?{
????????????????'User-Agent':?ua.random
????????????}
????def?get_html(self,url):
????????response=requests.get(url,headers=self.headers)
????????html=response.content.decode('utf-8')
????????return?html
????def?parse_html(self,html):
????????target=etree.HTML(html)
????????links=target.xpath('//ul[@class="cf"]/li/a/@href')
????????for?link?in?links:
????????????host='https:'+link
????????????#解析鏈接地址
????????????res=requests.get(host,headers=self.headers)
????????????c=res.content.decode('utf-8')
????????????target=etree.HTML(c)
????????????names=target.xpath('//span[@class="content-wrap"]/text()')
????????????results=target.xpath('//div[@class="read-content?j_readContent"]/p/text()')
????????????for?name?in?names:
????????????????print(name)
????????????????with?open('F:/pycharm文件/document/'?+?name?+?'.txt',?'a')?as?f:
????????????????????for?result?in?results:
????????????????????????#print(result)
????????????????????????f.write(result+'\n')
????def?main(self):
????????url=self.url
????????html=self.get_html(url)
????????self.parse_html(html)
if?__name__?==?'__main__':
????spider=qidian()
????spider.main()