數(shù)據(jù)自動(dòng)抓取到底有多簡(jiǎn)單?
說(shuō)到數(shù)據(jù)自動(dòng)抓取，很多人可能會(huì)聯(lián)想到網(wǎng)絡(luò)爬蟲，不自覺地就會(huì)認(rèn)為這超出了自己的能力圈邊界。但是，當(dāng)我們對(duì)互聯(lián)網(wǎng)上發(fā)現(xiàn)的某些數(shù)據(jù)感興趣的時(shí)候，我們知道如何自動(dòng)保存這些網(wǎng)頁(yè)數(shù)據(jù)以供日后數(shù)據(jù)分析，節(jié)省下來(lái)手工復(fù)制下載的時(shí)間，喝杯咖啡、刷會(huì)兒抖音，是不是很香？web 開發(fā)有兩個(gè)基本概念：前端和后端。簡(jiǎn)單來(lái)說(shuō)，瀏覽器端、客戶端就是前端，提供真實(shí)服務(wù)的服務(wù)器端就是后端。網(wǎng)頁(yè)開發(fā)模式主要分為兩類：前后端分離模式和非前后端分離模式。前后端分離模式，是指前端通過(guò) AJAX 向后端 API 服務(wù)發(fā)送數(shù)據(jù)請(qǐng)求，后端響應(yīng)請(qǐng)求并返回?cái)?shù)據(jù)給前端，前端使用后端返回的數(shù)據(jù)（通常是 JSON 數(shù)據(jù)）渲染呈現(xiàn)網(wǎng)頁(yè)。非前后端分離模式是由服務(wù)器端渲染生成 HTML 網(wǎng)頁(yè)，直接發(fā)送給瀏覽器端渲染。前后端分離模式是當(dāng)前網(wǎng)頁(yè)開發(fā)的主流模式，互聯(lián)網(wǎng)大廠幾乎都使用這種模式，但是一些政府事業(yè)單位的網(wǎng)站以外包為主，更新迭代緩慢，非前后端分離模式較多。關(guān)于這兩種開發(fā)模式，只需當(dāng)作科普理解即可。CSV vs JSONJSON 數(shù)據(jù)是 web 開發(fā)中經(jīng)常用到的數(shù)據(jù)格式，很多人經(jīng)常使用 CSV 格式，但是對(duì) JSON 數(shù)據(jù)格式不太熟悉?，F(xiàn)在有很多在線工具可以輕松實(shí)現(xiàn) CSV 格式與 JSON 格式之間的相互轉(zhuǎn)換。便于理解 JSON 數(shù)據(jù)，這里簡(jiǎn)單對(duì)比一下 CSV 和 JSON。
CSV 格式：
JSON格式：ID,Name,Season,Points1,LeBron James,2019-20,25.21,LeBron James,2018-19,27.41,LeBron James,2017-18,27.
[{ "id": 1, "name": "LeBron James", "season": "2019-20", "points": 25.2 },{ "id": 1, "name": "LeBron James", "season": "2018-19", "points": 27.4 },{ "id": 1, "name": "LeBron James", "season": "2017-18", "points": 27.5 }]
可以看出，相比CSV，JSON有更多的符號(hào)以及重復(fù)的表頭字段名。雖然這個(gè)例子是一種比較常見的 JSON 格式，但有時(shí)候 JSON 也會(huì)寫成 和 CSV 相似的樣式，比如 stats.nba.com 這個(gè)網(wǎng)站里使用的樣式:
直接使用 API 爬取數(shù)據(jù)現(xiàn)在流行的前后端分離開發(fā)模式下，web 網(wǎng)頁(yè)應(yīng)用一般通過(guò) AJAX 向后端 API 服務(wù)接口發(fā)送數(shù)據(jù)請(qǐng)求，前端使用 API 返回的數(shù)據(jù)（通常是 JSON 格式）渲染呈現(xiàn)網(wǎng)頁(yè)。因此，我們只要找到返回請(qǐng)求數(shù)據(jù)的 API 接口，并模擬發(fā)送請(qǐng)求給這個(gè) API，我們就能拿到想要的數(shù)據(jù)了。這里我們用 stats.nba.com 這個(gè)網(wǎng)站為例，使用 Node.js 來(lái)演示如何使用 API 自動(dòng)抓取 NBA 籃球運(yùn)動(dòng)員的年度職業(yè)數(shù)據(jù)。1. 檢查數(shù)據(jù)是否動(dòng)態(tài)加載首先，我們使用 Chrome 瀏覽器去 stats.nba.com 網(wǎng)站上找到詹姆斯的數(shù)據(jù)主頁(yè)https://www.nba.com/stats/player/2544/{"headers": ["id", "name", "season", "points"],"rows": [[1, "LeBron James", "2019-20", 25.2],[1, "LeBron James", "2018-19", 27.4],[1, "LeBron James", "2017-18", 27.5]]}
2.?尋找返回?cái)?shù)據(jù)的?API 接口數(shù)據(jù)不在 HTML頁(yè)面里，到底在哪里呢？我們可以使用瀏覽器的開發(fā)人員工具尋找線索。在網(wǎng)頁(yè)上鼠標(biāo)單擊右鍵，選擇?檢查?菜單調(diào)出開發(fā)人員工具。
3. 寫一個(gè) Node.js 腳本自動(dòng)爬取多頁(yè)面數(shù)據(jù)自動(dòng)爬取數(shù)據(jù)你可以使用任何語(yǔ)言，這里只是以 node.js 為例，給大家演示。安裝 request 庫(kù)
復(fù)制下面代碼到 index.js 文件npm init -ynpm install --save request request-promise-native
命令窗口運(yùn)行腳本const rp = require("request-promise-native");const fs = require("fs");async function main() {console.log("Making API Request...");// request the data from the JSON APIconst results = await rp({????uri:?"https://stats.nba.com/stats/playerdashboardbyyearoveryear?DateFrom=&DateTo=&GameSegment=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlayerID=2544&PlusMinus=N&Rank=N&Season=2020-21&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&Split=yoy&VsConference=&VsDivision=" ,headers: {"Connection": "keep-alive","User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36","x-nba-stats-origin": "stats","Referer": "https://stats.nba.com/player/2544/"},json: true});console.log("Got results =", results);// save the JSON to diskawait fs.promises.writeFile("output.json", JSON.stringify(results, null, 2));console.log("Done!")}// start the main scriptmain();
node index.jshttps://stats.nba.com/stats/playerdashboardbyyearoveryear?對(duì)比其他球員職業(yè)數(shù)據(jù)的 API 接口路徑，發(fā)現(xiàn)只要修改這個(gè) playerID 就可以變成對(duì)應(yīng)球員的 API 接口。根據(jù)球員主頁(yè)的路徑，能夠獲取球員對(duì)應(yīng)的 playerID。
DateFrom=&DateTo=&GameSegment=&LastNGames=0&LeagueID=00&
Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&
PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&
PlayerID=2544&
PlusMinus=N&Rank=N&Season=2019-20&SeasonSegment=&
SeasonType=Regular+Season&ShotClockRange=&Split=yoy&
VsConference=&VsDivision=
| Player ID | Player | URL |
| 2544 | LeBron James | https://stats.nba.com/player/2544/ |
| 1629029 | Luka Doncic | https://stats.nba.com/player/1629029/ |
| 201935 | James Harden | https://stats.nba.com/player/201935/ |
| 202695 | Kawhi Leonard | https://stats.nba.com/player/202695/ |
我們提取了一個(gè)請(qǐng)求數(shù)據(jù)的公共函數(shù)?fetchPlayerYearOverYear，遍歷 ID 數(shù)組，并調(diào)用這個(gè)函數(shù)就能一次獲取需要的全部數(shù)據(jù)。另外，出于對(duì)服務(wù)器的保護(hù)以及道德操守，我們一般要設(shè)置延遲來(lái)避免造成服務(wù)器堵塞，影響其他用戶的訪問。通過(guò)上述步驟，我們就成功的使用 API 實(shí)現(xiàn)了數(shù)據(jù)自動(dòng)抓取。那如果是非前后端分離模式的 web 網(wǎng)頁(yè)應(yīng)用，沒有 API 接口，我們?cè)趺磳?shí)現(xiàn)自動(dòng)抓取呢？自動(dòng)抓取服務(wù)端渲染的 HTML 網(wǎng)頁(yè)數(shù)據(jù)非前后端分離模式的 web 應(yīng)用是將數(shù)據(jù)直接呈現(xiàn)在 HTML 頁(yè)面內(nèi)，這種情況我們只要下載、解析 HTML 就能提取到我們的目標(biāo)數(shù)據(jù)。下面還是用案例來(lái)演示，目標(biāo)是從 espn.com 網(wǎng)站上爬取 NBA 比賽數(shù)據(jù)。1. 驗(yàn)證數(shù)據(jù)是否在?HTML?頁(yè)面內(nèi)和上一個(gè)案例相似，我們目標(biāo)是爬取籃球比賽得分?jǐn)?shù)據(jù)，地址：?https://www.espn.com/nba/boxscore?gameId=401160888。const rp = require("request-promise-native");const fs = require("fs");// helper to delay execution by 300ms to 1100msasync function delay() {const durationMs = Math.random() * 800 + 300;return new Promise(resolve => {setTimeout(() => resolve(), durationMs);});}async function fetchPlayerYearOverYear(playerId) {console.log(`Making API Request for ${playerId}...`);// add the playerId to the URI and the Referer header// NOTE: we could also have used the `qs` option for the// query parameters.const results = await rp({uri: "https://stats.nba.com/stats/playerdashboardbyyearoveryear?DateFrom=&DateTo=&GameSegment=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&" +`PlayerID=${playerId}` +"&PlusMinus=N&Rank=N&Season=2019-20&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&Split=yoy&VsConference=&VsDivision=",headers: {"Connection": "keep-alive","User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36","x-nba-stats-origin": "stats","Referer": `https://stats.nba.com/player/${playerId}/`},json: true});// save to disk with playerID as the file nameawait fs.promises.writeFile(`${playerId}.json`,JSON.stringify(results, null, 2));}async function main() {// PlayerIDs for LeBron, Harden, Kawhi, Lukaconst playerIds = [2544, 201935, 202695, 1629029];console.log("Starting script for players", playerIds);// make an API request for each playerfor (const playerId of playerIds) {await fetchPlayerYearOverYear(playerId);// be polite to our friendly data hosts and// don't crash their serversawait delay();}console.log("Done!");}main();
2. 找到可以定位目標(biāo)數(shù)據(jù)的 CSS 元素選擇器我們需要用某些標(biāo)識(shí)來(lái)識(shí)別出HTML里目標(biāo)數(shù)據(jù)，而 CSS 選擇器是一個(gè)非常好的標(biāo)識(shí)。鼠標(biāo)右鍵單擊頁(yè)面-檢查，調(diào)出開發(fā)人員工具。
document.querySelectorAll('tr')
document.querySelectorAll('.gamepackage-away-wrap tbody tr')
ok，大功告成了！接下來(lái)，我們要做的是使用Node.js 的 request 庫(kù)下載 HTML頁(yè)面，使用cheerio 庫(kù)解析HTML頁(yè)面并提取目標(biāo)數(shù)據(jù)。document.querySelectorAll('.gamepackage-away-wrap tbody tr:not(.highlight)')> NodeList(13) [tr, tr, tr, tr, tr, tr, tr, tr, tr, tr, tr, tr, tr]
3.?下載 HMTL 頁(yè)面和上個(gè)案例一樣，我們先按照 request 庫(kù)，命令行窗口輸入：
然后新建一個(gè) index.js 文件，輸入以下代碼npm init -ynpm install --save request request-promise-native
運(yùn)行這個(gè)程序const rp = require('request-promise-native');const fs = require('fs');async function downloadBoxScoreHtml() {// where to download the HTML fromconst uri = 'https://www.espn.com/nba/boxscore?gameId=401160888';// the output filenameconst filename = 'boxscore.html';// check if we already have the fileconst fileExists = fs.existsSync(filename);if (fileExists) {console.log(`Skipping download for ${uri} since ${filename} already exists.`);return;}// download the HTML from the web serverconsole.log(`Downloading HTML from ${uri}...`);const results = await rp({ uri: uri });// save the HTML to diskawait fs.promises.writeFile(filename, results);}async function main() {console.log('Starting...');await downloadBoxScoreHtml();console.log('Done!');}main();
node index.js4. 使用 Cheerio 解析 HTMLCheerio 是從 HTML 頁(yè)面里提取數(shù)據(jù)的最佳工具庫(kù)，跟 jQuery 非常類似。安裝 Cheerio
npm install --save cheerio第一步，使用?Cheerio.load 函數(shù)讀取 html文件內(nèi)容
第二步，使用 css 選擇器提取數(shù)據(jù)，跟?jQuery 語(yǔ)法一樣// the input filenameconst htmlFilename = 'boxscore.html';// read the HTML from diskconst html = await fs.promises.readFile(htmlFilename);// parse the HTML with Cheerioconst $ = cheerio.load(html);
const $trs = $('.gamepackage-away-wrap tbody tr:not(.highlight)')進(jìn)一步使用選擇器遍歷我們提取到的 html 片段，就能得到目標(biāo)數(shù)據(jù)。<tr><td class="name"><a name="&lpos=nba:game:boxscore:playercard" href="https://www.espn.com/nba/player/_/id/6440/tobias-harris" data-player-uid="s:40~l:46~a:6440"><span>T. Harris</span><span class="abbr">T. Harris</span></a><span class="position">SF</span></td><td class="min">38</td><td class="fg">7-17</td><td class="3pt">3-7</td><td class="ft">1-1</td><td class="oreb">0</td><td class="dreb">5</td><td class="reb">5</td><td class="ast">2</td><td class="stl">1</td><td class="blk">0</td><td class="to">2</td><td class="pf">3</td><td class="plusminus">-10</td><td class="pts">18</td></tr>...
數(shù)據(jù)結(jié)果：const values = $trs.toArray().map(tr => {// find all children <td>const tds = $(tr).find('td').toArray();// create a player object based on the <td> valuesconst player = {};for (td of tds) {// parse the <td>const $td = $(td);// map the td class attr to its valueconst key = $td.attr('class');const value = $td.text();player[key] = value;}return player;});
現(xiàn)在你學(xué)會(huì)了使用 cheerio 庫(kù)。接下來(lái)我們新建 index.js 文件，輸入以下代碼并運(yùn)行，即可獲取到我們的目標(biāo)數(shù)據(jù)。[{"name": "T. HarrisT. HarrisSF","min": "38","fg": "7-17","3pt": "3-7","ft": "1-1","oreb": "0","dreb": "5","reb": "5","ast": "2","stl": "1","blk": "0","to": "2","pf": "3","plusminus": "-10","pts": "18"}...]
小結(jié)總結(jié)一下，自動(dòng)抓取數(shù)據(jù)根據(jù) web 應(yīng)用開發(fā)模式的不同，分為兩種方法。前后端分離模式下，我們可以直接使用 API 自動(dòng)抓取數(shù)據(jù)；而非前后端分離模式下，我們需要先下載 HTML 網(wǎng)頁(yè)，再解析 HTML 并提取數(shù)據(jù)。你學(xué)會(huì)了嗎？參考網(wǎng)站：https://beshaimakes.com/js-scrape-data最后，如果覺得不錯(cuò)，不要忘記你的三連，留下你的三連是對(duì)我的最大支持！const rp = require('request-promise-native');const fs = require('fs');const cheerio = require('cheerio');async function downloadBoxScoreHtml() {// where to download the HTML fromconst uri = 'https://www.espn.com/nba/boxscore?gameId=401160888';// the output filenameconst filename = 'boxscore.html';// check if we already have the fileconst fileExists = fs.existsSync(filename);if (fileExists) {console.log(`Skipping download for ${uri} since ${filename} already exists.`);return;}// download the HTML from the web serverconsole.log(`Downloading HTML from ${uri}...`);const results = await rp({ uri: uri });// save the HTML to diskawait fs.promises.writeFile(filename, results);}async function parseBoxScore() {console.log('Parsing box score HTML...');// the input filenameconst htmlFilename = 'boxscore.html';// read the HTML from diskconst html = await fs.promises.readFile(htmlFilename);// parse the HTML with Cheerioconst $ = cheerio.load(html);// Get our rowsconst $trs = $('.gamepackage-away-wrap tbody tr:not(.highlight)');const values = $trs.toArray().map(tr => {// find all children <td>const tds = $(tr).find('td').toArray();// create a player object based on the <td> valuesconst player = {};for (td of tds) {const $td = $(td);// map the td class attr to its valueconst key = $td.attr('class');let value;if (key === 'name') {value = $td.find('a span:first-child').text();} else {value = $td.text();}player[key] = isNaN(+value) ? value : +value;}return player;});return values;}async function main() {console.log('Starting...');await downloadBoxScoreHtml();const boxScore = await parseBoxScore();// save the scraped results to diskawait fs.promises.writeFile('boxscore.json',JSON.stringify(boxScore, null, 2));console.log('Done!');}main();
如果你喜歡，請(qǐng)點(diǎn)點(diǎn)關(guān)注啦~
以上
評(píng)論
圖片
表情