aiohttp+motor

• motor
• 代码

好久没写过爬虫了，今天看到有群友在问就看了下他的代码，大概是一个异步的爬取影视作品链接的爬虫，看到有别的大佬给他解决了问题后他发了张存入mongodb的截图，于是我也正好学习也复习下asyncio。

motor

Motor 提供了一个基于协程的 API，用于从Tornado或asyncio对 MongoDB 的非阻塞访问。Motor 在连接到 MongoDB 或执行 I/O 时从不阻塞事件循环。Motor 封装了几乎所有 PyMongo 的 API 并使其成为非阻塞的。

简单的介绍

代码

import reimport asyncio
import aiohttp
from lxml import etree
import motor.motor_asyncioclient = motor.motor_asyncio.AsyncIOMotorClient()
collection = client.film.kkxz_testasync def index_page(url):page_url_list = []async with aiohttp.ClientSession() as session:async with session.get(url) as req:res = await req.json()for page_url in res['data']:page_url_list.append('https://kkxz.vip/d/' + page_url['attributes']['slug'])await save_data(page_url_list)async def save_data(urls):for url in urls:async with aiohttp.ClientSession() as session:async with session.get(url) as req:text = await req.text()movie_link = re.findall('href="(https://pan.quark.cn.*?)"', text)movie_title = re.findall('
(.*?)
', text)print(movie_title[0], movie_link)await collection.insert_one({'title': movie_title[0],'link': movie_link})async def main():url = 'https://kkxz.vip/api/discussions?include=user%2ClastPostedUser%2CfirstPost%2Ctags%2Ctags.parent&filter%5Btag%5D=tv&sort&page%5Boffset%5D={id}'tasks = [asyncio.create_task(index_page(url.format(id = i))) for i in range(0, 20, 30)]await asyncio.wait(tasks)if __name__ == '__main__':asyncio.run(main())