旧笔记本 做网站/中国数据统计网站

糗事百科url为https://www.qiushibaike.com/text/page/1/
其中后面的/1/表示第几页，这里只爬取前面7页，具体看下面的代码。
（在chrome浏览器中，利用开发者工具，查看到的元素和在pycharm中requests.get请求下来的网页不一样，标签变了，导致按照视频教程的正则匹配不到。于是，自己在pycharm中查看标签，重新写正则匹配式。）

#!/usr/bin/env python 
# -*- coding:utf-8 -*-import requests
import renum = 0
def parse_page(url):headers = {'User-Agent': "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) ""Chrome/72.0.3610.2 Mobile Safari/537.36",'Referer': "https://www.qiushibaike.com/text/"}response = requests.get(url, headers=headers)text = response.text#print(text)# contents = re.findall(r'<div\sclass="content">.*?<span>\s*(.*?)\s*</span>', text, re.DOTALL)contents = re.findall(r'<a href="/article/.*?>\s*(.*?)\s*</a>', text, re.DOTALL)#print(len(contents))duanzi = []for content in contents:content = content.replace('<br/>', '')content = re.sub(r'<span class="fullArtitle" >.*?</span>', '', content) #去掉“查看全文”content = re.sub(r'<div([\s\S]*)</div>', '', content)  # 去掉多余的divcontent = content.strip()print(content)duanzi.append(content)global numnum = num+1def main():base_url = 'https://www.qiushibaike.com/text/page/{}/'for x in range(1, 8):url = base_url.format(x)parse_page(url)print(num)if __name__ == '__main__':main()

使用正则表达式匹配任意字符包括空格和换行符，参考：
https://my.oschina.net/zchuanzhao/blog/849948