urllib3--爬取网页内容

简介

Urllib3是一个功能强大，条理清晰，用于HTTP客户端的Python包

urllib3网址：https://pypi.org/project/urllib3/

Urllib3提供了很多python标准库里所没有的重要特性：

连接池&线程安全

客户端SSL/TLS验证

文件分部编码上传

协助处理重复请求和HTTP重定位

支持压缩编码

支持HTTP和SOCKS代理

100%测试覆盖率

基本用法

1. 导入urllib3模块：

import urllib3

2. 创建一个PoolManager实例：

http = urllib3.PoolManager()

3、 通过request()方法创建一个请求，返回的r是一个HTTPResponse对象

r = http.request('GET', 'http://httpbin.org/get')

4、添加头部字段headers

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36 Edg/141.0.0.0"} r = http.request('GET', 'http://httpbin.org/get', headers=headers)

5、添加请求参数

data = {"username":"sunwukong","password":"123456"} r = http.request('GET', 'http://httpbin.org/get', fields=data)

典型用法

import urllib3 # 创建PoolManager实例 http = urllib3.PoolManager() # 发送GET请求 response = http.request('GET', 'https://httpbin.org/get') # 发送POST请求 response = http.request('POST', 'https://httpbin.org/post', fields={'key': 'value'}) # 获取响应数据 print(response.status) print(response.data.decode('utf-8')) # 记得关闭连接池 http.clear()

实训

1、get请求

import urllib3 http = urllib3.PoolManager() data = {"username":"sunwukong","password":"123456"} r = http.request('GET', 'http://httpbin.org/get', fields=data) print(r.data.decode('utf-8'))

2、配置headers

import urllib3 http = urllib3.PoolManager() headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36 Edg/141.0.0.0"} r = http.request('GET', 'http://httpbin.org/get', headers=headers) print(r.data.decode('utf-8'))

3、post请求

import urllib3 import json http = urllib3.PoolManager() data = {"username":"sunwukong","password":"123456"} data = json.dumps(data).encode("utf-8") headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) " "Chrome/141.0.0.0 Safari/537.36 Edg/141.0.0.0", "Content-Type":"application/json"} r = http.request('post', 'http://httpbin.org/post', headers=headers, body=data) print(r.data.decode('utf-8'))