python线程池 批量抓取蔬菜价格 并写入csv文件
2022/4/11 14:12:51
本文主要是介绍python线程池 批量抓取蔬菜价格 并写入csv文件,对大家解决编程问题具有一定的参考价值,需要的程序猿们随着小编来一起学习吧!
# http://www.xinfadi.com.cn/index.html # 接口 http://www.xinfadi.com.cn/getPriceData.html # 想提取单个页面的数据 # 上线程池,多页面同时抓取import requests from concurrent.futures import ThreadPoolExecutor import csv import time
def get_one_page(current): data = {"limit":20,"current":current} r = requests.post("http://www.xinfadi.com.cn/getPriceData.html",data=data) list = r.json()["list"] write_csv(list)
def write_csv(list): # 写数据 with open("蔬菜价格.csv","a",encoding="utf-8",newline="") as f: w = csv.writer(f) for v in list: w.writerow((v["prodName"],v["lowPrice"],v["avgPrice"],v["highPrice"],v["place"],v["unitInfo"],v["pubDate"])) print(v["prodName"],v["lowPrice"],v["avgPrice"],v["highPrice"],v["place"],v["unitInfo"],v["pubDate"]) time.sleep(1)
if __name__ == "__main__": # 写表头 with open("蔬菜价格.csv","a",encoding="utf-8",newline="") as f: w = csv.writer(f) w.writerow(("品名","最低价","平均价","最高价","产地","单位","发布时间"))
# 创建线程池 with ThreadPoolExecutor(20) as t: for i in range(100): # 投递100个任务 t.submit(get_one_page,current=i)
print("OK")
结果:
表格打开:
这篇关于python线程池 批量抓取蔬菜价格 并写入csv文件的文章就介绍到这儿,希望我们推荐的文章对大家有所帮助,也希望大家多多支持为之网!
- 2024-05-08有遇到过吗?同样的规则 Excel 中 比Python 结果大
- 2024-03-30开始python成长之路
- 2024-03-29python optparse
- 2024-03-29python map 函数
- 2024-03-20invalid format specifier python
- 2024-03-18pool.map python
- 2024-03-18threads in python
- 2024-03-14python Ai 应用开发基础训练,字符串,字典,文件
- 2024-03-13id3 algorithm python
- 2024-03-13sum array elements python