推荐学习书目 Learn Python the Hard Way Python Sites PyPI - Python Package Index http://diveintopython.org/toc/index.html Pocoo 值得关注的项目 PyPy Celery Jinja2 Read the Docs gevent pyenv virtualenv Stackless Python Beautiful Soup 结巴中文分词 Green Unicorn Sentry Shovel Pyflakes pytest Python 编程 pep8 Checker Styles PEP 8 Google Python Style Guide Code Style from The Hitchhiker's Guide
这是一个创建于 1650 天前的主题，其中的信息可能已经有所发展或是发生改变。
看到此贴 2021 年了，requests 内存泄露的问题解决了吗？如果没解决，怎么解决？ ， 便去测试了下
from concurrent.futures import ThreadPoolExecutor, wait, ALL_COMPLETED import requests from memory_profiler import profile s = requests.Session() def get(i): s.get('URL') @profile def test(): executor = ThreadPoolExecutor(max_workers = 100) task = [executor.submit(get, (i)) for i in range(5000)] wait(task, return_when = ALL_COMPLETED) s.close() if __name__ == '__main__': test() 5000 任务，116.3 MB
Line # Mem usage Increment Occurences Line COntents============================================================ 10 25.6 MiB 25.6 MiB 1 @profile 11 def test(): 12 25.6 MiB 0.0 MiB 1 executor = ThreadPoolExecutor(max_workers = 100) 13 115.9 MiB 90.3 MiB 5003 task = [executor.submit(get, (i)) for i in range(5000)] 14 116.3 MiB 0.3 MiB 1 wait(task, return_when = ALL_COMPLETED) 500 任务 10 次，141.3 MB，逐渐增长
Line # Mem usage Increment Occurences Line COntents============================================================ 10 25.7 MiB 25.7 MiB 1 @profile 11 def test(): 12 25.7 MiB 0.0 MiB 1 executor = ThreadPoolExecutor(max_workers = 100) 13 94.8 MiB 69.1 MiB 503 all_task = [executor.submit(get, (i)) for i in range(500)] 14 106.2 MiB 11.4 MiB 1 wait(all_task, return_when=ALL_COMPLETED) 15 109.6 MiB 3.4 MiB 503 all_task = [executor.submit(get, (i)) for i in range(500)] 16 111.7 MiB 2.1 MiB 1 wait(all_task, return_when=ALL_COMPLETED) 17 114.3 MiB 2.6 MiB 503 all_task = [executor.submit(get, (i)) for i in range(500)] 18 115.8 MiB 1.5 MiB 1 wait(all_task, return_when=ALL_COMPLETED) 19 120.0 MiB 4.1 MiB 503 all_task = [executor.submit(get, (i)) for i in range(500)] 20 121.0 MiB 1.0 MiB 1 wait(all_task, return_when=ALL_COMPLETED) 21 124.1 MiB 3.1 MiB 503 all_task = [executor.submit(get, (i)) for i in range(500)] 22 124.6 MiB 0.5 MiB 1 wait(all_task, return_when=ALL_COMPLETED) 23 126.7 MiB 2.1 MiB 503 all_task = [executor.submit(get, (i)) for i in range(500)] 24 127.4 MiB 0.8 MiB 1 wait(all_task, return_when=ALL_COMPLETED) 25 130.5 MiB 3.1 MiB 503 all_task = [executor.submit(get, (i)) for i in range(500)] 26 131.8 MiB 1.3 MiB 1 wait(all_task, return_when=ALL_COMPLETED) 27 135.2 MiB 3.4 MiB 503 all_task = [executor.submit(get, (i)) for i in range(500)] 28 136.7 MiB 1.5 MiB 1 wait(all_task, return_when=ALL_COMPLETED) 29 137.7 MiB 1.0 MiB 503 all_task = [executor.submit(get, (i)) for i in range(500)] 30 138.0 MiB 0.3 MiB 1 wait(all_task, return_when=ALL_COMPLETED) 31 139.8 MiB 1.8 MiB 503 all_task = [executor.submit(get, (i)) for i in range(500)] 32 141.3 MiB 1.5 MiB 1 wait(all_task, return_when=ALL_COMPLETED) 33 141.3 MiB 0.0 MiB 1 s.close()  | 1 joApioVVx4M4X6Rf 2021-06-18 07:59:48 +08:00 requests 的 session 是非线程安全的，这么用似乎也不太对 |
 | 2 4BVL25L90W260T9U 2021-06-18 09:21:40 +08:00 requests 就是个内存泄漏的坑爹货，建议尽早划成 httpx |
 | 3 warcraft1236 2021-06-18 11:22:33 +08:00 @ospider requests 为啥会有内存泄漏呢？ |
 | 4 Multicom OP @ospider 将 requests.Session() 更换为 httpx.Client() ，内存占用降低，但仍持续增长 ``` Line # Mem usage Increment Occurences Line Contents ============================================================ 10 25.2 MiB 25.2 MiB 1 @profile 11 def test(): 12 25.2 MiB 0.0 MiB 1 executor = ThreadPoolExecutor(max_workers = 100) 13 28.2 MiB 3.1 MiB 503 task = [executor.submit(get, (i)) for i in range(500)] 14 28.5 MiB 0.3 MiB 1 wait(task) 15 28.8 MiB 0.3 MiB 503 task = [executor.submit(get, (i)) for i in range(500)] 16 28.8 MiB 0.0 MiB 1 wait(task) 17 29.0 MiB 0.3 MiB 503 task = [executor.submit(get, (i)) for i in range(500)] 18 31.3 MiB 2.3 MiB 1 wait(task) 19 31.6 MiB 0.3 MiB 503 task = [executor.submit(get, (i)) for i in range(500)] 20 35.9 MiB 4.3 MiB 1 wait(task) 21 35.9 MiB 0.0 MiB 503 task = [executor.submit(get, (i)) for i in range(500)] 22 39.0 MiB 3.0 MiB 1 wait(task) 23 39.2 MiB 0.3 MiB 503 task = [executor.submit(get, (i)) for i in range(500)] 24 41.0 MiB 1.8 MiB 1 wait(task) 25 41.0 MiB 0.0 MiB 503 task = [executor.submit(get, (i)) for i in range(500)] 26 43.8 MiB 2.8 MiB 1 wait(task) 27 43.8 MiB 0.0 MiB 503 task = [executor.submit(get, (i)) for i in range(500)] 28 45.8 MiB 2.0 MiB 1 wait(task) 29 45.8 MiB 0.0 MiB 503 task = [executor.submit(get, (i)) for i in range(500)] 30 48.1 MiB 2.3 MiB 1 wait(task) 31 48.1 MiB 0.0 MiB 503 task = [executor.submit(get, (i)) for i in range(500)] 32 49.4 MiB 1.3 MiB 1 wait(task) 33 49.4 MiB 0.0 MiB 1 s.close() ``` |
Select Language