不懂就问: django 不是一个请求就是一个生命周期吗? 为什么还会有 Too many open files 的错误 - V2EX
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
firejoke
V2EX    问与答

不懂就问: django 不是一个请求就是一个生命周期吗? 为什么还会有 Too many open files 的错误

  •  
  •   firejoke 2019-04-22 12:01:38 +08:00 1886 次点击
    这是一个创建于 2431 天前的主题,其中的信息可能已经有所发展或是发生改变。

    一个从远程获取安装日志, 以文件流的形式返回前端的逻辑
    先贴代码:

    class LogFile(LoginRequiredMixin, View): """获取日志文件 """ def get(self, request, *args, **kwargs): data = json.loads(request.GET.get('data', '{}')) host = data.get('host', None) user = data.get('user', None) pwd = data.get('password', None) if host and user and pwd: c = Connection(host, user, connect_kwargs={'password': pwd}) try: logs = c.run( 'ls -t ' + DEPLOYMENT_LOG_PATH + ' | grep install', pty=True) logs = logs.stdout.split('\n') return JsonResponse({'logs': logs}) except (AuthenticationException, NoValidConnectionsError, UnexpectedExit) as e: return JsonResponse({'error': e.__str__()}) else: return HttpResponse(0) def post(self, request, *args, **kwargs): data = json.loads(request.POST.get('data', '{}')) host = data.get('host', None) user = data.get('user', None) pwd = data.get('password', None) seek = data.get('seek', None) log_name = data.get('logfile', None) log_name = log_name[:-1] if log_name else None if host and log_name and (isinstance(seek, int) or seek.isdigit()): if not (user and pwd) and (user or pwd): return JsonResponse({'error': 'args error'}) elif user and pwd: c = Connection(host, user, connect_kwargs={'password': pwd}) try: c.get(DEPLOYMENT_LOG_PATH + log_name, '%s/log/%s' % ( settings.BASE_DIR, log_name)) log_file = '%s/log/%s' % (settings.BASE_DIR, log_name) except (IOError, AuthenticationException, NoValidConnectionsError, UnexpectedExit) as e: return JsonResponse({'error': e.__str__()}) else: log_file = '/var/log/' + log_name def read_log(log_file): with open(log_file, 'rb') as f: f.seek(0, 2) if f.tell() == int(seek): yield '\nseek:' + seek else: f.seek(int(seek)) while 1: log = f.read(512) if log: yield log else: break yield '\nseek:' + str(f.tell()) os.remove(log_file) respOnse= StreamingHttpResponse(read_log(log_file)) return response else: return JsonResponse({'error': 'args error'}) 

    前端在迭代完之后,会等待 2 秒再发送下一个请求, 在匹配到安装完成的关键字后就不再请求
    上周五走的时候,放着测试
    21:13:22 开始查看日志
    21:53:18 返回的错误信息
    错误信息在这一段

    c = Connection(host, user, connect_kwargs={'password': pwd}) try: c.get(DEPLOYMENT_LOG_PATH + log_name, '%s/log/%s' % ( settings.BASE_DIR, log_name)) log_file = '%s/log/%s' % (settings.BASE_DIR, log_name) except (IOError, AuthenticationException, NoValidConnectionsError, UnexpectedExit) as e: return JsonResponse({'error': e.__str__()}) 

    查了一下说是达到了系统限制的打开文件句柄数量
    但奇怪的是,我的逻辑是每次获取请求后,
    拉取文件,读文件,关闭文件,删除文件,
    按说,一个请求结束,文件也就没了,打开的文件句柄也就没了呀,
    怎么会报这个错呢?
    难道说, 要整个 django 进程都关闭了才算?

    第 1 条附言    2019-04-22 14:21:43 +08:00

    午休的时候又跑了一遍, 又出这个问题

    Traceback (most recent call last): File "/usr/lib64/python2.7/wsgiref/handlers.py", line 85, in run File "/usr/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 157, in __call__ File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 124, in get_response File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template File "/usr/lib/python2.7/site-packages/django/template/loaders/cached.py", line 60, in get_template File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template File "/usr/lib/python2.7/site-packages/django/template/loaders/cached.py", line 28, in get_contents File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents IOError: [Errno 24] Too many open files: '/root/setupUI/templates/500.html' 
    15 条回复    2019-04-24 08:59:10 +08:00
    CEBBCAT
        1
    CEBBCAT  
       2019-04-22 13:59:05 +08:00
    traceback 贴一下?
        2
    firejoke  
    OP
       2019-04-22 14:23:35 +08:00
    @CEBBCAT
    贴了, 看起来就是整个 django 进程开了太多文件, 以至于连 html 文件都不能打开,
    必须把整个服务重启一遍
    PureWhiteWu
        3
    PureWhiteWu  
       2019-04-22 14:25:12 +08:00
    是不是你哪里资源泄露了?
    你代码中有打开文件的地方么?
    firejoke
        4
    firejoke  
    OP
       2019-04-22 14:32:32 +08:00
    @PureWhiteWu
    不会, 其他的地方涉及到文件操作的,基本都是用的 with 语法,
    而且, 中午再次测的时候, 从头到尾, 就只是访问部署接口, put 了以个压缩包到目标服务器,
    然后就访问这个接口看日志
    julyclyde
        5
    julyclyde  
       2019-04-22 16:05:30 +08:00
    出故障的时候别重启,检查一下 /proc/XXXX/fd/ 目录
    firejoke
        6
    firejoke  
    OP
       2019-04-22 16:49:31 +08:00
    @julyclyde 看了, 有 1024 个
    katsusan
        7
    katsusan  
       2019-04-22 17:15:31 +08:00 via iPhone
    lsof 看一下打开了哪些文件
    firejoke
        8
    firejoke  
    OP
       2019-04-22 17:15:55 +08:00
    @PureWhiteWu
    @julyclyde

    找到问题了, Fabric 的连接实例没关......
    大意了
    firejoke
        9
    firejoke  
    OP
       2019-04-22 17:16:18 +08:00
    @katsusan 找到原因了
    julyclyde
        10
    julyclyde  
       2019-04-23 15:58:13 +08:00
    @firejoke 是指 ssh 那个 Fabric 吗?似乎你上面的代码没提到这个
    firejoke
        11
    firejoke  
    OP
       2019-04-23 16:39:26 +08:00
    @julyclyde 是的, 一开始看到 files error, 完全没往这个方面想, 反复检查的时候才发现我没关连接,
    现在我只要能用 with 的都用 with 了
    firejoke
        12
    firejoke  
    OP
       2019-04-23 17:01:49 +08:00
    @julyclyde 奇怪的是, 我发现就算使用了 connection.close(), connection 实例竟然还是可以用, .run 方法也可以用
    firejoke
        13
    firejoke  
    OP
       2019-04-23 17:06:44 +08:00
    @julyclyde
    嗯, 看了源码, 每次调用 run 方法, 都被 open 方法装饰过......
    julyclyde
        14
    julyclyde  
       2019-04-23 18:10:38 +08:00
    @firejoke 没太能理解呢,run 和 open 到底啥关系?
    firejoke
        15
    firejoke  
    OP
       2019-04-24 08:59:10 +08:00
    @julyclyde
    run 被加了一个装饰器, 该装饰器的作用是在调用之前, 先调用 open 方法
    @decorator
    def opens(method, self, *args, **kwargs):
    self.open()
    return method(self, *args, **kwargs)


    @opens
    def run(self, command, **kwargs):
    return self._run(self._remote_runner(), command, **kwargs)
    关于     帮助文档     自助推广系统     博客     API     FAQ     Solana     892 人在线   最高记录 6679       Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 26ms UTC 22:01 PVG 06:01 LAX 14:01 JFK 17:01
    Do have faith in what you're doing.
    ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86