请教 scrapy 的 cookie 导入方式？

headers = { 'Accept-Language':'zh-CN', 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36', # dummy: "cookie":"__cfduid=d2ea661e4cddec1eebedc2c54d78e6156698409;__utma=50351329.433346.1527221979.1527221979.1527221979.1;__utmc=501329;__utmz=501329.15271979.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);__51cke__=;__51laig__=1;__tins__3878067=%7B%22sid%22%3A%201527221980323%2C%20%22vd%22%3A%201%2C%20%22expires%22%3A%201527223780323%7D;91username=regf3qc;CLIPSHARE=n9u906378onmf990kdlpq3;DUID=15fdcQq0vvU0U10%2FdVsdg5smxiBM18RO%2FQ5O3uuV%2F3pe;EMAILVERIFIED=no;language=cn_CN;level=1;remainclosed=1;user_level=1;USERNAME=1a9dkmjCzR6TueiJdjZzM6gIUE8AXAHzxIuxb4x6vEk;", } r = requests.get('http://example.com',headers=headers)

但是看scrapy文档的话是把cookie和header分开输入的，而且cookie还只能用 dict 输入。上面的cookie直接用EditThisCookie导出就可以了。cookie不是包含在了header里面了吗？在scrapy里这样写似乎也没用：

class PornSpider(scrapy.Spider): name = 'pron' def start_requests(self): urls = ['http://pron.com/page={}'.format(i) for i in range(1,3)] for url in urls: yield Request(url,callback=self.parse,headers=headers) #这样似乎是没有用的, 返回的应该是不带 cookie 的页面

scrapy看起来比自己写方便很多，那想请教下能不能用requests库取代scrapy自带的Request呢？或者有没有更优雅的cookie导入方法呢？

谢谢大家

第 1 条附言 2018 年 6 月 2 日

Resigned to fate.

cookie_raw = "__cfduid=d2ea661e4cddec1eebedc2c54d78e6156698409;__utma=50351329.433346.1527221979.1527221979.1527221979.1;__utmc=501329;__utmz=501329.15271979.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);__51cke__=;__51laig__=1;__tins__3878067=%7B%22sid%22%3A%201527221980323%2C%20%22vd%22%3A%201%2C%20%22expires%22%3A%201527223780323%7D;91username=regf3qc;CLIPSHARE=n9u906378onmf990kdlpq3;DUID=15fdcQq0vvU0U10%2FdVsdg5smxiBM18RO%2FQ5O3uuV%2F3pe;EMAILVERIFIED=no;language=cn_CN;level=1;remainclosed=1;user_level=1;USERNAME=1a9dkmjCzR6TueiJdjZzM6gIUE8AXAHzxIuxb4x6vEk;" cookie = {} for i in cookie_raw.split(';')[:-1]: cookie[i.split('=')[0]] = i.split('=')[1]

Scrapy

Headers

Python

2 条回复 2018-06-02 22:24:34 +08:00

Leigg

2018 年 6 月 2 日

def start_requests(self):
cookies = settings.weibo_cookies （字典）
#cookie 登录
return [FormRequest(self.start_urls[1], cookies=cookies, formdata={}:这个参数是请求的 body,callback=self.parse)]

Leigg

2018 年 6 月 2 日

注意缩进。