Skip to content

Comments

feat: automated cookie grabbing for config.json (must be logged in on chrome)#614

Merged
dataabc merged 1 commit intodataabc:masterfrom
ahuangg:alan-branch
Dec 6, 2024
Merged

feat: automated cookie grabbing for config.json (must be logged in on chrome)#614
dataabc merged 1 commit intodataabc:masterfrom
ahuangg:alan-branch

Conversation

@ahuangg
Copy link
Contributor

@ahuangg ahuangg commented Dec 6, 2024

current approach:
user has to go to the website manually to grab the cookie

requirements:
user has to be logged in to weibo.cn on the Chrome browser

solution:
grabs the cookie from the browser and adds it to the cookie section in config.json when the crawler is ran for the first time

future:
if the cookie becomes invalid we can run this functionality

@ahuangg ahuangg changed the title automated cookie grabbing for config.json (must be logged in on chrome) feat: automated cookie grabbing for config.json (must be logged in on chrome) Dec 6, 2024
@dataabc dataabc merged commit 7844dfa into dataabc:master Dec 6, 2024
@dataabc
Copy link
Owner

dataabc commented Dec 6, 2024

感谢贡献代码,非常实用的功能。

@songzy12
Copy link
Collaborator

songzy12 commented Jan 2, 2025

目前有一点点问题,如果是在 Windows 的 WSL 下运行的话会报错:

Failed to obtain weibo.cn cookie from Chrome browser: [Errno 2] No such file or directory
Check for cookie failed: [Errno 2] No such file or directory
[Errno 2] No such file or directory
Traceback (most recent call last):
  File "/home/songzy/crawler/weiboSpider/weibo_spider/spider.py", line 387, in main
    config = _get_config()
  File "/home/songzy/crawler/weiboSpider/weibo_spider/spider.py", line 376, in _get_config
    config_util.check_cookie(config_path)
  File "/home/songzy/crawler/weiboSpider/weibo_spider/config_util.py", line 209, in check_cookie
    cookie = get_cookie()
  File "/home/songzy/crawler/weiboSpider/weibo_spider/config_util.py", line 181, in get_cookie
    chrome_cookies = browser_cookie3.chrome(domain_name='weibo.cn')
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/browser_cookie3/__init__.py", line 1188, in chrome
    return Chrome(cookie_file, domain_name, key_file).load()
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/browser_cookie3/__init__.py", line 653, in __init__
    super().__init__(browser='Chrome', cookie_file=cookie_file,
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/browser_cookie3/__init__.py", line 416, in __init__
    self.__add_key_and_cookie_file(**kwargs)
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/browser_cookie3/__init__.py", line 431, in __add_key_and_cookie_file
    USE_DBUS_LINUX).get_password(os_crypt_name)
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/browser_cookie3/__init__.py", line 232, in get_password
    return self.__get_secretstorage_password(os_crypt_name)
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/browser_cookie3/__init__.py", line 247, in __get_secretstorage_password
    return self.__methods_map.get('secretstorage')(schema, os_crypt_name)
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/browser_cookie3/__init__.py", line 303, in __get_secretstorage_item_jeepney
    with _JeepneyConnection(*args) as connection:
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/browser_cookie3/__init__.py", line 195, in __enter__
    self.__connection = open_dbus_connection()
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/jeepney/io/blocking.py", line 342, in open_dbus_connection
    sock = prep_socket(bus_addr, enable_fds, timeout=auth_timeout)
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/jeepney/io/blocking.py", line 310, in prep_socket
    with_sock_deadline(sock.connect, addr)
  File "/home/songzy/.virtualenvs/crawler/lib/python3.10/site-packages/jeepney/io/blocking.py", line 307, in with_sock_deadline
    return meth(*args)
FileNotFoundError: [Errno 2] No such file or directory

原因在于 WSL 下确实不会有 Chrome 的 cookie.

我准备稍后加一个 try except 来让这种情况下显示的报错信息更明确一些。

@songzy12 songzy12 self-assigned this Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants