Ng ddlmZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZerdd lmZdd lmZdd lmZmZdd lmZe e Gd dZy)) annotationsN) defaultdict) TYPE_CHECKING) implementer) ISpiderLoader) walk_modules)iter_spider_classes) ModuleType)Self)RequestSpider) BaseSettingscZeZdZdZd dZd dZd dZd dZeddZ ddZ ddZ dd Z y ) SpiderLoaderzZ SpiderLoader is a class which locates and loads spiders in a Scrapy project. c|jd|_|jd|_i|_t t |_|jy)NSPIDER_MODULESSPIDER_LOADER_WARN_ONLY) getlistspider_modulesgetbool warn_only_spidersrlist_found_load_all_spiders)selfsettingss 3lib/python3.12/site-packages/scrapy/spiderloader.py__init__zSpiderLoader.__init__sH)1)9)9:J)K'//0IJ13 ?J4?P   c6g}|jjD]A\}}|j|Dcgc]\}}t|dkDr d|d|d|d!c}}C|r1dj |}t j d|dt yycc}}w) Nz z named z (in )z z/There are several spiders with the same name: z' This can cause unexpected behavior.category)ritemsextendlenjoinwarningswarn UserWarning)rdupesname locationsmodcls dupes_strings r_check_name_duplicatesz#SpiderLoader._check_name_duplicates&s#{{002OD) LL%.$-S9~)WTHE#a8$-  3 !;;u-L MMC. IK$  s$B ct|D]Y}|j|jj|j|jf||j |j<[yN)r rr.append__name__r)rmodulespclss r _load_spiderszSpiderLoader._load_spiders9sK(0E KK # * *FOOU^^+L M(-DMM%** %1r c 8|jD]$} t|D]}|j|&|jy#ttf$rE|j r5t jdtjd|dtnYwxYw)N z$Could not load spiders from module 'z#'. See above traceback for details.r$) rrr: ImportError SyntaxErrorrr*r+ traceback format_excRuntimeWarningr3)rr.r8s rrzSpiderLoader._load_all_spiders>s''D *40F&&v.1( ##% - >>MMY11345((,v.;;"0  s!AABBc||Sr5)r1rs r from_settingszSpiderLoader.from_settingsOs 8}r cX |j|S#t$rtd|wxYw)z Return the Spider class for the given spider name. If the spider name is not found, raise a KeyError. zSpider not found: )rKeyError)r spider_names rloadzSpiderLoader.loadSs8  ?==- - ?/ }=> > ?s)c|jjDcgc]\}}|j|s|c}}Scc}}w)zT Return the list of spider names that can handle the given request. )rr&handles_request)rrequestr.r1s rfind_by_requestzSpiderLoader.find_by_request]sD #'--"5"5"7 "7YT33;N;Nw;WD"7   s AAcHt|jjS)zW Return a list with the names of all spiders available in the project. )rrkeys)rs rrzSpiderLoader.listesDMM&&())r N)rr)returnNone)r8r rOrP)rrrOr )rGstrrOz type[Spider])rKr rO list[str])rOrR) r7 __module__ __qualname____doc__rr3r:r classmethodrDrHrLrrCr rrrs> !&. &"? *r r) __future__rr?r* collectionsrtypingrzope.interfacerscrapy.interfacesrscrapy.utils.miscrscrapy.utils.spiderr typesr typing_extensionsr scrapyr r scrapy.settingsrrrCr rrbsR"# &+*3 '&, ]P*P*P*r