Ng!,dZddlmZddlZddlZddlmZmZddlm Z ddl m Z m Z ddl mZddlmZdd lmZdd lmZdd lmZmZdd lmZdd lmZddlmZddlmZm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)erddl*m+Z+GddZ,ddZ-ddZ.y)z:Scrapy Shell See documentation in docs/topics/shell.rst ) annotationsN) TYPE_CHECKINGAny)is_item)deferthreads) threadable) any_to_uri)Crawler) IgnoreRequest)RequestResponse)Settings)Spider) get_config)DEFAULT_PYTHON_SHELLSstart_python_console)SequenceExclude) load_object)is_asyncio_reactor_installedset_asyncio_event_loop)open_in_browser)CallableceZdZUeeeeefZde d< d ddZ d ddZ ddZ ddZ d ddZ d dd Zdd Zdd Zdd Zy)Shellztuple[type, ...]relevant_classesNc||_|xsd|_t|jd|_d|_t j |_||_ i|_ y)NcyN)xs ,lib/python3.12/site-packages/scrapy/shell.pyz Shell.__init__..-sdDEFAULT_ITEM_CLASS) crawler update_varsrsettings item_classspiderr isInIOThreadinthreadcodevars)selfr&r'r-s r"__init__zShell.__init__%s` !( =H>  !,G,<,<=Q,R S%) ","9"9";; $ $& r$ctjtjtj|r|j|||nG|r|j||n2|r |j}|j |||n|j |j r3tt|j t|jy t}d\}}tjjd} g} | r1| | jj!j#dz } n\|j%||r3| |j||jj!gz } n| t'j(z } | dgz } t+|j| |jj-ddy) N)redirect)r(shellSCRAPY_PYTHON_SHELL,pythonbanner)shellsr7)signalSIGINTSIG_IGNfetchrequest populate_varsr-printevalglobalsr.rosenvirongetstriplowersplit has_optionrkeysrpop) r/urlr>responser*r2cfgsectionoptionenvr9s r"startz Shell.start5s^  fmmV^^4  JJsFXJ 6  JJw ' &&G   x& 9    99 $tyy')TYY7 8 ,C1OGV**..!67CF#))+++-33C8803777F399;AACDD/4466 xj F  &x1L r$c@tr$|jjd}t||j |t |}|j fd|jjsJ|jjj||S)NASYNCIO_EVENT_LOOPc |fSrr )r!r*s r"r#z!Shell._schedule..ks F r$) rr&r(r _open_spider_request_deferred addCallbackenginecrawl)r/r>r*event_loop_pathds ` r" _schedulezShell._scheduleds ' )"ll334HIO "? 3""7F3 g & +,||"""" !!'*r$cL|jr |jS|2|jjxs|jj}||j_|jjsJ|jjj |d||_|S)NF) close_if_idle)r*r&_create_spiderrY open_spider)r/r>r*s r"rVzShell._open_spiderps ;;;;  >\\((IDLL,G,G,IF$ ||"""" ''e'D  r$c dddlm}t|tr|}nMt |}t|fddi|}|r#t t dd|jd<nd|jd<d} tj||j||\}}|j|||y#t$rYwxYw) Nr)reactor dont_filterTi,ihandle_httpstatus_listhandle_httpstatus_all) twisted.internetrc isinstancer r rrangemetarblockingCallFromThreadr]r r?) r/request_or_urlr*r2kwargsrcr>rLrMs r"r=z Shell.fetch}s - ng .$G^,Cc>t>v>G9H#sO: 569= 45 &==&  Hf 8Wf5   s*%B## B/.B/cbddl}||jd<|j|jd<|j|jd<|jj|jd<||jd<||jd<||jd<|j r|j |jd <t|jd <|j|jd <|j|j|js|j|jd <yy) Nrscrapyr&itemr(r*r>rMr=viewshelpr7) ror.r&r)r(r,r=r print_helpr'r-get_help)r/rMr>r*ros r"r?zShell.populate_varss $ (#|| ) OO- & $ 5 5 *$ (& ) ( * ==!%DIIg + &!__ ' #yy"&--/DIIh r$c6t|jyr)r@rt)r/s r"rszShell.print_helps dmmor$cg}|jd|jdt|jjD]/\}}|j |s|jd|dd|1|jd|j r"|jd|jd|jd |jd d j d |DS) NzAvailable Scrapy objects:zJ scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)z z<10 zUseful shortcuts:ze fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)zN fetch(req) Fetch a scrapy.Request and update local objects z0 shelp() Shell help (print this help)z. view(response) View response in a browser c3&K|] }d| yw)z[s] Nr ).0lines r" z!Shell.get_help..s5144v1s)appendsortedr.items _is_relevantr,join)r/bkvs r"rtzShell.get_helps  ,-  X 499??,-DAq  #2aWAaS)*. $% == HHZ  HHC  CD AByy51555r$cHt||jxs t|Sr)rhrr)r/values r"rzShell._is_relevants%!6!67I75>Ir$)NN)r&r r'z'Callable[[dict[str, Any]], None] | Noner- str | None)NNNNT) rLrr>Request | NonerMResponse | Noner* Spider | Noner2boolreturnNone)r>r r*rrdefer.Deferred[Any])r>r r*rrr)NT) rlz Request | strr*rr2rrmrrr)NNN)rMrr>rr*rrr)rr)rstr)rrrr)__name__ __module__ __qualname__r rr rrr__annotations__r0rRr]rVr=r?rsrtrr r$r"rr"s"*167Hh)W&W @D ''=' '$"&$( $ - - -" -  -  - -^   !% 6%66 6  6  6>%)"& $ 2!2 2 2  2.60Jr$rctjtj}t|jj ||tjtj|y)z*Open a shell to inspect the given response)rMr*N)r: getsignalr;rr&rR)rMr*sigint_handlers r"inspect_responsersF%%fmm4N &..&A MM&--0r$cxjjdfd }tj}|j |jr|j jjr|j j|j|jc__|S)aWrap a request inside a Deferred. This function is harmful, do not use it until you know what you are doing. This returns a Deferred whose first pair of callbacks are the request callback and errback. The Deferred also triggers when the request callback/errback is executed (i.e. when the request is downloaded) WARNING: Do not call request.replace() until after the deferred is called. c$__|Sr)callbackerrback)resultr>request_callbackrequest_errbacks r"_restore_callbacksz-_request_deferred.._restore_callbackss+) r$)rrrr)rrrDeferredaddBothrX addErrback)r>rr\rrs` @@r"rWrWs''ooO #^^-AII ! g&&' W__%() AII%Ggo Hr$)rMrr*rrr)r>r rr)/__doc__ __future__rrCr:typingrr itemadapterrrgrrtwisted.pythonr w3lib.urlr scrapy.crawlerr scrapy.exceptionsr scrapy.httpr rscrapy.settingsrscrapy.spidersrscrapy.utils.confrscrapy.utils.consolerrscrapy.utils.datatypesrscrapy.utils.miscrscrapy.utils.reactorrrscrapy.utils.responsercollections.abcrrrrWr r$r"rsd # %+% "+)$!(L2)U1(kJkJ\1 r$