3dpddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z mZmZddlmZmZddlmZddlmZdd lmZdefd Zd Z dd eeejfdeeeejfdefdZddZddZdde efdZ dZ! dde edee efdZ"dS)N) ConfigParser) itemgetter)Path)AnyDictListOptionalUnion)ScrapyDeprecationWarning UsageError) BaseSettings)update_classpath)without_none_valuescfdfd}d}t|ttfr/|t|fd|DS|||||t ||}dt |tdDS) zzBbuild_component_list.._check_components..s!---q ---Some paths in 8 convert to the same object, please update your settings)len ValueError)complistrs r_check_componentsz/build_component_list.._check_componentsse ----H--- . .#h-- ? ....   rct|trt}|D]\}}||}|||kr2t dt |d|||||S|fd|DS)Nrr)priorityc.i|]\}}||Srr)rkvrs r z;build_component_list.._map_keys..)s';;;$!Q A;;;r) isinstancer items getpriorityrlistkeysset)compdictcompbsr$r%prior rs r _map_keysz'build_component_list.._map_keyss h - - !^^F (( = =1++A..%%ggajj11T9=$>hmmoo)>)>>>> JJwwqzz1tJ<<<<M(###;;;;(..*:*:;;;;rc|D]7\}}|0t|tjst d|d|d8dS)zDFail if a value in the components dict is not a real number or None.NzInvalid value z for component z., please provide a real number or None instead)r(r'numbersRealr)r-namevalues r_validate_valuesz.build_component_list.._validate_values+s#>>++  KD% E7<)H)H  CUCC4CCC  rc3.K|]}|VdS)Nrrs r z'build_component_list..6s+771GGAJJ777777rNcg|]\}}|Srr)rr$r%s r z(build_component_list..=s F F F$!QA F F Fr)key) r'r*tupletypeupdatersortedr(r)r-customrr0r6r s ` @rbuild_component_listrBs<<<<<<"&4-((8&!!!tF||7777777777  X"99X#6#677H F F&!1!1z!}}EEE F F FFrc4td|DS)zSConvert a list of arguments like ['arg1=val1', 'arg2=val2', ...] to a dict c3BK|]}|ddVdS)=r;N)split)rxs rr8z"arglist_to_dict..Ds.11AQ111111r)dict)arglists rarglist_to_dictrJ@s! 11111 1 11r.pathprevpathreturnc|"t|t|krdSt|}|dz }|rt|St |j|S)zkReturn the path to the closest scrapy.cfg file by traversing the current directory and its parents N scrapy.cfg)strrresolveexistsclosest_scrapy_cfgparent)rLrMcfgfiles rrUrUGszD S]] :r ::    D\!G~~7|| dk4 0 00rdefaultTc`t}|d|r#|d|tjd<t }|rRt t|j}|r1|tj vr%tj |dSdSdSdS)zInitialize environment to use command-line tool from inside a project dir. This sets the Scrapy settings module and modifies the Python path to be able to locate the project module. settingsSCRAPY_SETTINGS_MODULEN) get_config has_optiongetosenvironrUrRrrVsysrLappend)project set_syspathcfgclosestprojdirs rinit_envrhWs ,,C ~~j'**L/2wwz7/K/K +, ""G%d7mm*++  %7#(2 % HOOG $ $ $ $ $%% % % % %rcjt|}t}|||S)z(Get Scrapy config file as a ConfigParser) get_sourcesrread) use_closestsourcesres rr\r\fs/+&&G ..CHHW Jrcjtjdp td}ddt t|dz t tdg}|r!|t|S)NXDG_CONFIG_HOMEz ~/.configz/etc/scrapy.cfgzc:\scrapy\scrapy.cfgrQz ~/.scrapy.cfg)r_r`r^r expanduserrRrbrU)rlxdg_config_homerms rrjrjns ())KT+->->-I-I-K-K  D ! !L 011 D ! ! , , . .// G -)++,,, NrcT|}|d|d|d|d|d|dpd|d|d|d |d |d i|d |d dn)|d |d |S)Nbatch_item_countFEED_EXPORT_BATCH_ITEM_COUNTencodingFEED_EXPORT_ENCODINGfieldsFEED_EXPORT_FIELDS store_emptyFEED_STORE_EMPTY uri_paramsFEED_URI_PARAMSitem_export_kwargsFEED_EXPORT_INDENTindent)copy setdefaultgetint getdictorlistgetbool)feedrZouts r*feed_complete_default_values_from_settingsr}s ))++CNN%x7U'V'VWWWNN:x(>?@@@NN8X334HIIQTRRRNN=("2"23E"F"FGGGNN<*;!<===NN',,,$%H x&&&& x1E!F!FGGG Jroutputoverwrite_outputc t|d fd}d}|r&|rtd|rtd|}d}|rWt |dkr5||d}t j|td |d d |iiStd i}|D]~} |dd\} } | dkrd} n:#t$r-|} t|j dd} YnwxYw|| d | i|| <|r d|| d<| |d|S)z Receives feed export params (from the 'crawl' or 'runspider' commands), checks for inconsistencies in their quantities and returns a dictionary suitable to be used as the FEEDS setting. FEED_EXPORTERScV|vr#td|dtddS)NzUnrecognized output format 'z'. Set a supported one (z`) after a colon at the end of the output URI (i.e. -o/-O :) or as a file extension.)r r=) output_formatvalid_output_formatss rcheck_valid_formatz8feed_process_params_from_cli..check_valid_formatsY  4 4 :}::(-.B(C(C:::   rFz:). Example working in the tutorial: scrapy crawl quotes -O quotes.json:jsonTr;aThe -t/--output-format command line option is deprecated in favor of specifying the output format within the output URI using the -o/--output or the -O/--overwrite-output option (i.e. -o/-O :). See the documentation of the -o or -O option or the following examples for more information. Examples working in the tutorial: scrapy crawl quotes -o quotes.csv:csv or scrapy crawl quotes -O quotes.json:json) stacklevelrformatzOThe -t command-line option cannot be used if multiple output URIs are specified:-zstdout:rKrP overwriteFEEDS)r getwithbaser+r rwarningswarnr rsplitrrsuffixreplacer?getdict) rZrrrrrmessageresultelementfeed_uri feed_formatrs @rfeed_process_params_from_clirs/-.. dffI  N   : "  v;;!  :  } - - -:  M'#; J J J J1I-89 9 !   )+F 1 1 %$+NN3$:$: !Hk 3 %$  @ @ @Hw--.66sB??KKK @ ;'''$k2x  1,0F8 [ ) MM(""7++,,, MsC''4DD)rKN)rXT)T)NN)#r2r_rar configparserroperatorrpathlibrtypingrrrr r scrapy.exceptionsr r scrapy.settingsr scrapy.utils.deprecaterscrapy.utils.pythonrrBrJrRPathLikerUrhr\rjrrrrrrs  %%%%%%33333333333333BBBBBBBB((((((333333333333+/8H-G-G-G-G`222%(26 1 1 R[ ! 1uS"+-./ 1  1 1 1 1 % % % %  T#Y       $,0 OO IOtCy) OOOOOOr