3 d% d Z ddlZddlmZ ddlmZmZ ddlmZ ddl Z ddl
m Z m Z m
Z
mZmZ ddlmZ ddlmZ dd lmZ dd
lmZmZ dd lmZ e Z G d d
e Z G d de Z d Z dS )z
This module implements the TextResponse class which adds encoding handling and
discovering (through HTTP headers) to base Response class.
See documentation in docs/topics/request-response.rst
N)suppress) GeneratorTuple)urljoin)html_body_declared_encodinghtml_to_unicodehttp_content_type_encodingread_bomresolve_encoding)strip_html5_whitespace)Request)Response)memoizemethod_noargs
to_unicode) get_base_urlc e Zd ZU dZeZej dz Zee df e
d< fdZ fdZ fdZ
ed Zd Zd
Zed Zd Zed
Zd Zd Zed Zed Zed Zd Zd Z d de f fd
Z d de e ddf f fd
Z xZ!S ) TextResponseascii)encoding.
attributesc | dd | _ d | _ d | _ d | _ t j |i | d S )Nr )pop _encoding _cached_benc
_cached_ubody_cached_selectorsuper__init__)selfargskwargs __class__s 9lib/python3.11/site-packages/scrapy/http/response/text.pyr zTextResponse.__init__& sR J55 ! $ $)&))))) c t |t r t || j | _ d S t | d S N)
isinstancestrr r _urlr _set_url)r urlr" s r# r* zTextResponse._set_url- sJ c3 ""3
66DIII GG S ! ! ! ! !r$ c d| _ t |t rM| j %t dt | j d | | j | _ d S t | d S )Nr$ z Cannot convert unicode body - z has no encoding)
_bodyr' r( r TypeErrortype__name__encoder _set_body)r bodyr" s r# r2 zTextResponse._set_body3 s
dC $~
=Dzz*= = = T^44DJJJ GG d # # # # #r$ c R | p| S r& )_declared_encoding_body_inferred_encodingr s r# r zTextResponse.encoding? s% &&((JD,H,H,J,JJr$ c | j p;| p'| p| S r& )r
_bom_encoding_headers_encoding_body_declared_encodingr7 s r# r5 z TextResponse._declared_encodingC sJ N
.!!##
.%%''
. ++--
r$ c h | j t u r t j | j | _ | j S )z`
.. versionadded:: 2.2
Deserialize a JSON document to a Python object.
)_cached_decoded_json_NONEjsonloadstextr7 s r# r? zTextResponse.jsonK s1 $ - >(,
49(=(=D %((r$ c v | j }| j %d| }t || j d | _ | j S )zBody as unicodeNzcharset= )r r r r3 )r benccharsets r# rA zTextResponse.textU sF
} H'''G!0$)!D!DQ!GD !!r$ c < t t | | S )zoJoin this Response's url with a possible relative url to form an
absolute interpretation of the latter.)r r )r r+ s r# r zTextResponse.urljoin` s |D))3///r$ c p | j dd }t t | S )N Content-Typer$ )headersgetr r )r content_types r# r: z TextResponse._headers_encodinge s/ |''== )*\*B*BCCCr$ c | j [t | j dd }t || j | j | j \ }}|| _ || _ | j S )NrH r$ )auto_detect_fundefault_encoding) r r rI rJ r r3 _auto_detect_fun_DEFAULT_ENCODINGr )r rK rD ubodys r# r6 z$TextResponse._body_inferred_encodingj ss '%dl&6&6&L&LMML) $ 5!%!7 KD% !%D !&D r$ c | j ddfD ]9} | | n# t $ r Y %w xY wt | c S d S )Nzutf-8cp1252)rP decode UnicodeErrorr )r rA encs r# rO z TextResponse._auto_detect_funw ss *GX> ) )C
C
#C(( ( ( ( ) )s $
11c * t | j S r& )r r3 r7 s r# r; z$TextResponse._body_declared_encoding s *49555r$ c 6 t | j d S )Nr )r
r3 r7 s r# r9 zTextResponse._bom_encoding s ""1%%r$ c J ddl m} | j || | _ | j S )Nr )Selector)scrapy.selectorrZ r )r rZ s r# selectorzTextResponse.selector s8 ,,,,,, 3$,HTNND !$$r$ c ( | j j |fi |S r& )r\ xpath)r queryr! s r# r^ zTextResponse.xpath s "t}"533F333r$ c 6 | j | S r& )r\ css)r r_ s r# ra zTextResponse.css s } '''r$ NGETr Freturnc t |t j rt | }n)t |t j rt d || j n|}t ||||||||| |
| | |
S )a
Return a :class:`~.Request` instance to follow a link ``url``.
It accepts the same arguments as ``Request.__init__`` method,
but ``url`` can be not only an absolute URL, but also
* a relative URL
* a :class:`~scrapy.link.Link` object, e.g. the result of
:ref:`topics-link-extractors`
* a :class:`~scrapy.selector.Selector` object for a ```` or ```` element, e.g.
``response.css('a.my_link')[0]``
* an attribute :class:`~scrapy.selector.Selector` (not SelectorList), e.g.
``response.css('a::attr(href)')[0]`` or
``response.xpath('//img/@src')[0]``
See :ref:`response-follow-example` for usage examples.
z SelectorList is not supportedN)
r+ callbackmethodrI r3 cookiesmetar priority dont_filtererrback cb_kwargsflags) r' parselrZ _url_from_selector SelectorList
ValueErrorr r follow)r r+ re rf rI r3 rg rh r ri rj rk rl rm r" s r# rr zTextResponse.follow s @ c6? + + >$S))CC
V 0
1
1 > <== =$,B4==(ww~~ #
r$ c d |||fD }t | dk rt d |s.|r| | }|r| | }t |t
j rW|}g }|D ]P}t t 5 | t | ddd n # 1 swxY w Y Qt ||||||||| |
| | |
S )a
A generator that produces :class:`~.Request` instances to follow all
links in ``urls``. It accepts the same arguments as the :class:`~.Request`'s
``__init__`` method, except that each ``urls`` element does not need to be
an absolute URL, it can be any of the following:
* a relative URL
* a :class:`~scrapy.link.Link` object, e.g. the result of
:ref:`topics-link-extractors`
* a :class:`~scrapy.selector.Selector` object for a ```` or ```` element, e.g.
``response.css('a.my_link')[0]``
* an attribute :class:`~scrapy.selector.Selector` (not SelectorList), e.g.
``response.css('a::attr(href)')[0]`` or
``response.xpath('//img/@src')[0]``
In addition, ``css`` and ``xpath`` arguments are accepted to perform the link extraction
within the ``follow_all`` method (only one of ``urls``, ``css`` and ``xpath`` is accepted).
Note that when passing a ``SelectorList`` as argument for the ``urls`` parameter or
using the ``css`` or ``xpath`` parameters, this method will not produce requests for
selectors from which links cannot be obtained (for instance, anchor tags without an
``href`` attribute)
c g | ]}||S r& ).0xs r#
z+TextResponse.follow_all.. s DDD1aDQDDDr$ rC zFPlease supply exactly one of the following arguments: urls, css, xpathN)
urlsre rf rI r3 rg rh r ri rj rk rl rm )
lenrq ra r^ r' rn rp r _InvalidSelectorappendro r
follow_all)r ry re rf rI r3 rg rh r ri rj rk rl rm ra r^ arguments selectorsselr" s r# r} zTextResponse.follow_all sy R EDsE 2DDD y>>Q X
)
%xx}}
)zz%(( dF / 0 0 9 ID
9
9 .// 9 9KK 23 7 78889 9 9 9 9 9 9 9 9 9 9 9 9 9 9ww!! # "
s #CC C ) Nrb NNNNNr FNNN)NNrb NNNNNr FNNNNN)"r0
__module__ __qualname__rP r> r= r r r r( __annotations__r r* r2 propertyr r5 r? rA r r r: r6 rO r; r9 r\ r^ ra r
rr r r}
__classcell__)r" s @r# r r st "*"5
"EJc3hEEE* * * * *" " " " "
$
$
$
$
$ K K XK
) ) ) " " X"0 0 0
D D D ! ! !) ) ) 6 6 6 & & & % % X%4 4 4( ( (
3
3
3
3
3
3
3
3
n
!G
G
"
7D$& '#G
G
G
G
G
G
G
G
G
G
r$ r c e Zd ZdZdS )r{ z>
Raised when a URL cannot be obtained from a Selector
N)r0 r r __doc__ru r$ r# r{ r{ s r$ r{ c t | j t rt | j S t | j d st d| | j j dvr t d| j j d | j d }| t d| j j d| t | S ) NtagzUnsupported selector: )alinkz1Only and elements are supported; got <>href element has no href attribute: )r' rootr( r hasattrr{ r rJ )r r s r# ro ro s #(C 0%ch/// 38U # # ? = = =>>>
x|=(
R38< R R R
8<< D Y W38< W WRU W WXXX !$ ' ''r$ )!r r?
contextlibr typingr r urllib.parser rn w3lib.encodingr r r r
r
w3lib.htmlr scrapy.httpr
scrapy.http.responser scrapy.utils.pythonr r scrapy.utils.responser objectr> r rq r{ ro ru r$ r# r s # # # # # # # #
. - - - - - ) ) ) ) ) ) @ @ @ @ @ @ @ @ . . . . . .r
r
r
r
r
8 r
r
r
j z ( ( ( ( (r$