a-ddZddlZddlZddlZddlmZddlmZmZddlm Z ejdejZ ejdejZejejdej Zejejd ejejzZejd ejZd Zdd Zdd ZddZddZejdejZddZddZddZ ddZddZd dZd!dZ dZ!y)"z( Functions for dealing with markup text N)moves)to_bytes to_unicode)safe_url_stringzI&((?P[a-z\d]+)|#(?P\d+)|#x(?P[a-f\d]+))(?P;?)z<[a-zA-Z\/!].*?>z5]*href\s*=\s*[\"\']\s*([^\"\'\s]+)\s*[\"\']z}]*http-equiv[^>]*refresh[^>]*content\s*=\s*(?P["\'])(?P(\d*\.)?\d+)\s*;\s*url=\s*(?P.*?)(?P=quote)z<((?P.*?)(?P\]\]>))z cRtjdtt||||S)z .. warning:: This function is deprecated and will be removed in future. Please use :func:`replace_entities` instead. z`w3lib.html.remove_entities` function is deprecated and will be removed in future releases. Please use `w3lib.html.replace_entities` instead.)warningswarnDeprecationWarningreplace_entities)textkeepremove_illegalencodings *lib/python3.12/site-packages/w3lib/html.pyremove_entitiesrs- MM 1   D$ AAcRfd}tj|t||S)uRemove entities from the given `text` by converting them to their corresponding unicode character. `text` can be a unicode string or a byte string encoded in the given `encoding` (which defaults to 'utf-8'). If `keep` is passed (with a list of entity names) those entities will be kept (they won't be removed). It supports both numeric entities (``&#nnnn;`` and ``&#hhhh;``) and named entities (such as `` `` or ``>``). If `remove_illegal` is ``True``, entities that can't be converted are removed. If `remove_illegal` is ``False``, entities that can't be converted are kept "as is". For more information see the tests. Always returns a unicode string (with the entities removed). >>> import w3lib.html >>> w3lib.html.replace_entities(b'Price: £100') u'Price: \xa3100' >>> print(w3lib.html.replace_entities(b'Price: £100')) Price: £100 >>> c|j}|jdrt|dd}n|jdrt|dd}n|jdr|d}|jvr|j dSt j jj|xs7t j jj|j}H d|cxkrdkr'nn$tj|jd Stj|Sr|jd rd S|j dS#t$rY1wxYw) Ndec hexnamedrcp1252 semicolon) groupdictgetintlowergroupr html_entitiesname2codepointsixint2bytedecodeunichr ValueError)mgroupsnumber entity_namer rs rconvert_entityz(replace_entities..convert_entityEs9 ::e +F ZZ +F ZZ  /K  "d*wwqz!--<<@@MP''66::;;L;L;NO    6)T)<</66x@@::f--%K)@sPaggajP  s21E$E E+*E+)_ent_resubr)r r rrr/s `` rr r )s#8Q8 ;;~z$'A BBrcRttjt||SN)boolr0searchrr rs r has_entitiesr7cs z$9: ;;rcBtj|t||S)afReplace all markup tags found in the given `text` by the given token. By default `token` is an empty string so it just removes all tags. `text` can be a unicode string or a regular string encoded as `encoding` (or ``'utf-8'`` if `encoding` is not given.) Always returns a unicode string. Examples: >>> import w3lib.html >>> w3lib.html.replace_tags(u'This text contains some tag') u'This text contains some tag' >>> w3lib.html.replace_tags('

Je ne parle pas fran\xe7ais

', ' -- ', 'latin-1') u' -- Je ne parle pas -- fran\xe7ais -- -- ' >>> )_tag_rer1r)r tokenrs r replace_tagsr;fs( ;;ujx8 99rz|$)cFt||}tjd|S)z Remove HTML Comments. >>> import w3lib.html >>> w3lib.html.remove_comments(b"test whatever") u'test whatever' >>> r)r_REMOVECOMMENTS_REr1r6s rremove_commentsr>~s# dH %D  ! !#t ,,rcfr rJdDchc]}|jc}Dchc]}|jc}fdfd}d}tj|tjtjz}|j |t ||Scc}wcc}w)a1 Remove HTML Tags only. `which_ones` and `keep` are both tuples, there are four cases: ============== ============= ========================================== ``which_ones`` ``keep`` what it does ============== ============= ========================================== **not empty** empty remove all tags in ``which_ones`` empty **not empty** remove all tags except the ones in ``keep`` empty empty remove all tags **not empty** **not empty** not allowed ============== ============= ========================================== Remove all tags: >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags(doc) u'This is a link: example' >>> Keep only some tags: >>> w3lib.html.remove_tags(doc, keep=('div',)) u'
This is a link: example
' >>> Remove only specific tags: >>> w3lib.html.remove_tags(doc, which_ones=('a','b')) u'

This is a link: example

' >>> You can't remove some and keep some: >>> w3lib.html.remove_tags(doc, which_ones=('a',), keep=('p',)) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/w3lib/html.py", line 101, in remove_tags assert not (which_ones and keep), 'which_ones and keep can not be given at the same time' AssertionError: which_ones and keep can not be given at the same time >>> z5which_ones and keep can not be given at the same timec8|j}r|vS|vSr3)r")tagr which_oness r will_removez remove_tags..will_removes&iik *$ $d? "rc\|jd}|rdS|jdS)Nrr)r#)r+rArCs r remove_tagzremove_tags..remove_tags*ggaj!#&s6AGGAJ6rz/]+).*?>)r"recompileDOTALL IGNORECASEr1r) r rBr rrArFregexretagsrCs `` @r remove_tagsrMs^t]&]] $)34##))+4J#' (4CCIIK4 (D#7 E ZZryy2==8 9F ::j*T8"< ==#5 (s B)B.c  t||}|rpdj|Dcgc]}d|d|d|dc}}tj|tjtj z}|j d|}|Scc}w)aRemove tags and their content. `which_ones` is a tuple of which tags to remove including their content. If is empty, returns the string unmodified. >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags_with_content(doc, which_ones=('b',)) u'' >>> ||r)rjoinrGrHrIrJr1)r rBrrAtagsrLs rremove_tags_with_contentrSsr dH %DxxPZ[PZc3DPZ[\D"))bmm";<zz#t$ K\sBcdt||}|D]}|j|t||} |S)a$Remove escape characters. `which_ones` is a tuple of which escape characters we want to remove. By default removes ``\n``, ``\t``, ``\r``. `replace_by` is the string to replace the escape characters by. It defaults to ``''``, meaning the escape characters are removed. )rreplace)r rB replace_byrecs rreplace_escape_charsrXs6 dH %D||B :x @A Krcd}t||}d}||tD]B}t|tjr|t |||z }/||j dz }D|S)a` This function receives markup as a text (always a unicode string or a UTF-8 encoded string) and does the following: 1. removes entities (except the ones in `keep`) from any part of it that is not inside a CDATA 2. searches for CDATAs and extracts their text (if any) without modifying it. 3. removes the found CDATAs c3Kd}|j|D]#}|jd\}}|||||}%||dyw)NrrE)finditerspan)txtpatternoffsetmatchmatch_smatch_es r_get_fragmentsz&unquote_markup.._get_fragmentssW%%c*E$zz!} GWfW% %KF + &'lsAAr)r rcdata_d)r _cdata_re isinstancer& string_typesr r#)r r rrrcret_textfragments runquote_markuprjsk dH %DH"43 h 0 0 1 (^\ \H y1 1H 4 Orct||}tj|}|rMtjj j t|t|jd|St|S)zReturn the base url if declared in the given HTML `text`, relative to the given base url. If no base url is found, the given `baseurl` is returned. rE)r) r _baseurl_rer5rurllibparseurljoinrr#)r baseurlrr+s r get_base_urlrqsg dH %D4 A||!!)) G $ AGGAJ :  w''rctjr t||} t||}t ||}tt|}tj|}|rrt|jd}t|jdjd|}tj j"j%||}||fSy#t$r t |wxYw)aYReturn the http-equiv parameter of the HTML meta element from the given HTML text and return a tuple ``(interval, url)`` where interval is an integer containing the delay in seconds (or zero if not present) and url is a string with the absolute url to redirect. If no meta redirect is found, ``(None, None)`` is returned. r!urlz "')NN)r&PY2rrUnicodeDecodeErrorprintrSr>r _meta_refresh_rer5floatr#rstriprrmrnro)r rpr ignore_tagsr+intervalrss rget_meta_refreshr|(s ww7H-$) $D+ 6D +D1 2D%A(aggen226:HEll  ((#6}  d  s CC*c,|jtS)a Strip all leading and trailing space characters (as defined in https://www.w3.org/TR/html5/infrastructure.html#space-character). Such stripping is useful e.g. for processing HTML element attributes which contain URLs, like ``href``, ``src`` or form ``action`` - HTML5 standard defines them as "valid URL potentially surrounded by spaces" or "valid non-empty URL potentially surrounded by spaces". >>> strip_html5_whitespace(' hello\n') 'hello' )ryHTML5_WHITESPACE)r s rstrip_html5_whitespacerEs ::& ''r)Tutf-8r3)rN)rrN)rN))   rN)rTN)rr)rr)scriptnoscript)"__doc__rrGr&r w3lib.utilrr w3lib.urlrrHrJr0rIr9uIrlrwrer~rr r7r;r=r>rMrSrXrjrqr|rrrrrsQ +% "**acecpcp q "**(")) 4bjjWXZ\Z^Z^_ 2::ecee%efhjhqhqtvtAtAhAB BJJVXZXaXa b  B$8Ct<:. RZZ 3RYY? -B>H,JM"@($: (r