a-dZddlZddlZddlZddlmZddlmZmZddlm Z ej dej Z ej dej Zej ejdejZej ejd ej ej zZej d ej Zd ZddZddZd dZd!dZej dej Zd dZd"dZd#dZ d$dZd%dZd&dZd'dZ dZ!dS)(z( Functions for dealing with markup text N)moves)to_bytes to_unicode)safe_url_stringzI&((?P[a-z\d]+)|#(?P\d+)|#x(?P[a-f\d]+))(?P;?)z<[a-zA-Z\/!].*?>z5]*href\s*=\s*[\"\']\s*([^\"\'\s]+)\s*[\"\']z}]*http-equiv[^>]*refresh[^>]*content\s*=\s*(?P["\'])(?P(\d*\.)?\d+)\s*;\s*url=\s*(?P.*?)(?P=quote)z<((?P.*?)(?P\]\]>))z Tutf-8cZtjdtt||||S)z .. warning:: This function is deprecated and will be removed in future. Please use :func:`replace_entities` instead. z`w3lib.html.remove_entities` function is deprecated and will be removed in future releases. Please use `w3lib.html.replace_entities` instead.)warningswarnDeprecationWarningreplace_entities)textkeepremove_illegalencodings *lib/python3.11/site-packages/w3lib/html.pyremove_entitiesrs8 M 1   D$ A AAcdfd}t|t||S)uRemove entities from the given `text` by converting them to their corresponding unicode character. `text` can be a unicode string or a byte string encoded in the given `encoding` (which defaults to 'utf-8'). If `keep` is passed (with a list of entity names) those entities will be kept (they won't be removed). It supports both numeric entities (``&#nnnn;`` and ``&#hhhh;``) and named entities (such as `` `` or ``>``). If `remove_illegal` is ``True``, entities that can't be converted are removed. If `remove_illegal` is ``False``, entities that can't be converted are kept "as is". For more information see the tests. Always returns a unicode string (with the entities removed). >>> import w3lib.html >>> w3lib.html.replace_entities(b'Price: £100') u'Price: \xa3100' >>> print(w3lib.html.replace_entities(b'Price: £100')) Price: £100 >>> c8|}|drt|dd}n|drt|dd}n|dr|d}|vr|dSt jj|p5t jj|}|\ d|cxkrdkr*nn'tj | d Stj |S#t$rYnwxYwr|d rd n|dS) Ndec hexnamedrcp1252 semicolon) groupdictgetintlowergroupr html_entitiesname2codepointsixint2bytedecodeunichr ValueError)mgroupsnumber entity_namerrs rconvert_entityz(replace_entities..convert_entityEs ::e   Q++FF ZZ   Q++FF ZZ  Q /K  ""d**wwqzz!-<@@MMP'6::;;L;L;N;NOO    6))))T)))))<//66x@@@:f---    %PK)@)@PssaggajjPs6E E E+*E+)_ent_resubr)rrrrr1s `` rr r )sH8QQQQQQ8 ;;~z$'A'A B BBrclttt||SN)boolr2searchrrrs r has_entitiesr9cs& z$99:: ; ;;rr cTt|t||S)afReplace all markup tags found in the given `text` by the given token. By default `token` is an empty string so it just removes all tags. `text` can be a unicode string or a regular string encoded as `encoding` (or ``'utf-8'`` if `encoding` is not given.) Always returns a unicode string. Examples: >>> import w3lib.html >>> w3lib.html.replace_tags(u'This text contains some tag') u'This text contains some tag' >>> w3lib.html.replace_tags('

Je ne parle pas fran\xe7ais

', ' -- ', 'latin-1') u' -- Je ne parle pas -- fran\xe7ais -- -- ' >>> )_tag_rer3r)rtokenrs r replace_tagsr=fs"( ;;ujx88 9 99rz|$)cXt||}td|S)z Remove HTML Comments. >>> import w3lib.html >>> w3lib.html.remove_comments(b"test whatever") u'test whatever' >>> r )r_REMOVECOMMENTS_REr3r8s rremove_commentsr@~s) dH % %D  ! !#t , ,,rcr r JddDdDfdfd}d}tj|tjtjz}||t ||S)a1 Remove HTML Tags only. `which_ones` and `keep` are both tuples, there are four cases: ============== ============= ========================================== ``which_ones`` ``keep`` what it does ============== ============= ========================================== **not empty** empty remove all tags in ``which_ones`` empty **not empty** remove all tags except the ones in ``keep`` empty empty remove all tags **not empty** **not empty** not allowed ============== ============= ========================================== Remove all tags: >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags(doc) u'This is a link: example' >>> Keep only some tags: >>> w3lib.html.remove_tags(doc, keep=('div',)) u'
This is a link: example
' >>> Remove only specific tags: >>> w3lib.html.remove_tags(doc, which_ones=('a','b')) u'

This is a link: example

' >>> You can't remove some and keep some: >>> w3lib.html.remove_tags(doc, which_ones=('a',), keep=('p',)) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/w3lib/html.py", line 101, in remove_tags assert not (which_ones and keep), 'which_ones and keep can not be given at the same time' AssertionError: which_ones and keep can not be given at the same time >>> z5which_ones and keep can not be given at the same timec6h|]}|Srr$.0tags r zremove_tags..s 444##))++444rc6h|]}|SrrCrDs rrGzremove_tags..s ( ( (CCIIKK ( ( (rc@|}r|vS|vSr5rC)rFr which_oness r will_removez remove_tags..will_removes-iikk  #*$ $d? "rcr|d}|rdn|dS)Nr r)r%)r-rFrKs r remove_tagzremove_tags..remove_tags5ggajj!k#&&6ssAGGAJJ6rz/]+).*?>)recompileDOTALL IGNORECASEr3r)rrJrrrNregexretagsrKs `` @r remove_tagsrUs^]t]]&]]]]44444J ( (4 ( ( (D######77777 E Zry2=8 9 9F ::j*T8"<"< = ==rct||}|radd|D}tj|tjtjz}|d|}|S)aRemove tags and their content. `which_ones` is a tuple of which tags to remove including their content. If is empty, returns the string unmodified. >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags_with_content(doc, which_ones=('b',)) u'' >>> |c &g|]}d|d|d|dS)|rrDs r z,remove_tags_with_content..s-[[[ccc333D[[[rr )rjoinrOrPrQrRr3)rrJrtagsrTs rremove_tags_with_contentr]sn dH % %D%xx[[PZ[[[\\D")bm";<<zz#t$$ Kr   cxt||}|D]&}||t||}'|S)a$Remove escape characters. `which_ones` is a tuple of which escape characters we want to remove. By default removes ``\n``, ``\t``, ``\r``. `replace_by` is the string to replace the escape characters by. It defaults to ``''``, meaning the escape characters are removed. )rreplace)rrJ replace_byrecs rreplace_escape_charsrfsH dH % %DBB||B :x @ @AA Krcd}t||}d}||tD]J}t|tjr|t |||z }2||dz }K|S)a` This function receives markup as a text (always a unicode string or a UTF-8 encoded string) and does the following: 1. removes entities (except the ones in `keep`) from any part of it that is not inside a CDATA 2. searches for CDATAs and extracts their text (if any) without modifying it. 3. removes the found CDATAs c3Kd}||D],}|d\}}|||V|V|}-||dVdS)NrrM)finditerspan)txtpatternoffsetmatchmatch_smatch_es r_get_fragmentsz&unquote_markup.._get_fragmentss{%%c**  E$zz!}} GWfWn% % % %KKKFF&''lrr )rrcdata_d)r _cdata_re isinstancer( string_typesr r%)rrrrrqret_textfragments runquote_markuprxs dH % %DH"N43322 h 0 1 1 2 (^\\\ \HH y11 1HH Orc t||}t|}|rTtjjt|t|d|St|S)zReturn the base url if declared in the given HTML `text`, relative to the given base url. If no base url is found, the given `baseurl` is returned. rM)r) r _baseurl_rer7rurllibparseurljoinrr%)rbaseurlrr-s r get_base_urlrs dH % %D4  A(|!)) G $ $ AGGAJJ : : :   w'''rscriptnoscriptc0tjrt||} t||}n#t$rt |wxYwt ||}tt|}t |}|rt| d}t| dd|}tjj||}||fSdS)aYReturn the http-equiv parameter of the HTML meta element from the given HTML text and return a tuple ``(interval, url)`` where interval is an integer containing the delay in seconds (or zero if not present) and url is a string with the absolute url to redirect. If no meta redirect is found, ``(None, None)`` is returned. r#urlz "')NN)r(PY2rrUnicodeDecodeErrorprintr]r@r _meta_refresh_rer7floatr%rstriprr{r|r})rr~r ignore_tagsr-intervalrs rget_meta_refreshr(s w.7H--$))  d   $D+ 6 6D +D11 2 2D%%A((aggenn226::HEEl ((#66}zs /A c6|tS)a Strip all leading and trailing space characters (as defined in https://www.w3.org/TR/html5/infrastructure.html#space-character). Such stripping is useful e.g. for processing HTML element attributes which contain URLs, like ``href``, ``src`` or form ``action`` - HTML5 standard defines them as "valid URL potentially surrounded by spaces" or "valid non-empty URL potentially surrounded by spaces". >>> strip_html5_whitespace(' hello\n') 'hello' )rHTML5_WHITESPACE)rs rstrip_html5_whitespacerEs ::& ' ''r)rTrr5)r N)rrN)rN)r^r N)rTN)r r)r rr)"__doc__r rOr(r w3lib.utilrr w3lib.urlrrPrRr2rQr;uIrzrrsrrr r9r=r?r@rUr]rfrxrrrrrrrsP ++++++++%%%%%% "*acecp q q "*(") 4 4bjWXXZ\Z^__ 2:ece%effhjhqtvtAhABB BJVXZXa b b  BBBB$8C8C8C8Ct<<<<::::. RZ 3RY?? - - - -B>B>B>B>H,JM"@(((($: ( ( ( ( (r