c=dZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z mZmZmZmZddlZejeZd-dZ d-d Zd Zd.d ZGd d e Zedkr^ejdejejejeddZ e!dej"dz Z#e $dddde $ddde $ddde%e# e $d!d"d#e%d e $d$d%d&d'(e &Z'e(d)d*)ej*ee'j+e'j,e'j-e'j.e'j/+e(d,ej*ddSdS)/aThis script using for extracting plain text out of a raw Wikipedia dump. Input is an xml.bz2 file provided by MediaWiki that looks like wiki--pages-articles.xml.bz2 or wiki-latest-pages-articles.xml.bz2 (e.g. 14 GB of https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2). It streams through all the XML articles using multiple cores (#cores - 1, by default), decompressing on the fly and extracting plain text from the articles and their sections. For each extracted article, it prints its title, section names and plain text section contents, in json-line format. How to use ---------- #. Process Wikipedia dump with this script :: python -m gensim.scripts.segment_wiki -i -f enwiki-latest-pages-articles.xml.bz2 -o enwiki-latest.json.gz #. Read output in simple way: .. sourcecode:: pycon >>> from gensim import utils >>> import json >>> >>> # iterate over the plain text data we just created >>> with utils.open('enwiki-latest.json.gz', 'rb') as f: >>> for line in f: >>> # decode each JSON line into a Python dictionary object >>> article = json.loads(line) >>> >>> # each article has a "title", a mapping of interlinks and a list of "section_titles" and >>> # "section_texts". >>> print("Article title: %s" % article['title']) >>> print("Interlinks: %s" + article['interlinks']) >>> for section_title, section_text in zip(article['section_titles'], article['section_texts']): >>> print("Section title: %s" % section_title) >>> print("Section text: %s" % section_text) Notes ----- Processing the entire English Wikipedia dump takes 1.7 hours (about 3 million articles per hour, or 10 MB of XML per second) on an 8 core Intel i7-7700 @3.60GHz. Command line arguments ---------------------- .. program-output:: python -m gensim.scripts.segment_wiki --help :ellipsis: 0, -10 N) ElementTree)partial)IGNORED_NAMESPACES WikiCorpus filter_wikifind_interlinks get_namespaceutilsFc#Ktj|d5}t||||}d|_|}|D]}|V ddddS#1swxYwYdS)aIExtract article titles and sections from a MediaWiki bz2 database dump. Parameters ---------- file_path : str Path to MediaWiki dump, typical filename is wiki--pages-articles.xml.bz2 or wiki-latest-pages-articles.xml.bz2. min_article_character : int, optional Minimal number of character for article (except titles and leading gaps). workers: int or None Number of parallel workers, max(1, multiprocessing.cpu_count() - 1) if None. include_interlinks: bool Whether or not interlinks should be included in the output Yields ------ (str, list of (str, str), (Optionally) list of (str, str)) Structure contains (title, [(section_heading, section_content), ...], (Optionally) [(interlink_article, interlink_text), ...]). rb)min_article_character processesinclude_interlinksTN)gensimr open_WikiSectionsCorpusmetadataget_texts_with_sections) file_pathrworkersr xml_fileobjwiki_sections_corpuswiki_sections_textarticles ;lib/python3.11/site-packages/gensim/scripts/segment_wiki.pysegment_all_articlesrIs2   9d + +{2 /DPW1 3 3 3)-%1IIKK)  GMMMM s8A))A-0A-c|&ttjdtj}n tj|d} t ||||}t|D]\}}|d|d} } |r|d} | ggd} |r| | d <| D];\} }| d | | d |<|dzd zdkrt d |dz| | tj | dzd ||dSdS#||wwxYw)aWrite article title and sections to `output_file` (or stdout, if output_file is None). The output format is one article per line, in json-line format with 4 fields:: 'title' - title of article, 'section_titles' - list of titles of sections, 'section_texts' - list of content from sections, (Optional) 'section_interlinks' - list of interlinks in the article. Parameters ---------- file_path : str Path to MediaWiki dump, typical filename is wiki--pages-articles.xml.bz2 or wiki-latest-pages-articles.xml.bz2. output_file : str or None Path to output file in json-lines format, or None for printing to stdout. min_article_character : int, optional Minimal number of character for article (except titles and leading gaps). workers: int or None Number of parallel workers, max(1, multiprocessing.cpu_count() - 1) if None. include_interlinks: bool Whether or not interlinks should be included in the output Nbufferwb)rrr)titlesection_titles section_texts interlinksr$r%iz"processed #%d articles (at %r now) zutf-8)getattrsysstdoutrr rr enumerateappendloggerinfowritejsondumpsencodeclose)r output_filerrroutfilearticle_streamidxr article_titlearticle_sectionsr& output_datasection_headingsection_contents rsegment_and_write_all_articlesr=ms:7#*h ;;,##K66-i9NX_ASUUU%n55 L LLC.5aj'!*+M! ($QZ '"$!#K " 7,6 L)4D E E0,-44_EEEO,33ODDDDa6!Q& Z @#'=YYY MM4:k22T9AA'JJ K K K K' L,   MMOOOOO  ;  MMOOOO s C5EE3c#Kdtj|dD}t|}t|j}d|i}d|z}|D]7}|j|kr*tj|V|8dS)zExtract pages from a MediaWiki database dump. Parameters ---------- f : file File descriptor of MediaWiki dump. Yields ------ str XML strings for page tags. c3 K|] \}}|V dSN).0_elems r z$extract_page_xmls..s& K KgaT K K K K K K)end)eventsnsz {%(ns)s}pageN)r iterparsenextr tagtostringclear)felemsrD namespace ns_mappingpage_tags rextract_page_xmlsrTs L K!6q!J!J!J K K KE ;;Ddh''I "J *H   8x  &t,, , , , JJLLL  rFctj|}d}t|j}d|i}d|z}d|z}d|z}d} d} d} ||j} ||j} ||j}||vrd } | n|rt | }tj| | }| gtj | | z}d |D}t|t|ksJng}g}g}d |D}tt||}|r| ||fS| |fS) aParse the content inside a page tag Parameters ---------- page_xml : str Content from page tag. include_interlinks : bool Whether or not interlinks should be parsed. Returns ------- (str, list of (str, str), (Optionally) list of (str, str)) Structure contains (title, [(section_heading, section_content), ...], (Optionally) [(interlink_article, interlink_text), ...]). 0rIz./{%(ns)s}revision/{%(ns)s}textz./{%(ns)s}titlez ./{%(ns)s}ns Introductionz\n==[^=].*[^=]==\nz\n==([^=].*[^=])==\nNc6g|]}|SrA)strip)rBheadings r zsegment..s LLLGMMOOLLLrFc,g|]}t|SrA)r)rBr<s rr\zsegment..s ]]] O44]]]rF) r fromstringr rLfindtextrresplitfindalllenlistzip)page_xmlrrDfilter_namespacesrQrR text_path title_pathns_pathlead_section_headingtop_level_heading_regextop_level_heading_regex_capturer#r`rIr&section_contentssection_headingssectionss rsegmentrrs$  !( + +Ddh''I "J1J>I"Z/Jz)G)3&=# IIj ! ! &E 99Y   $D 7   B ""   /(..J8$;TBB01BJ?^`d4e4eeLL;KLLL#$$,<(=(====== ]]L\]]]C(*:;;<wiki--pages-articles.xml.bz2 or wiki-latest-pages-articles.xml.bz2) as a (read-only) corpus. The documents are extracted on-the-fly, so that the whole (massive) dump can stay compressed on disk. r NrVFc|td||_||_d|_|$t dt jdz }||_||_||_ dS)aE Parameters ---------- fileobj : file File descriptor of MediaWiki dump. min_article_character : int, optional Minimal number of character for article (except titles and leading gaps). processes : int, optional Number of processes, max(1, multiprocessing.cpu_count() - 1) if None. filter_namespaces : tuple of int, optional Enumeration of namespaces that will be ignored. include_interlinks: bool Whether or not interlinks should be included in the output NzThe lemmatize parameter is no longer supported since Gensim 4.0.0. If you need to lemmatize, use e.g. https://github.com/clips/pattern to preprocess your corpus before submitting it to Gensim.Fr!) NotImplementedErrorfileobjrhrmaxmultiprocessing cpu_countrrr)selfrvrr lemmatizerhrs r__init__z_WikiSectionsCorpus.__init__s"  %L   !2  @A8::Q>??I"%:""4rFc#\ Kd\}}}d\}}t|j}tj|j}t j|d|jzdD]}|tt|j |D]} | d| dc } t fdtDr|dz }9| rE| dd d r|dz }t!d | D|jkr|dz }|dz }|t%| z }|j r| d } | | fV܉ | fVt&d ||||||||_d S)aIterate over the dump, returning titles and text versions of all sections of articles. Notes ----- Only articles of sufficient length are returned (short articles & redirects etc are ignored). Note that this iterates over the **texts**; if you want vectors, just use the standard corpus interface instead of this function: .. sourcecode:: pycon >>> for vec in wiki_corpus: >>> print(vec) Yields ------ (str, list of (str, str), list of (str, str)) Structure contains (title, [(section_heading, section_content), ...], (Optionally)[(interlink_article, interlink_text), ...]). )rrr)rr r!) chunksizemaxsize)rrc3HK|]}|dzVdS):N) startswith)rBignorer8s rrEz>_WikiSectionsCorpus.get_texts_with_sections..Rs5__&}// ==______rFz #redirectc3^K|](\}}t|V)dSr@)rdrZ)rBrCbodys rrEz>_WikiSectionsCorpus.get_texts_with_sections..Xs6CCYas4::<<((CCCCCCrFr"zhfinished processing %i articles with %i sections (skipped %i redirects, %i stubs, %i ignored namespaces)N)rTrvrxPoolrr chunkizeimaprrrranyrlstriplowerrsumrrdr-r. terminatelength) rzskipped_namespaceskipped_lengthskipped_redirecttotal_articlestotal_sections page_xmlspoolgrouprrqr&r8s @rrz+_WikiSectionsCorpus.get_texts_with_sections/s .?F;>+;)-&%dl33 #DN33^Idn9LVWXXX 4 4E99WWI`%a%a%a%*,, 4 4*1!*gaj' x____L^_____%*%8A;q>#8#8#:#:#@#@#B#B#M#Mk#Z#Z$)$CC(CCCCCdF``"a'N!##h--/*4!(J((J?????((33333- 40  v N,rs(11f  !!!!!!xxxxxxxxxxxxxxxx  8 $ $!!!!H_c6;<<<<~D5555pc%c%c%c%c%*c%c%c%L z%4GW_f_kllll $X $X5R`ghmimhm`n o o oFc!6_6881<==O h-[fjkkk  jPQQQ  k W     ' e     $u      D KK chhsx00111"" 4;"8 2  KK%sx{33333K%4%4rF