tag:blogger.com,1999:blog-3105702472983275135.post6067489649117249546..comments2023-04-11T16:17:22.348+02:00Comments on Truly Madly Wordly: Creating a text corpus from WikipediaStephen Marquardhttp://www.blogger.com/profile/06185718122117108334noreply@blogger.comBlogger27125tag:blogger.com,1999:blog-3105702472983275135.post-72910400062942667892017-12-20T06:29:27.824+02:002017-12-20T06:29:27.824+02:00what are the command line arguments for mkvocab.pl...what are the command line arguments for mkvocab.pl file . Please help me !!Johnhttps://www.blogger.com/profile/04065764598700402366noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-25215549700929829712017-08-03T15:43:19.770+02:002017-08-03T15:43:19.770+02:00Hi, first of all thank you for your blog, it has r...Hi, first of all thank you for your blog, it has really helped me out. Secondly, I am a bit stuck trying to use the perl script, would you have an example of how you converted the html output from the java code to clean text? atm the moment I'm running the Java script on the xml dump file and thought whilst its running id try get the next step working before it finishes with some smaller data.Anonymoushttps://www.blogger.com/profile/15273789773578492843noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-50063882253726636092017-08-03T15:42:23.562+02:002017-08-03T15:42:23.562+02:00Hi, first of all thank you for your blog, it has r...Hi, first of all thank you for your blog, it has really helped me out. Secondly, I am a bit stuck trying to use the perl script, would you have an example of how you converted the html output from the java code to clean text? atm the moment I'm running the Java script on the xml dump file and thought whilst its running id try get the next step working before it finishes with some smaller data.Anonymoushttps://www.blogger.com/profile/15273789773578492843noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-83010197199365218772017-05-27T11:46:29.375+02:002017-05-27T11:46:29.375+02:00no sir.. actually i am trying to prepare the corpu...no sir.. actually i am trying to prepare the corpus manually. it will be helpful if somebody share the info.Anonymoushttps://www.blogger.com/profile/13663301549512777861noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-91847249144277147062017-05-08T11:24:15.532+02:002017-05-08T11:24:15.532+02:00@habi did u get any help ??@habi did u get any help ??Anonymoushttps://www.blogger.com/profile/07323320684799384795noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-13752822183797577762017-05-04T14:23:50.291+02:002017-05-04T14:23:50.291+02:00Sir, I want to extract some books or articles in M...Sir, I want to extract some books or articles in Malayalam language from web and also want to create the language model. can you please help me to do this? what are the tool kits available for pre-processing the text?Habihttps://www.blogger.com/profile/00220616470621837583noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-38663574727004769942017-03-17T17:24:43.360+02:002017-03-17T17:24:43.360+02:00Hi,
i need to recognize only 100 specific words so...Hi,<br />i need to recognize only 100 specific words so i don't want other unnecessary words in my dictionary .SO how could i do it.Please help meAnonymoushttps://www.blogger.com/profile/07943823404138358485noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-42517426928034965702016-11-01T11:13:19.409+02:002016-11-01T11:13:19.409+02:00Hi,
Do you get any solution?
I need to convert int...Hi,<br />Do you get any solution?<br />I need to convert into Korean language. So if you could help me it would be good for me.<br />can you please knock me at banna.kbet@gmail.com?Anonymoushttps://www.blogger.com/profile/13284692189357865324noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-43951187722717871052016-11-01T11:11:46.893+02:002016-11-01T11:11:46.893+02:00This comment has been removed by the author.Anonymoushttps://www.blogger.com/profile/13284692189357865324noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-23324288831937366862016-10-01T14:43:47.902+02:002016-10-01T14:43:47.902+02:00Hi
I found the link of this page from CMUsphinx an...Hi<br />I found the link of this page from CMUsphinx anguage model building page: http://cmusphinx.sourceforge.net/wiki/tutoriallm<br /><br />I have downloaded and installed gwtwiki, but don't know how to use it? I have also downloaded the wiki dump file in .xml.<br /><br />May you explain more the stem number 2?<br /><br />thanks a lotrezaeehttps://www.blogger.com/profile/11850426714503462150noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-47199184441005654512014-12-12T11:15:08.573+02:002014-12-12T11:15:08.573+02:00Id your final model or cleaned wiki corspus availa...Id your final model or cleaned wiki corspus available for downladAbhishek Sharmahttps://www.blogger.com/profile/01953303813794594914noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-44850830510411612322014-05-25T02:42:28.449+02:002014-05-25T02:42:28.449+02:00This comment has been removed by the author.Anonymoushttps://www.blogger.com/profile/03154413332601786113noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-23905880128715414042014-03-05T08:02:31.141+02:002014-03-05T08:02:31.141+02:00Out of interest, what version of mitlm did you use...Out of interest, what version of mitlm did you use? I ask because I get segmentation faults when using the same commands (estimate-ngram).<br /><br />Thanks<br /><br />ChrisAnonymoushttps://www.blogger.com/profile/02334166423314511196noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-32623973009197705452014-01-22T23:21:33.093+02:002014-01-22T23:21:33.093+02:00How do I need to change Wikipedia2Txt.java to leav...How do I need to change Wikipedia2Txt.java to leave paragraphs in one piece, i.e. not break them into separate sentences?Anonymoushttps://www.blogger.com/profile/10054922042569997298noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-65841191496205018612014-01-02T14:33:19.840+02:002014-01-02T14:33:19.840+02:00Hey,
I dig up this article because I try to make...Hey, <br /><br />I dig up this article because I try to make pocketsphinx working in french.<br />I was thinking about using different french books agregated into a single file and jumping right to step 3. My understanding is that it would do the trick right ?<br /><br />Regards<br />Anonymoushttps://www.blogger.com/profile/04984251207307448025noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-70749734509056118962013-08-16T11:31:03.126+02:002013-08-16T11:31:03.126+02:00naive solution will run in O(n) : suppose phrases ...naive solution will run in O(n) : suppose phrases are made of 2 words, then split a setence(for example) of n words into n-1 "phrases" and calculate..<br />nob0dyhttps://www.blogger.com/profile/13047526917296676326noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-52951487047431309202013-06-12T19:43:27.969+02:002013-06-12T19:43:27.969+02:00We can assume this could be done in any of the wik...We can assume this could be done in any of the wikipedias, for example, French or Spanish.<br />This would appear to pick out single morphemes well. Is it adaptable to pulling out multiple word collocations?<br />For example: "food poisoning," "an easy read," "light snack," and "absorbed in her book," while made up of adjective and noun or verb and prepositional or adverbial phrase, act as commonly occurring sequences of words whose collocation means more or other than the "sum of its parts."<br />Any algorithm(s) available to extract collocative phrases?Anonymoushttps://www.blogger.com/profile/15049663028976205535noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-264028213280746142013-05-25T01:55:30.744+02:002013-05-25T01:55:30.744+02:00This comment has been removed by the author.Aon GoltzCrankhttps://www.blogger.com/profile/06043460093470396424noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-17040297516406477282013-03-23T13:49:30.753+02:002013-03-23T13:49:30.753+02:00Thanks for your post. could you upload the lm file...Thanks for your post. could you upload the lm file you just created with this steps.Vignesh Chttps://www.blogger.com/profile/06051835728051213867noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-66296179983380514182013-03-20T12:28:32.697+02:002013-03-20T12:28:32.697+02:00Thank you very much! With your help, I solved the ...Thank you very much! With your help, I solved the whole problem in just one hour :)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-13764672636812806132011-04-30T16:41:37.254+02:002011-04-30T16:41:37.254+02:00Marwen and danajaatcse,
To do that you probably d...Marwen and danajaatcse,<br /><br />To do that you probably don't want to use the same technique here of processing the entire large wikipedia dump file, because most of the articles are not relevant.<br /><br />To construct a corpus on a specific topic, you'd need a collection of articles that are related to the topic. One approach would be to use Wikipedia categories or Wikipedia Book pages, and retrieve all linked articles.<br /><br />Another approach, which I am working on at the moment, is to start with Wikipedia search with some keywords, and then follow links based on the similarity of articles to each other. I will post a new article about that when I'm done.<br /><br />My approach uses vector space modelling and LSA to establish similarity, using the gensim toolkit (http://nlp.fi.muni.cz/projekty/gensim/).Stephen Marquardhttps://www.blogger.com/profile/06185718122117108334noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-63714913912301901012011-04-30T14:19:14.361+02:002011-04-30T14:19:14.361+02:00hi! i want to know too if there is a way to extrac...hi! i want to know too if there is a way to extract a corpus from only articles that are specific to a given domain (e.g. water domain [water, river, tsunami, purification, freshwater...]Marwenhttps://www.blogger.com/profile/14921868532005695805noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-18468349229850647752011-04-08T07:38:45.387+02:002011-04-08T07:38:45.387+02:00say I wanted to extract articles on history or com...say I wanted to extract articles on history or computer science?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-900480957707527212011-04-07T21:51:37.668+02:002011-04-07T21:51:37.668+02:00How do you define "a specific domain" in...How do you define "a specific domain" in the Wikipedia context?Stephen Marquardhttps://www.blogger.com/profile/06185718122117108334noreply@blogger.comtag:blogger.com,1999:blog-3105702472983275135.post-82435629788752827742011-04-07T20:41:16.751+02:002011-04-07T20:41:16.751+02:00Hi,
Do you know a quick way to extract and creat...Hi, <br /><br />Do you know a quick way to extract and create the corpus for only the articles of a specific domain?Anonymousnoreply@blogger.com