python - NLTK - when to normalize the text? -

-

i've finished gathering data plan use corpus, i'm bit confused whether should normalize text. plan tag & chunk corpus in future. of nltk's corpora lower case , others aren't.

can shed light on subject, please?

by "normalize" mean making lowercase?

the decision whether lowercase dependent of plan do. purposes, lowercasing better because lowers sparsity of data (uppercase words rarer , might confuse system unless have massive corpus such statistics on capitalized words decent). in other tasks, case information might valuable.

additionally, there other considerations you'll have make similar. example, should "can't" treated ["can't"], ["can", "'t"], or ["ca", "n't"] (i've seen 3 in different corpora). 7-year-old? 1 long word? or 3 words should separated?

that said, there's no reason reformat corpus. can have code make these changes on fly. way original information still around later if ever need it.


Comments

Post a Comment

Popular posts from this blog

javascript - Iterate over array and calculate average values of array-parts -

-
i have array this: ["2011-06-16 16:37:20",23.2], ["2011-06-21 16:37:20",35.6], ["2011-06-26 16:37:20",41.8], ["2011-07-01 16:37:20",25], ["2011-07-06 16:37:20",22.8], ["2011-07-11 16:37:20",36.4], ["2011-07-16 16:37:20",34], ["2011-07-21 16:37:20",20] [...] format: [$date,count] now need add 3rd 4th value every array element, average of n counts before or after. example if n=3 the 3rd value should average of 3 count-values before , 4th value should average of 3 count-values after current array element. result that: ["2011-06-16 16:37:20",23.2, null, 34.13], ["2011-06-21 16:37:20",35.6, null, 29.86], ["2011-06-26 16:37:20",41.8, null, 28.06], ["2011-07-01 16:37:20",25, 33.53, 31.06], ["2011-07-06 16:37:20",22.8, 34.13, 30.13], ["2011-07-11 16:37:20",36.4, 29.86, null], ["2011-07-16 16:37:20",34, 28.06, null], ["2...
Read more

iphone - Using nested NSDictionary with Picker -

-
newbie here, i have single component picker set plist, each item being array, multiple strings in each array app uses. currently plist structure this: nsdictionary -> nsarray -> nsstring | | items in picker data each item but now, want: nsdictionary -> nsdictionary -> nsarray -> nsstring | | | different picker data sets items in picker data each item so there multiple sets of picker components show using segmented control etc... i'm not sure if possible, , hoping save me making many different separate picker controllers. what has me stumped getting ingested properly this have now, builds crashes (debug info below): nsbundle *bundle = [nsbundle mainbundle]; nsstring *plistpath = [bundle pathforresource:@"camerasdatabase" oftype:@"plist"]; nsdictionary *dictionary = [[nsdictionary alloc] initwithcontentsoffi...
Read more

php - How can I edit my code to echo the data of child's element where my search term was found in, in XMLReader? -

-
i new in xmlreader, , think kind of hard find tutorials/code or advanced examples. question how can transform code have now, if search (through form) term jquery give me opportunity output value of <info></info> (and later other elements) in every <name></name> found in ? this code output names of books. <?php $reader = new xmlreader(); $reader->open("books.xml"); while ($reader->read()) { switch ($reader->nodetype) { case (xmlreader::element): if ($reader->localname == "name") { $reader->read(); echo $reader->value; break; }}} ?> the xml file <?xml version="1.0" encoding="iso-8859-1"?> <library> <book isbn="781"> <name>scjp 1.5</name> <info>sun certified java programmer book</info> </book> <bo...
Read more