python - NLTK - when to normalize the text? -

-

i've finished gathering data plan use corpus, i'm bit confused whether should normalize text. plan tag & chunk corpus in future. of nltk's corpora lower case , others aren't.

can shed light on subject, please?

by "normalize" mean making lowercase?

the decision whether lowercase dependent of plan do. purposes, lowercasing better because lowers sparsity of data (uppercase words rarer , might confuse system unless have massive corpus such statistics on capitalized words decent). in other tasks, case information might valuable.

additionally, there other considerations you'll have make similar. example, should "can't" treated ["can't"], ["can", "'t"], or ["ca", "n't"] (i've seen 3 in different corpora). 7-year-old? 1 long word? or 3 words should separated?

that said, there's no reason reformat corpus. can have code make these changes on fly. way original information still around later if ever need it.


Comments

Post a Comment

Popular posts from this blog

linux - Using a Cron Job to check if my mod_wsgi / apache server is running and restart -

-
my group , running server based upon django , uses mod_wsgi run apache server. not working on project after over, attempting set cronjob similar functionality check if apache server has shut down(system restart or power failure), , if has, restart server me. i've found documentation on how check if apache server down , restart server if is, our server uses https , our start command pretty verbose. can use functionality provided in these examples: https://askubuntu.com/questions/277389/cron-job-to-restart-apache https://www.digitalocean.com/community/tutorials/how-to-use-a-simple-bash-script-to-restart-server-programs or need more complicated process make happen? the command use start server is python manage.py runmodwsgi --host 0.0.0.0 --port 8001 --https-port 8000 --ssl-certificate (certificate location) --server-name (domain name) i'm pretty new linux , using both mod-wsgi apache appreciated. you better off using --setup-only option mod_wsgi-expres...
Read more

actionscript 3 - TweenLite does not work with object -

-
i got following problem: i have object called tempscore game. this object blitted canvas renderer via copypixels method. object not display object. it's score-object (self made). score-object extends object called basicblitarrayobject. basicblitarrayobject extends eventdispatcher (therefore no display object). i tried apply several different tweenlite-plugins tempscore-object (i.e. transformaroundcenter, colormatrixfilter, etc.). nothing happens. absolutely nothing. sometimes error messages (when plugin requires display object , object not display object). far good. according greensock (maker of tweenlite) engine can tween numeric property of object. when plugin transformaroundcenter requires display object tweening have modify plugin working non-display object (tempscore). can't because it's way hard me. my game rests upon code: http://www.8bitrocket.com/book/ch11_blastermines.zip try apply tweenlite object called tempmine inside game class blastermines. ...
Read more

jQuery Ajax Render Fragments OR Whole Page -

-
i'm in throes of building application wraps mess of legacy code. page working on, need substitute fragment divs constructed on back-end -- or else need replace entire page altogether. idea there dynamically controlled flow needs satisfied before can forward legacy product. substituting fragments works fine, seen in below my_body_content swap. trouble comes when i'm trying render not fragment, whole page, in "body" swap. @ point page goes blank. i want similar errors returned server. want nice rest 404 error messages displayed on screen, legacy-product 404s show in legacy-product 404 page. yes, requirements project weird. that's not problem can fix. here's jquery invocation, names changed protect guilty: $.ajax({ url: "places/things", type: "post", data: json.stringify(somebadassobject), datatype: "text", accepts: "text/html", contenttype: "application/json; charset=...
Read more