nested - Use python to crawl a website -

-

so looking dynamic way crawl website , grab links each page. decided experiment beauitfulsoup. 2 questions: how do more dynamically using nested while statements searching links. want links site. don't want continue put nested while loops.

    toplevellinks = self.getalluniquelinks(baseurl)     listoflinks = list(toplevellinks)             length = len(listoflinks)     count = 0             while(count < length):          twolevellinks = self.getalluniquelinks(listoflinks[count])         twolistoflinks = list(twolevellinks)         twocount = 0         twolength = len(twolistoflinks)          twolinks in twolistoflinks:             listoflinks.append(twolinks)          count = count + 1          while(twocount < twolength):             threelevellinks = self.getalluniquelinks(twolistoflinks[twocount])               threelistoflinks = list(threelevellinks)              threelinks in threelistoflinks:                 listoflinks.append(threelinks)              twocount = twocount +1        print '--------------------------------------------------------------------------------------'     #remove duplicates     finallist = list(set(listoflinks))       print finallist

my second questions there anyway tell if got links site. please forgive me, new python (year or so) , know of processes , logic might childish. have learn somehow. want more dynamic using nested while loop. in advance insight.

the problem of spidering on web site , getting links common problem. if google search "spider web site python" can find libraries you. here's 1 found:

http://pypi.python.org/pypi/spider.py/0.5

even better, google found question asked , answered here on stackoverflow:

anyone know of python based web crawler use?


Comments

Post a Comment

Popular posts from this blog

linux - Using a Cron Job to check if my mod_wsgi / apache server is running and restart -

-
my group , running server based upon django , uses mod_wsgi run apache server. not working on project after over, attempting set cronjob similar functionality check if apache server has shut down(system restart or power failure), , if has, restart server me. i've found documentation on how check if apache server down , restart server if is, our server uses https , our start command pretty verbose. can use functionality provided in these examples: https://askubuntu.com/questions/277389/cron-job-to-restart-apache https://www.digitalocean.com/community/tutorials/how-to-use-a-simple-bash-script-to-restart-server-programs or need more complicated process make happen? the command use start server is python manage.py runmodwsgi --host 0.0.0.0 --port 8001 --https-port 8000 --ssl-certificate (certificate location) --server-name (domain name) i'm pretty new linux , using both mod-wsgi apache appreciated. you better off using --setup-only option mod_wsgi-expres...
Read more

actionscript 3 - TweenLite does not work with object -

-
i got following problem: i have object called tempscore game. this object blitted canvas renderer via copypixels method. object not display object. it's score-object (self made). score-object extends object called basicblitarrayobject. basicblitarrayobject extends eventdispatcher (therefore no display object). i tried apply several different tweenlite-plugins tempscore-object (i.e. transformaroundcenter, colormatrixfilter, etc.). nothing happens. absolutely nothing. sometimes error messages (when plugin requires display object , object not display object). far good. according greensock (maker of tweenlite) engine can tween numeric property of object. when plugin transformaroundcenter requires display object tweening have modify plugin working non-display object (tempscore). can't because it's way hard me. my game rests upon code: http://www.8bitrocket.com/book/ch11_blastermines.zip try apply tweenlite object called tempmine inside game class blastermines. ...
Read more

jQuery Ajax Render Fragments OR Whole Page -

-
i'm in throes of building application wraps mess of legacy code. page working on, need substitute fragment divs constructed on back-end -- or else need replace entire page altogether. idea there dynamically controlled flow needs satisfied before can forward legacy product. substituting fragments works fine, seen in below my_body_content swap. trouble comes when i'm trying render not fragment, whole page, in "body" swap. @ point page goes blank. i want similar errors returned server. want nice rest 404 error messages displayed on screen, legacy-product 404s show in legacy-product 404 page. yes, requirements project weird. that's not problem can fix. here's jquery invocation, names changed protect guilty: $.ajax({ url: "places/things", type: "post", data: json.stringify(somebadassobject), datatype: "text", accepts: "text/html", contenttype: "application/json; charset=...
Read more