robots.txt - How to disallow bots from a single page or file -

-

how disallow bots single page , allow allow other content crawled.

its important not wrong asking here, cant find definitive answer elsewhere.

is correct?

    user-agent:*     disallow: /dir/mypage.html     allow: /

the disallow line that's needed. block access starts "/dir/mypage.html".

the allow line superfluous. default robots.txt allow: /. in general, allow not required. it's there can override access disallowed. example, want disallow access "/images" directory, except images in "public" subdirectory. write:

allow: /images/public disallow: /images

note order important here. crawlers supposed use "first match" algorithm. if wrote 'disallow` first, crawler assume access "/images/public" blocked.


Comments

Post a Comment

Popular posts from this blog

linux - Using a Cron Job to check if my mod_wsgi / apache server is running and restart -

-
my group , running server based upon django , uses mod_wsgi run apache server. not working on project after over, attempting set cronjob similar functionality check if apache server has shut down(system restart or power failure), , if has, restart server me. i've found documentation on how check if apache server down , restart server if is, our server uses https , our start command pretty verbose. can use functionality provided in these examples: https://askubuntu.com/questions/277389/cron-job-to-restart-apache https://www.digitalocean.com/community/tutorials/how-to-use-a-simple-bash-script-to-restart-server-programs or need more complicated process make happen? the command use start server is python manage.py runmodwsgi --host 0.0.0.0 --port 8001 --https-port 8000 --ssl-certificate (certificate location) --server-name (domain name) i'm pretty new linux , using both mod-wsgi apache appreciated. you better off using --setup-only option mod_wsgi-expres...
Read more

actionscript 3 - TweenLite does not work with object -

-
i got following problem: i have object called tempscore game. this object blitted canvas renderer via copypixels method. object not display object. it's score-object (self made). score-object extends object called basicblitarrayobject. basicblitarrayobject extends eventdispatcher (therefore no display object). i tried apply several different tweenlite-plugins tempscore-object (i.e. transformaroundcenter, colormatrixfilter, etc.). nothing happens. absolutely nothing. sometimes error messages (when plugin requires display object , object not display object). far good. according greensock (maker of tweenlite) engine can tween numeric property of object. when plugin transformaroundcenter requires display object tweening have modify plugin working non-display object (tempscore). can't because it's way hard me. my game rests upon code: http://www.8bitrocket.com/book/ch11_blastermines.zip try apply tweenlite object called tempmine inside game class blastermines. ...
Read more

php - How can I edit my code to echo the data of child's element where my search term was found in, in XMLReader? -

-
i new in xmlreader, , think kind of hard find tutorials/code or advanced examples. question how can transform code have now, if search (through form) term jquery give me opportunity output value of <info></info> (and later other elements) in every <name></name> found in ? this code output names of books. <?php $reader = new xmlreader(); $reader->open("books.xml"); while ($reader->read()) { switch ($reader->nodetype) { case (xmlreader::element): if ($reader->localname == "name") { $reader->read(); echo $reader->value; break; }}} ?> the xml file <?xml version="1.0" encoding="iso-8859-1"?> <library> <book isbn="781"> <name>scjp 1.5</name> <info>sun certified java programmer book</info> </book> <bo...
Read more