Sitemaps.xml tells the search engines what pages exist in your website, but ~/robots.txt tells them what to index. The two files are complementary. Almost all search engines will respect your robots.txt instructions.
http://www.robotstxt.org/robotstxt.html
You can also use rel="nofollow" attributes on hyperlinks, or include <meta name="robots" content="noindex, nofollow" /> in your page metadata to prevent search engines indexing specific individual pages.
http://www.robotstxt.org/robotstxt.html
You can also use rel="nofollow" attributes on hyperlinks, or include <meta name="robots" content="noindex, nofollow" /> in your page metadata to prevent search engines indexing specific individual pages.