To remove content from all search engines, consider using one of the methods described in this article. Note: The methods conveyed do not make content on your website private. It is up to individual search engines to honor your "robots" instructions to exclude public content from their indexes. And it is up to the site maintainer to properly secure private content.
To exclude individual pages from search engines' crawls, include the following meta tag between the <head> and </head> tags on your page:
<meta name="robots" content="noindex, nofollow">
This will prevent crawlers (robots) from indexing the page or following any links on the page. If the page has already been indexed, it will be removed from the index the next time search engines like Google crawl the page. If you remove this tag, your page will be indexed the next time search engines crawl the page.
To exclude an entire site or directory from search engines' crawls, insert a robots.txt file at the top level of the site. The contents of the robots.txt file should resemble the following lines:
User-agent: * Disallow: /
More detailed instructions about robots.txt files can be found in this article The Web Robots Pages.