Posts tagged domains

Filtering Tags

Search.gov Home > Admin Center > YourSite > Manage Content > Domains > Advanced > Filter Tags

Note: this feature is only available to sites that are enabled for i14y.

Tags are used in the full text searching of i14y documents. If a tag is added to a document, the document will appear in search results when a query matches the tag, even if the term does not appear in the document’s full text.

Use the Filter Tags feature to scope your search site based on tags. You can exclude documents that have a given tag from all search results. Or, you can require that a tag is present in order for a document to be shown.

For example:

  • You can have a “News Search” site that will only show results that have the tag “news”.
  • You can exclude all documents that have the “blog” tag from your search results, if you don’t want them to appear.

Adding Supplemental Content to Your Search Configuration

Search.gov Home > Admin Center > YourSite > Manage Content > Domains > Advanced

Is content missing from your search results? Do you want to have RSS content appear in your main page search results?

We offer two ways for you to tell us about content that you want us to fetch and include in our web index: via RSS feed (Supplemental Feed) or manually (via the Supplemental URLs section).

Note for customers using Collections: We serve Collections results from commercial indexes; the instructions below only apply to your main search results.

Note to our i14y customers: Supplemental content will not appear for customers using our i14y content indexing API.

Adding Content via Supplemental Feed

You can use an RSS feed to add URLs. The feed is useful if you’d like to automate the process, add multiple URLs, or both.

Enter the URL of your RSS feed. Click Submit. We’ll fetch each URL in your feed and index the title, description (optional), and the full text of the document/webpage for the items you provide. Please be sure to follow our schema, and note that we will only index the items that are listed in the feed.

You can see the list of URLs we have indexed from your feed by viewing the Supplemental URLs section.

Delete a URL added via RSS feed by deleting the item from your RSS feed. We’ll remove the URL from our index the next time we fetch your RSS feed (we fetch saved feeds once per day, in the evening in Eastern Time; for brand new feeds, we fetch as soon as you hit Submit on the feed URL).

You can also delete all of the URLs added via the RSS feed by deleting the RSS feed itself.

Note: We can’t restore URLs that don’t exist within your current feed. Please don’t remove items from the feed unless you want them to be removed from the index. You can add a feed that contains up to 1,000 items on your own. If you have a feed that will include more than 1,000 items, please contact us.

Adding Content via the Supplemental URLs Section

You can also manually add a specific URL on the Supplemental URLs page. We’ll fetch each URL you add manually and we’ll index the title, description, and the full text of the document/webpage for the link you provide. Note: You can’t manually add a Supplemental URL if it has already been added by via Supplemental feed.

Delete a manually added URL by selecting Remove within the list of supplemental URLs.

The source column in the crawl report shows how you added the URL, via an RSS feed or manually.

How Supplemental Content Appears on Your Search Results Page

We display titles as they are provided in the feed or as you entered them manually.

We display descriptions as follows:

  • If a searcher’s term matches terms in the description you provided: We’ll display the description exactly as you provided it.
  • If a searcher’s term only matches terms in the full text of the document: We’ll display snippets taken from the full text.

By default, supplemental content will appear after all Bing commercial search results have been shown. For example, a search on DigitalGov.gov for “serverless architecture” displays results from Bing’s index on page 1.

After hitting ‘next’ to view page 2, supplemental content is served.

Page 1:

Commercial Results are Displayed on Page One of the DigitalGov.gov Search Results Page

Page 2:

Supplemental Feed Results are Displayed on Page Two of the DigitalGov.gov Search Results Page

If there are no commercial results for a query but there are supplemental content results, we will display the supplemental content on page one.

Getting Supplemental Content To Always Appear First

If you would like your supplemental content to always appear first in search results, please contact our team. Searchers will first be served any supplemental content results, and are then offered the chance to “search again,” which will lead them to commercial index results.

Please note: if you have any regular RSS feeds set up in your search site, this regular RSS content will also appear first with the supplemental content, prior to commercial results. If content is present in both types of feed, duplicates will appear in your results.

In the example below, a Supplemental URL appears first, and clicking “Try your search again” will lead to a page of Bing results.

A Supplemental URL is Displayed on Page One of the DigitalGo Search Search Results Page

Getting Supplemental Content To Be Your Only Search Results

If you would like your supplemental content to be the only results served from your site, please contact our team. Searchers will not be offered the chance to “search again” on a commercial index after they exhaust your supplemental content results (see above example).

Please note: if you have any regular RSS feeds set up in your search site, this regular RSS content will also appear with the supplemental content. If content is present in both types of feed, duplicates will appear in your results.


Troubleshooting tip: We support RSS 2.0 and Atom feeds. Learn more and validate your feeds at:

Troubleshooting tip: Are you seeing an error message in the crawled URLs report for your PDF that says, “No content found in document”? Your PDF is likely an image-only, non-searchable file that was created from a paper document using a scanner. See the resources below for more information on how to create searchable PDF files.

Troubleshooting tip: Are you seeing an error message in the crawled URLs report that says your page is taking too long to load? Use Pingdom Tools Full Page Test (External link) to test the load time of your page, analyze it, and find the bottlenecks.

Did you know? To help the public find your web pages when they search on Bing.com, we notify Bing about any URLs you add. While this helps with search engine optimization (SEO), it is not a cure-all. You should also register for and use commercial search engines’ webmaster tools.

How to Edit Your Domains

Search.gov Home > Admin Center > YourSite > Manage Content > Domains

Enter the applicable domains, subdomains, or folders to return results specific to your website. You do not need to include http:// as we recognize domains with or without it.

Entire Site(s)

  • List yoursite.gov (without www) to search your entire site, including all subdomains.
  • List yoursite.gov and yoursite2.gov to search multiple sites.

Specific Subdomains

  • List www.yoursite.gov to include only the primary ‘www’ domain.
  • List subdomain.yoursite.gov to include subdomains.

Specific Directories

List yoursite.gov/folder to include folders. Note: this will include all sub-folders (if they exist), i.e. yoursite.gov/folder/folder2.

You cannot specify individual URLs.

Searches that start three or more subfolders deep cannot use our default configuration. Email us for assistance.


Troubleshooting Tip: You can exclude specific subdomains or folders using any of the following options.

  1. List the excluded subdomains or folders in your robots.txt (External link) file. This blocks them from all crawlers that follow the robots exclusion protocol, including Google.com and Bing.com.

  2. Exclude them by omission. For example, to exclude subdomain3, list only subdomain1.yoursite.gov, subdomain2.yoursite.gov, and subdomain4.yoursite.gov.

  3. Email us at search@support.digitalgov.gov to filter them from your site’s search results. Note that this option should be used only if the other options aren’t feasible as it can have an undesired effect on other features such as the display of spelling suggestions and relevance ranking.

Troubleshooting Tip: You can remove or filter specific URLs using any of the following options.

  1. Remove the page from your site and return a 404 (Not Found) 401 (Gone), or 301 (Permanently Moved) HTTP status.

  2. Add a noindex robots metatag or x-robots tag (External link). This blocks them from all crawlers that follow the robots exclusion protocol, including Google.com and Bing.com.

  3. Request the page’s removal using the Block URLs option in Bing Webaster Tools (External link) and Remove URLs option in Google Webmaster Tools (External link).

  4. Use the Filter URLs option under the Advanced tab. Note that this option should be used only if the other options aren’t feasible as it can have an undesired effect on other features such as the display of spelling suggestions and relevance ranking.

Troubleshooting Tip: Are specific web pages missing from your Bing web results? You can tell us about any missing URLs via an RSS feed or manually (the Supplemental Feed and Supplemental URLs options under the Advanced tab, respectively). We’ll fetch and index the pages and append them to your web results.

Filtering URLs

Search.gov Home > Admin Center > YourSite > Manage Content > Domains > Advanced > Filter URLs

You can filter a single URL from the results on your site. Use it for emergency reasons only (such as spillage of classified information or unintentional publication of a page). Note that this option should be used only if the other options aren’t feasible as it can have an undesired effect on other features such as the display of spelling suggestions and relevance ranking.

It is not meant to fix the results for entire site, such as for launch or redesign problems. For tips on how to fix these more systemic problems, read our Checklist for a Successful Website Redesign.

Entire the full path of the URL, e.g. use https://www.usa.gov/Citizen/Topics/Benefits.shtml, not usa.gov/citizen/topics.

You can later use the delete option to remove the filter.

Please note: this filtering only affects your search results page that is hosted by our service - it does not affect commercial web results on Google.com, Bing.com, etc.