Skip to main content
U.S. flag

An official website of the United States government

Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.


Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

June 2020 Newsletter

Everything Old Is New Again

Search and Website Redesign

If you are planning a website redesign, whether it is an upgrade or changing content management systems,  we encourage you to think about its impact on your search results. A good way to start factoring search into your redesign project is by letting us know ahead of time of your plans and if possible, the timeline you have set for them. Being aware of your redesign plans allows us to work together with you and prepare for this transition.

To get started, we suggest you read our checklist for a successful website redesign.

Don’t Get Locked Out

Take Control of Your User Account

Now our help manual includes step-by-step instructions on how to create and edit your user account with the service.

SEO Pro Tip

Robots.txt Crawl-Delays Aren’t What They Used to Be

Robots.txt files have traditionally included an option to direct spiders to crawl your website at a pace of your choosing, the crawl-delay, though it’s not part of the formal protocol. A robots.txt file for a Drupal website defaults to a 10 second crawl-delay. However, Google announced last year that they would stop honoring crawl-delay, due to widespread misconfiguration issues.

We do honor crawl-delays, by popular demand, but as we point out on our Robots.txt explainer page, they have a significant impact on the length of time it takes to index your content, especially for large websites. We plan to keep honoring crawl-delay settings for the foreseeable future, but ask that you add a provision that would allow us to work more swiftly in your site. We will never work faster than one request per second, but if you want a slower pace we suggest a two second crawl-delay. For example: user-agent: usasearch crawl-delay: 2.

Release Notes

Want to learn about the latest features, fixes, and focuses of the Search team? We post monthly Release Notes on our website. Here are some highlights:

  • We are thrilled to report that all the major system upgrades we’ve been working so hard on this year are complete!
  • We’ve upgraded all of’s elasticsearch components and are now operating on Elasticsearch 6.8.
  • We now have routed queries available for our results API users. If you use the API and have questions about how to implement this feature on your site, reach out.

Read the Latest Notes

June 01, 2020