Crawling and Indexing
We are exploring how we can use our own crawler for a subset of our customers’ search configurations. The goal is to serve more results directly from our own indexes, rather than leveraging commercial indexes. We recognize that not all agency customers are able to send us their content via API or RSS, and we also recognize that agencies are interested in having more control over when their content gets indexed, which is not possible with the commercial crawlers.
- In August, our development team successfully migrated Search’s infrastructure to AWS. We continue to make updates to the new environment, including ClamAV scanning and new VPCs. Other work around our ATO includes:
- Nessus scanner work
- Re-IP’ing our system to integrate with GSA’s new log archiving system
- Password management and handling updates, now requiring complex passwords, updates every 90 days, and displaying ambiguous login error messages.
- We continue to assist customers with transitioning to HTTPS, adding sites’ domain masks to our SSL certificate and preparing to update our help manual and some backend configurations.
- When you clone a search configuration, your type-ahead suggestions will no longer clone to the new configuration. This enables your new site to populate type-ahead suggestions based on its own traffic.
- In December, Bing is deprecating 2 of its APIs. We’re handling this transition for all search configurations that use our hosted search results page. We began preparing for the transition this month and will complete work prior to the December 15th deprecation.
- Email verification links were redirecting infinitely, causing some new users to be unable to get into the system.
- We fixed a conflict for our customers who may share machines with other users, so now if a user is logged in and another user attempts to verify their account, the original user will be logged out so the second user can complete their verification.