Set up and administer Microsoft Code Search

Team Services | TFS 2017

In this topic:

• Configure Code Search in Team Services

  • Install the Code Search extension
  • Uninstall the Code Search extension

• Configure Code Search in Team Foundation Server

  • Configure Code Search
  • Secure Code Search
  • Manage Code Search
  • Uninstall Code Search
  • Limitations of Code Search
  • Troubleshoot Code Search

Also see Install and configure TFS and TFS requirements and compatibility.

Users with at least a Basic access can use Code Search. Stakeholders do not have access to code, and therefore no access to Code Search.

Install Code Search in Team Services

Go to Visual Studio Marketplace to install the extension in your Team Services account as an administrator. Non-administrative users can also go here to request the extension be added to Team Services.

For more details, see Install an extension in the Marketplace documentation.

Uninstall Code Search in Team Services

See Uninstall or disable an extension in the Marketplace documentation.

Configure Code Search in Team Foundation Server

Code Search is available in Microsoft Team Foundation Server 2017 and later. Configure Code Search using the dedicated pages in the TFS Configuration Wizard as you install TFS. You can also configure and unconfigure Code Search afterwards by running the TFS Configuration Wizard again and lauching the Search Configuration Wizard.

Hardware recommendations

Code Search can be used on any size physical server or virtual machine that runs TFS 2017 or above. It can be configured on the same server as TFS, or on a separate server dedicated to Search. When configuring Code Search on the same server as TFS, you must take into account in the existing CPU utlization factor due to TFS itself. In most cases you should consider configuring Code Search on a separate server.

For acceptable performance in multi-user scenarios, consider the following recommendations:

• Less than 250 users with Code Search co-located on the TFS server:

  • Quad core processor, 8 GB (minimum) RAM
  • CPU Utilization factor less than 50%
  • Fast hard drive backed by Solid State Drive (SSD) storage

• Less than 500 users with Code Search located on a separate server:

  • Dual core processor, 8 GB (minimum) RAM
  • Fast hard drive backed by Solid State Drive (SSD) storage

• Less than 1,000 users with Code Search located on a separate server:

  • Quad core processor, 16 GB (minimum) RAM
  • Fast hard drive backed by Solid State Drive (SSD) storage

• More than 1,000 users with Code Search located on a separate server:

  • Quad core processor, 16 GB (minimum) RAM
  • Fast hard drive backed by Solid State Drive (SSD) or Storage Area Network (SAN) storage

• TFS server CPU utilization greater than 50% before installing Code Search:

  • Install Code Search on a separate server

Disk space requirement:

The amount of disk space taken up by Search depends mainly on the type of code files in version control that will be indexed. As a general guideline, allocate upto 30% of the size of all the collections that will be indexed.

Software Dependencies

Code Search has the following dependencies, which are installed automatically as part of the configuration:

• Elasticsearch by Elasticsearch BV (see Note 1)
• Elasticsearch NEST client
• Oracle Server JRE 8 Update 20 or higher Java runtime environment (see Note 2)
• Markdowndeep by Topten Software
• Roslyn compiler platform
• ANTLR language recognition parser

NOTES:

1. A modified version of Elasticsearch ships with TFS. Code Search will work only with this version of Elasticsearch.
2. The system or TFS administrator must ensure that Server JRE is maintained and updated in line with the software provider's recommendations. Also see the installation notes that follow.

Java installation notes

If the Search configuration wizard does not detect a working installation of Java Server JRE, it provides an option to download and install the latest version. Internet connectivity is required to download this from the Java website. If the target server does not have Internet connectivity, you must download and install Server JRE manually before attempting to install Code Search.

During installation, the wizard sets the JAVA_HOME environment variable to point to the Server JRE installation folder. The configuration wizard may fail to detect an existing Server JRE installation if it is not correctly configured, or if the JAVA_HOME setting points to an earlier version than that required by Code Search.

If there is a version of Server JRE earlier than the minimum required by Code Search, and the JAVA_HOME variable is set to that version, we recommend you install Search on a separate server because changing the value of the JAVA_HOME variable may cause other installed software to fail.

If there is a version of Server JRE equal to or later than the minimum required by Code Search, and it is not recognized by the configuration wizard, you must set the value of the JAVA_HOME variable to that version as described in the Java troubleshooting guide, and then rerun the configuration wizard.

If you cannot install the version of Java required by Code Search due to other dependencies, you can:

• Install TFS and Search together on a different server that does not have Java installed (not recommended for more than 250 users or CPU utilization greater than 50%).

• Install Search and Java on a separate server from TFS.

Code Search does not use or support any of the commercial features of Server JRE 7 or 8 as outlined here. Therefore, during Code Search configuration on Team Foundation Server 2017, the commercial features of the Server JRE are not activated or unlocked. See the Oracle documentation for examples of unlocking the commercial features of Java 7 or Java 8.

Installation considerations

Consider the following when configuring Code Search:

• The Code Search extension must be installed for each TFS collection where you want to use it. When initially configuring Code Search, you can set a checkbox to Automatically install Code Search extension for existing and new Tem Project Collections to automate this process.

• If you do not set the checkbox to install the Code Search extension for all your Team Project Collection, while configuring Search, your Project Collection administrator can install it from Visual Studio Marketplace. Make sure you navigate to the Marketplace from your TFS portal page.

• The search index folder should be located on a separate fast hard drive backed by fast storage such as Solid State Drive (SSD) or Storage Area Network (SAN) to maximize search performance. As a general guide, the Code Search index for a collection can be a maximum of 50% the size of the collection itself. That is the worst case scenario; the actual space consumed is dictated by the amount and type of code files in that collection.

• The indexing service and Elasticsearch engine use the account you specify during installation to create and access the index files. This account must have Log on as a service permission.

• Restrict the permissions for the index disk and folder to protect the index from accidental or malicious modification or deletion, and configure appropriate security settings for the service.

• If you are performing a pre-production upgrade on a TFS server where Search is already configured, you must fully reconfigure Search again to avoid corrupting your production instance of Search. For this reason there is no option to configure Search as part of a pre-production upgrade. Instead, configure it after the pre-production upgrade is complete. As this is a pre-production upgrade, you can choose to uncheck Automatically install and configure Code Search for all existing and new collections during configuration, and instead install the Code Search extension for just one or two of your collections after configuration is complete.

• If you are performing a production upgrade on a TFS server where Search is already configured, and want to retain the Search feature, you must set the checkbox to Install and Configure Search. At this point the wizard will detect your existing Search instance and automatically select the Use existing Search instance option, and pre-populate the option with your current Code Search URL. Use the Install a new Search instance option only if you want to set up a new instance of Search on the same TFS server. Setting up a new instance causes all your code to be indexed again, which - depending on the size of the collections - can take some time. During indexing, users may see partial search results.

• If you are upgrading your TFS server to new hardware, depending on how Search was previously configured, you have two options:

  • If Search was configured on a separate server, you must select Install and Configure Search in the TFS wizard, and subsequently select Use an existing Search instance and provide the URL of your existing Search instance to complete the Search configuration.

  • If Search was configured alongside your TFS instance on the old server, you must select Install and Configure Search in the TFS wizard, and subsequently select Install a new Search instance again on the new TFS server if you want to continue to co-host Search and TFS. This will cause all Search indexes for all collections to be re-created which, depending on the size of each collection, might take some time.

• If you are detaching a collection from one TFS instance in order to attach it to another TFS instance, ensure that:

  • Search has been configured on the target TFS instance.
  • Before you detach the collection, you have uninstalled the Code Search extension for that collection from the Manage Extensions page of your source TFS instance.
  • After you attach the collection to the target TFS instance, you install the Code Search extension for that collection from the Marketplace by browsing to it from that TFS instance.

Installing Code Search on a separate server

To install Code Search on a separate (remote) server, typically when you have more than 250 users, follow these steps:

1. As you install TFS on the primary server, set the Install and configure Search checkbox in the Search page of the TFS Configuration Wizard.

2. Select the option to Use an existing Search Service.

3. Use the Search Service package link to access a set of Search installer files on the local machine, and then copy these files to the remote server.

   

4. Follow the instructions in the Readme.md file located in the set of installer files to install the Search service on the remote server.

5. After the installation of the Search service on the remote server is complete, copy the resulting Search server URL into the Search URL field of the configuration wizard running on the TFS server.

6. When both installations are complete, configure appropriate security settings for both servers.

Secure Code Search

The Search service uses a modified version of Elasticsearch (the terms "Search" and "Elasticsearch" are used interchangeably for the remainder of this section). Elasticsearch does not perform authentication or authorization, so it is vital you configure appropriate security settings based on your corporate security and compliance requirements.

Typically you should aim to limit access to both searching and indexing to specific users or user groups - potentially with encryption.

When the Search service is installed locally on the TFS server, it is protected by the security settings you configure for that server. However, if you set up Search on a remote server you must ensure that only the TFS App Tier can access the Elasticsearch cluster by creating an appropriate firewall rule to allow incoming requests on the Elasticsearch port on the remote server.

By default, this open port allows anonymous access to the server. To prevent this, consider the following techniques for using IPSec to secure Elasticsearch on a Windows server.

Configure security with authentication only

This ensures only authorized users can access the Elasticsearch port. It requires only service-side rules (firewall rules on only on the server running Elasticsearch).

Prerequisite: TFS must be configured with a domain account.

Follow the steps in Creating Firewall Rules that Allow IPsec-protected Network Traffic.

Configure security with authentication, integrity protection, and encryption

This ensures encryption and integrity protection are applied along with authentication. It requires both client-side and service-side rules (firewall rules on the server running Elasticsearch and all of TFS App Tier servers).

Prerequisite: TFS must be configured with a domain account.

Follow the steps in Isolating a Server by Requiring Encryption and Group Membership.

Manage Code Search in Team Foundation Server

Code Search is managed by running PowerShell and SQL scripts. All of these scripts are available to download from this GitHub repository. You may wish to download all of the scripts into a local folder on your TFS server using the Download ZIP option.

The PowerShell scripts require the SQL script files, so ensure the SqlScripts folder and its contents is present, along with the PowerShell scripts.

Check indexing status

To check the indexing status after Search is configured, or after the extension is installed for a collection:

1. Execute the CheckIndexingStatus.ps1 script with administrative privileges. You will be prompted to enter:

   • The SQL server instance name where the TFS configuration database resides.
   • The name of the TFS collection database.
   • The name of the TFS configuration database.
   • The name of the collection.
   • The number of previous days to check indexing status.

2. Check the following outputs:

   • Collection indexing was triggered successfully: Indicates that indexing is in progress. If it is displayed, check the following outputs. If it is not displayed, go to step 3 below.

   • Repositories Indexing Completed: The repositories whose indexing has been completed and are now searchable.

   • Repositories in File Discovery Phase: The repositories where files are yet to be discovered. The files are indexed after this stage. Repositories in this state are not yet searchable. The number of files already discovered for indexing in each repository is shown, and this number should be increasing as more files are discovered.

   • Repositories Indexing In Progress: These repositories are partially indexed and should be searchable now, even if the results are only partial.

3. It takes some time for indexing to complete. Execute the CheckIndexingStatus.ps1 script at intervals to check indexing progress.

4. If indexing is not occurring, or indexing is in progress but the number of files pending or the number of files discovered has not changed for some time, or no results are returned for a search, trigger indexing again by running the TriggerCollectionIndexing.ps1 script in a PowerShell window with administrative permission.

5. If the problem persists, contact customer support at the address shown at the end of this topic.

Pause indexing

To pause all indexing, execute the script PauseSearchIndexing.ps1 with administrative privileges. Useful if you see spikes in CPU utilization after configuring Search.

You will be prompted to enter:

• The SQL server instance name where the TFS configuration database resides.
• The name of the TFS configuration database.

Resume indexing

If indexing was paused, execute the script StartSearchIndexing.ps1 with administrative privileges, to start indexing again. You will be prompted to enter:

• The SQL server instance name where the TFS configuration database resides.
• The name of the TFS configuration database.

Re-index a repository or collection

To re-index a Git or TFVC repository, execute the script Re-IndexingRepository.ps1 with administrative privileges. You will be prompted to enter:

• The SQL server instance name where the TFS configuration database resides.
• The name of the TFS collection database.
• The name of the TFS configuration database.
• The type of re-indexing to execute. This can be one of the values:
  • Git_Repository
  • Tfs_Repository
• The name of the collection.
• The name of the repository to re-index.

To re-index a collection, execute the script TriggerCollectionIndexing.ps1 with administrative privileges. You will be prompted to enter:

• The SQL server instance name where the TFS configuration database resides.
• The name of the TFS collection database.
• The name of the TFS configuration database.
• The name of the collection.

Re-indexing a collection can take from a few minutes to a few hours, depending on the size of the collection.

Also see Troubleshoot Code Search.

Unconfigure Code Search in Team Foundation Server

In cases such as a pre-production upgrade, production upgrade, new hardware migration, cloning, or other maintenance operation, the TFS wizard will unconfigure Code Search in a way that makes it easy to re-configure it after the TFS maintenance operation is complete.

However, there might be cases where you no longer want to use Code Search, or want to perform a new and clean install. This requires multiple steps, depending on whether Code Search is configured on the same server as TFS or on a separate server.

Unconfigure Code Search on the machine configured as your TFS server

1. Uninstall the Code Search extension for each collection where it is installed. Do this by navigating to the Manage Extensions page of each collection in your Team Foundation Server:

   

2. Remove the Search feature:

   1. Open the Team Foundation Server Administration Console.
   2. In the left pane, select the name of the TFS server.
   3. In the right pane, choose Remove Feature.
   4. In the Remove Feature dialog, select Team Foundation Search Service and choose Remove.

3. Remove the Elasticsearch service:

   1. Open Command Prompt as an administrator
   2. Change directory: cd "C:\Program Files\Microsoft Team Foundation Server 15.0\Search\ES\elasticsearch-1.7.1-SNAPSHOT\bin"
   3. Remove the service: "service.bat remove"

4. Remove Search data:

   • Delete the contents of the location described by the environment variable SEARCH_ES_INDEX_PATH

5. Remove environment variables:

   1. Delete the environment variable "SEARCH_ES_INDEX_PATH"
   2. Delete the environment variable "ES_HEAP_SIZE"

Unconfigure Code Search when its configured on a separate server

1. Uninstall the Code Search extension for each collection where it is installed. Do this by navigating to the Manage Extensions page of each collection in your Team Foundation Server.

   

2. Remove the Search feature:

   1. Open the Team Foundation Server Administration Console.
   2. In the left pane, select the name of the TFS server.
   3. In the right pane, choose Remove Feature.
   4. In the Remove Feature dialog, select Team Foundation Search Service and choose Remove.

3. Remove the Elasticsearch service and data

   1. Open Powershell as an administrator
   2. Go to the folder where ConfigureTFSSearch.ps1 is installed along with the rest of the files required for a remote install of Search.
   3. Run the script again with the remove option: "ConfigureTFSSearch.ps1 -RemoveTFSSearch"

Limitations of Code Search in Team Foundation Server

Code Search for Team Foundation Server 2017 has the following limitations:

• If you perform a disaster recovery operation and move your server back to an earlier snapshot of your SQL database, Code Search will not reflect the changes unless you re-index the collections.

Troubleshoot Code Search in Team Foundation Server

• Search is configured but the Search box is not displayed
• No search results are shown after installing or configuring Search
• How do I know if indexing was triggered for all the collections?
• Search stops working and no results are shown
• Search does not show the expected results
• TFS server overall performance is affected

Search is configured but the Search box is not displayed

1. The search box is shown only in the context of a project page. Navigate to a project and check if the search box is displayed at the top right.

   

2. If the search box is not shown, verify that the extension is installed for the collection. If not, install or configure the extension.

No search results are shown after installing or configuring Search

1. Wait until you are sure a sufficient time has elapsed after installing or configuring Search. It typically takes less than one hour for Search to index a collection, but it may take up to 12 hours depending on the size and number of code files.
2. If no results are shown after this period, check indexing status.

How do I know if indexing was triggered for all the collections?

• Check indexing status separately for each collection.

Search stops working and no results are shown

Follow these steps. Replace "SearchServer" with the name of the server where Search is installed:

1. Access the URL http://SearchServer:9200 from a web browser on a computer in the same domain as the server running Search.

   • If the status returned is 200 - OK, go to step 2.
   • If any other status is returned, contact support at the address shown at the end of this topic.
   • If you don't get a response, verify that the elasticsearch-service-x64 service is running on the server where Search is configured. If the service is stopped, start it and access the Search server again.
     If you still get no response, or a response other than 200 - OK, contact support at the address shown at the end of this topic.

2. If the status is 200, access the URL http://SearchServer:9200/_cat/health?v from a web browser on a computer in the same domain as the server running Search.

   • If the status column shows green/OK, and Search is still not working, contact support at the address shown at the end of this topic.
   • If the status column shows red/fault, look at the value in the init or unassigned columns. If these are greater than zero, wait for 30 minutes and then repeat this step. If the values are unchanged, go to step 3.

3. Access the URL http://SearchServer:9200/_cat/shards?v from a web browser on a computer in the same domain as the server running Search.

   • Make a note of the values in the Shard column for the rows with a state value of unassigned and contact support at the address shown at the end of this topic.

Search does not show the expected results

1. If the files were added in the last few minutes, wait for ten minutes or so while they are indexed.
2. Check indexing status for the collection.
3. If the files are still not shown in the results, re-index the repository or collection where the files are located.

TFS server overall performance is affected

1. Pause all indexing and see if performance recovers.
2. If performance does recover, consider locating Code Search on a separate server if you have not already done so.

See also

• Install and configure TFS
• TFS requirements and compatibility
• Get started with Code Search
• Choose your search scope
• Advanced Code Search options
• Rich Code Search results

Got feedback?

Send your feedback to vsosrch@microsoft.com. Post your ideas and feature requests on UserVoice.

Back to Search overview