| ||||||
| Search Microsoft.com for: |
| ||||||
| Search Microsoft.com for: |
MS HTML Mail Interface Performance AnalysisApril 1999 Microsoft Corporation  On This Page
Definition of Terms
IntroductionThis document describes the performance of Microsoft® Commercial Internet System (MCIS) HTML Mail Interface (HMI), which is one of the components included with MCIS version 2.5. The capacity and performance analysis of the HMI service is based on configuration points of reference and identifies factors that drive system scaling characteristics. Generally, the performance of a system depends on a number of factors, including:
These factors will be discussed in detail in this document for the HMI system. In this document, the capacity of the HMI service was measured against two reference models. Analyzing the performance of these two reference models provides insight into how well the HMI system behaves, how high its load and throughput can be pushed, what kind of response time can be expected, and where the potential bottlenecks are. This information is particularly useful to system administrators who will deploy HMI with MCIS 2.0 mail services. This document is organized as follows. The "Service Description" section provides a brief introduction to the HMI service and its interconnection with other component services in the MCIS 2.5 e-mail system. The "Component Interactions" section captures the dynamic behavior of the HMI service using event sequence charts. The "Reference Model" section presents two reference models that were used in this HMI performance study. The "Performance Analysis" section discusses the service-level requirements of the HMI service, reports the capacity of the reference models, and identifies potential bottlenecks. Appendix A lists important Performance Monitor counters for measuring performance, and Appendix B documents the InetMonitor scripts used to simulate the usage profile and generate the work load to the HMI service. Note: The HMI performance results presented in this document were derived under laboratory conditions. These results and the sample usage profile may not represent your site conditions. It is recommended that you use this document as a guide only as you test the performance and capacity of your own system. Service DescriptionIntroductionMCIS 2.5 HTML Mail Interface (HMI) is essentially MCIS 2.0 Mail with a Web front end. Using the IMAP4 client-server protocol with a server-side mail store as the back end, HMI service providers can offer their customers Web-based e-mail access from any computer anywhere on the Internet. Service Architecture A complete HMI system configuration has five components:1
Each of these services can be installed on separate computers, as shown in Figure 2. However, for small systems, one server can support multiple services to make the system more cost-effective. An HMI system can span multiple servers for increased scalability and reliability. For example, if it is determined that in a particular configuration the HMI service is a bottleneck to performance, adding another HMI Web server to the system can increase throughput. Component InteractionsUser OperationsWhen users log on to HMI, they view message headers in the Inbox, read messages, reply to messages, save them in folders or delete them, then log out. Each user operation is a transaction that requires some response from the system. That response can trigger a series of interactions between component services within the system. Understanding these interactions is the theme of this section. Event Sequence Charts Individual HMI operations involve multiple component services within the system. For example, opening a mail folder on an HMI site requires processing to take place on the Web and IMAP4 servers before retrieving the file from the mail store. In this sequence, a message passes from the front-end Web server to the IMAP4 server, then to the mail store and back to the IMAP4 server before finally returning to the Web server. From the standpoint of performance analysis, it is crucial to understand:
Event sequence charts depict the interactions between system components by showing how messages pass between them as the operation moves toward completion. Once you identify which messages are going to which components, and how much of the component resources are needed to process each message, you can predict component resource usage under load, and you can predict where bottlenecks might occur. Following are event sequence charts for some basic HMI operations. HMI Logon The first step a user takes in using the HMI service is logging on. In Figure 3, this operation is broken down into a number of sub-transactions corresponding to the periods of time each service is used during the authentication process. For example, in Figure 3, a sub-transaction at the Web server begins with the authorization request from the Web client and ends with the authentication results being returned to the Web server. Note: Interactions between the LDAP server and SQL Server computer are not shown in this diagram. Also not shown are sub-transactions to retrieve the personal address book and the folder list. These two sub-transactions are initiated by the HMI Web Server as part of the logon process, and the component interactions are similar to that in the "Get Message List of Inbox" sub-transaction that is included in figure 3. In a more detailed analysis, the resources consumed at each involved component would be measured (for example, LDAP server CPU real-time consumption). If further analysis shows that most of the latency for this authorization operation occurs when the HMI Web server initially contacts the Membership Directory, for example, further performance optimization should focus on that area. Browse Folder When an HMI user opens a message folder, the request is sent from the HMI Web server to the IMAP4 server, which in turn requests information from the mail store. This process is outlined in Figure 4. All e-mail messages and folders are located on the mail store and all retrieval requests pass through the IMAP4 server. Read Message Reading and sending messages are the two basic transactions of e-mail services. Other operations, such as replying to or forwarding messages, are combinations of these basic operations. The process of reading a message is similar to the process of opening a folder. The request is sent from the HMI Web server to the IMAP4 server, which in turn requests information from the mail store. This process is outlined in Figure 5. Note: The component interactions of other user operations, such as adding and deleting contacts in the personal address book, creating and deleting folders, and moving messages into a folder, are all similar to the process of opening a folder. Send Message Sending a message involves the service on the SMTP server. As shown in Figure 6, an outgoing message is initially stored on the Web server in a Drop directory, at which point the message is forwarded to the SMTP server and then to the IMAP4 server. The SMTP server sends the message out to the Internet, and the IMAP4 server saves the message in the user Sent Items folder on the mail store. Reference ModelThis section describes the reference model used in this performance study. There are two configurations and one user profile. Hardware ConfigurationThe first hardware configuration, referred to as the baseline configuration, establishes a baseline for comparing scalability, capacity, and economy with other configurations. The baseline configuration is depicted in Figure 7. The second configuration, as shown in Figure 8, is referred to as HMI/IMAP4 co-located. That is, the HMI and IMAP4 services are both running on the same server. This is an economical configuration in which no additional hardware is required to add the HMI service when the service provider has already deployed the IMAP4 service. Note that in both configurations there is excess capacity in the other services, including LDAP, SQL Server, SMTP and the mail store, so that the HMI/IMAP4 servers can be pushed to their limit before bottlenecks will be encountered in other components. Pushing the HMI Web server to its limit helps identify potential bottlenecks within the HMI service. Usage Profile A usage profile characterizes the behavior of a typical user accessing an HMI site. A usage profile for an HMI service includes the following information:
The following table shows a usage profile of an average user. It contains 12.5 operations of various types being performed during a 25.5-minute session. Table 1 HMI Usage Profile for this Paper
Note: The total session time is calculated as the sum of the weighted thinking times. The thinking times are weighted by the ceiling of the corresponding transaction per session value. For a general discussion of how a usage profile is defined and used, refer to the Capacity Model for Internet Transactions white paper included with the MCIS Resource Kit, version 2.5. InetMonitor Limitations Due to certain InetMonitor limitations, the length of a user session was reduced to a quarter of the full length. All 12.5 operations were performed in a session of 6.4 minutes. Most tests described in this paper were run with the shortened script. When the full-length script was run to verify the results, capacity was unchanged. In practice, a full user session is four times as long as the shortened user session described in the previous paragraph. A typical user will spend more time thinking and their user session will have longer idle intervals. Thus, in a real production environment, more CPU processing resources will be available, and the response time should be faster than predicted by this paper. Performance AnalysisService-Level RequirementBy definition, the capacity of a system is its maximum throughput while meeting the target service-level requirement. Service-level requirements specify the quality of service provided by the system, such as response time, request-failure ratio, and so on. In the case of HMI, the average ASP request response time is selected as the service-level requirement and is designated to be less than or equal to five seconds. Users tend to lose patience when the response time is substantial. Note that a user-level operation consists of one or more ASP requests and that the network round-trip delay should be factored in to the response times that users experience in the real world. Capacity Analysis This section describes the capacity of both the baseline configuration and the HMI/IMAP4 co-located configuration. The numbers presented in all the graphs in this paper were created using the shortened script. This indicates that response times and CPU utilization are higher than might be seen in a production environment with the full-length usage profile. Note: These capacity estimates are based on the usage profile provided in the "Usage Profile" section. If your usage profile is different, these estimates may not apply to your deployment. Baseline Configuration The capacity of the baseline configuration is 1,100 users. Throughput is limited by the HMI ASP thread pool size, as explained in the "Bottleneck Analysis" section. The load-service curve of the HMI Web server, measured on the baseline configuration, is plotted in Figure 9. It shows the average ASP request response time as a function of the offered load, that is, current logon users. As indicated in Figure 9, the current HMI system can support up to 1,100 concurrent users. The average ASP request response time would be longer than five seconds when more concurrent users log on. HMI/IMAP4 Co-located Configuration As shown in Figure 10, the capacity of the HMI/IMAP4 co-located configuration is approximately 1,000 concurrent users. The HMI ASP thread pool size limits the throughput. Bottleneck Analysis During the testing performed for this analysis, the system was deliberately configured in such a way that the HMI/IMAP4 server became the bottleneck. However, in the real world, a number of other potential bottlenecks may occur when deploying the HMI service. Service providers should watch for the following bottlenecks. Open File Limitation In Windows NT 4.0, the server message block (SMB) limits the number of open network files between any pair of client and server computers to 2,048. In HMI, this limits the number of files that an IMAP4 server can open on a mail store server. For every logon user, the IMAP4 server opens two files at the mail store. When you have only one IMAP4 server and one mail store server, SMB effectively limits the total number of concurrent users that can be supported to 1,024. To get around this limitation, install more than one IMAP4 and mail store computer pair. IMAP4 Server Capacity Limitation Baseline Configuration For the baseline configuration, the CPU utilization of all servers is plotted in Figure 11. Note that neither the HMI Web server nor the IMAP4 server has reached its CPU limit in this figure. HMI/IMAP4 Co-located Configuration The CPU utilization of servers in the HMI/IMAP4 co-located configuration is plotted in Figure 12. The CPU utilization of the HMI/IMAP4 server approaches its limit at 1,000 concurrent users.2 Discussion When the HMI and IMAP4 servers get busy as more subscribers sign on to the service, load-sharing mechanisms can be deployed to mitigate this bottleneck. To scale the capacity for an HMI/IMAP4 co-located configuration, use either DNS round robin or Windows NT Load Balancing Service (WLBS). In a baseline configuration, with HMI and IMAP4 services on separate computers, using DNS round robin to load balance IMAP4 is not effective, because an HMI Web server caches the IP address of an IMAP4 server and subsequently uses it whenever it contacts an IMAP4 server. Use the Windows NT Load Balancing Service instead. HMI Web Server Thread Pool Limitation While measuring performance of the baseline configuration during testing, it was observed that increasing the offered load to the HMI Web server does not necessarily increase the system throughput, although none of the system resources was used up. When the offered load reaches 1,000 users, both ASP execution time and waiting time at the HMI Web server start to increase, and the ASP request queue starts to build up, as shown in Figures 13 and 14. The ASP request queue begins to build up because the HMI Web server has run out of its ASP thread pool. These threads are synchronous. That is, when one thread passes a request to another thread, it waits for the other thread to respond before continuing. Therefore, when the back-end service is relatively slow, all threads in the HMI Web server end up waiting for the response from the IMAP4 server. The HMI Web server CPU has no more active threads to work on the incoming ASP requests in the queue, causing the HMI Web server CPU to idle and HMI ASP request waiting time to increase. Because most threads are waiting for the IMAP4 server, their execution time, which includes the response time of the back-end servers, also increases when the IMAP4 server is loaded with work. This thread pool size prevents the CPU of the HMI and IMAP4 servers from being fully utilized. However, the HMI ASP thread pool also prevents the back-end servers from being overloaded. ASP requests above the capacity of the thread pool are queued by the IIS server. This explains why the IMAP4 CPU utilization does not increase further with more simulated users.3 Ideally, the thread pool size of the HMI Web server should be engineered according to the response time of the back-end IMAP4 servers to keep their CPU(s) busy at 75 to 80 percent but not overload them. Of the three following ways to keep the Web server open, only one is recommended for fully utilizing the HMI and IMAP4 server CPU resources:
Other Considerations
Deployment Planning Based on the evidence presented thus far, it is clear that the HMI/IMAP4 co-located configuration is more economical and better utilizes resources than the baseline configuration. Customers who already have the IMAP4 service do not need additional hardware to add the HMI service using an HMI/IMAP4 co-located configuration. Note that if HMI usage is substantial, then some additional hardware might be appropriate because the capacity of a co-located configuration is slightly below that of IMAP4 deployed separately. Larger mail service deployments are usually complex combinations of POP3, IMAP4, SMTP, and HMI services. Configurations to support larger numbers of mail users are particularly sensitive to usage patterns and peak loads. When planning a large HMI configuration, it is important to note that the HMI capacity limitation is ASP thread pool size and not rising CPU or memory demand. This suggests that, with careful planning, configurations might use less powerful servers and still have sufficient processor and memory resources to reach maximum HMI throughput. Assuming that, at most, 4 percent of HMI service subscribers use the HMI service concurrently at any peak period of traffic, an HMI/IMAP4 co-located server can support 25,000 users with the usage profile specified in the "Reference Model" section. For a one million subscriber base, assuming the usage profile described earlier, and with at most 4 percent of the subscribers using the HMI service concurrently during the peak period, an HMI build-out based on multiple instances of the co-located configuration would require 40 HMI/IMAP4 co-located servers. Because the HMI capacity limitation is ASP thread pool, these servers are likely to have substantial unused resources that could be applied to other uses. SummaryThe capacity of the HMI service is approximately 1,000 concurrent users for both the baseline and HMI/IMAP4 co-located configurations. ASP thread pool size is the bottleneck for the HMI Web server itself. There are three potential bottlenecks that can limit the scalability of the HMI service:
Appendix A - Important Monitoring CountersPerformance Monitor (PerfMon) is a Microsoft® Windows NT® tool that charts and logs performance counters over time.4 Basic performance counters are built into Windows NT, for example, percent of processor time. Other service-related counters are service specific and are defined when the service is developed. The values of these counters help system analysts to understand the state of the service being tested and to identify potential bottlenecks. PerfMon counters used in monitoring HMI system performance are shown in the following table. The counters can be used to monitor capacity. Table 2 PerfMon Counters for the HMI Service
Appendix B - Work Load GenerationLoad Simulator ToolInetMonitor 3.0 is a performance-testing tool that is provided in the MCIS 2.0 Resource Kit. InetMonitor simulates user loads on a service, which helps shed light on how a particular system configuration will perform prior to deploying the system in a production environment. InetMonitor 3.0 uses a proprietary scripting language to simulate specific HMI requests made by a Web browser. It is capable of increasing user load to simulate a large number of concurrent users. When used in conjunction with Windows NT Performance Monitor (PerfMon) counters, resource utilization and transaction throughput can be monitored as user load is increased to determine when maximum throughput and maximum resource capacity have been reached. For more information about InetMonitor, refer to the InetMonitor Operations Guide Version 3, which is also part of the MCIS 2.0 Resource Kit. Load Simulation ScriptThe following script was used with InetMonitor to simulate the behavior of an average HMI user. The script approximates the mix of transactions found in the Usage Profile in Table 1 in the "Reference Model" section. Using this script in conjunction with the PerfMon counters identified in Table 2 in Appendix A will help determine maximum transaction throughput and resource utilization for an HMI system. Increasing offered load by increasing the number of simulated users until the usage of some resources is maximized shows the maximum number of users that an HMI system can support.
REM WebMail load simulator script based on usage profile
REM Assumption: every user already has at least 3 messages in folders ALT 0-6
REM -------------------------------------------------------
REM 100% login (0.5 minute)
REM -------------------------------------------------------
REM login
SLEEP RANDNUMBER(15000,25000)
USER abcdRANDNUMBER(30001,49999) pass
SLEEP RANDNUMBER(15000,25000)
GET URL:/email
GET URL:/email/navmail.asp
GET URL:/email/blank.htm
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
GET URL:/email/Messages.asp?PageNumber=1
GET URL:/email/webmail.css
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
GET URL:/email/pfi.gif
GET URL:/email/pofi.gif
GET URL:/email/fi.gif
GET URL:/email/ofi.gif
GET URL:/email/tp.gif
GET URL:/email/lp.gif
GET URL:/email/tm.gif
GET URL:/email/lm.gif
GET URL:/email/spacer.gif
GET URL:/email/t.gif
GET URL:/email/l.gif
GET URL:/email/i.gif
GET URL:/email/prev.gif
GET URL:/email/next.gif
SLEEP RANDNUMBER(15000,25000)
REM -------------------------------------------------------
REM 100% opens a folder (0.5 minutes)
REM reads 3 messages (2 minutes each)
REM -------------------------------------------------------
REM opens a folder ALT 0 - 5
GET URL:/email/selectfolder.asp?CurrentFolder=%2FALT+RANDNUMBER(0,5)
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(15000,45000)
REM reads 3 messages
GET URL:/email/ViewMail.asp?MessageSequenceNumber=1
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(60000,180000)
GET URL:/email/Viewmail.asp?MessageSequenceNumber=2
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(60000,180000)
GET URL:/email/Viewmail.asp?MessageSequenceNumber=3
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(60000,180000)
REM -------------------------------------------------------
REM 100% composes a message (5 minutes)
REM sends the message (0.5 minute)
REM moves message #1 to another folder (0.5 minute)
REM opens the sent items folder (0.5 minute)***
REM deletes message #2 (0.5 minute)
REM -------------------------------------------------------
REM composes a message
GET URL:/email/ComposeDetail.asp
GET URL:/email/header.asp?ViewID=0&CurrentTask=1
REM sends the composed message
SLEEP RANDNUMBER(150000,450000)
POST URL:/email/sendmail.asp?PROPERTIES:todo=1&To=abcdRANDNUMBER(30001,49999)@wm
25.org&Cc=&Bcc=&Subject=Verification1&messagebody=Verification1
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(15000,45000)
REM moves the 1st mail to another folder ALT 0 - 5
GET URL:/email/MoveMessages.asp?Folder=%2FALT+RANDNUMBER(0,5)&SelectedItems=1
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(15000,45000)
REM opens the sent items folder
GET URL:/email/selectfolder.asp?CurrentFolder=%2FSent+Items
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(15000,45000)
REM deletes the 1st message
Get URL:/email/DeleteMessages.asp?SelectedItems=1
Get URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(15000,45000)
REM -------------------------------------------------------
REM 50% composes a multi-recipient message (5 minutes)
REM sends the message (0.5 minute)
REM deletes message #1 (0.5 minute) from the sent items box
REM opens another folder (0.5 minute)
REM -------------------------------------------------------
50% SKIP 13
REM composes a multi-recipient message
GET URL:/email/ComposeDetail.asp
GET URL:/email/header.asp?ViewID=0&CurrentTask=1
SLEEP RANDNUMBER(150000,450000)
POST URL:/email/sendmail.asp?PROPERTIES:todo=1&To=abcdRANDNUMBER(30001,49999)@wm
25.org&Cc=abcdRANDNUMBER(30001,49999)@wm25.org&Bcc=abcdRANDNUMBER(30001,499
99)@wm25.org&Subject=Verification2&messagebody=Verification2
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(15000,45000)
REM deletes the sent message
Get URL:/email/DeleteMessages.asp?SelectedItems=2
Get URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(15000,45000)
REM opens a folder
GET URL:/email/selectfolder.asp?CurrentFolder=%2FALT+RANDNUMBER(0,5)
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(15000,45000)
skip 1
REM makes up the SLEEP time
SLEEP RANDNUMBER(200000,580000)
REM -------------------------------------------------------
REM 20% moves a message from the current folder to another (0.5 minute)
REM creates a new folder (1 minute)
REM deletes the folder mike (0.5 minute)
REM -------------------------------------------------------
80% SKIP 17
REM moves a mail from the current folder to another
GET URL:/email/MoveMessages.asp?Folder=%2FALT+RANDNUMBER(0,5)&SelectedItems=1
GET URL:/email/header.asp?ViewID=1&CurrentTask=0
SLEEP RANDNUMBER(15000,45000)
REM creates a new folder
GET URL:/email/GetCreateFolder.asp
GET URL:/email/header.asp?ViewID=0&CurrentTask=2
SLEEP RANDNUMBER(30000,50000)
GET URL:/email/CreateFolder.asp?FolderName=mike&ParentFolder=
GET URL:/email/header.asp?ViewID=0&CurrentTask=2
GET URL:/email/navmail.asp
SLEEP RANDNUMBER(15000,25000)
REM deletes the new folder
GET URL:/email/GetDeleteFolder.asp
GET URL:/email/header.asp?ViewID=0&CurrentTask=4
SLEEP RANDNUMBER(15000,45000)
GET URL:/email/DeleteFolder.asp?FolderName=/mike
GET URL:/email/header.asp?ViewID=0&CurrentTask=4
GET URL:/email/navmail.asp
skip 1
REM makes up the SLEEP time
SLEEP RANDNUMBER(100000,140000)
REM -------------------------------------------------------
REM 20% add an alias to PAB (2 minutes)
REM deletes an alias from PAB (0.5 minute)
REM -------------------------------------------------------
REM adds an alias test = foo@wm25.org to the personal address book
80% SKIP 14
GET URL:/email/Addresses.asp
GET URL:/email/header.asp?ViewID=0&CurrentTask=5
GET URL:/email/GetNewContact.asp
SLEEP RANDNUMBER(40000,80000)
POST URL:/email/newcontact.asp?PROPERTIES:txtNick=test+pab&txtEmailOrig=&txtEmail
=foo@wm25.org&txtName=&txtHomAdd=&txtHomPhone=&txtOrg=&txtTitle=&txtWorkPhone
=&txtMobPhone=&txtPager=&txtWorkFax=&txtBirthday=&txtPersURL=&txtComments=
GET URL:/email/header.asp?ViewID=0&CurrentTask=5
SLEEP RANDNUMBER(40000,80000)
REM deletes the alias test = foo@wm25.org from the personal address book
GET URL:/email/Addresses.asp
GET URL:/email/header.asp?ViewID=0&CurrentTask=5
SLEEP RANDNUMBER(15000,25000)
GET URL:/email/DeleteContact.asp?Name=foo@wm25.org
GET URL:/email/header.asp?ViewID=0&CurrentTask=5
SLEEP RANDNUMBER(8000,12000)
skip 1
REM makes up the SLEEP time
SLEEP RANDNUMBER(120000,180000)
REM -------------------------------------------------------
REM 100% logout (0.5 minutes)
REM -------------------------------------------------------
SLEEP RANDNUMBER(15000,45000)
GET URL:/email/logout.asp
1 POP3, which is part of MCIS 2.0, is not part of HMI. With POP3, messages and folders are stored on the client. HMI supports roaming, using a Web Browser as the e-mail client, and stores messages on the mail server. 2 In both configurations, the CPU utilization of servers drops when there are more than 1,000 concurrent users because the ASP request error rate becomes significant after this point and because it takes less CPU real time to drop failed requests in HMI. Also, dropping these failed requests effectively reduces the workload on other services such as IMAP4, LDAP, and the mail stores. 3 To keep the system from overloading, it is recommended that you queue jobs at the front-end servers instead of at the back-end servers, because that is where overload requests can be dropped without wasting back-end server resources. 4 In Windows NT Server and Windows NT Workstation, click Start, click Run, type perfmon, and then click OK. |
