I have decided to move my blog to a different server. 

My new blog is located here:

 http://www.arash.cc/

Or

http://blog.arash.cc/

I will continue to blog about the same topics that you see here and more new stuff as we get green light to talk about them in public; Microsoft Live Services and developer platform, CRM, Program Management, etc, etc.  I also try to answer questions about my posts, in my spare time.

Come and pay a visit……I’ll be fun…..

By now, you should be familiar with the day to day Powershell commands, so here is one that uses the foreach command.  Foreach is used to loop through a given set of items (an array), and perform an action on each one.

As an example, let’s say that you want to perform a Cross-Org mailbox move, but you want to use Get-Mailbox first to get a list of mailboxes.  Just using the standard filtering, you have limited options.  You can filter 1. By a specific Organizational Unit; 2. By a specific Exchange server; 3. Don’t filter at all - return all users; 4. Specify an identity to only return a single mailbox.  Now, that’s nice that these filters exist, and they are pretty easy to use.  You can reference the help for Get-Mailbox for more details (get-help get-mailbox -full).  However, what if you only want to move a handful of mailboxes?  You need to do some more work.

First, you’ll need to define an array by specifying a variable.  You also need to know the alias of the users that you are wanting to move.

$mboxes = “user1″,”user2″,”user3″,”user4″

You have now successfully created an array.  Now, you can perform an action on that array using the Foreach command.  The basic usage of the foreach command is as follows.

foreach ($<item> in $<collection>) {<command_block>}

One thing to remember is that $<item> also references a variable, but it is dynamically created, and you can call it whatever you want.  In our case, the resulting command would look something like this.  I’ve chosen to use $mb for the$<item> variable.  We’ll start out by simply getting the mailbox information for those users to make sure it works, then we’ll move on to the more complex move portion.

Note that unless your current account has rights in both forests, you will still need to pass the -credential string, and you will need to define a variable for the credentials using get-credential.  To make things easier, I always use $s for the Source Forest credentials, and $t for the Target Forest credentials.

foreach ($mb in $mboxes) {get-mailbox $mb -DomainController <source DC FQDN> -Credential $s}

You should now have a list on your screen that looks something like this.

Now that we are sure that the results are being returned properly, let’s include the move-mailbox command.  First, I’ll show you what doesn’t work.

foreach ($mb in $mboxes) {get-mailbox $mb -DomainController <source DC FQDN> -Credential $s} | move-mailbox -TargetDatabase “Server\Storage Group\Database” -GlobalCatalog <target GC FQDN> -SourceForestCredential $s -SourceForestGlobalCatalog <source GC FQDN> -TargetForestCredential $t -AllowMerge -SourceMailboxCleanupOptions deletesourcemailbox

You might think this would work, because previous experience with Powershell has taught you about Pipelining, but it will return an error indicating you have an empty pipe element, which is not permitted.  In order to get this to work, you actually have to include the Pipeline within the command block.  I’m going to move the end brace } to the end of the statement.

foreach ($mb in $mboxes) {get-mailbox $mb -DomainController <source DC FQDN> -Credential $s | move-mailbox -TargetDatabase “Server\Storage Group\Database” -GlobalCatalog <target GC FQDN> -SourceForestCredential $s -SourceForestGlobalCatalog <source GC FQDN> -TargetForestCredential $t -AllowMerge -SourceMailboxCleanupOptions deletesourcemailbox}

The above command will loop through each user in your array, and perform the move-mailbox command.  You should note that the command I used assumes that the user account has already been migrated to the target domain by using a migration tool such as the Active Directory Migration Tool, and that you have migrated SID History (The SID is used to match the source mailbox to a target account).  The other option would be to include the -NTAccountOU parameter, and have Exchange 2007 create a disabled user account, but this is much less desirable, as it ends up creating a linked mailbox instead of a normal user account.

[update]
Thanks to comments from some of you, who pointed out that using foreach, while it *does* work as it is shown above, it isn’t optimal because it passes the results serially.  In other words, it means that it will pass the first result to move-mailbox, then it will wait to pass the next result until the first loop completes.  This will result in move-mailbox being called 4 times (for the example above), more if you have a larger list.  It also doesn’t take advantage of the multi-threading capability of move-mailbox.

In order to make it work properly, you can either simply specify the array, or you can pass the array into the foreach loop (instead of specifying each item in the array).  Examples below.

$mboxes | get-mailbox $mb -DomainController <source DC FQDN> -Credential $s | move-mailbox -TargetDatabase “Server\Storage Group\Database” -GlobalCatalog <target GC FQDN> -SourceForestCredential $s -SourceForestGlobalCatalog <source GC FQDN> -TargetForestCredential $t -AllowMerge -SourceMailboxCleanupOptions deletesourcemailbox

foreach { $mboxes } | get-mailbox $mb -DomainController <source DC FQDN> -Credential $s | move-mailbox -TargetDatabase “Server\Storage Group\Database” -GlobalCatalog <target GC FQDN> -SourceForestCredential $s -SourceForestGlobalCatalog <source GC FQDN> -TargetForestCredential $t -AllowMerge -SourceMailboxCleanupOptions deletesourcemailbox

What I’m learning here is that there is a LOT to learn about Powershell!!!

Thanks to Devin Ganger for pointing out the limitation, and thanks to Evan Dodds for helping with the correct usage.  I’ll be posting a followup to this with some more information on moving mailboxes using a csv file, and using other filters to get the list you want.

As you no doubt know by now, Exchange 2007 introduces different types of clustering.  It still retains the traditional Single Copy cluster where all data is stored in a central location (i.e. SAN).  It also introduces the concept of Cluster Continuous Replication, or CCR, where centralized storage is not required, rather each server contains a separate copy of the data.  This is achieved by an initial “seeding”, and is then maintained by replication of log files.  Service Pack 1 for Exchange 2007 includes yet a third type of cluster a new technology for managing server failures, called Standby Continuous Replication, or SCR, but we’ll leave that for another post.

[edit] as was pointed out to me, SCR isn’t actually a clustered solution, as it does not use Windows clustering.  I’ve thus modified the original post to reflect that.  However, it can be used as the target for an Exchange 2007 cluster.  More details to come when I post about SCR.

For today, I’ve installed a new CCR cluster, but I didn’t allow it to complete.  Let’s have a look at the setup log.  This is located in the c:\ExchangeSetupLogs directory.  The one that contains the most information is ExchangeSetup.log.  Since this log file contains a lot of information, I’ll only post part of it.

[8/3/2007 4:46:12 PM] [0] Starting Microsoft Exchange 2007 Setup
[8/3/2007 4:46:12 PM] [0] **********************************************
[8/3/2007 4:46:12 PM] [0] Setup version: 8.0.685.24.
[8/3/2007 4:46:12 PM] [0] Logged on user: E12CCR\Administrator.
[8/3/2007 4:46:12 PM] [0] Command Line Parameter Name=’mode’, Value=’Install’.
[8/3/2007 4:46:12 PM] [0] Command Line Parameter Name=’sourcedir’, Value=’D:\i386′.
[8/3/2007 4:46:12 PM] [0] Command Line Parameter Name=’fromsetup’, Value=”.
[8/3/2007 4:46:12 PM] [0] ExSetupUI was started with the following command: ‘-mode:install -sourcedir:D:\i386 /FromSetup’.
[8/3/2007 4:46:17 PM] [0] Setup is choosing the domain controller to use
[8/3/2007 4:46:18 PM] [0] Setup is choosing a local domain controller…
[8/3/2007 4:46:20 PM] [0] Setup has chosen the local domain controller e12dc.E12CCR.com for initial queries
[8/3/2007 4:46:21 PM] [0] PrepareAD has been run, and has replicated to this domain controller; so setup will use e12dc.E12CCR.com
[8/3/2007 4:46:21 PM] [0] Setup is choosing a global catalog…
[8/3/2007 4:46:21 PM] [0] Setup has chosen the global catalog server e12dc.E12CCR.com.
[8/3/2007 4:46:21 PM] [0] Setup will use the domain controller ‘e12dc.E12CCR.com’.
[8/3/2007 4:46:21 PM] [0] Setup will use the global catalog ‘e12dc.E12CCR.com’.
[8/3/2007 4:46:21 PM] [0] Exchange configuration container for the organization is ‘CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=E12CCR,DC=com’.
[8/3/2007 4:46:21 PM] [0] Exchange organization container for the organization is ‘CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=E12CCR,DC=com’.
[8/3/2007 4:46:21 PM] [0] This machine is part of a Windows failover cluster.
[8/3/2007 4:46:21 PM] [0] This server is not an active node for any Clustered Mailbox servers.
[8/3/2007 4:46:21 PM] [0] Setup will search for an Exchange Server object for the local machine with name ‘E12CCR2-N1′.
[8/3/2007 4:46:21 PM] [0] No Exchange Server with identity ‘E12CCR2-N1′ was found.
[8/3/2007 4:46:36 PM] [0] The following roles are unpacked:
[8/3/2007 4:46:36 PM] [0] The following roles are installed:
[8/3/2007 4:46:36 PM] [0] The local server does not have any Exchange files installed.
[8/3/2007 4:46:36 PM] [0] Setup will use the path ‘D:\i386′ for installing Exchange.
[8/3/2007 4:46:36 PM] [0] The server is cluster type: ‘None’.
[8/3/2007 4:46:36 PM] [0] The requested cluster type: ‘None’.
[8/3/2007 4:46:36 PM] [0] The installation mode is set to: ‘Install’.
[8/3/2007 4:46:36 PM] [0] An Exchange organization with name ‘First Organization’ was found in this forest.
[8/3/2007 4:46:36 PM] [0] Active Directory Initialization status : ‘True’.
[8/3/2007 4:46:36 PM] [0] Schema Update Required Status : ‘False’.
[8/3/2007 4:46:36 PM] [0] Organization Configuration Update Required Status : ‘False’.
[8/3/2007 4:46:36 PM] [0] Domain Configuration Update Required Status : ‘False’.
[8/3/2007 4:46:37 PM] [0] Applying default role selection state

This simply shows an overview of the setup choices that have been made.

[8/3/2007 4:52:39 PM] [1] Setup launched task ‘test-setuphealth -DomainController ‘e12dc.E12CCR.com’ -DownloadConfigurationUpdates $false -ExchangeVersion ‘8.0.685.24′ -Roles ‘ClusterMailbox’ -ScanType ‘PrecheckInstall’ -SetupRoles ‘AdminTools’,'Mailbox’,'ClusterMailbox’ -CmsName ‘CCR-EVS2′ -CmsDataPath ‘E:\Exchsrvr\MDBData’ -CmsIPAddress ‘10.10.201.230′ -CmsSharedStorage $false -CreatePublicDB $false’

This is showing the pre-requisite check tests that are being performed, and shows that the setup roles that have been selected are AdminTools, Mailbox, ClusterMailbox.  Most of the rest of the log file just shows the progress of each role, so I won’t bore you with that.  Let’s get on to the interesting part.  Once the mailbox role finishes installing, setup runs the new-ClusteredMailboxServer to create the actual cluster.  One nice thing is that you don’t have to worry about creating the Exchange group in Cluster Admin as you had to do with Exchange 2003.  Setup now creates it for you automatically.  Ok - here is the failure.  Note the section in Bold.

[8/6/2007 5:36:17 PM] [1] Processing component ‘Mailbox System Attendant Dependent Tasks’ (Configuring tasks dependent on System Attendant service).
[8/6/2007 5:36:17 PM] [1] Executing ’start-ClusteredMailboxServer -Identity:$RoleName’, handleError = False
[8/6/2007 5:36:17 PM] [2] Launching sub-task ‘$error.Clear(); start-ClusteredMailboxServer -Identity:$RoleName’.
[8/6/2007 5:36:18 PM] [2] Beginning processing.
[8/6/2007 5:36:18 PM] [2] Administrator Active Directory session settings are: View Entire Forest: ‘True’, Configuration Domain Controller: ‘e12dc.E12CCR.com’, Preferred Global Catalog: ‘e12dc.E12CCR.com’, Preferred Domain Controllers: ‘{ e12dc.E12CCR.com }’
[8/6/2007 5:36:18 PM] [2] Searching objects “ccr-evs2″ of type “Server” under the root “$null”.
[8/6/2007 5:36:18 PM] [2] Previous operation run on domain controller ‘e12dc.E12CCR.com’.
[8/6/2007 5:36:18 PM] [2] Start-ClusteredMailboxServer is trying to start clustered mailbox server ccr-evs2.
[8/6/2007 5:37:05 PM] [2] Start-ClusteredMailboxServer finished starting (bringing online) clustered mailbox server ccr-evs2.
[8/6/2007 5:37:05 PM] [2] [ERROR] Unexpected Error
[8/6/2007 5:37:05 PM] [2] [ERROR] Clustered mailbox server ‘ccr-evs2′ is not in a started (online) state (Failed). The cluster resource ‘First Storage Group/Mailbox Database (ccr-evs2)’ is in state (Failed).
[8/6/2007 5:37:05 PM] [2] Ending processing.
[8/6/2007 5:37:05 PM] [1] The following 1 error(s) occurred during task execution:
[8/6/2007 5:37:05 PM] [1] 0. ErrorRecord: Clustered mailbox server ‘ccr-evs2′ is not in a started (online) state (Failed). The cluster resource ‘First Storage Group/Mailbox Database (ccr-evs2)’ is in state (Failed).
[8/6/2007 5:37:05 PM] [1] 0. ErrorRecord: Microsoft.Exchange.Management.Tasks.NewCmsNotOnline: Clustered mailbox server ‘ccr-evs2′ is not in a started (online) state (Failed). The cluster resource ‘First Storage Group/Mailbox Database (ccr-evs2)’ is in state (Failed).
[8/6/2007 5:37:05 PM] [1] [ERROR] Clustered mailbox server ‘ccr-evs2′ is not in a started (online) state (Failed). The cluster resource ‘First Storage Group/Mailbox Database (ccr-evs2)’ is in state (Failed).
[8/6/2007 5:37:05 PM] [1] Setup is halting task execution because of one or more errors in a critical task.
[8/6/2007 5:37:05 PM] [1] Finished executing component tasks.
[8/6/2007 5:37:05 PM] [1] Ending processing.
[8/6/2007 5:40:48 PM] [0] End of Setup

The Setup UI will now show that setup has completed with an error.  Ok, so let’s re-run setup to allow it to complete.  What’s that?  The Setup UI only allows you to Add or Remove roles.  Oh yeah - since this was set up as a Cluster, you can’t have any other roles on it, and you can’t de-select the Active Cluster role.  Now at this point, I can actually go in to Cluster Admin, and verify that all the resources are online.  In the above case where my First Storage Group/Mailbox Databases (ccr-evs2) was in a failed state, I simply brought it online, and it came online just fine.  Since all the resources are online, you might think that everything is hunky dory.  Not so, my friend.  We’ve got a BIG problem.  The System Attendant Object was never fully provisioned.  If you leave your cluster running in the above state, you’ll see lots of issues related to many tasks that the System Attendant Mailbox is used for.  In my above setup, if we look at the System Attendant object using ADSIEdit, we will find that there are no proxy addresses (BAD BAD BAD), and msExchPoliciesIncluded is not set.  This will cause massive problems with Free/Busy, and you’ll see NDR’s being generated for any messages sent to your System Attendant mailbox (duh - it doesn’t have an e-mail address)

OK - so how do you fix it?  Well, before we get there, there is one more thing you need to look at that will tell you if setup encountered any problems that need to be fixed.  Open up the registry, and navigate to

HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange\v8.0

and see which subkeys exist.  You should see a separate key for Each role that is installed.  For the above cluster, if setup had completed successfully, you would only see the following keys.

v8.0
   AdminTools
   MailboxRole
   Setup

In my case, however, I saw one more (bolded)

v8.0
   AdminTools
   ClusteredMailboxServer
   MailboxRole
   Setup

Viewing that key showed a Value with a name of “WaterMark” that contained a hex string.  This Watermark is very important!  It is how setup notes where it left off, and what still needs to be completed.

Ok - now on to the resolution.  If you experience a failure during setup (clustered or not), the best way to resolve it in almost all cases is to run setup from the command prompt.  The command-line version of setup is called from setup.com.  For my cluster issue, since the Mailbox role completed, I don’t have to actually run setup and select a role to install, rather I need to re-setup the cluster.  The command I used was

setup.com /newcms /cmsname:ccr-evs2 /cmsipaddress:10.10.21.230 /cmsdatapath:e:\exchsrvr\mdbdata

When this command is run on an existing cluster with this same name, all setup will do is determine if there are any actions that have not been completed.  In other words, it checks for a Watermark.  In my case, it picked up at about 75% or so, and ran through to completion successfully.  Inspecting the setup log after completion shows the following tasks (which I’d consider fairly important!)

[8/6/2007 5:51:55 PM] [1] Executing ‘enable-SystemAttendantMailbox -Identity:$RoleFqdnOrName -DomainController $RoleDomainController’, handleError = True
[8/6/2007 5:51:55 PM] [2] Launching sub-task ‘$error.Clear(); enable-SystemAttendantMailbox -Identity:$RoleFqdnOrName -DomainController $RoleDomainController’.
[8/6/2007 5:51:55 PM] [2] Beginning processing.
[8/6/2007 5:51:55 PM] [2] Searching objects “ccr-evs2″ of type “ADSystemAttendantMailbox” under the root “First Organization”.
[8/6/2007 5:51:55 PM] [2] Previous operation run on domain controller ‘e12dc.E12CCR.com’.
[8/6/2007 5:51:55 PM] [2] Processing object “Microsoft System Attendant”.
[8/6/2007 5:51:55 PM] [2] Applying RUS policy to the given recipient “Microsoft System Attendant” with the home domain controller “e12dc.E12CCR.com”.
[8/6/2007 5:52:17 PM] [2] The RUS server that will apply policies on the specified recipient is “CCR-EVS2.E12CCR.com”.
[8/6/2007 5:52:17 PM] [2] Saving object “Microsoft System Attendant” of type “ADSystemAttendantMailbox” and state “Changed”.
[8/6/2007 5:52:17 PM] [2] Previous operation run on domain controller ‘e12dc.E12CCR.com’.
[8/6/2007 5:52:17 PM] [2] Ending processing.

If you aren’t installing a cluster, you may still see a Watermark value for one of the other roles (ClientAccess, HubTransport, Mailbox, UnifiedMessaging, etc.

If you have a watermark for one of those roles, you need to note which role has a watermark, then run the following

setup.com /roles:mb (or insert the name of the role that shows the watermark)

This will allow setup to process that role, and based on the watermark in the registry, it will continue where it left off.  Once setup completes successfully, the watermark value will be removed.

I’m hoping and expecting that Service Pack 1 for Exchange 2007 will be able to handle issues like this one much better, as it should include a “Reinstall” option.  If not, you can always fall back to the trusty command-line options.

Sorry folks.  I’m starting to see some spam comments with nasty stuff in them.  As of now, if you want to leave a comment here, you can no longer do so anonymously.