Office 365 / Exchange Hybrid Security Issue

I have recently determined a major security issue related to deleted mailboxes in an Office 365 Hybrid environment. Our former decommission process for accounts was to allow Office 365 to delete the mailboxes of “Former” users no longer at the company.  We simply move the AD account to an OU that is not currently being synced with Office 365 with the Azure AD connect program.  This marks the mailbox and account for deletion within Office 365.

In the past I noticed that the On-Premise Exchange servers still showed these users had “Office 365” as their Mailbox Type.


I also have not configured Azure AD connect for any writeback functionality.

With some recent AD account compromises, I have noticed that these accounts were somehow sending Spam email to outbound email addresses. How could this happen? This mailbox had been deleted by Office 365, there is no way that you can send mail without an actual mailbox, right? Wrong! If you look at your On-Premise receive connectors’ security settings, you will see the following: Permission Groups -> Exchange users


This means that the On-Premise Exchange Servers allows these accounts that have been removed from Office 365 to connect and send email!! Even though they technically don’t have a mailbox on-premise or in Office 365. So it is now imperative to disable the remote mailboxes on your On-premise exchange servers. I fortunately have these AD accounts in a particular OU within AD, so I could execute the following command to immediately disable all of these mailboxes.

Get-RemoteMailbox -ResultSize Unlimited -OnPremisesOrganizationalUnit “OU=someOU,OU=anotherOU,DC=contoso,DC=com” | Disable-RemoteMailbox

I hope this helps anyone before it becomes an issue within your organization. Cheers!

Posted in Exchange 2013, Office 365 | Leave a comment

New Active Directory Replication Status Tool

It’s that time of the year for another Exchange 2013 CU upgrade.  Usually the first part of the upgrade process is to check to see if you need to extend the AD schema etc. Before I extend the schema I always check AD replication using the command line tools. This time however, I found a really awesome GUI tool that simplifies this process.

Active Directory Replication Status Tool

Run the tool and refresh replication status before and after each step in the process.  The tool also has really cool error guide that links directly to Microsoft support articles for replication issues.  Enjoy!




Posted in Active Directory | Leave a comment

RHEL7 extend a LVM managed XFS File System

Most virtualization/linux administrators will have to continually expand certain virtual disks from time to time to increase free space.  Here is my simple step-by-step procedure to expand these volumes.  This process assumes that the volume you want to expand is contained within a LVM, and the file system used is XFS (New default file system for RHEL 7).

#SSM currently does not the +100%FREE option. So we use a combination of ssm and lvm commands.  System Storage Manager commands are available when you run the following command:

yum install system-storage-manager

Add new space to virtual machine drive.  If using VMWare, open vCenter and locate the desired hard drive. Expand the drive to size needed.

ssm list //list volumes, devices, pools…

partprobe //informs the operating system kernel of partition table changes, by requesting that the operating system re-read the partition table

pvdisplay  //find the physical device that you want to extend

pvresize <Device Path>

pvdisplay  //verify the pv has been extended, in my case I had increased it by 1 TB of space

LVM <LV Path> example = /dev/rhel_data_pool/rhel_data_volume

lvextend -L +100%FREE <LV Path>


ssm list //verify ssm can now see volume size increase

#Now we extend the file system

xfs_info <LV Path>  //write down blocks
xfs_growfs <LV Path> // Grow XFS file system to the largest possible size
xfs_info <LV Path>  //verify block number increases

ssm list //verify FS size matches Volume size
df -h //verify FS size

Posted in Linux, RHEL7 | Leave a comment

Office 365 Mailbox Migration – RemoteRoutingAddress Issues

I recently started an Office 365 Exchange migration batch job with several thousand mailboxes.  The migration of the mailboxes was working just fine, but we heard  reports of bounced messages to a few migrated mailboxes.

Remote Server returned ‘554 5.4.6 Too many hops’

After troubleshooting this problem for a few hours, I determined that the problem was with an incorrect setting of the RemoteRoutingAddress.  Our Exchange 2013 Email Address Policy is pretty simple:

Email Address Policy

  • Primary:
  • Address 2:
Address 2 was added to our policy when the Office365, Exchange Hybrid wizard was successfully run for the first time. Now my current problem is several migrated mailboxes had the wrong RemoteRoutingAddress.
For some reason, after migrating this mailbox to Exchange Online, the incorrect RemoteRoutingAddress was set on the mailbox. Simply selecting the correct address of: in the drop down list resolved the issue. Now I am curious, how many other mailboxes had the exact same problem, and why?  I created the following powershell script to search all remote mailboxes for this same problem.

Get-RemoteMailbox -ResultSize Unlimited | Where-Object {$_.RemoteRoutingAddress -notlike “*”}

This script will search all remote mailboxes that have this problem. Now the question is why is this happening? Since I don’t have the address in my email address policy, why does the mailbox migration add this smtp address and set it as the remoteroutingaddress?  At this point I am not sure, but I am looking into it. If anyone knows why this is happening, or has a more permanent solution let us know. Thanks!

Posted in Exchange 2013, Office 365 | Tagged , | 3 Comments

Office 365 Hybrid Check List

Implementing Office 365 is not easy for an a large organization. Here is a current summary of the issues that I have had to deal with in order to get from point A to point B. This will most certainly be an evolving list that I add to over time as new “features” are turned on or are no longer applicable.

  • Office 365 Licensing

During our initial implementation of Office365, we chose to sync all user accounts (Dirsync and now replaced by Azure AD Connect) and assign the appropriate E1 license.

The first trick is to figure out what license to assign each user. The Faculty E1 license or the Student E1 license? My approach was to use powershell to query the MSOL users that do not have a license assigned. Then loop through these users and then query our On-Premise active directory to see if this user was a “student” or “staff” or “faculty” in simplistic terms. Based on this response, the appropriate Student E1 or Faculty E1 license would be assigned in Office 365.

We did not want to include the Exchange Online Plan as part of the E1 license. We did this because if your mailbox has not been moved to Exchange Online, then all users would get an error message when clicking on the Mail icon within Office 365.  This is because there is no corresponding Exchange Online mailbox associated with the account.

Once we decided to start moving mailboxes to Exchange Online, a second script was written to look for the LicenseReconciliationNeeded setting for each migrated mailbox

LicenseReconciliationNeeded: Whether or not the user currently has a mailbox without a license. In this case, the user should be licensed with 30 days to avoid losing their mailbox.

This script then is run on a certain schedule and assigns the Exchange Online plan to MSOL users that have the LicenseReconciliationNeeded = $true.  This means that we either migrated the mailbox to Exchange Online or the New-RemoteMailbox cmdlet has been executed On-Premise from another script.

  • Exchange Online Address Lists

Our Exchange On-Premise environment uses Address Lists for each department. We have created a “\Departments” Address List structure that then contains 1 or even 2 levels of departments address lists in the structure. Unfortunately, Exchange Online as of March 2016 does not allow Address List management within the Admin Center. Powershell is the only way to create and manage Address Lists.  This is fine, because our existing Address Lists were created by Powershell.  Before we could create address lists, we needed to create the Mail Enabled Distribution groups that the Address Lists would use as the members.  Our On-Premise environment simply used Active Directory Security Groups as the membership of the AddressLists.  We were actually deprecating the use of Distribution Groups entirely.  Using Azure AD Connect, we were even syncing these AD security groups with Office 365 for use within Sharepoint Online etc. However, Exchange Online could not see these groups!! So we enabled these groups as Exchange distribution groups and then after our next sync, they became available within Exchange Online.

A new script was created to create Address Lists within Exchange Online using our newly created synced Distribution Groups as the membership.

  • Exchange Online “Tickle” Mail Recipients
  • Message Size Limits – Exchange On-Premise and Online
  • Exchange Online Public Folder Contacts
  • Exchange Online Other Contacts
  • Exchange Online Spam Email Removal
  • Exchange Online Disable Clutter Feature
  • Office 365 Disable Yammer License Plan
Posted in Active Directory, Exchange 2013, Office 365 | Leave a comment

Silent OWA Redirection for Exchange 2013 / Office 365 Hybrid

If you have Exchange 2013 on-premise and configured in hybrid mode with office 365, users with office 365 mailboxes who login to the on-premise Exchange owa website receive a static link that they must click on manually.

Steve Goodman’s post goes a long way to solve this problem for Exchange 2010, but not for Exchange 2013. Here is a non Microsoft approved solution to this issue.

By default the Exchange 2013 Hybrid wizard will set the TargetOwaURL to the portal.  This address forces end users to type in their email address in the microsoft portal before being directed back to the federated login page.  Because the user has already logged in to owa, this extra step is not needed. Run the following command to set the appropriate TargetOwaURL from the Exchange on-premise servers.

#run this command to get the correct identity of the Exchange Online relationship


#Choose a targetowaurl simliar to:

Set-OrganizationRelationship -Identity "On-premises to O365 - dce3beca-eaad-43dc-939e-2f41135hj317ee" -TargetOwaURL ""

Now on to the good stuff. Let’s modify the errorFE.aspx file found at:Exchange 2013 install location

Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\errorFE.aspx

Make a copy of this file before modifying it.  Now enter the following:

<% if (ErrorInformation.RedirectionUrl == "") { Response.Redirect(""); }%> 

directly after this line:

<div class="errorDetails"><%RenderErrorDetails();%></div>

That is it!! The ErrorInformation.RedirectUrl variable does not get set until the RenderErrorDetails() function is called.  Hope this helps!

Posted in Exchange 2013, Office 365 | 23 Comments

Exchange 2013 Monitoring Health Mailboxes Growing Out of Control

Exchange 2013 Monitoring mailboxes do not have a set quota.  What you may find is that there are several thousand items in each of these mailboxes. I currently have 16 mailbox databases, with 2 health mailboxes per database.  I was curious to find out how many items are in each mailbox to try to get a handle on Exchange database growth. Here is what I found:
[PS] C:\Windows\system32>Get-Mailbox -Monitoring | Get-MailboxStatistics
DisplayName               ItemCount    StorageLimitStatus                                                 LastLogonTime
———–               ———    ——————                                                 ————-
HealthMailbox9905ed923… 3                                                                       7/16/2014 10:48:34 AM
HealthMailbox9905ed923… 3                                                                        7/10/2014 1:47:10 PM
HealthMailboxfbc60c8a0… 1                                                                       7/16/2014 10:48:09 AM
HealthMailbox2007dbd50… 193369                                                                    3/4/2015 5:24:13 AM
HealthMailbox0af257470… 2                                                                       7/16/2014 10:52:24 AM
HealthMailboxa2757c02f… 33                                                                      7/16/2014 10:48:21 AM
HealthMailbox8d3070263… 126                                                                     7/16/2014 10:04:29 AM
HealthMailbox09c0df914… 194872                                                                    3/4/2015 5:23:36 AM
HealthMailbox49ef0e29a… 194875                                                                    3/4/2015 5:26:26 AM
HealthMailboxa9182c1be… 191485                                                                    3/4/2015 5:24:08 AM
HealthMailbox8d3070263… 32                                                                       7/13/2014 9:25:03 AM
HealthMailbox0af257470… 2
HealthMailbox0fdbdcd48… 42                                                                      7/16/2014 10:35:32 AM
HealthMailboxd25cfc384… 194860                                                                    3/4/2015 5:27:00 AM
HealthMailbox2e01e9b1b… 8                                                                        7/7/2014 12:03:26 PM
HealthMailboxac2e240d9… 1711                                                                    7/16/2014 10:01:08 AM
HealthMailbox2e01e9b1b… 1                                                                       7/16/2014 10:54:20 AM
HealthMailbox647c626f4… 2                                                                       7/16/2014 10:50:19 AM
HealthMailbox6582ca589… 194771                                                                    3/4/2015 5:24:34 AM
HealthMailbox5f7c59aa8… 1                                                                       7/16/2014 10:04:12 AM
HealthMailboxf71de36bd… 192205                                                                    3/4/2015 5:24:15 AM
WARNING: The user hasn’t logged on to mailbox ‘Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailbox9189a2dd33424890912362c395bc79b5′ (‘f3bc348b-0c6b-4e4b-931a-4f20e11659e4’), so there is no data
to return. After the user logs on, this warning will no longer appear.
HealthMailboxe069b07d1… 1627                                                                    7/16/2014 10:03:01 AM
HealthMailbox9ac9811ab… 1                                                                       7/16/2014 10:54:36 AM
HealthMailboxa8396d773… 194896                                                                    3/4/2015 5:25:21 AM
HealthMailboxdc27ef642… 1                                                                       7/16/2014 10:46:05 AM
HealthMailboxad4f6c557… 194908                                                                    3/4/2015 5:27:46 AM
HealthMailbox9ac9811ab… 2                                                                       7/15/2014 10:53:34 PM
WARNING: The user hasn’t logged on to mailbox ‘Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailboxffc569e756924769944073cf6e5bf3f4′ (‘d1097e38-7751-442a-b940-4be07ba753fc’), so there is no data
to return. After the user logs on, this warning will no longer appear.
HealthMailboxf6664dfc4… 191961                                                                    3/4/2015 5:23:29 AM
HealthMailbox4cc71caff… 191684                                                                    3/4/2015 5:26:25 AM
HealthMailboxe35c04a03… 192581                                                                    3/4/2015 5:26:43 AM
HealthMailboxfe76a7905… 2                                                                       7/16/2014 10:05:08 AM
HealthMailbox54f54c0d9… 1                                                                       7/16/2014 10:00:19 AM
HealthMailbox1c0f9ade6… 35                                                                      7/16/2014 10:41:21 AM
HealthMailbox1af838489… 2                                                                       7/16/2014 10:45:54 AM
HealthMailbox9810a8c59… 192714                                                                    3/4/2015 5:28:08 AM
HealthMailboxb4c7412e9… 194873                                                                    3/4/2015 5:27:24 AM
HealthMailbox02f0ea61d… 192622                                                                    3/4/2015 5:27:47 AM
HealthMailbox282d34047… 1                                                                         6/5/2014 4:18:31 PM
HealthMailbox3870d3244… 192705                                                                    3/4/2015 5:27:00 AM
HealthMailbox-excas1-001  10835                                                                    1/13/2016 9:43:38 AM
HealthMailbox-excas1-004  21932                                                                   1/13/2016 10:51:20 AM
HealthMailbox-excas1-006  17005                                                                   1/13/2016 10:07:30 AM
HealthMailbox-excas1-008  10910                                                                   1/13/2016 10:54:11 AM
HealthMailbox-excas1-007  4372                                                                    1/13/2016 10:58:11 AM
HealthMailbox-excas1-002  11740                                                                   1/13/2016 10:40:51 AM
HealthMailbox-excas1-005  2796                                                                    1/13/2016 10:50:11 AM
HealthMailbox-excas2-002  7435                                                                   12/16/2015 12:22:18 AM
HealthMailbox-excas2-004  9397                                                                   12/16/2015 12:29:34 AM
HealthMailbox-excas2-006  2240                                                                   12/16/2015 12:33:57 AM
HealthMailbox-excas2-008  5943                                                                    1/13/2016 11:00:01 AM
HealthMailbox-excas2-001  9110                                                                   12/16/2015 12:39:11 AM
HealthMailbox-excas2-003  5964                                                                   12/16/2015 12:16:56 AM
HealthMailbox-excas2-007  22840                                                                   1/13/2016 10:57:20 AM
HealthMailbox-excas2-005  8968                                                                   12/16/2015 12:41:35 AM
HealthMailbox-excas3-001  8054                                                                     1/13/2016 1:16:04 AM
HealthMailbox-excas3-002  5813                                                                     1/13/2016 1:24:07 AM
HealthMailbox-excas3-005  2045                                                                     1/13/2016 9:40:09 AM
HealthMailbox-excas3-006  29248                                                                    1/13/2016 5:17:20 AM
HealthMailbox-excas3-008  1442                                                                     8/28/2014 4:22:50 AM
HealthMailbox-excas3-010  4362                                                                    1/13/2016 10:57:47 AM
HealthMailbox-excas3-007  6430                                                                    1/13/2016 10:11:17 AM
HealthMailbox-excas3-009  9127                                                                    1/13/2016 10:06:12 AM
WARNING: The user hasn’t logged on to mailbox ‘Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailbox4cc8c29af43640fbba96680a43d808ef’ (’69f8debf-fbee-4cc9-bae9-d506f66beefa’), so there is no data
to return. After the user logs on, this warning will no longer appear.
HealthMailbox-excas3-004  6810                                                                     1/13/2016 9:52:26 AM
HealthMailbox-excas3-003  2                                                                        1/13/2016 6:05:45 AM
WARNING: The user hasn’t logged on to mailbox ‘Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailbox379e3cea56b44fdb86f7736acec13fa2′ (‘c0e35de2-a91d-4910-8105-c9034d562cf6’), so there is no data
to return. After the user logs on, this warning will no longer appear.
WARNING: The user hasn’t logged on to mailbox ‘Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailbox406d390b801046789d4a130017ae9e24′ (‘7e04fcea-d618-4bee-b59f-8653f893bc24’), so there is no data
to return. After the user logs on, this warning will no longer appear.
HealthMailbox-excas2-009  8011                                                                   12/16/2015 10:34:27 AM
WARNING: The user hasn’t logged on to mailbox ‘Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailbox2b9a0b3b8f6c468190a3dfd4eb84b1e4′ (‘7f67afc1-71a6-4328-b715-22bd4e1c84ea’), so there is no data
to return. After the user logs on, this warning will no longer appear.
HealthMailbox-mbx1-exdb8  452671                                                                  1/13/2016 11:15:41 AM
HealthMailbox-mbx1-exdb3  452710                                                                  1/13/2016 11:14:06 AM
HealthMailbox-mbx1-exdb6  2383                                                                    1/13/2016 11:11:03 AM
HealthMailbox-mbx1-exdb2  2366                                                                    1/13/2016 11:12:53 AM
HealthMailbox-mbx1-exdb5  452694                                                                  1/13/2016 11:14:38 AM
HealthMailbox-mbx1-exdb1  2351                                                                    1/13/2016 11:16:03 AM
HealthMailbox-mbx1-exdb4  2775                                                                    1/13/2016 11:15:41 AM
HealthMailbox-mbx1-exdb7  2388                                                                    1/13/2016 11:13:29 AM
HealthMailbox-mbx2-exdb14 452807                                                                  1/13/2016 11:15:47 AM
HealthMailbox-mbx2-exdb13 452857                                                                  1/13/2016 11:14:24 AM
HealthMailbox-mbx2-exdb17 452596                                                                  1/13/2016 11:15:27 AM
HealthMailbox-mbx2-exdb18 452817                                                                  1/13/2016 11:14:50 AM
HealthMailbox-mbx2-exdb16 455523                                                                  1/13/2016 11:15:28 AM
HealthMailbox-mbx2-exdb15 313142                                                                  1/13/2016 11:15:57 AM
HealthMailbox-mbx2-exdb11 2232                                                                    1/13/2016 11:15:46 AM
HealthMailbox-mbx2-exdb12 51408                                                                   1/13/2016 11:13:59 AM
As you can see, there are several Health Mailboxes that have around 452,807 items in them. After reading several articles, there appear to be 4 options on how to deal with high number of items in the Health Mailboxes.
1. Leave them alone, and allow the mailbox to grow unlimited
2. Apply a retention policy on the health mailboxes to delete messages older than 30 days
3. Run an export-mailbox command on the health mailboxes with the -DeleteContent parameter
4. Delete the Health Mailboxes and recreate them
I decided to assign a retention policy to our health mailboxes to delete messages older than 30 days.  WARNING!!!!!This process will tag each message and create significate logging in your transaction logs for each database.  I recommend applying the retention policy to one health mailbox at a time and waiting for successful results before applying to additonal health mailboxes.  This could potentially fill up your log directories and/or cause backup issues if your mailbox servers are virtual.
Get-Mailbox –Monitoring | Set-Mailbox –RetentionPolicy ‘Health Mailboxes Retention Policy’
or for only 1 mailbox
Get-Mailbox HealthMailbox9905ed92377398783df | Set-Mailbox –RetentionPolicy ‘Health Mailboxes Retention Policy’
Good Luck!
Posted in Exchange 2013 | Leave a comment

Exchange 2013 CU9 Upgrade Issues

Earlier this morning I had to upgrade Exchange Server 2013 to CU9, The exchange upgrade Failed with the following error:
Start-SetupProcess -Name “iisreset” -Args “/noforce /timeout:120”
Process execution failed with exit code 1052

Microsoft states that you need to have your servers Powershell excution policy set to “Unrestrictive” before running the upgrade.  I did set the execution policy to “Unrestrictive” using the following command from an Administrative Powershell console:
set-executionpolicy “Unrestrictive”
However, my execution policy reverted back to “RemoteSigned” after running the first part of the upgrade GUI installer.  I therefor attempted to create a GPO that forces the executionpolicy to “Unrestrictive” so I would not have to run this command before every upgrade. However, the exchange pre-req analysis will complain that GPO is setting the executionpolicy.  You therefor have to set the GPO to “Not Configured” which negates the use of creating the GPO in the first place.
I found that simply running the command: set-executionpolicy “Unrestrictive” multiple times during the install will fix the installation issue. Maybe someone at Microsoft or someone smarter than me can tell us what the appropriate solution might be. Thanks!
Posted in Exchange 2013 | Leave a comment

LDAPS certificate process

If you ever need to create a CA signed certificate for active directory, follow this procedure for each domain controller.

#Create a file called request.inf in the c:\temp directory

;—————– request.inf —————–
Signature=”$Windows NT$

Subject= "CN=yourFQDN, OU=YourCompany, O=YourDepartment, L=YourCity, S=YourState, C=US."
KeySpec = 1 
KeyLength = 2048 ; Can be 1024, 2048, 4096, 8192, or 16384. ; Larger key sizes are more secure, but have ; a greater impact on performance.
Exportable = TRUE
MachineKeySet = TRUE
SMIME = False
PrivateKeyArchive = FALSE
UserProtected = FALSE
UseExistingKeySet = FALSE
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"
ProviderType = 12
RequestType = PKCS10
KeyUsage = 0xa0

OID= ; this is for Server Authentication

Hashalgorithm = sha256

Then run the following commands:

cd c:\temp

certreq.exe -new .\request.inf youserver.domain.2015.req

#wait for certificate from CA

certreq.exe -Accept .\yourservername_yourdomain_edu-2015-SHA2.cer

#The certreq utility will put this cert in the personal store of the local computer. I open up both the local computer certificate store, as well as the certificate store for “Active Directory Domain Services”. Once both personal stores are open, I simply drag the certificate into the “Active Directory Domain Services” personal store.  Make sure you open up the cert and verify the certificate chain is ok etc.  Active Directory will immediately see the new certificate and start using it if it is expire date is newer than the previous cert. So you don’t have to remove the old cert first.

Update 7/29/2019
If you are copying this certificate over from another domain controller, make sure that you include the private key with the certificate. If needed, export this certificate, include the private key, certificate chain etc. into a .pfx file. Then inport this file into the new DC’s “Active Directory Domain Services” personal store.


I always use the ldp.exe command installed by default on all DCs to test the AD SSL connection. See the following screenshot:ldp-test

You should immediately connect to your DC using the new certificate. Good luck!

This process has been tested on Active Directory 2008 R2 and Active Directory 2016

Posted in Active Directory, ldaps | Leave a comment

Exchange 2013 – Public Mailbox Size Increase

If you have an instance where you want to increase a public mailbox in Exchange 2013, follow these directions:
1. Determine which mailbox needs to be increased.
2. Get-Mailbox -PublicFolder -Identity Mailbox1 | fl prohibitsendquota
3. Now we will go ahead and increase the prohibitsendquota
4. get-Mailbox -PublicFolder -Identity Mailbox1 | set-Mailbox -PublicFolder -ProhibitSendQuota 21000000000
This increases the prohibitsendquota to approx 19.56GB. You may have to increase the prohibitsendreceivequota first to a value larger than 19.56GB. You can see that I set it to a value of 25GB first, before increasing the prohibitsendquota to 19.56GB

Posted in Exchange 2013 | 1 Comment