Ondrej Žilinec – IT Blog

Last night I removed Domain Controller Role from Exchange 2007 server and we had problem in the morning with Exchange Outlook Web Access (OWA). We couldn’t log in at all. Symptomps:

• Form based authentification was enabled, but Basic was proposed to clients instead
• When users logged in they received 440 Login Timeout error

After couple minutes of googling I found this article which solved problem.

IMHO when computer was demoted from domain controller role it created local SAM database and didn’t use domain created accounts IUSR_ComputerName and IWAM_ComputerName.

Today I was solving problem with open relay Exchange server. I came to server and it was full of SPAM e-mail messages. I needed to clean all this mess from Submission queue. I used following command:

Get-Message -Filter {FromAddress -eq “<>”} -Server MAILSERVER | Remove-Message

That’s all folks 🙂

I had to solve interesting problem today at one of our customer. Here is a short preview of customer’s environment:

AD Topology

Customer has following 5 sites:

• Site1 – containing 1 DC
• Site2 – containing 1 DC (one has PDC FSMO role)
• Site3 – containing 2 DCs. Let’s call this central site.
• Site4 – containing 2 DCs. This site represents one datacenter (datacenter 1)
• Site5 – containing 2 DCs. This site represents one datacenter (datacenter 2)

All domain controllers are Global Catalogs. Replication was set manually. It’s configured to be in star topology with median in Site3. For each connection was defined newInter-Site Transport in AD Sites.

AD Topology

Replication works fine. Exchange servers are able to resolve all domain controler. I have checked this using DNS and also nltest.

Exchange Topology

There are four Exchange 2012 servers. Two are CAS/HUB servers put into CAS Array. CAS Servers and CAS Array IP addresses belong to Site4 IP Subnet. And two Mailbox server that are put into DAG. Both mailbox server and DAG IP addresses are in Site4. Problem is that one CAS/HUB and one Mailbox server are physically located in Site4 and one CAS/HUB and one Mailbox server are located physically in Site5. Between Site4 and Site5 are L2 networks for CAS/HUB and Mailbox server.

Exchange topology

Everything works fine. All IP subnets are assigned to Site4 which means all Exchange servers use primary Global Catalog functionality from domain controller from Site4. Idea from network/security guys was to allow Exchange servers to use Global Catalog just from domain controllers located in datacenters – Site4 and Site5. So firewalls don’t let Exchange server to use Global Catalog from other domain controller besides those located in Site4 and Site5.

Problem

Problem appeared when domain controllers in Site4 went down. Exchange servers didn’t want to start and mount databases.

When we looked into Events we could see event 2080 which stated that Exchange AD Topology service sees just four domain controllers:

• Two in-site domain controllers from same site IP subnet are in (Site4)
• Two out-of-site domain controllers. Controllers only from central site Site3

Exchange didn’t use those out-of-site domain controllers, because firewalls blocked it – regarding network/security guys recomendations. Question was why exchange servers didn’t see and use other domain controllers? It sees and uses only those four domain controllers (two in same AD site and two from central site).

After couple of minutes discusing with my coleague we find out that Exchange copies AD topology and it uses domain controllers in following way:

• Primary uses domain controllers in same site as Exchange services are located – in-site DC
• Secondary uses only domain controller which are directly replicating with domain controllers from primary site  – out-of-site DC

My colleague tried to convince me to believe it’s good idea and Exchange tries to protect you from some problems. But I don’t see any point of Exchange not contacting all domain controllers and contacing only domain controllers in the site and contacting domain controlers which replicate with domain controllers in site. I don’t see a poing of Exchange not trying to connect to Global Catalogs in Site1, Site2 and Site5. So this is the way Exchange looks for Global Catalog servers by design.

Proof of problem 🙂

I’ve done couple testing scenarios.

Exchange servers in Site4

• In-site DCs: DCs from Site4
• Out-of-site DCs: DCs only from Site3

Exchange servers in Site5

• In-site DCs: DCs from Site5
• Out-of-site DCs: DCs only from Site3

Exchange servers in Site1

• In-site DCs: DC from Site1
• Out-of-site DCs: DCs only from Site3

Exchange servers in Site3

• In-site DCs: DCs from Site3
• Out-of-site DCs: all DCs from all sites

This is really proof of problem with Exchange locating DCs.

Solution

To solve this issue we could make two things:

• Create new AD Site only for all Exchange IP Subnets and add two domain controllers into this new created AD Site. One DC would be located in physical location 1/datacenter 1 (with CAS1 and MBX1 servers) and other DC would be located in physical location 2/datacenter 2 (with CAS2 and MBX2 servers).

• Create new AD Inter-site Transport between Site4 and Site5.

We decided to create new AD Inter-site Transport.

I still don’t understand why Exchange doesn’t use all domain controllers in AD domain as I would think it would 🙁

More and more are people working with Outlook 2007+, using Lync clients, … all of these applications have ability to show pictures of users in their interface. Let’s look how to implement picture into AD for some users.

Picture in AD is stored in one AD attribute called thumbnailPhoto. We need to put picture into this attribute. Exchange servers and other services use Global Catalog Domain Controllers to resolve attributes for objects in domains. So first of all we need to make sure attribute thumbnailPhoto is propagated into Global Catalog database. We need to modify properties of attribute thumbnailPhoto:

• Register MMC snap-in for Active Directory Schema running command:

Regsvr32 schmmgmt.dll

• Open up Active Directory Schema MMC snap-in
• Under Attributes look for thumbnailPhoto and open its properties
• Check option “Replicate this attribute to the Global Catalog”

Edit thumbnailPhoto properties

Read more…

At one of my customer I had following request. They have Exchange server inside their network. They have two Internet connections. All mail communication goes via primary Internet connection. Second Internet connection is used for Internet browsing.

Internal IP network is 10.0.0.0/24. First Internet connection is called primary and there is default gateway at 10.0.0.1/24. Second Internet connection is called backup Internet connection and it’s default dateway is 10.0.0.2/24. Exchange server has IP address 10.0.0.21/24 and it had default gateway 10.0.0.1/24. MX records are registered to public IP address of primary Internet connection. Problem is when primary line goes down. E-mail communication stops completely – e-mails don’t come and don’t leave company.

To solve this problem we need to make two changes:

• Change MX records for Internet domain
• Change routing for internal Exchange server

Read more…

Today one customer sent me an e-mail with a following problem. They wanted to use e-mail address microsoft@DOMAIN.TLD, but when they assigned this e-mail address to someone they received error that e-mail address is already in use.

Read more…

When I was checking events today at one of our customers I mentioned one weird event. It was:

Event 2070

Process MSExchangeMailboxReplication.exe () (PID=1388).  Exchange Active Directory Provider lost contact with domain controller OLD_DC_NAME.  Error was 0x51 (ServerDown) (Active directory response: The LDAP server is unavailable.).  Exchange Active Directory Provider will attempt to reconnect with this domain controller when it is reachable. 

Read more…

We have Exchange 2007 environment which contains 6 Failover Exchange clusters and each with 24 mailbox databases (total 144 mailbox databases). When there is a problem with replication of databases we needed to click every Failover cluster and check Copy Status of databases. To get rid of this hassle I wrote little script which lists all databases which Copy Status is not “Healthy”:

Get-MailboxServer | % { Get-StorageGroupCopyStatus -Server $_.Identity | ? {$_.SummaryCopyStatus -NotLike “Healthy” } }

For Exchange 2010 it would change little bit to:

Get-MailboxServer | % { Get-MailboxDatabaseCopyStatus -Server $_.Identity | ? {$_.Status -NotLike “Healthy” } }

Now I’m ready to check in couple of seconds the health of databases 🙂

Today one of our biggest customer with the biggest Exchange environment called with weird problem. Customer’s exchange environment is scalled for 60k users in future.

E-mail from outside to inside were going through. So people could receive mails from Internet. When they tried to send an e-mail following appeared:

• When user sent e-mail from OWA, all e-mail messages stayed as Draft folder
• When user sent e-mail from Outlook, all e-mail message were at Sent Items, but no mails were delivered

Read more…

One Exchange 2010 server didn’t mount databases after each reboot. When I looked to Event viewer I saw following events:

The Microsoft Exchange Information Store service terminated with service-specific error %%-2137221213

Read more…