Ondrej Žilinec – IT Blog

Problem

Today I wanted to have day off, but customers with Exchange environments (2013/2016 and 2019) started to have a problem with e-mails delivery. They have started to receive following error in event viewer:

This event started to show couple minutes after 1.1.2022.

Messages stucked at queue and with error “Message deferred by categorized agent”:

Solution

So Antimalware has problem with date “22010100009” and error says it’s “too long”. It looks like Microsoft programmers use to record date 32b numbers and now this value is over variable limits. I think there is will hotfix pretty soon. Until hotfix we need to solve a problem. I just disabled Antimalware engine running following powershell script:

& $env:ExchangeInstallPath\Scripts\Disable-AntimalwareScanning.ps1

and restart MSExchangeTransport service afterwards:

Restart-Service MSExchangeTransport

And that’s all folks 🙂

I have deployed WPA2-Enterprise authentication on network. Computer should use computer certificate to authenticate to Wireless network. I deployed NPS server, configured all requirements. Windows 10 didn’t have problem to connect to Wifi network. Windows 7 computers have problem.

Problem

In eventviewer (Applications and Services Logs\Microsoft\Windows\WLAN-AutoConfig) I found following error:

Network Adapter: Intel(R) Centrino(R) Advanced-N 6205
Interface GUID: {c2c428cb-76cc-4474-a043-33ce2bfe9f0d}
Local MAC Address: 60:67:20:a6:42:63
Network SSID: SECURE_WIFI
BSS Type: Infrastructure
Peer MAC Address: 34:1F:a2:AF:C8:1E
Identity: DOMAN\USER1
User: USER1
Domain: DOMAIN
Reason: Explicit Eap failure received
Error: 0x80074005
EAP Reason: 0x4005
EAP Root cause String:
EAP Error: 0x4005

Solution

Problem is very weird. When you install NPS server, NPS server will use certificate to proof of identity. By default it uses certificate created from Windows CA Template called Kerberos Authentication. This certificate has empty attribute called Subject.

Windows 7 has problem that it doesn’t accept certificate with empty Subject attribute. Other OS’s worked fine. Solution to make it work is to populate this attribute into certificate. You can do it following way:

• Go to Certificate Template console (use MMC, add snap in –> Certificate Templates) and change template called Kerberos Authentication.
• Open properties of this template
• Go to Subject Name tab
• Change Subject name format to DNS name

Now go to NPS server. Open the Certificates MMC (Computer), go to Personal / Certificates and right click in the details pane – All Tasks / request New Certificate. Request new certificate from template Kerberos Authentication for computer account:

After new certificate is installed under computer certification store, you can find attribute Subject populated with DNS name of NPS server:

When certificate is OK, you can change certificate which should be used to proof identity.

Let’s make new certificate work on NPS server. Open NPS console. Go to Network Policies and open up your policy which handles 802.1x requests. Go to the Constraint tab, Authentication Methods / PEAP and click on Edit.

The certificate expiry date now shown should tie in to the one you just created:

Now also Windows 7 will connect to wireless network with no problem and not error event 🙂

That’s all fokls,

I didn’t know that it’s possible in Windows TCP/IP stack make port forwarding. I knew it’s possible in Linux using iptables. In windows we have powerful tool called netsh.

Let’s have a example. Some service is listening on port TCP/10000. If I want to make this service listen on other port than TCP/10000 and there is no configuration to change I will use netsh to make it happen. Let’s look at listening ports on TCP/10000 and TCP/20000:

We can see there is no port listening on TCP/20000. Let’s make a magic and run command:

netsh interface portproxy add v4tov4 listenport=20000 listenaddress=0.0.0.0 connectport=10000 connectaddress=192.168.100.118

You cannot use loopback or 0.0.0.0 in connectaddress parameter. You can even use remote server IP address in connectaddress parameter.

Let’s look at netstat commands:

If your command doesn’t work, please, check if service called IP Helper is Running.

To see all configuration of portproxy settings just run following command:

netsh interface portproxy dump

To delete rule just run following command:

netsh interface portproxy delete v4tov4 listenport=20000 listenaddress=0.0.0.0

And that’s all folks,

Once upon the time I had problem with Windows Update. All updates got downloaded and installed. When user click on button Restart Windows got error 0x80070005.

After couple minutes of debugging with Process Monitor I found out that process called explorer.exe had problem with accessing directory C:\Windows\System32\Tasks. That means that if you click on button Restart in Windows Update, Windows doesn’t just restart system. It creates Scheduled Task to reboot. Weird, but it does it this way.

So I have created GPO with security settings for directory C:\Windows\System32\Tasks and allow BUILTIN\Users have Modify rights to this directory.

And that’s the way we make it 🙂

Problem: When I do migration from Exchange 2010 to Exchange 2016 most of the time I have problem with old style Public Folders hosted on Exchange 2010. When you migrate user from Exchange 2010 to Exchange 2016 she cannot access Public Folders hosted on Exchange 2010.

Solution: You need to setup Exchange 2016 environment to be able to proxy Public Folders hosted on Exchange 2010. You need to create proxy mailbox which Exchange 2016 will use to proxy Public folders. Here are couple steps you need to do:

• Create new mailbox:

New-Mailbox -Name PFMBX1 -Database {YOUR  EX2010 DATABASE NAME}

• I preffer to hide this mailbox from other users

Set-Mailbox -Identity PFMBX1 -HiddenFromAddressListsEnabled $true

• Set on Exchange 2016 server following proxying

Set-OrganizationConfig -PublicFoldersEnabled Remote -RemotePublicFolderMailboxes PFMBX1

Now when Exchange 2016 users restart their Outlook clients they will see Public folders hosted on Exchange 2010.

Have a nice day,

Problem: There is a known error in Exchange 2013/2016 when you migrate user’s mailbox. After the migration is done use cannot access her new mailbox. It is possible after some time (max 15 minutes).

Cause: This is caused by outdated data cached by Autodiscover process on Exchange server. It’s called MSExchangeAutodiscoverAppPool.

Solution: You need to set application pool responsible for creation autodiscover XML file to recycle more often. You can do Recycle manually from IIS console after migration is done:

Or you can configure automatic recycling on specific condition. I most of the time configure Recycling on every minute. When all users are migrated I disable this rule:

Have a nice day,

In long time I was asked to decommission one domain controller. It’s kinda straightforward process. But, yes again, I had a problem. When decommission process started I received error stated:”The operation failed because: Active Directory Domain Services could not configure the computer account DC_ACCOUNT$ on the remote Active Directory Domain Controller DC2_ACCOUNT$: “Access Denied.””.

 

 

I have checked my account and I was domain admin. My rights were alright.

Error “Access Denied” was interesting. Problem was checkbox called “Protect object from accidental deletion” on domain controller object which denied system to delete domain controller object:

 

To see this option you need to enable Advanced Features in Active Directory Users and Computers (dsa.msa) console.

Have a nice day,

From some update there is no relevant information about version in Exchange 2010 SP3. Let’s try it in Powershell:

It says it’s 123.4 build which should coresponds to plaint Microsoft Exchange Server 2010 SP3 (Exchange Update Rollup numbers). This is not corrent information, because those servers have UR installed.

Using EMC I can see:

Which are also incorrect information about server versions. I suppose it’s just nicer output of Powershell cmdlet 🙂

Best way to find out it’s to use EMC console on server and use “About”:

You will get information which are correct:

It’s Update Rollup 11 for Exchange Server 2010 SP3.

It’s really bad that you cannot find Exchange version of all your Exchange servers in one place (Powershell or EMC). Let’s hope they will fix it in new Update Rollup 🙂

That’s all fokls for today.

Problem: One of our customer has about 30 locations across our country and they wanted to use DFSR to replicate folder content to all locations. At each location there is one Hyper-V host running Windows Server 2012 R2 OS with DFSR installed. Each server has three disks – C:\, D:\ and E:\. I have configured DFS Replication in DFS console, replicated AD across environment, and … and nothing happened. DFSR didn’t do anything. It did NOT even create its own private folders in “System Volume Information” folder. No error event in Event viewer. DFSR was set to replicate folder on disk D:\ – didn’t work. I tried to replicate directory on E:\ disk – didn’t work. So I tried to replicate folder on C:\ disk – it did work 🙂

Solution: I enabled DFSR debug logging. There was no error mentioned, but what I noticed was that disks D:\ and E:\ had same serial numbers and also same volume IDs. It was weird. But after some discussion I found out that those two disks were “copied” and “cloned” in Hyper-V environment. So that’s why they had same serial numbers and volume IDs. I found out that it can be a problem. So I had to change volume ID using Microsoft utility VolumeID. After I changed one disk volume ID and rebooted system, DFSR started to work as expected.

So never do disk cloning. Or if you do, change at least volume ID for those disks so Windows services don’t get confused. Looking into this problem took me one and half day! Thank you Microsoft 🙂

Today I played with Windows Server 2016 and its new feature called “Server to Server Storage Replication”. First of all I had to create two virtual server with Windows Server 2016 installed. I also created new domain. After all was done I installed two features on both servers using powershell:

Install-WindowsFeature -Name Storage-Replica -IncludeAllSubFeature -IncludeManagementTools -Restart

There is couple conditions that have to be met before you can use Server Storage Replication:

• You must create two volumes on each enclosure: one for data and one for logs.
• Log and data disks must be initialized as GPT, not MBR.
• The two data volumes must be of identical size.
• The two log volumes should be of identical size.
• All replicated data disks must have the same sector sizes.
• All log disks must have the same sector sizes.
• The log volumes should use flash-based storage, such as SSD.
• The data disks can use HDD, SSD, or a tiered combination and can use either mirrored or parity spaces or RAID 1 or 10, or RAID 5 or RAID 50.
• The data volume should be no larger than 10TB (for a first test, we recommend no more than 1TB, in order to lower initial replication sync times).
• The log volume must be at least 8GB and may need to be larger based on log requirements.

So I have created two new disks on both VMs. One was E: (DATA) 15 GB and other F: (LOGS) 10 GB.

Here is a list of all cmdlets which came with Storate Replication:

Let’s use Test-SRTopology to test if our VMs are ready for Storage Replication:

Test-SRTopology -SourceComputerName W2016-01 -SourceVolumeNames E: -SourceLogVolumeName F: -DestinationComputerName W2016-02 -DestinationVolumeNames E: -DestinationLogVolumeName F: -DurationInMinutes 10 -IntervalInSeconds 1 -ResultPath C:\tmp\

This cmdlet checks all the prerequirements and also test performance between servers:

When everything is alright and working you get nice report. Now you are ready to create new Storate Replication Partnership and Group.

Let’s create SR partnership:

New-SRPartnership -SourceComputerName W2016-01 -SourceRGName RG01 -SourceVolumeName E: -SourceLogVolumeName F: -DestinationComputerName W2016-02 -DestinationRGName RG02 -DestinationVolumeName E: -DestinationLogVolumeName F:

When new partnership is created you can see the result in powershell:

and it also created logfiles on log volume. Default size is 8GB. You can change it if you want based on report after Test-SRTopology:

Other interesting thing is that data volume on destination server is disconnected/dismounted. So replicated data are not accessible. So let copy some files on Source volume. When I copied something on source data volume I noticed activity on network between nodes:

We can also see some events saying about replication between nodes:

When we want to check if anything was copied on other side we need to switch replication other way around. It’s because replicated data disk is not accessible on destination server. This can look very odd, but replication is only one-way. If we want to switch replication direction we do this using powershell:

Set-SRPartnership -NewSourceComputerName W2016-02 -SourceRGName RG02 -DestinationComputerName W2016-01 -DestinationRGName RG01

There is also problem with low memory servers (2GB and less). They stop replicate because of low memory problem. I would expect some GUI console to this feature even Powershell is fine 🙂

If you want to see events from Storate Replication provider you can use following cmdlet:

Get-WinEvent -ProviderName Microsoft-Windows-StorageReplica | select timecre*,id,messa*

So let’s wait what will be in the final version of Windows Server 2016.

That’s all folks.