Microsoft Exchange Server 2003  – Troubleshooting Eseutil

Introduction to Exchange Server 2003 – Troubleshooting Eseutil

Eseutil reminds me of a knife.  How do you feel about a knife in the hand of a lunatic – uneasy?  How about a knife in you hand at the supper table – happier?   My message is this, please be aware that eseutil is a dangerous tool and that you ought to practice on a test Exchange machine before taking a stab at eseutil /r on your production server.  Now that I have warned you of the dangers, there will be situations where eseutil is a life saver. (Or at least a mail saver.)

Topics for Exchange Server 2003  – Troubleshooting Eseutil

• Eseutil for Exchange 2007 Server
• Eseutil Commands for Exchange 2010 Server
• Eseutil /mh  /ml  /mk
• Eseutil /k to check for damaged headers
• Eseutil /cc for troubleshooting
• Eseutil /d to defrag the .edb database
• Eseutil /r to repair Exchange 2003 log files
• Eseutil /p will attempt to repair a corrupted store database

 ♠

Eseutil for Exchange Server

By spelling it ESEutil, two thoughts spring to my mind; firstly, I am reminded that here is a tool that manipulates Exchange’s Extensible Storage Engine.  Secondly, ESEutil is a relative of NTDSutil which I use to manipulate Windows Active Directory from the command line.  Whether you spell it ESEutil, Eseutil or plain eseutil, this executable is really three tools in one. 

A different switch controls each aspect of eseutil.  The first and harmless aspect, is shown by the eseutil /k, /mh and /cc switches.  These gentle commands give you the ability to re-run procedures that occur naturally in Exchange, for example, when you remount a store, or replay the logs after a backup.

The second side of eseutil is to defrag Exchange 2003’s databases with eseutil /d switch.  This /d switch shrinks the .edb files and recovers disk space.  Eseutil /d performs a specialist database compaction which is not the same as Windows 2003’s built-in disk defragmenter.

The third and most dangerous side of eseutil is the repair function with /r or /p.  Regard eseutil /r or /p as a last resort to repair your damaged mailstore.  If the repair fails then it can leave the store in an unusable state, so always backup your Exchange server before you unleash the /r or /p switches.

My advice is to begin by practicing with the harmless switches, for example eseutil /mh or /k.  To get started go to the command prompt and then navigate to the Exchsrvr\Bin folder.  Because this \bin folder is not in the file ‘Path’, beware of the notorious: ‘not recognised as an internal or external command ‘ error.  This does not necessarily mean there is no eseutil on the Exchange server, just that you are not executing the command from the Exchsrvr\Bin folder.

Navigate to the \exchsrvr\bin folder before typing any eseutil commands.  An old trick is to copy the Address as seen in Explorer and then go to the command prompt, right-click and paste that path.  (See diagram opposite.)

Alternatively, if you are going do a lot of command line troubleshooting, then it’s worth editing the Path in the System Icon, Environmental Variables.

Guy Recommends: The SolarWinds Exchange Monitor

Here is a free tool to monitor your Exchange Server.  Download and install the utility, then inspect your mail queues, monitor the Exchange server’s memory, confirm there is enough disk space and check the CPU utilization.

This is the real deal – there is no catch.  SolarWinds provides this fully-functioning freebie, as part of their commitment to supporting the network management community.

Free Download of SolarWinds Exchange Monitor

Eseutil /mh

Here is a simple switch to verify the state of an Exchange database.  All that eseutil /mh does is to determine whether the last shutdown was clean or dirty.  Eseutil /mh is ideal to practice getting to the right path and executing eseutil without doing any harm to the mailstore databases.

To start with, familiarise your self with the names and location of the Exchange 2003 databases, for example priv1.edb is usually in the \exchsrvr\mdbdata folder.  My suggestion is to type this command from the exchsrvr\bin folder:
eseutil /mh "d:\program files\exchsrvr\mdbdata\priv1.edb"  (Assuming Exchange 2003 is installed on the d:\.)

Examine the output for this line, ‘State: Clean Shutdown’ (or Dirty Shutdown).  In passing, you can also see when the last backup occurred.  Notice how the first line of the output changes when you substitute priv1.STM for priv1.edb.  Note the phrase: ‘Streaming File’.

Another use of eseutil /mh is in disaster recovery where you want to see if eseutil /p has already been run.  If ‘Repair Count’ is greater than zero, then you can see how many times eseutil /p has been tried already.  In general, the greater the Repair Count, the less chance of a successful repair.

Eseutil /ml
Similar to the /mh, except this switch performs an integrity check on log files, for example, E00.log.

Eseutil /mm
Dumps metadata from the database file (not the logs).  Specialist use only, I find the output fascinating but not very useful.

Eseutil /mk
Provides information about the checkpoint file.  Handy for troubleshooting backup / restore problems.  Where /mh used priv1.edb, remember to substitute the name of the checkpoint file E00.chk with /mk.

Eseutil /k to check for damaged headers

This switch, eseutil /k is new in Exchange 2003.  The keyword is: check.  Just as checksum verifies a file’s size, so eseutil /k checks the integrity of Exchange 2003’s information stores.  One application of eseutil /k is to troubleshoot an Exchange 2003 database after an unscheduled shutdown of the Windows 2003 server.  The only downside with eseutil /k is that it does not recover the database.  (For recovery try /r or /p – but be careful.)

If you create additional mailbox stores, then check their corresponding .edb filenames.   Example: to check the default mailbox store = priv1.edb go to the command prompt and type:
eseutil /k "c: \program files\exchsrvr\mdbdata\priv1.edb" 
(This time I assumed that Exchange 2003 has been installed on the c:\).

Do not worry about uninititialized pages, it’s normal to have several hundred in this category.  However, what you don’t want is bad checksums or wrong page numbers.

Another scenario is that you wish to check the transaction logs, in which case here is the command:

eseutil /k c:\exchsrvr\mdbdata\e00.log

As there are no spaces in the above file or folder names, you do not need to enclose the command with speech marks.  However, to save disappointment, pay special attention to the path where the databases are stored.

Guy Recommends: SolarWinds Free Wake-On-LAN Utility

Encouraging computers to sleep when they’re not in use is a great idea – until you are away from your desk and need a file on that remote sleeping machine!

WOL also has business uses for example, rousing machines so that they can have update patches applied.  My real reason for recommending you download this free tool is because it’s so much fun sending those ‘Magic Packets’. Give WOL a try – it’s free.

Download your free copy of SolarWinds Wake-On-LAN

Eseutil /cc for replaying the logs

Key terms: Hard and Soft recovery.  Checkpoint file, Transaction Logs

A common scenario for this /cc switch is that you have just restored an Exchange mailstore from last night’s backup and you want to replay today’s logs.  Eseutil /cc would achieve your goal provided you issue the command from the folder that contains the Restore.env file.  This special file (Restore.env) carries information about the restore in general and the log sequence numbers in particular.

Command: eseutil /cc path to restore.env

There is a sister command just to check the contents of restore.env : eseutil /cm path to restore.env

Likely contents of restore.env would include paths to source files.  Names of databases .edb and .stm files.  See more on restore.env here.

Terminology check
Cases where you force a replay of the transaction logs are referred to as a hard recovery.  Eseutil /cc gets the restored database up-to-date through such a hard recovery process.  Hard recovery replays the transaction logs after a backup, either select the ‘Last Backup Set’ checkbox, or use eseutil /cc.  Remember that eseutil /cc looks for instructions in the Restore.env file.  Perhaps you can see what I mean when I say that some aspects of Eseutil are just command line methods of controlling Exchange 2003.

In cases where you are short of disk space, call for the temp switch.  Eseutil /cc "name of temp folder" /t.  Naturally you would need to substitute "name of temp folder" for a real folder.

To be sure that the recovery is complete, wait until you see an ESE event ID 205 in the Event Viewer, Application Log.

Soft recovery replays the logs – but only after the last checkpoint.  The normal routine at startup is for uncommitted transactions to be written to the database.  Just remounting the store triggers a built-in soft recovery routine.

With a soft recovery, Exchange processes a few recent transactions after the last checkpoint.  Soft recovery reads pointers in E00.chk, from this information it knows which transactions to commit or roll-back in order to get the database into a consistent state.  One such soft recovery scenario could be a sudden ‘dirty’ store shutdown, which resulted in transactions being interrupted.

If you delve more deeply, you find that eseutil /c has a whole family of commands e.g. cc /ch

Eseutil /cm – Read Restore.env

Guy Recommends:  A Free Trial of the Network Performance Monitor (NPM) v11.5

SolarWinds’ Network Performance Monitor will help you discover what’s happening on your network.  This utility will also guide you through troubleshooting; the dashboard will indicate whether the root cause is a broken link, faulty equipment or resource overload.

What I like best is the way NPM suggests solutions to network problems.  Its also has the ability to monitor the health of individual VMware virtual machines.  If you are interested in troubleshooting, and creating network maps, then I recommend that you try NPM now.

Download a free trial of SolarWinds’ Network Performance Monitor

Eseutil /d to defragment the .edb database

Eseutil / d is probably the commonest and the safest of eseutil’s switches.  This switch works in the same way that Disk Keeper defrags a physical disk.  Take the problem where Exchange’s mailstore is huge and does not shrink even after you have deleted several mailboxes.  You would like to recover the space occupied by the deleted mailboxes.  So this is a job eseutil /d.

To prepare for eseutil /d, first dismount the store.  There is no need to stop the Information Store service, just dismount the individual stores in the Exchange System Manager.  Next, make sure that you have plenty of free disk space, at least as much as the priv.edb or store.edb that you wish to defrag.  Navigate in the cmd window to the \exchsrver\bin folder and issue a command such as this:

Example: eseutil /d  e:\exchsrvr\mdbdata\priv1.edb (Or other path to your store)

If you really do not have enough free space try the Eseutil /d /t "f:\temp.edb".  Where the f drive has enough free space.  Always remember to remount the store once the defrag has finished.

  Take a reading of the store size before and and after running eseutil /d.

Eseutil /r to repair Exchange 2003 log files

Typical Scenario: you have restored an Exchange 2003 database but you cannot mount the store.  When you examine the event log, you see errors: ESE ID 494 – Recovery failed with error -1216.  Further down in the Application Log you may see ESE BACKUP ID 904 and ID 905.

Do not run /r just for fun or  merely to see what happens, eseutil /r is strictly an emergency measure when all else fails to get the restored server working.

What can you do?  Really, you should backup the Exchange database as it is NOW.  Then Try eseutil /r e00 /i .  Note the sequence /r e00 /i is correct.  This assumes that your first, or base log is e00 not some other number.  If you have a storage group with multiple stores, I am afraid that you have to dismount all stores before running the /r switch.  Perhaps this reminds you that all members of a storage group share the same transaction log.

Eseutil /p will attempt to repair a corrupted store database

Scenario.  You try to recover a store.edb database.  However it fails, possibly because the corresponding transaction logs are missing.  For example, you may get an error: ‘The database files in this storage are inconsistent’.  To gather more information try eseutil /mh.  You determine that the state is inconsistent, after backing up the current database, you try eseutil /p.  Follow up with isinteg -fix.

Another nasty problem is that you cannot backup the store.  The worst cases are errors caused by hardware malfunction.  As a last ditch, do or die measure, you could try eseutil /p.  I was going to say backup before you try, but of course in this particular case, backup is the problem.  How about a little lateral thinking and try to copy the store before you run eseutil /p.

Guy Recommends:  SolarWinds’ Free Bulk Mailbox Import Tool

Import users from a spreadsheet, complete with their mailbox.  Just provide a list of the users with the fields in the top row, and save as .csv file.  Then launch this FREE utility, match your Exchange fields with AD’s attributes, click and import the users.  Optionally, you can provide the name of the OU where the new mailboxes will be born.

There are also two bonus tools in the free download, and all 3 have been approved by Microsoft:

1. Bulk-import new users and mailboxes into Active Directory.
2. Seek and zap unwanted user accounts.
3. Find inactive computers.

Download your FREE bulk mailbox import tool.

Summary

Eseutil is a powerful utility.  It has at least three separate jobs, defragging stores, checking the .edb database files, repairing corrupted priv1.edb files.  My advice is to practice with the /cc switch before you have to use the /r (repair) switch on a live network.

Full list of Eseutil switches for Windows Exchange

Eseutil /cc Performs a hard recovery after a database restore.

Eseutil /d Performs an offline compaction of a database.

Eseutil /g Verifies the integrity of a database.

Eseutil /k  Verifies the checksums of a database.

Eseutil /m Generates formatted output of various database file types. e.g. /mh

Eseutil /p Repairs a corrupted or damaged database.

Eseutil /r Performs soft recovery to bring a single database into a consistent or clean shutdown state.

Eseutil /y Copies a database, streaming file, or log file.

See Also

• BPA – Best Practice Analyzer
• Eseutil
• Isinteg
• Network Monitoring
• Performance Monitoring Advice
• Review Solarwinds Exchange Monitor
• Exchange Logs
• Diagnostic Logging
• Free Kiwi Syslog Analyzer
• SMTP Logging