Using Eseutil 2007 on Exchange Server

Using Eseutil 2007 for Troubleshooting

Using eseutil reminds me of using a knife.  How would you feel about a carving knife in the hand of a madman – nervous?  How do you feel about the same knife in your hand at the dinner table – happier?

My message is this, please realize that eseutil is a dangerous instrument and that you ought to practice on a test Exchange machine before taking a stab at eseutil /r on your production Exchange 2007 server.  Now that I have alerted you to possible danger, there will be circumstances where eseutil is a life-saver. (Or at least an email saver.)

Topics for Exchange Eseutil 2007

• Eseutil Switches for Exchange 2007 Server
• Eseutil Commands for Exchange 2010 Server
• Eseutil Getting Started
• Eseutil /k – To Check for Damaged Headers
• Eseutil /d – To Defragment the .edb Database
• Eseutil /r – To repair Exchange 2007 log files
• Eseutil /p – To Repair a Corrupted Exchange Store Database
• Eseutil /cc and /a – To Replay Legacy Logs (Superseded by LLR)

 ♠

Eseutil Switches for Exchange 2007 Server

By spelling this tool – ESEutil, three thoughts spring to my mind; firstly, I am reminded that here is a tool that manipulates the Extensible Storage Engine (ESE).  Secondly, ESEutil is a sister of NTDSutil which I use to troubleshoot Active Directory from the command line.  Thirdly spelling it with capitals highlights the ‘util’ part of the word, and this produces a vision of a Suisse army knife / utility.  Whether you spell it ESEutil, Eseutil or plain eseutil, this Exchange executable is really three tools in one.  A different switch controls each aspect of eseutil.  Incidentally, eseutil’s switches are not case sensitive.

Mild Commands to Replay Natural Actions
The first, and harmless aspect, is illustrated by using the eseutil switches: /k, and /mh.  These mild commands give us the ability to replay actions that occur naturally on an Exchange 2007 server, for example, when you replay the logs after a backup, or remount a store.

Classic Database Defrag
The second side of eseutil is to defrag Exchange 2007’s databases with eseutil /d switch.  This purpose of the /d switch is to shrink the .edb files and thus reclaim disk space.  However, eseutil /d carries out a specialist database compaction, which is not the same as a Windows Server 200x built-in disk defragmenter.

Troubleshooting – Death or Glory
The third, and the most risky side of using eseutil, is the repair function accessed with the /r or /p switch.  I must emphasise that if you need to repair a damaged Exchange mailstore, then eseutil /r or /p is your last resort.  If the database repair fails then it can scramble the messages in your store, therefore, always backup your Exchange 2007 server before you try the /r or /p switches.

Getting Started with Eseutil
To get started with Eseutil go to the command prompt and then navigate to the Program Files\Microsoft\Exchange Server\Bin folder.  (See screenshot) Since this \bin folder is not in the file ‘Path’, beware of the infamous: ‘not recognised as an internal or external command’ error message.  This does not necessarily mean there is no eseutil on the Exchange server, merely that you are not executing the eseutil command from the Exchange Server\Bin folder.  Begin by using eseutil /?

  An old trick is to copy the Address as seen in Explorer and then go to the command prompt, right-click and paste that path.  (See diagram opposite.)

Alternatively, if you are going do a lot of command line troubleshooting, then it’s worth appending the Path in the System Icon, Environmental Variables.

Scope of Eseutil in Exchange 2007 Server
A new development in Exchange is that you can use eseutil 2007 not only on Mailbox servers, but also on the Hub Transport and Edge servers.

Guy Recommends:  SolarWinds’ Log & Event Management Tool

LEM will alert you to problems such as when a key application on a particular server is unavailable.  It can also detect when services have stopped, or if there is a network latency problem.  Perhaps this log and event management tool’s most interesting ability is to take corrective action, for example by restarting services, or isolating the source of a maleware attack.

Yet perhaps the killer reason why people use LEM is for its compliance capability, with a little help from you, it will ensure that your organization complies with industry standards such as CISP or FERPA.  LEM is a really smart application that can make correlations between data in different logs, then use its built-in logic to take corrective action, to restart services, or thwart potential security breaches – give LEM a whirl.

Download your FREE trial of SolarWinds Log & Event Management tool.

Eseutil /mh

The eseutil /m mode does not repair any of your Exchange files.  Its main purpose is to provide you with information about the state of the database files.  In particular to analyse a repair of the Exchange database file using the /p switch.  However, I recommend that you start with this command to get a feel of eseutil database management ability.

Here is a simple switch to verify the state of an Exchange database.  All that eseutil /mh does is to determine whether the last shutdown was clean or dirty.  Eseutil /mh is also ideal to practice getting to the right path and executing eseutil without doing any harm to the mailstore databases.

To start with, familiarise your self with the names and location of the Exchange 2007 databases.  My suggestion is to type this command from the Exchange Server\bin folder:

eseutil /mh "D:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\Mailbox Database.edb"
(Assuming Exchange 2007 is installed on the D:\.)

Examine the output for this line, ‘State: Clean Shutdown’ (or Dirty Shutdown).  In passing, you can also see when the last backup occurred.

Another use of eseutil /mh is in disaster recovery where you want to see if eseutil /p has already been run.  If ‘Repair Count’ is greater than zero, then you can see how many times eseutil /p has been tried already.  In general, the greater the Repair Count, the less chance of a successful repair.

Eseutil /ml
Similar to the /mh, except this switch performs an integrity check on log files, for example, E00.log.

Eseutil /mm
Dumps metadata from the database file (not the logs).  Specialist use only, I find the output fascinating but not very useful.  If you do try this command, best to redirect the output to a file thus:

Eseutil / filename.edb > C:\logs\ExchHead.txt

Eseutil /mk
Provides header information about the checkpoint file.  Handy for troubleshooting backup / restore problems.

Guy’s Challenge – Download this free device backup utility

(CatTools)

Kiwi CatTools is a free program for backing up configuration settings on hardware devices.  Here is Guy’s challenge.  If you download CatTools, then it will not only take care of backups, but also it will show you something new about the hardware on you network. I could give you a money back guarantee – but CatTools is already free!  Thus, I just make a techie to techie challenge, you will learn more about your network if you:

Download your free Kiwi CatTools configuration backup tools

Eseutil /k – To Check for Damaged Headers

The key verb is ‘check’, while the key noun is ‘header’.  Just as checksum verifies a file’s size, so by using eseutil /k you can verify the integrity of Exchange 2007’s information stores.  Another job for eseutil /k is to troubleshoot an Exchange 2007 database after an unscheduled shutdown of the Windows 2007 server.  One point to note is that eseutil /k does not recover the database, for that you need the /r or /p switch – but be careful.

If you create additional mailbox stores, then check their corresponding .edb filenames.  Example: to check the default mailbox store = Mailbox Database.edb go to the command prompt and type:
eseutil /k "c: \Program Files\Microsoft\Exchange Server\Mailbox\xyz Storage Group\Mailbox Database.edb" 
(This time I assumed that Exchange 2007 has been installed on the c:\).

Do not worry about uninititialized pages, it’s normal to have several hundred in this category.  However, what you don’t want is bad checksums or wrong page numbers.

Another scenario is that you wish to check the transaction logs, in which case here is the command:

eseutil /k e00.log

As there are no spaces in the above file or folder names, you do not need to enclose the command with speech marks.  However, to save disappointment, pay special attention to the path where the databases are stored.

You can also run checksum against a database on a transport server queue.  By default, the database file name is mail.que.

Guy Recommends:  A Free Trial of the Network Performance Monitor (NPM) v11.5

SolarWinds’ Network Performance Monitor will help you discover what’s happening on your network.  This utility will also guide you through troubleshooting; the dashboard will indicate whether the root cause is a broken link, faulty equipment or resource overload.

What I like best is the way NPM suggests solutions to network problems.  Its also has the ability to monitor the health of individual VMware virtual machines.  If you are interested in troubleshooting, and creating network maps, then I recommend that you try NPM now.

Download a free trial of SolarWinds’ Network Performance Monitor

Eseutil /d – To Defragment the .edb Database

Eseutil /d is probably the commonest, and possibly the safest of eseutil’s switches.  Firstly, think of ‘d’ for eseutil database.  Secondly, realise that this /d switch works in the same way that Diskkeeper defrags a physical disk.  Take the problem where Exchange’s mailstore is huge and does not shrink even after you have deleted several mailboxes.  You would like to recover the space occupied by the deleted mailboxes.  Thus logon as a local administrator and run: eseutil /d.

• To prepare for eseutil /d, make sure that you have plenty of free disk space, at least as much as the database file that you wish eseutil to defrag. 
• There is no need to stop the Information Store service, just dismount the individual stores in the Exchange Management Console, then right-click the store and select ‘Dismount Database’. 

  Alternatively, you could try a PowerShell command such as:
  dismount-Database -identity "Worcester\First Storage Group\Your Database" 

• Navigate in the cmd window to the \Exchange Serverer\bin folder and issue a command such as this:

  Example: eseutil /d  e:\Exchange Server\mdbdata\Mailbox Database.edb (Or other path to your store)

If you really do not have enough free space try the Eseutil /d /t "f:\temp.edb".  Where the f drive has enough free space. 

  Take a reading of the store size before and after running eseutil /d.  Naturally, remember to remount the store once the defrag has finished.

Defrag Queue Database
The defrag procedure for a Hub Transport or Edge Transport server is slightly different.  As a preliminary step dismount the Queue database.  Navigate to the operating system’s Services snap-in, stop the Microsoft Exchange Transport Service.  Now set eseutil /d on to the database.

Eseutil /r – To repair Exchange 2007 log files

Hard Recovery
In troubleshooting when you force the ESE engine to replay the transaction logs, this is know as a hard recovery; hence Eseutil /r.

Soft Recovery
Soft recovery replays the logs – but only those after the last checkpoint.  If a the Mailbox database is dismounted or stopped, then the logs pile up.  One such soft recovery scenario could be a sudden ‘dirty’ store shutdown, which resulted in transactions being interrupted.

Once that Mailbox store is started again, uncommitted transactions in those logs are written to the database.  Also remounting the store triggers a built-in soft recovery routine. 

Controlling the Checkpoint File
With a soft recovery, Exchange processes a few recent transactions after the last checkpoint.  Soft recovery reads pointers in E00.chk, from this information it knows which transactions to commit or roll-back in order to get the database into a consistent state. 

When you run a soft recovery, usually you want to move (or delete) the checkpoint file.  This is because you will want to be safe and replay all the transaction logs.  It is possible to control the checkpoint file during a soft recovery, by adding the /S switch to the recovery command:

Eseutil /r E00 /Sd:\checkpoint\

Typical Scenario for Eseutil /r
Do not run /r just for fun or  merely to see what happens, eseutil /r is strictly an emergency measure when all else has failed to get the server working.  However if you have restored an Exchange 2007 database but you cannot mount the store then consider Eseutil /r.

Firstly, make the best of a bad situation and backup the Exchange database as it is NOW.  Then navigate to the folder containing the transaction logs, now try: eseutil /r e00 /i .  Note the sequence /r e00 /i is correct.  This assumes that your first, or base log is e00 not some other number.  If you have a storage group with multiple stores, I am afraid that you have to dismount all stores before running the /r switch.  Perhaps this reminds you that all members of a storage group share the same transaction log.

Eseutil /p – To Repair a Corrupted Exchange Store Database

Scenario.  You need to recover a store.edb database, you tried the last backup, but that was no good.  Perhaps the root cause is the corresponding transaction logs are missing.  You may see the error message: ‘The database files in this storage are inconsistent’.  Next step, gather more information try eseutil /mh.  You determine that the state is inconsistent, after backing up the current database, try eseutil /p. 

Follow up you repair of the database with Eseutil /d.  This not only defrags but also rebuilds indexes.  Finally switch to the Isinteg.exe utility and try: isinteg -fix.  This Information Store Integrity Checker can repair the database at the application level.

Another spiteful problem is that you cannot backup the store.  Perhaps the root case was a hardware malfunction.  As a last ditch measure, you could try using eseutil /p.  I was going to say backup before you try, but of course, in this instance, backup is the problem!  How about a little lateral thinking and try to copy the store before you run eseutil /p.

In Exchange 2007 Eseutil /p can also repair the transport queue database on the Hub server.

Eseutil /a and /cc –  To Replay Legacy Logs (Superseded by LLR)

The purpose of the /cc in Exchange 2003 was to force the ESE engine to replay the log files.  In Exchange 2007 server, this capability has been superseded by LLR (Lost Log Resilience).  The built-in LLR feature which protects Exchange databases from losing the most recent log files.  LLR is most important when CCR (Cluster Continuous Replication) is in use.

You can run Eseutil in log recovery mode with the /a switch.  Navigate to the folder containing the database where the LLR-protected log file is missing then experiment with this command:

Eseutil /r E00 /a    (E00 maybe different on your server)

Here follows the Exchange 2003 server routine

A common scenario for this /cc switch is that you have just restored an Exchange mailstore from last night’s backup and you want to replay today’s logs.  Eseutil /cc would achieve your goal provided you issue the command from the folder that contains the Restore.env file.  This special file (Restore.env) carries information about the restore in general and the log sequence numbers in particular.

Command: eseutil /cc path to restore.env

Eseutil /cc gets the restored database up-to-date through a hard recovery process.  Hard recovery replays the transaction logs after a backup, either select the ‘Last Backup Set’ checkbox, or use eseutil /cc.  Remember that eseutil /cc looks for instructions in the Restore.env file.  Perhaps you can see what I mean when I say that some aspects of Eseutil are just command line methods of controlling Exchange 2007.

In cases where you are short of disk space, call for the temp switch.  Eseutil /cc "name of temp folder" /t.  Naturally you would need to substitute "name of temp folder" for a real folder.

There is a sister command just to check the contents of restore.env : eseutil /cm path to restore.env

Likely contents of restore.env would include paths to source files.  Names of databases .edb and .stm files.  See more on restore.env here.

If you delve more deeply, you find that eseutil /c has a whole family of commands e.g. cc /ch

Eseutil /cm – Read Restore.env

Eseutil Database and Log Errors

Error -501 (0xfffffe0b) JET_errLogFileCorrupt
This error indicates physical to a transaction log file.  Error -501 is similar to error -1018 in a database file.   Unfortunately, there is not much you can o to repair the log file.

Error -515 (0xfffffdfd) JET_errInvalidLogSequence
The most likely reason is the log file is missing.

Error -530 (0xfffffdee) JET_errBadLogSignature
Another error message I received, but in this case I could not find the root cause.  Suspect missing log files.

More Exchange 2007 Troubleshooting Tools

Database Recovery Management
This utility executes a set of pre-determined troubleshooting steps to identify Exchange 2007 database problems, for example inconsistency on the log files.  Then magically wizards appear to guide you to solutions for the root cause, such as running one of the above Eseutil switches.

Database Troubleshooter
This utility is a son of ExTRA (Exchange Troubleshooting Assistant).   It calls for built-in routines to identify the reasons why the ESE engine cannot mount the Exchange 2007 database.

To access these tools:
Launch the Exchange Management Console
Drill down to the Toolbox folder.

Guy Recommends:  SolarWinds’ Free Bulk Mailbox Import Tool

Import users from a spreadsheet, complete with their mailbox.  Just provide a list of the users with the fields in the top row, and save as .csv file.  Then launch this FREE utility, match your Exchange fields with AD’s attributes, click and import the users.  Optionally, you can provide the name of the OU where the new mailboxes will be born.

There are also two bonus tools in the free download, and all 3 have been approved by Microsoft:

1. Bulk-import new users and mailboxes into Active Directory.
2. Seek and zap unwanted user accounts.
3. Find inactive computers.

Download your FREE bulk mailbox import tool.

Full list of Eseutil Switches for Exchange 2007 Server

Eseutil /? The best way to learn more about these Eseutil switches.

Eseutil /a (New switch for Exchange 2007)  LLR replays logs.

(Eseutil /cc Performs a hard recovery after a database restore.  Exchange 2003)

Eseutil /d Performs an offline compaction of a database.

Eseutil /g Verifies the integrity of a database.

Eseutil /k  Verifies the checksums of a database.

Eseutil /m Generates formatted output of various database file types. e.g. /mh

Eseutil /p Repairs a corrupted or damaged database.

Eseutil /r Performs soft recovery to bring a single database into a consistent or clean shutdown state.

Eseutil /y Copies a database, or log file.

Summary – Microsoft Exchange Server 2007 Eseutil

Eseutil is a powerful command-line utility.  It has at least three jobs, defragging stores, checking the .edb database files and repairing corrupted mailbox files or logs.  My advice is to practice with the /m switch before you have to use the eseutil /r (repair) switch on a live network.

See more Microsoft Exchange Server 2007 topics:

• Exchange 2007 Home   • Compatibility  • Recipients   • Eseutil  • Edge  • Free Syslog Analyser

• Install   • Server Roles   • CAS Role   • Hub Transport  • SMTP Connector   • NDRs  • Exchange CCR

• Solarwinds Exchange Monitor   • Mailbox Role   • Create Mailbox• OWA   • GAL  • Mailbox Stores