|RSS | MY PROFILE | PRIVACY|
Vol. 18, #5 - February 4, 2013 - Issue #915
Last August in issue #891 Snapshot Snags of this newsletter, we served up a bunch of tips and recommendations concerning managing virtual machine snapshots in a Microsoft Hyper-V environment. Some of our readers asked if we could do something similar for VMware, and since we aim to please we've decided to devote this issue to the topic of VMware snapshots with a guest editorial by well-know expert Erik Zandboer, who is a vSpecialist Technical at EMC Computer Systems (Benelux).
But first, here's a cartoon to waste some of your valuable time at work:
Don't forget to click and drag on the final panel...
Before we dig into our mailbag, I just wanted to let readers know that I'm now on LinkedIn and you can feel free to add me to your list of connections:
The response from readers to my suggestion that I share some of my weightloss tips that have helped transform me from a fat IT pro to an increasingly fit IT pro has been overwhelmingly positive. In fact, I don't think we've received this much mail from any issue we've published! Here's a small sampling of what we've received:
As a result of the widespread interest from our readers in this subject, I'm going to dedicate a couple of editorials in WServerNews over the next several months to sharing some of my weightloss and fitness "secrets". The first of these editorials will be in the March 18th issue and will focus on the eating side of the equation. Then sometime in April I'll talk about the exercising side of things, targeting specifically middle-aged IT pros like myself since we need to be careful before embarking on any new strenuous physical activities. And if there's continued interest I might talk later about things like supplementation and motivation and how these play into the mix of things that can help one successfully transform one's body from stereotypical fat IT pro to OMG is that person REALLY an IT pro??? (lol)
In the Mailbag section of the previous newsletter we included an email from a reader named Roger who works at Verizon about an issue we've discussed several times in this newsletter, namely, the reliability (or lack thereof) of today's hard disk drives. Several readers had more to say on this topic as follows:
I would have to fairly well agree with what Rodger from Verizon stated. I would like to point out that there is a rather nice computer case available which addresses these cooling issues. It is the Corsair Obsidian 800d. Place your liquid cooling solution, such as a Corsair H100i on the roof of the case, with fans sucking air out of the case through the radiator. The case has separate cooling zones to keep everything nice and cool. Definitely the case to have if you want your system to run cool and consequently last. --a reader named Marvin
For more information on the case Marvin is talking about, see here:
I've had this same suspicion for about 5 years or more. The old drives which were slower and cooler never seemed to give up; the newer ones by contrast seem to have much higher failure rates. That's why my home made gamer system has a direct HDD cooler, along with tons of case fans. Seems to me that the major manufacturers of either the drives or systems need to account for the higher temperatures that these drives generate. Thanks for a great e-newsletter! --Robert, a network engineer and Microsoft Certified Professional
Got more feedback on the topic of hard drive reliability, especially in server environments? Email us at firstname.lastname@example.org
And now on to our guest editorial by Erik Zandboer...
VMware snapshots explained
VMware snapshots. Virtually every vSphere administrator uses them, either direct or indirect via backup software (Avamar, vSphere Data Protection, Veeam, PHD virtual and many others). Even products like VMware View and vCloud director use some form of snapshotting (linked clones). But what exactly happens under the covers when we take or remove a VMware snapshot?
Snapshotting in general
The purpose of snapshotting is always the same: You want to create something that appears to be a copy of an original, which you can modify without impacting the original. But there is an extra trick in the snapshot: The data isn't actually duplicated (that would be called a clone), but the original data is still used for both the original and the snapshot. The nice part: You do not need to copy all of the data. The not-so-nice part: You need to do extra work to keep the copy and the original separated, especially as you will continue to receive writes to the data.
This opens up a new discussion: Where and how do you store the data that is written without impacting the original? There seem to be two popular ways of doing this: Copy-on-write and Redirect-on-write.
Copy-on-write will modify the original, but copy the original content to some other place first. So if we write a block of data, the original data in the block we are about to overwrite will be copied out to a safe place, and then the original block is overwritten with the new data.
Redirect-on-write will not touch the original, but will store any writes in a separate place. So any block to be written to the original will simply be written to some safe place, and the original stays intact.
VMware choose the Redirect-on-write method for its snapshots. Looking at how we use these snapshots today, it makes perfect sense: While the snapshot is active, all writes are deflected to a snapshot file and the original remains intact. Perfect for making backups of that original!
How VMware snapshots work under the cover
When you create a snapshot of a virtual disk in vSphere, an empty snapshot file will be created next to the original virtual disk or vDisk. In the VM's configuration there will be a reference to both snapshot file(s) and base disk. Writes go to the snapshot file, while reads will (initially!) come from the original vDisk which we normally call the base disk.
As more writes come in, the snapshot file will grow. In the old days (ESX 2.5 and earlier) this snapshot file would be a true redo log: Each write was stored in the snapshot file no matter what, allowing the snapshot file to grow indefinite. In the new approach (ESX 3.0 and above), the snapshot file will keep just one copy of each block. This means that in the new approach, a snapshot file can only grow up to the same size as its base disk. There has been a persistent rumor, and even today some people will claim snapshots can grow beyond the size of the base disk. See "Quick dive: ESX and maximum snapshot sizes" for a quick debunk of this myth:
A more interesting thought is reading back from a snapshotted vDisk. You cannot just read back from the original disk, as blocks in there may have been rewritten after the snapshot was made. So for every read performed on the vDisk, vSphere will have to check if this block is present in the snapshot file(s). If it is, it is read back from the snapshot file. If it is not, the block will be read back from the base disk.
Because the base disk is untouched in this scenario (nothing is written to it as long as the snapshot is active), it makes perfect sense to use this feature for backing up your data, as this base disk is the point-in-time situation when you took the snapshot. It is a simple and effective way of making backups; basically you just need to copy the file out (and you can as it is only read and not written to at this time).
A vSphere technology called "Changed Block Tracking" or CBT for short makes backups even more effective: CBT is able to deliver a list of all block numbers that were changed between two snapshots. So backup software can make a successful backup on day one from a snapshot, then snapshot again on day 2 and post a request for list of blocks that were touched between the first and the second snapshot. After that it is merely a matter of only reading the changed blocks, as the backup software now knows the rest hasn't changed anyway.
Impact of having snapshots
As you can imagine, having snapshots is never without impact. I already described writes and reads that need to be performed by vSphere when you have a snapshot active on a vDisk:
Even though vSphere does some smart caching under the covers to quickly discover what sits where, there still is quite some impact in having snapshots. But we still haven't touched on the hardest part: removing the snapshot.
Deleting a VMware snapshot
I think at least 99.9% of all snapshots ever taken with VMware are deleted after some time instead of reverted. I still am not completely happy with the naming convention that is used by VMware here:
The older ESX 2.5 naming convention was clearer from a geek's perspective -- the naming convention then was COMMIT or REVERT which is clearer in my opinion.
Reverting is easy -- you delete the snapshot file. Deleting the snapshot ("commit") on the other hand is a pretty hard thing to do. Imagine: You need to commit all of the writes inside the snapshot into the base disk again. To make things even more complex -- you still need to accept writes as well as you are working on the commit…
How VMware deletes/commits a snapshot -- the gory details
So how do you think VMware commits a snapshot to the base disk? Exactly, by making a second snapshot. And a third, a fourth… and so on if needed. So how does this work exactly?
During the removal of a snapshot, VMware needs to somehow accomplish two things:
VMware solved this problem by creating a snapshot of the snapshot. This second snapshot is used to store the new writes (#2). Following the same logic, the original snapshot now is never written to (like the base disk), so vSphere is free to copy the original snapshot back into the base disk and finally delete that initial snapshot.
Now I hear you thinking: What changed? After the initial snapshot was committed to the base disk and removed, I now have an updated base disk with yet again a snapshot. Actually, nothing changed but the size of the snapshot file.
vSphere will repeat this process as long as it takes, up to a point that the snapshot is extremely tiny. At that point vSphere will hold new writes (like a SCSI BUSY), commit the last few bits from the snapshot into the base disk, remove the last snapshot and then resume I/O. Success: The snapshot is now removed!
As you have seen committing a snapshot ("delete" in vSphere) is a complex matter. Can this go wrong? You bet (for an idea on things that can go wrong, see "Ghostly snapshots: Failed to remove snapshot":
As another cool example, what do you think would happen if I feed the vDisk more writes from the VM than the snapshot commit process can move into the base disk? Exactly, the new snapshot would grow every cycle instead of shrink, and the snapshot would never remove. I did some interesting measurements on this some time ago, but they still are current, see "Performance impact when using VMware snapshots":
In this post large amounts of IOPS can be seen while removing snapshots. I actually managed to configure a system that dropped its read performance to a tiny 5% (!!!) while removing a snapshot. That is where the story comes from that a VM can "freeze" while removing a snapshot. The VM is actually not freezing, but the removal process causes a storage overload that in turn appears to freeze the VM (and any other VMs on those disks!).
A second tricky thing can be forgetting to remove a snapshot. All of a sudden you end up with a 200GB snapshot file, and committing of a snapshot of this size may take days, see "Ye Olde Snapshot":
A third and really scary one is this message in vSphere: "The parent virtual disk has been modified since the child was created". This message has scared many people. It means that the base disk was somehow written to while a snapshot was in place. This is something that should never happen and it will keep VMware from being able to remove the snapshot.
In many cases concerning issues with VMware snapshots I have successfully used VMware Converter: You can actually P2V a virtual machine (V2V) if it is still running ok. The V2V'ed VM will not have any snapshots attached, so you can just use that one and throw away the source VM with the messed-up snapshots after the V2V is completed.
Best practices when using VMware snapshots
Looking at the way VMware snapshots operate, there certainly are some best practices to consider when working with snapshots:
About Erik Zandboer
I live in the Netherlands and technology has always had my interest, in any shape or form. I do a lot of photography which seems to be strangely connected to techies, and I also have a "weird" hobby building and automating a small piece of jungle in my living room.
I have actually been involved in technology for the past 20 years now and have done many different things professionally. Before I started my career I did some programming for the Atari ST and Amiga systems. Then I got my first job designing embedded hard- and software for Siemens. That was REAL embedded hardware, not Linux on ITX boards ;) . Later on I switched more to software, and worked for companies like ASML, KPN and Imtech. As VMware became more and more popular, I was drawn to this technology and I started to go virtual.
As I gathered knowledge on VMware and related products, I also got more and more into the storage business. I got a consultancy job in virtualization and storage, and I started to evangelize VMware through blogging on www.vmdamentals.com. This blogging site mostly contains technical deepdives on various subjects related to VMware and storage. I was honored to make it into the VMware vExpert program through my blogging. Finally I made the move to work for EMC as a vSpecialist. Since then I have been gaining knowledge on a speed I normally could never have.
You can find more about Erik here:
Here are a few tips on how to keep the lithium ion battery in your laptop, tablet or other mobile device working well for as long as possible (these have been recommended to me by colleagues who are hardware geeks):
Contact me at email@example.com if you have a tip you'd like to share with our readers.
Hours of fun (and possibly profit!) from O'Reilly:
Getting Started with MakerBot:
This book is a hands-on introduction to affordable 3D printing can help you get onboard the rapidly growing "prefabrication movement" sweeping across both hobbyists and industry.
"All men dream: but not equally. Those that dream by night in the dusty recesses of their minds wake in the day to find that it was vanity: but the dreamers of the day are dangerous men, for they may act their dreams with open eyes, ti make it possible."
--T. E. Lawrence a.k.a. "Lawrence of Arabia"
Note to subscribers: If for some reason you don’t receive your weekly issue of this newsletter, please notify us at firstname.lastname@example.org and we’ll try to troubleshoot things from our end.
Free Trial: NetWrix Change Reporter Suite, a simple IT infrastructure auditing tool that tracks changes made to all critical IT systems and reports on the "4W detail" – Who changed What, Where and When.
Download SolarWinds free Exchange Monitor that continuously monitors Microsoft Exchange to deliver real-time insight into Exchange services, mail queue sizes, and host server health.
Email Archiving made easy – Exclaimer Mail Archiver provides you with all the benefits of email archiving in a package that’s simple to install, easy to maintain and low cost to own.
Organizations now need the combined power of a scalable on-premise AND cloud based enterprise class email archive to improve productivity, lower TCO and address regulatory requirements.
Iomega StorCenter ix4-200d Network Storage, Cloud Edition offers content sharing with advanced security, and is ideal for small and remote offices, workgroups or home networks:
Contact Michael Vella at email@example.com to get your conference or other event listed in our Events Calendar.
Contact Michael Vella at firstname.lastname@example.org to get your webcast listed in our Webcasts Calendar.
We'll start with some links to articles and blog posts that include tips about VMware snapshots:
Next, here's some more VMware articles...
From BizTech Magazine comes five tips to build a virtual environment that will become the platform for all your IT projects:
From VirtualizationAdmin.com comes this article where David Davis looks at the most recent free tool from VMware labs called vBenchmark:
From NetApp Tech on Tap comes this post about how the new and updated technologies from VMware and NetApp enable you to virtualize demanding business-critical applications:
Describes the pros and cons of using large volumes with VMware virtual machines:
And now for some other stuff...
From the TechNet Flash newsletter comes news that System Center 2012 SP1, the management component of the Cloud OS, now available. Learn what's new in System Center 2012 SP1 and download evaluation software:
Ben Hunter of The Deployment Guys explains how to customize the start screen for users when deploying Windows 8 using Microsoft Deployment Toolkit:
Ben Hunter also shows how to customize the default lock screen for users when deploying Windows 8:
Deb Shinder on WindowsNetworking.com explores the new features in Hyper-V network virtualization:
The private cloud is a secure way to take advantage of all the benefits the public cloud has to offer. However, networking and security concerns have kept many IT pros from adopting this cloud model. Fortunately, leveraging key tips can help you overcome these private cloud barriers. Learn more inside.
Before deploying VDI in your organization, be sure your network infrastructure is well-prepared for the move. In this resource, review key insights and tips on checking for VDI network connectivity and security so that you can eliminate potential pain points.
The Hyper-V extensible virtual switch, offered in the recent release of Windows Server 2012, is opening the door to a variety of new virtual machine (VM) security products. Access this tip to discover how this change will impact the overall virtualization market, as well as your individual organization.
Leveraging VMware Workstation encryption can help you protect the data on your VMs from unauthorized users. Inside this exclusive guide, review essential tips and tricks for determining how – and when – to take advantage of this advanced feature.
"Let’s just drop it and hope it floats." Launch of the Alaska Region Research Vessel 'Sikuliaq' at Marinette Marine in Wisconsin on October 13, 2012.
Aerobatic pilot Martin Sonka manages to maneuver his airplane in a tilted position, hovering like a helicopter alongside parachutist Petr Mestak.
21 extraordinary dancers deliver a visual spectacle that is at once intricate and stirring.
Some amazing, awesome and funny moments of 2012..
Got fun videos or other links to suggest for this section? Email us at email@example.com
Mitch Tulloch is Senior Editor of WServerNews and is a widely recognized expert on Windows administration, deployment and virtualization. Mitch was lead author of the bestselling Windows 7 Resource Kit from Microsoft Press and has published hundreds of articles for IT pros. Mitch is also a seven-time recipient of Microsoft's Most Valuable Professional (MVP) award for his outstanding contributions in support of the global IT pro community. Mitch owns and runs an information technology content development business based in Winnipeg, Canada. For more information see www.mtit.com
Ingrid Tulloch is Associate Editor of WServerNews and was co-author of the Microsoft Encyclopedia of Networking from Microsoft Press. Ingrid is also Head of Research for our content development business and has co-developed university-level courses in Information Security Management for a Masters of Business Administration program.