Vol. 21, #39 - September 26, 2016 - Issue #1099
|
Recently in Issue #1096 Security Briefs we provided some information about ramsomware and how your business can avoid getting caught by it. In this week's newsletter we're going to hear a bedtime story (a nightmare actually) by one of our readers, Bill Bach who runs Goldstar Software, about how he helped a client recover data from their mission-critical PSQL database which had been rendered useless by a Cryptolocker-style, data-encrypting malware attack. What made it a nightmare for the client was that their brand new hybrid/cloud backup solution failed them at this critical moment, so read on and hope this doesn't happen to you (and call Bill if it does).
Dilbert's four-footed companion Dogbert gets nightmares sometimes--here's one of them:
http://www.wservernews.com/go/dbmrcns3/
Man's best friend, eh? We've also got tips, tools and other stuff both useful and fun in this issue, so be sure to read it all! Also, we've received a ton of helpful feedback concerning last week's Issue #1098 Windows 7 updating pain so we want to give you a heads-up that we'll be devoting the next issue of WServerNews to sharing our readers' insights into the problems many have been experiencing with Windows Update and Windows 7 computers, so stay tuned!
Ask Our Readers: WServerNews has almost 100,000 subscribers worldwide. That's a lot of expertise to tap into. Do you need help with some issue or need advice on something IT-related? Got a question you'd like us to toss out to our readers to try and answer? Email us at wsn@mtit.com
Darby, a reader whose story we featured in Issue #1094 Good IT is anything that works sent us the following comment in response to our recent Issue #1096 Security Briefs:
This week's articles on security are timely to say the least. I have been giving much thought to AV lately, as you are well aware, and while I still consider it a key component of a comprehensive defense strategy, I must also admit that it is not really the area I lose the most sleep over, as it were. Frankly, it seems to me from what I see happening with my clients that the bad guys have figured out how to treat most AV software as a proverbial Maginot Line using increasingly clever social engineering techniques.
I am not anywhere near ready to chuck my AV software, but I can tell you that I did start a subscription with Stu's new company KnowBe4 some months back and I have been recommending it to anyone who will listen ever since. If anyone has not looked at it, I strongly urge them to do so. If you assume, as I now do, that the most vulnerable attack surface for most organizations is the end user, then it is really a question of education and training. This is exactly what KnowB4 does. Our organization, though not large by any means, showed a dramatic improvement in our incident of 'clickers' after a few months by requiring our staff to watch the educational videos provided by KnowBe4 and then using the very robust Phishing Campaigns feature provided by KnowBe4 to test them and drive home the points of the videos. I know there is no such thing as perfect, but I have to say, in my humble opinion, this is one of the most well thought out and executed offerings I've seen in a long time.
EDITORS' NOTE: For readers who aren't aware of it, this newsletter was originally founded by Stu Sjouwerman way back in 1997 and we took over editing it when TechGenix acquired the newsletter from Stu in 2011. We interviewed Stu in October 2014 when WServerNews reached its milestone 1000th issue and you can read the interview here:
http://www.wservernews.com/newsletters/archives/issue-1000-12576.html
As you can see from the interview, Stu is the founder and CEO of KnowBe4, LLC, which provides web-based Security Awareness Training (employee security education and behavior management) to small and medium-sized enterprises. You can find KnowBe4 online here:
http://www.wservernews.com/go/2hw3o9rr/
Not all of our readers agreed with the observations I expressed in Issue #1094 Good IT is anything that works and we published some of their feedback in Issue #1097 Revisited: Good IT is anything that works. Michael Hallsted, a long-time IT pro who I interviewed on WindowsNetworking.com in a two-part article titled Interview: Maintaining Legacy Software sent us the following email last week:
Good IT is anything that works... well, this put a smile on my face. Just ask anyone who has to support legacy systems. Technology moves so fast that it is easy to be overwhelmed by the sheer volume of change. Most businesses just want to get their work done, and when they find something that they like and fits their needs, they tend to stick with it, come heck and high water.
An "Act of God" is about the only way you can get some people to update/upgrade. If it works and does everything I need, I'm perfectly comfortable with my "old" technology, and helping other people get extended use out of their comfortable, and well-worn "shoes".
And now on to our guest editorial by Bill Bach...
Recently, we were tasked with helping a client recover data from their mission-critical database which had been rendered useless by a Cryptolocker-style, data-encrypting, malware attack. Normally, this would be no big deal, as we would recommend that they just restore from the latest full backup, take the loss of data involved, and go on about the day normally. However, this was not going to be a normal day.
This client had recently moved to a new virtual server, and they had also implemented a brand new hybrid/cloud backup solution. This particular solution was configured to scan their server for data changes every hour, take a VSS snapshot to get the data into a consistent state, and then copy the latest changes to a local data repository. After the hourly snapshot completed, the local device pushed the changed data to the cloud, providing a complete disaster-recovery solution, just in case the entire data center melted down. With local data on the repository, local restores could be done very quickly, and with the data in the cloud, a new virtual machine could be stood up in the cloud data center, recovering from a full-fledged disaster in minutes. What a great idea!
What they had not known is that the backup solution was NOT capturing every disk write. Instead, it was ONLY looking at changes to the size and timestamp of each file (often called an incremental backup), and then using that information to decide if a file needed to be backed up. It then took the changed data files and applied them to the original "full" backup of the server on the local device (and in the cloud), and then reported to have successfully built a full, restorable copy of the server. This solution was silently doing its job every hour of every day, typically backing up 30-40MB of data each hour, and nobody questioned it.
While a backup like this would have been fine for a gaggle of spreadsheets and other such documents, the Actian PSQL database required by their mission-critical application is a very different animal. In order to optimize performance, PSQL writes directly to the database files themselves, bypassing the OS cache. The net result is that the file timestamps are ONLY updated when the file is completely closed by the database engine, meaning that ALL users are done using that database file. While this wouldn't be an issue for a typical application used only during the day (as all files would be closed at night and thus backed up nightly), a mission-critical system can be accessed by users for weeks or months at a time.
So, what happens when these two worlds collide? Each PSQL table timestamp is updated ONLY when all users are out of the application. For some files, this means that the changes are immediately written, the timestamp is updated, and the file is backed up the next hour. Other files were only closed at the end of the day, and therefore they were only backed up during the LAST backup of the day. Indeed, for this environment, some of the files were NEVER closed, and thus were never backed up by the incremental backup process. This meant that the only valid backup of these files was from immediately after the last server reboot -- some 5 weeks earlier!
If you know anything about a relational database, you can already see where this is heading. After the malware attack, they opted to perform a full restore of their server locally. They saw the varying timestamps of the files, but didn't think anything of it at the time. However, after they tried to launch the application, several of the files turned out to be corrupted, because they were stored in multiple extents -- file segments -- and each extent was from a different point in time. They also found that the relationships between some database tables were completely mangled. In fact, we found two related tables where the parent (header) table was dated a week older than the child (detail) table, leaving over 300 sets of orphan records in the child table after the restore!
Needless to say, restoring the data from the cloud wasn't any better, because the core problem was that the database was never backed up properly in the first place. Because of this core problem, even though they could restore a bootable copy of the server in the cloud, the data stored thereon was just as useless from a database perspective.
So, after much consternation and discussion, we finally were able to determine that the ONLY usable backup would be the FIRST one immediately following the last system reboot. Although this was a full 5 weeks prior, the database would be the "closest" to actually being in sync, because all of the files would have been closed during the reboot process, and the file timestamps would be as close as they could ever be to each other.
Now I'm not one to blame Murphy's Law for things like this, but you all know what's coming by now, right? Because of the amount of data being backed up on all of the various servers being protected, there was not enough storage on the local appliance to go back that far. Further, the data retention in the cloud was set to only three weeks, so there was no way to restore the data from 5 weeks prior! Argh!
Now, with no backups, what other options were there? Go back to the old server? Oops -- the cutover was 5 weeks prior to all of this excitement, and the old server was already obliterated, effectively eliminating that option. What about restoring a backup of the old server? They were long gone, too. The ONLY remaining option was to restore the mangled data to an off-line folder for reference purposes, and then create an entirely NEW database for the mission-critical application and start entering ALL of their system data over again from scratch. Ugh...
Read the rest of the story
To read the rest of this story and Bill's recommendations concerning Bill's recommendations for how to back up PSQL databases, you can download Bill's whitepaper "Validating your PSQL Database Backups" from this link on his company website:
http://www.wservernews.com/go/1kezau2o/
You can also download other whitepapers from Bill's site here:
http://www.wservernews.com/go/z192xtaq/
About Bill Bach
Bill is President of Goldstar Software and specializes in support and training for the PSQL database community, used (and often embedded) by applications in just about any vertical market imaginable, including airlines, banks, doctor's offices, steel mills, funeral homes, insurance companies, and more. You can find more information about PSQL on Bill's website:
http://www.wservernews.com/go/f620vk2j/
Got feedback about anything in this issue of WServerNews? Email us at wsn@mtit.com
From the Microsoft Press Blog comes this announcement:
New book--Azure Security Infrastructure
In this book three leading experts show how to plan, deploy, and operate Microsoft Azure with outstanding levels of control, security, and compliance. You'll learn how to prepare infrastructure with Microsoft's integrated tools, prebuilt templates, and managed services—and use these to help safely build and manage any enterprise, mobile, web, or Internet of Things (IoT) system. The authors guide you through enforcing, managing, and verifying robust security at physical, network, host, application, and data layers. You'll learn best practices for security-aware deployment, operational management, threat mitigation, and continuous improvement—so you can help protect all your data, make services resilient to attack, and stay in control no matter how your cloud systems evolve.
http://www.wservernews.com/go/2zrw8b8r/
Create a Dynamic Datacenter with Hybrid Software-Defined Networking
Microsoft pioneered the Software-Defined Networking (SDN) revolution in its own datacenters more than a decade ago. Today, with Microsoft Azure, Windows Server 2012 R2, and System Center 2012 R2, customers can leverage a distributed virtual switch, network virtualization, and edge gateways to enable multi-cloud workloads spanning their own private cloud, Azure, and service provider clouds.
In this session, get an overview of the set of capabilities Microsoft offers today and learn how we are bringing the hyper-scale Azure networking infrastructure, comprehensive security, rich edge gateways, first- and third-party services, and a high-performance data plane for simplifying and transforming your datacenter. Plus, hear an overview of how Microsoft will enable a new breed of cloud-born apps that just work across your datacenter and Azure.
http://www.wservernews.com/go/041wsrk6/
"Son, if you really want something in this life, you have to work for it. Now quiet! They're about to announce the lottery numbers." --Abraham "Grampa" Simpson, the father of Homer Simpson
Until next week,
Mitch Tulloch
GOT ADMIN TOOLS or other software/hardware you'd like to recommend? Email us at wsn@mtit.com
Back up your Hyper-V and VMware Virtual Machines for FREE. Grab your copy now!
http://www.wservernews.com/go/mlplvy9q/
With a multitude of sensors and a vendor agnostic platform, PRTG Network monitor enables you to use ONE solution to monitor your entire infrastructure including applications, software, hardware, cloud & virtual environments.
http://www.wservernews.com/go/xxf1653r/
SPSFarmReport is a free SharePoint Farm Reporting tool:
http://www.wservernews.com/go/38164ig9/
Directory Lister is a tool for generating listings of files from user-selected directories on hard disks, cd-roms, floppys, usb storages etc:
http://www.wservernews.com/go/68lx20jc/
GetFoldersize is a handy tool that will quickly let you know exactly which folders are using most of your hard drive space:
http://www.wservernews.com/go/dd9jqpas/
GOT TIPS you'd like to share with other readers? Email us at wsn@mtit.com
Microsoft MVP Greg Carmack has posted a Wiki article called "Windows 10 Install Integrity Checklist" to the Microsoft Answers forums. Greg says, "For those with performance problems in Windows 10, these can often be resolved by doing basic housekeeping and Troubleshooting steps. Please don’t give up before trying these steps, then if needed start your own thread for more help." You can read his article here:
http://www.wservernews.com/go/bknmbl7l/
There's a helpful thread on the Microsoft Answers forums about how Windows 10 may freeze after installing the Anniversary Update and how to resolve this problem. Note especially the comment by forum moderator Yaqub K which is flagged as "Answer" and can be found here:
http://www.wservernews.com/go/jg1at7rw/
The Windows Club has an excellent post that explains the difference between Safe Mode and Clean Boot State in Windows and how to perform a clean boot for different Windows versions:
http://www.wservernews.com/go/t79b7ni9/
Microsoft Ignite Australia on February 14-17, 2017 at the Gold Coast Convention & Exhibition Centre, Broadbeach, QLD
http://www.wservernews.com/go/zzb8ckyb/
Microsoft Worldwide Partner Conference (WPC) on July 9-13. 2017 in Washington, D.C.
http://www.wservernews.com/go/8819wfmp/
PLANNING A CONFERENCE OR OTHER EVENT you'd like to tell our 100,000 subscribers about? Contact info@techgenix.com
4 security risks you might not know exist
Think your network is secure? Several commonplace items and devices, including wireless keyboards and printers, could be posing a security risk.
http://www.wservernews.com/go/5b0w5ywp/
Tip: Configuring sign out in Citrix ShareFile with ADFS
Stuck with integrating Citrix ShareFile within an Active Directory Federation Service? Perhaps we can fix that for you.
http://www.wservernews.com/go/6dbqailr/
Why Windows 10 fired Mr. Fix It
Microsoft's Mr. Fix It is not available for Windows 10 OS. Let's find out why we have to say goodbye to this troubleshooting tool.
http://www.wservernews.com/go/qtqd1me1/
Shadow IT: threat or opportunity?
Shadow IT spreads across your workplace like the shadow of a tree does over the ground. Is it sunset for the IT profession as we traditionally know it?
http://www.wservernews.com/go/q4uni5jd/
8 tips to optimize your industrial wireless network
If you are facing network, connectivity or speed issues, don’t despair: there are several potential solutions that can lead to a faster industrial network.
http://www.wservernews.com/go/uqp9u94c/
How to update the DNS root hints (IT Pro Central)
http://www.wservernews.com/go/wms5agud/
Further simplifying servicing models for Windows 7 and Windows 8.1 (TechNet)
http://www.wservernews.com/go/nhry6bs1/
Considerations on DMZ Design in 2016, Part 1 (Insinuator)
http://www.wservernews.com/go/oau8hmc6/
Windows 10 Information Protection / Intune Quick Setup Guide (Ken Lince)
http://www.wservernews.com/go/tbwknsip/
Operating System Deployment Logs (1e)
http://www.wservernews.com/go/kr70i0ec/
ConfigMgr Current Branch - real world migration from ConfigMgr 2012R2 (Gerry Hampson)
http://www.wservernews.com/go/gwmsro6p/
Bulk enrollment for Windows 10 devices (More than just ConfigMgr)
http://www.wservernews.com/go/hk8tkv28/
A Bit About the Windows Servicing Model (AskPFE)
http://www.wservernews.com/go/v1s5618p/
How to run Best Practices Analyzer in Windows Server 2012 R2 (IT Pro Central)
http://www.wservernews.com/go/nvo0gw2g/
Work Folders and Offline Files support for Windows Information Protection (filecab)
http://www.wservernews.com/go/4c9gwis3/
How application layering reinvents remote app delivery
VDI shops don't need to install applications within virtual desktops for apps to interact with the host operating system. Instead, they can deliver and manage applications separately using app layering. Find out how with this complimentary tip from our editors:
http://www.wservernews.com/go/3t2zw0hk/
Learn the many benefits of SDN for clouds and clusters
The software-defined infrastructure effort is gaining traction and will soon be commonplace because of the many benefits of SDN, such as flexibility, control, reduced costs and increased performance. Learn more about the many benefits of SDN for clouds and clusters by clicking the link below:
http://www.wservernews.com/go/gnrb8j35/
Is your SLA in cloud computing negotiable?
An SLA in cloud computing sets expectations around performance and availability – and while some are more flexible than others, not all SLAs are set in stone. Review your requirements with your cloud provider to see if there are areas you can negotiate
http://www.wservernews.com/go/v1nr07jl/
Creating a plan for VMware upgrades
There are a number of benefits and risks to consider when deciding whether or not to upgrade, especially when it comes to VMware upgrades – and therefore, it is vital that you carefully craft a plan before just jumping right in. Find out a number of things admins should keep in mind when upgrading:
http://www.wservernews.com/go/5e2jl4rb/
GOT FUN VIDEOS or other fun links to suggest you'd like to recommend? Email us at wsn@mtit.com
New York City, Summer 1939. Rarely seen recently surfaced amateur movie, filmed by a French tourist, Jean Vivier, in 16mm Kodachrome:
http://www.wservernews.com/go/h0jtpech/
Some aspects of London have changed a lot in over 80 years - others have changed very little:
http://www.wservernews.com/go/phq667ov/
A film taken from a streetcar traveling down Market Street in San Francisco in 1906, a few days before the earthquake/fire destroyed the area:
http://www.wservernews.com/go/6ciqzz8w/
A ride through the streets of Barcelona in 1908. Filmed by Ricardo Baños, a pioneer of Spanish cinema:
http://www.wservernews.com/go/aoe52ka0/
|
Mitch Tulloch is Senior Editor of WServerNews and is a widely recognized expert on Windows administration, deployment and virtualization. Mitch was lead author of the bestselling Windows 7 Resource Kit and has been author or series editor for almost fifty books mostly published by Microsoft Press. Mitch is also a ten-time recipient of Microsoft's Most Valuable Professional (MVP) award for his outstanding contributions in support of the global IT pro community. Mitch owns and runs an information technology content development business based in Winnipeg, Canada. For more information see www.mtit.com.
Ingrid Tulloch is Associate Editor of WServerNews and was co-author of the Microsoft Encyclopedia of Networking from Microsoft Press. Ingrid is also manages research and marketing for our content development business and has co-developed university-level courses in Information Security Management for a Masters of Business Administration program.
