Vol. 19, #12 - March 24, 2014 - Issue #972
Proactive System Maintenance for Improved Security
- Editor's Corner
- From the Mailbag
- Proactive System Maintenance for Improved Security
- Tip of the Week: When Not to Use Virtual Machine Snapshots
- Recommended for Learning
- Microsoft Virtual Academy
- Quote of the Week
- Admin Toolbox
- Admin Tools We Think You Shouldn't Be Without
- Events Calendar
- Asia Pacific
- Webcast Calendar
- WindowsNetworking.com Webinar: How to Recover Exchange and SharePoint in Seconds
- Register for Webcasts
- Tech Briefing
- Enterprise IT
- Windows Server
- Windows Server News
- Stop the insanity: Private cloud benefits for the enterprise
- Four options for third-party VDI assessment tools
- Breaking the taboo of mixing test and production environments
- Got ravenous VMs? Check your virtual memory management configuration
- WServerNews FAVE Links
- Compilation of Aircraft Struggling With The Wind
- Supercar Street Race - Top Gear
- The Catwall Acrobats - Monte-Carlo Festival
- World's Fastest Water Car
- WServerNews - Product of the Week
- Solve the problem. Be a hero. Try FactFinder Express.
- SAVE THIS NEWSLETTER so you can refer back to it later for helpful tips, tools and resources!
- FORWARD THIS NEWSLETTER to a colleague who you think might find it useful!
- SEND YOUR FEEDBACK to [email protected] if you have any comments or suggestions!
This week's newsletter is all about the security benefits that come from taking a proactive approach to IT systems maintenance. We welcome Mike Doyle of Cigital who has contributed this week's guest editorial on this subject.
I've always thought of "proactive" as one of this Dilbert words i.e. something that marketing people use frequently but which hardcore techies tend to sneer at. A quick search on Dilbert.com does indeed suggest that "being proactive" can lead to some nasty consequences sometimes as this comic illustrates:
From the Mailbag
Back in Implementing Wi-Fi in Enterprise Environments (Issue #967) I shared some thoughts about implementing reliable WiFi networking in business and industrial environments. I also mentioned that Meru seemed to be a popular solution for enterprise WiFi networking. The week following in the Mailbag of Enterprise Monitoring Strategies (Issue #968) we included a comment from a reader named Chris who recommended Ubiquiti UniFi as a reliable, low-cost alternative. This was later seconded by another reader named Matt in BitTorrent in the Enterprise (Issue #970) who manages IT for an educational organization. I guess third time's a charm for Ubiquiti as yet another reader named Bob, a Computer System Manager, who sent us the following feedback:
Matt made a couple of very significant points in his piece about Ubiquiti. The first is that the use of IT in an Educational environment is quite different from normal business, and quite often much more demanding on the infrastructure. As well as Matt's point about hundreds of simultaneous logins at several specific times throughout the day, often during quite a short window (our lessons are 40 minutes) those users are quite likely to open three or four different applications (as they are taught, perhaps, how to construct a graph using a spreadsheet, modify pictures selected from the web using a graphics package, and then construct a document or poster combining these).
I would also like to add our endorsement of the Ubiquiti Point to Point wireless solutions which we have been using to connect our outlying buildings back to the campus. Again their solutions 'just work' for us, and are reasonably priced. I well remember advising one colleague in another school here in UK that he could buy a pair of devices cheaper than someone had quoted him to come in and do a feasibility study to connect two building (and thus do his own feasibility study)! Thanks for another interesting issue.
Great feedback. I actually have a colleague who bought a couple of Ubiquiti NanoStation Loco and used them for point to point wireless in a scenario where running Cat6 cabling wasn't feasible. You can actually buy the NanoStation Loco from Amazon here:
Note that Ubiquiti's website shows the NanoStation Loco as an end-of-life product, but it still might be a good deal:
It's pretty cool that a number of different Ubiquiti products are available through Amazon. Here are a few more you might want to check out:
Ubiquiti Networks UniFi AP Enterprise WiFi System:
Ubiquiti UniFi Long Range Access Point:
Ubiquiti AirRouter Indoor WiFi router:
Ubiquiti UniFI AP Outdoor 2x2 MIMO Access Point 802.11b/g/n:
And now on to this week's guest editorial by Mike Doyle...
Proactive System Maintenance for Improved Security
I'm not a system administrator anymore; I am a security consultant. I've been asked to share some of my incident response and penetration testing experiences to illustrate the security benefits of proactivity and diligence.
I hope this is worthwhile, or at least enjoyable. Penetration testing is adversarial in nature and many of my adversaries are overworked sysadmins who don't get the opportunity to fight back. This is clearly unfair and if I'm going to be a glorified cyber bully, I'd rather not further demean myself by bragging or preaching to my victims.
Nonetheless, I have a friend named Bob. Bob is a well-respected Windows Systems Engineer for a global fleet services provider. His job is classified as "works with computers," although they happen to be servers running Citrix, VMware, Exchange, SQL, or something like that. If you met him, you'd like him.
Bob suggested the topic of this article. Kinda. Actually, here's our exchange:
I don't question Bob. If he enjoys this sort of thing, I hope you will to.
As you might expect, certain details have been omitted or scribbled over to protect the confidentiality of my clients. Don't worry though; it shouldn't spoil the schadenfreude.
Opportunity 1: Apply Software Updates
Server updates take forever, I get it. You have to read the advisory, push out the update, bounce the systems, go back and check for install errors, push out any remaining updates, bounce again. Then you wait to see who complains. When they complain, you have to roll back and if that fails you have to restore the system from back up. Never pretty.
Nonetheless, the best way to protect your systems is to keep them updated with the latest security patches.
Once while testing a network, I found a UNIX server running an ancient version of Oracle, written before software developers gave a thought about who might be on their networks. The process that listened for client connections had a user-friendly feature to allow DBAs to perform a handful of functions remotely: you connected to the listener and issued it commands. It didn't let you manipulate the data, mind you, just the configuration of that one listener process. For example, you could change the location of the log file. You could also shut the listener down entirely, preventing anyone from connecting to the database if you didn't mind getting caught. Oracle had long since added a patch that allowed admins to protect the listener with a password.
When I tried explaining the issue to the UNIX and Oracle teams, they blew it off as being a theoretical risk only. Who cares where the logs go?! No one looks at those things. And if you shut down the database, the DBA team will get notified and turn it back on. As long as the data is safe, there's no need to risk hurting the system by putting a patch on it.
Most Windows sysadmins I've met understand the value in patching their servers. UNIX guys think they're invulnerable. They never met a patch they didn't like to ignore. And worse, most Oracle DBAs never met a patch.
A few days later I had them swing by my desk for a quick demo.
"I just connected to the listener," I said, showing them my screen.
"I just redirected the log file. Now the listener is going to send all logs to /home/oracle/.rhosts," I said to a pair of quizzical faces.
tap, tap, tap, tap.
"I just executed a nonsensical command. The log should say something to the effect of
LST-2468: Command not recognized:
With that, the UNIX SA bolted out the door. The .rhosts is the file used to list all the trusted login users on all the other servers on the network. These are the users that can connect without having to enter a password. "+ +" is the string to indicate that ALL users on ALL machines are trusted. The DBA, not familiar with this file, stayed put.
tap, tap, click.
"I just logged in as Oracle. The system let me in without a password."
tap, tap, tap, tap, tap, tap, tap.
"I just dumped the contents of the users table. Want I should drop it?"
With that, the DBA bolted out of the room. Sometimes my work gets lonely.
So patch those servers, right? Right. It would be naïve to think that the answer is as simple as that. Sometimes patches break systems. To defend yourself against bad patches you need to develop a routine of deploying patches to test systems, testing them, and then rolling them out to your network. At any point you might discover a problem patch, so you have to roll the patch back, research, and contact the vendor for a resolution. No doubt, this process takes equipment and time. Windows admins have WSUS and SCCS to take care of all their Microsoft software, but few use products like Big Fix and Shavlik to handle automated patching of third party software. If you don't have updates for third party packages like Adobe Reader and Java updated, I suggest you look into these products.
Opportunity 2: Review Logs
I had a client once who was able to get everything patched within a day or two of the patches coming out. Sadly, he got hacked anyway. It's how we met.
"Winston" was the system administrator of a company that ran only one application: a web bulletin board for hobbyists. It was staggeringly popular. Advertising revenue paid the salaries of about half a dozen people. The work was fun and easy; volunteer moderators did most of the heavy lifting.
BBSes are big targets for phishers and spammers, though, and Winston had to deal with all sorts of automated attacks and brute-force attempts. It was during such a campaign that he saw a post in one of the moderators-only forums: "I am Ivan. I need to get a job with Hobby Haven as security. Their software has holes. Yours too. I can help."
The moderator's name wasn't Ivan; it was Phil. And Phil hadn't logged in for a few years. Ivan must have popped his account. Winston locked Phil's account and went about his day.
A few days later, another message in another moderators-only forum from a different account: "This is Ivan. You have partnership with Hobby Haven. Tell them talk to me. They don't answer my emails. They and you have software hole. Theirs is worse, is in payment system."
Hobby Haven was one of Winston's advertisers. He became a little more worried, but deleted the account and the message.
The next day, Ivan logged in as another mod and posted the usernames and hashed passwords of about five percent of the user base. Winston's company went offline. Then he called me in to assist.
Most data breaches aren't discovered until months after the fact, even though bad guys almost always leave evidence in log files. On a normal day, Winston's web servers generated about a gig of access logs an hour, and never backed them up. This doesn't count the traffic caused by the Denial of Service attack. These consumed so much space that he deleted the files automatically after a week. Fortunately Ivan let his presence be known, and we were able to copy off the logs for analysis.
Buried amongst the normal accesses we found rows and rows of this:
200 GET hobbyforum.com/index.do?forum=3&topicid='IF((SELECT 1 FROM USERS where userid=1 and substr(name,0,1)=a,'1','x')'
404 GET hobbyforum.com/index.do?forum=3&topicid='IF((SELECT 1 FROM USERS where userid=1 and substr(name,1,1)=a,'1','x')'
404 GET hobbyforum.com/index.do?forum=3&topicid='IF((SELECT 1 FROM USERS where userid=1 and substr(name,1,1)=b,'1','x')'
404 GET hobbyforum.com/index.do?forum=3&topicid='IF((SELECT 1 FROM USERS where userid=1 and substr(name,1,1)=c,'1','x')'
200 GET hobbyforum.com/index.do?forum=3&topicid='IF((SELECT 1 FROM USERS where userid=1 and substr(name,1,1)=d,'1','x')'
404 GET hobbyforum.com/index.do?forum=3&topicid='IF((SELECT 1 FROM USERS where userid=1 and substr(name,2,1)=a,'1','x')'
This is what scripted blind, boolean-based SQL injection looks like in your logs. Each request manipulates a SQL query in a surprising but syntactically correct way. Each query reveals the answer to one true/false question.
The first request asked whether the user with userid of 1 had a user name that started with an 'a'. It did, because the userid 1 is 'admin'. The server returned a 200 code, http for request successful, here's your page. The second request asked if the username's second letter is also 'a'. It isn't, and the server returned 404 not found. Ivan tried 'b', then 'c', and so on until he got the right letter and then moved on trying to guess the third letter. By laboriously asking repetitive true/false questions, Ivan managed to read every username and password hash out of the forum database. But it took him days!
Once we found the evidence, we contacted the software vendor and demanded a patch. The SQL injection issue was news to them, what we call a zero-day vulnerability. They responded quickly with a patch. We re-imaged all the machines, of course, and forced all the affected users to change their passwords.
Do you think Winston waited for Ivan to find another zero-day? No. He tackled his log review problem.
Log review doesn't necessarily have to mean scrolling through event viewer until your eyes bleed. There are sophisticated log servers and security event managers that can reduce eye-hemorrhage by a factor of a bazillion.
In the case of Winston, he outsourced log review to a managed security services provider. By shipping his web and systems logs out to a third party, he let them correlate his spurious traffic with that of hundreds of other clients across the internet.
A few months later, Winston emailed me to let me know that Ivan had come back. He didn't post any messages this time; he just started extracting their database bit-by-bit with another zero-day! Fortunately, Winston's MSSP caught the activity within half an hour. They were able to block Ivan before he could do any more harm.
Opportunity 3: Change Default Passwords
Logs don't find everything. One of the most popular hacking techniques can let a bad guy enter your systems without a trace. It takes no special software besides the web browser or remote access client you are already using. The technique is called 'logging in to a privileged account with a default password.'
One time I was testing a stand-alone information delivery system, like a sort of kiosk. The makers of the kiosk wanted to expand the market for their product to discriminating clientele who were going to use the device to access sensitive personally identifiable information. They previously hadn't had to factor security into a lot of their design decisions. They knew some changes were going to have to be made before the next generation of their product went to market, so they sent me a demo unit for security testing.
During a lengthy kickoff meeting it was revealed that the device would automatically boot right into the kiosk software with no username or password. The software ran as a regular user, and the software had been rewritten in such a way that it could only write sensitive information, not view it. Only a kiosk administrator could view sensitive data. To do that, they simply had to press a special key combination, which brought in a login screen. By putting in a secret password, they would be able to view sensitive information and export it in a variety of formats. Further, sensitive information was stored in a permission-protected folder that the normal kiosk account couldn't even access. This appeared to be sound, and I was optimistic that the system would pass muster.
The kiosk company hadn't historically had a need to worry about product security before but they were wise enough to disable ALT-TAB, CTRL-TAB, and CTRL-ALT-DELETE, explaining proudly that it was a design decision they made early on. I'm sure they were telling the truth, because their security decision seems to have pre-dated the invention of the Windows key. By pressing Windows-R, I got a trusty "Run…" dialog.
From there I started exploring the file system. C:\Data was protected; I couldn't get in there, but I could get to all the kiosk software files. I found a file called 'autologin.xml' which contained these two lines:
Looking in C:\Users, I saw that the administrator account's user name was "kioskadmin".
It was time to try out my secret hacker technique. I brought up the login screen.
I entered the username 'kioskadmin'.
I entered the password 'kioskadminpass'.
tap, tap, click.
I was in. It had taken two minutes to get to the data.
Optimism can be a liability in my line of work. If I trust the system too much, I start to make certain assumptions. I get careless. I don't check the basics. But then something like this will happen and ennui sets in, a cold reminder that security vulnerabilities exist in everything. Then I have to start testing basic assumptions. Go back to blocking and tackling. What if I tried with a wrong password? Or no password at all? How many invalid passwords will the software let me try?
Before long I realized that I couldn't find the kiosk's password reset function anywhere. I put in a call to my client, only to discover that the software didn't come with one.
"How does the password get reset in the field then?"
"Wait. You mean that this so-called 'secret' administrator password is the same on every kiosk in every other retail location? And everyone who has ever administered one will know the password to every other kiosk? Even the ones at their competitor's locations?"
You might guess that I advised my client against unchangeable, simple default passwords.
When it comes to the software you administer, it only takes two things. First, google "default passwords" and read through some of the results. Second, check the lists here:
If you find that your software uses default passwords, see what it takes to change it. Most of the time, it will be a simple change, but some funky middleware likes to store default passwords.
Taking proactive steps isn't free; it costs the most precious resource a system administrator has: time. You can't set up a log server or roll out system updates if you're stuck putting out fires and closing request tickets most of the time. If you can't get out of the weeds, I recommend checking out Tom Limoncelli's book and videos on time management for system administrators. He gives great advice on setting up adequate time for projects, prioritizing the requests you get, and establishing the metrics that let you justify additional head count:
Otherwise, patch, review logs, and change default passwords.
About Mike Doyle
Mike Doyle helps people build secure software at Cigital.
You can follow him on Twitter here:
Be sure to check out his blog:
For more information about Cigital, see their website:
Send us feedback
Got feedback about anything in this issue? Let us know at [email protected]
The following tip is excerpted from my book Training Guide: Installing and Configuring Windows Server 2012 from Microsoft Press:
Although snapshots are not recommended for use in production environments, they might have value in certain limited scenarios. For example, you might consider performing a snapshot of a production virtual machine just before you apply a critical software update to the guest operating system of the virtual machine. That way, if something goes wrong after applying the update, you can quickly revert to the virtual machine to its previous state (that is, before the update was applied). However, there are certain scenarios where you should never perform snapshots, specifically:
- Don't perform snapshots on virtualized domain controllers.
- Don't perform snapshots on virtualized workloads that run time-sensitive services.
- Don't perform snapshots on virtualized workloads that use data distributed across multiple databases.
Also, don't try to restore snapshots older than 30 days because the computer password for the guest operating system might have expired, which will cause the guest to dis-join itself from the domain.
Finally, if you do plan on performing snapshots, make sure the host has sufficient storage for all the snapshot files you might create. Snapshots can consume a lot of disk space, and you could end up running out of storage space if you perform too many of them.
GOT TIPS you'd like to share with other readers? Email us at [email protected]
When I was studying Physics in university I had huge poster of David Hilbert the famous mathematician taped onto the inside door of my bathroom since I did most of my studying there. True story! Well the part about the poster anyways...
Do you like posters? You can download some helpful and informative ones from the Microsoft Download Center here:
Microsoft Virtual Academy
Become the solutions expert, with limited-time certification offers & free exam prep from Microsoft
Microsoft training and certification provides IT professionals the opportunity to expand skills and gain knowledge directly from the source. By proving these skills you can grow your career and make yourself indispensable as businesses deploy solutions and evolve more quickly than ever before. Take advantage of a limited-time certification offer & free exam prep from Microsoft to help you succeed. Start now:
Programming for Absolute Beginners
C# Fundamentals for Absolute Beginners:
VB Fundamentals for Absolute Beginners:
Quote of the Week
"Whoever you are, wherever you are stationed, these are the cards you got dealt. You can't spend your time wringing your hands about it. You play the cards you have. You accept the burdens in the context of which you came from and enjoy the privileges and don't be guilty and either one of them." - Goldman Sachs CEO Lloyd Blankfein
Note to subscribers: If for some reason you don’t receive your weekly issue of this newsletter, please notify us at [email protected] and we’ll try to troubleshoot things from our end.
Admin Tools We Think You Shouldn't Be Without
Try FactFinder Express- a new kind of server monitoring that offers you everything you need to find the source of performance problems. Solve the problem, be a hero. Try it free for 30 days.
Migrating to SharePoint causes broken file links. This is a hidden problem that results in costly disruptions. LinkFixer Advanced automatically fixes broken links. Try a free trial version.
#1 backup tool for Hyper-V. Veeam Backup Free Edition is the must-have tool for VMware and Hyper-V. Use Veeam Backup Free Edition for as long as you like. Download now.
You can use SendLater to set a schedule and recurrence for automatic e-mail messaging with Microsoft Outlook:
HJSplit is a popular freeware program to split and recombine files:
Microsoft Exchange Conference (MEC 2014) on March 30-April 2, 2014 in Austin, Texas
Microsoft Build Developer Conference (Build 2014) on April 2-4, 2014 in San Francisco, California
TechEd North America on May 12-15, 2014 in Houston, Texas
Microsoft Worldwide Partner Conference (WPC 2014) in July, 2014 in Washington, D.C.
Microsoft SQL Server PASS Summit 2014 on November 4-7, 2014 in Seattle, Washington
European SharePoint Conference on May 5-8, 2014 in Barcelona, Spain
TechEd Europe on October 27-31, 2014 in Barcelona, Spain
TechEd New Zealand on September 9-12, 2014 in Auckland, New Zealand
Add your event
PLANNING A CONFERENCE OR OTHER EVENT you'd like to tell our 95,000 subscribers about? Contact [email protected]
WindowsNetworking.com Webinar: How to Recover Exchange and SharePoint in Seconds
Microsoft's Hyper-V and VM protection are two of the hottest topics in the IT world. Join our webinar for an inside look on how together, Veeam and Microsoft can help to improve your Modern Data Protection Strategy.
Join Microsoft MVP Brien Posey and Chris Henley, Veeam Product Strategy Specialist for this complimentary, interactive webinar on Thursday, March 27th, 2014 at 2PM EDT / 7PM GMT to learn how to save precious recovery time and maintain your reputation as a good and lazy administrator!
Register for Webcasts
Add your Webcast
PLANNING A WEBCAST you'd like to tell our subscribers about? Contact [email protected]
Zoning a Linux Server to a Storage Array (WindowsNetworking.com)
Microsoft's non-Windows based RDP clients (VirtualizationAdmin.com)
Disk2VHD Version 2.0 AKA Disk2VHDX! (Aidan Finn, IT Pro)
Build your own lab with Free Hyper-V 2012 R2, Windows PowerShell and Trial Microsoft Software (EnergizedTech)
Web Browser Security Revisited (Part 1) (WindowSecurity.com)
Best practices for installing Updates and Service Packs in Forefront UAG and TMG (ISAServer.org)
Windows NIC Teaming using PowerShell (WindowsNetworking.com)
Virtual Desktop Infrastructure - A Deployment Guide for Education (Microsoft Download Center)
Maximizing the potential cloud computing benefits in a migration
You'd be shocked by how many cloud projects fail due to unrealistic goals and false assumptions. Fortunately, this exclusive guide provides tips for navigating around potential problems and controlling networking costs to help ensure your goals are met.
Four options for third-party VDI assessment tools
Is your enterprise ready for a comprehensive VDI assessment? There are many services that can help, but it can be complex to sort through the saturated market. Access this essential guide on four of the best third-party VDI assessment tools available to ensure you make an informed buying decision.
Breaking the taboo of mixing test and production environments
Keeping testing and production environments separate may be the traditional approach to take, but for some organizations, integrating the two can offer significant benefits. Access this exclusive resource to determine if the unconventional method of mixing these environments is right for your organization.
Got ravenous VMs? Check your virtual memory management configuration
There are many ways a virtualization administrator can improve VM responsiveness – but it's not always a simple task to achieve. Inside this valuable tip, explore step-by-step recipes on how to build and run a robust virtual infrastructure to ensure that VMs perform at the highest level.
This Week's Links We Like. Tips, Hints And Fun Stuff
GOT FUN VIDEOS or other fun links to suggest you'd like to recommend? Email us at [email protected]
Compilation Of Aircraft Struggling With The Wind
Landing and take-off highlights in awkward wind conditions at Birmingham Airport in England.
Supercar Street Race - Top Gear
Jeremy in a McLaren MP4-12C Spider, James in an Audi R8 Spyder and Richard in a Ferrari 458 Spider race against the Stig in a Jaguar XKR-S convertible.
The Catwall Acrobats - Monte-Carlo Festival
The Catwall Acrobats 360 degrees transparent wall act was the crowd's favorite at the 37th International Circus Festival In Monte Carlo.
World's Fastest Water Car
Water Car Panther is the fastest amphibious car in the World - capable of 80 mph (127 km/h) on the road and 44 mph (70 km/h) on water.