- Sign-up Now!
 - Current Issue
 - Edit Your Profile/Unsubscribe

Subscribe | Media Kit | About Us | All Issues | Subscriber Feedback | Contact Us | Privacy Statement
Sunbelt W2Knews™ Electronic Newsletter
The secret of those "who always seem to know" - Over 500,000 Readers!
Mon, Apr 22, 2002 (Vol. 7, #32 - Issue #363)
Sunbelt Unscheduled Downtime: Postmortem
  This issue of W2Knews™ contains:
    • Sunbelt Unscheduled Downtime: Postmortem
    • "OK, This Is What Happened". By The Man Who Did It.
    • What Is The Source Of 2005 W2K Support Stop?
    • HotFix Hint #1
    • HotFix Hint #2
    • How UpdateEXPERT Checks For HotFixes
    • Comprehensive Group Policy Management With Fazam 2000
    • Opalis And Sonic Mobility Work Together
  5. W2Knews 'FAVE' LINKS
    • This Week's Links We Like. Tips, Hints And Fun Stuff
    • The Hacker Diaries
  SPONSOR: Wireless Microsoft Enterprise Manager
The #1 Wireless Systems Management Tool, Now Starting at $149.
Limited Time Offer for W2Knews Subscribers only! Save Time while keeping
your systems running at top performance by managing them anytime and
from anywhere, using just your web-enabled mobile phone, PDA (PalmOS/
PocketPC/Symbian) or RIM Wireless Handheld. With StarAdmin, you start
working on system problems immediately because the MMC and all other
things like processes, services, events logs, etc), User Management,
File System, SQL, Telnet, SSH, and much more are in your hand. Demo
StarAdmin on PocketPC, RIM, or WAP. Download a 15-day eval copy now:
Visit Wireless Microsoft Enterprise Manager for more information.

Sunbelt Unscheduled Downtime: Postmortem

We were off the air for just under half an hour last night. Despite all the precautions we take. We lost some data as well. Ouch. The Tech Briefing explains what happened and how YOU can prevent this from happening. In the mean time, the links to W2Knews stopped working for a while, but we reconstructed them. If you tried to read your W2Knews but could not get anywhere and found yourself in an old W2Knews issue instead of the link destination, our apologies. Here are all the links that now work again. And learn from the Tech Briefing so this does not happen to you!!

Microsoft Patch Management? It's A Mess

Gartner: Securing Windows Takes 15% Longer Than Unix

Already Own One Half Of The Bundle? The Other One Now Discounted!

NEW: ScriptLogic V4.0- No More Messing With Logon Scripts!

PentaSafe OnLine Security Library Opened

Throwing More Hardware At It: "Storage Insanity"

Self Service For User Passwords!

The "HammerOfGod" site has a few good utilities

A new Linux desktop that's a spitting image of WinXP

New threats force intrusion-detection vendors to rearm.

Cool & scary at the same time: Key Katcher- record all keystrokes and URLS

Here is something to gross out some of the girls in the office

And when you think that your day is bad, take a look at THIS man's Monday morning. This is one of the best ads for furniture I have seen in years and every time I see it I'm in stitches. Forward this one to your friends!

Book Of The Week: Hacker Diaries

UNDO Dept: Last issue I said that ScriptLogic generates KiXtart. Bzzzzz. Wrong! It is not a "code generator" nor a "graphical wrapper" for KiXtart. Rather, ScriptLogic is a turnkey product that utilizes the KiXtart language. Since the ScriptLogic engine is compiled and the ScriptLogic Manager does not generate raw code, it cannot accidentally generate syntax errors: the type of problem that could easily shut down your enterprise one morning.

Warm regards,
Stu Sjouwerman (email me with feedback: [email protected])

Sunbelt Remote Admin Now Comes With "HelpDesk License"
You might want to give Remote Admin a try too. It's as if you are there,
with all the tools to control multiple remote systems. This puppy was
"made-by-and-for" system administrators: super fast and works great for
telecommuters. Pricing is very low: Just 35 bucks for Client/Server! (and
dirt cheap site-, helpdesk and corporate licenses). Available with instant
online delivery and mainframe quality Sunbelt Tech Support. Get your (eval)
copies here:
Visit RAdmin for more information.

"OK, This Is What Happened". By The Man Who Did It.

Last night at about 6:15pm we started working on the SQL Server that is the backend of our website. The purpose of the maintenance was to add 2 additional drives and reconfigure an existing set of drives that had been put in temporarily for storage purposes. The drive configuration to begin with was as follows:

Drive 0/1/2 = Array 0 = RAID 5 array with the OS and SQL installed
Drive 4/5 = Array 1 = RAID 0 stripe for temp storage of database backups

The addition of the two new drives, drives 6/7 as Array 2 in a RAID 1 configuration, went without any problems. After they had been setup and installed the members database was copied from Array 0 and attached.

Then came the procedure of recreating Array 1 as a RAID 1 array. After about 90 minutes of time spent working within the raid controller software, working in the raid controller's BIOS, and reading the colorful (but useless) documentation for the raid controller software things went bad, very bad.

Luckily we had replicated all the website SQL data to our co-location servers in Texas, so we told IIS to redirect everyone over there, and we got to work fixing things.

In the BIOS of the RAID controller I had selected Array 1 and hit initialize, nothing else had worked up to this point. After being prompted if I wanted to initialize and saying yes a progress screen popped up showing Array 0 and 2 being initialized, not Array 1. The initialization took about 1 second to complete and that was the end of any data that had existed on Array 0 and 2. As an FYI for those that care, the RAID controller in question is a Dell PERC2 with an AMI chipset. Up to this point I had been less than thrilled with the AMI based controller, the Dell PERC2 Adaptec controller had always seemed to be a superiorly designed controller.

After the RAID controller finished its business a reboot confirmed that it had in fact completely wiped both of the crucial arrays, time to call Dell. Call their support line, punch in the system service tag, follow the prompts for a PowerEdge 6400 server support. After holding for a few minutes shy of a full hour a tech finally came on the line. The conversation in a nutshell went along the lines of:

#1. If the drives initialized there is nothing you can do, grab a backup. If your backup isn't enough find a place that does data recovery.

#2. You must have selected an option to initialize all the drives.

#3. In order to remove an array on that type of controller you have to clear the configuration and then recreate the containers.

Needless to say this was a completely useless call and we started to rebuild the SQL server. Since I had opportunity to recreate the arrays I once again went into the controller BIOS. Now that I could run from a clean slate I setup the following configuration:

Drive 0/1 = Array 0 = RAID 1 mirror for OS, SQL binaries, master/model/tempdb/msdb databases.
Drive 4/5 = Array 1 = RAID 1 mirror for most of the other databases
Drive 6/7 = Array 2 = RAID 1 mirror for members database

While setting up these arrays I did confirm for my own piece of mind that there is no way that I could have selected all the drives for initialization by accident, the controller had in fact just freaked out.

Then we reinstalled the OS, installed SQL, and ran through all the patches and updates. Then we went to restore the database backups from tape. Now for the second surprise, the most current database backup was not data from Thursday morning. The data was from Wednesday morning. Some quick research explained what happened. On Monday the nightly integrity checks and backups had been changed from 11pm to 3:30am.

This was because the performance of the site would become extremely poor during this time. However the backups still ran at 1 am and completed backing up the SQL server at 2:25 am, missing the SQL backup files by just over an hour.

As of about 1 am this morning everything crucial was restored but any changes since Wednesday at 3:30am were lost. Augh! This included the last newsletter, database processing done in the mean time, subscribe and unsubscribe requests.

Today's project is going to come up with policies and procedures to keep this sort of incident from occurring again. Last time we got burned by a RAID failure was with the firewall and that was resolved by getting 2 PIX firewalls that have been > 99.999% reliable.


What Is The Source Of 2005 W2K Support Stop?

A bunch of people asked me that, because it cannot be found on any of the know MS webpages for that. Well, this is kinda new news and was disclosed by Gartner vice president and research director Tom Bittman at their "Nothing but .Net" Conference. The people from SearchWin2000 were there and reported on it. Here is the source:

HotFix Hint #1

When I run into a hotfix that doesn't support the -z switch, I just wait until the hotfix finishes installing. Then, when the dialog box pops up telling you to reboot, go into task manager and kill the hotfix. e x e process. You can then continue installing other hotfixes with the same method and then run Qchain and reboot.

HotFix Hint #2

"The Patch Crashed My Server"... You sometimes run into this apparent situation. Apply patch, reboot, problems! Would be easy to jump conclusions, but hold your horses for a moment.

Russ Cooper pointed this out the other day, and MS has told me as well, when a patch appears to cause a problem with your server it's more likely that it wasn't the patch that caused a problem, it was something else prior to the patch that caused the problem. So, the trick here would be to reboot that server once before you apply the patch(es). This would show you if any problems were there but yet undetected. But, if you really run into troubles with a patch, call PSS, Microsoft Product Support Services. They will help you out.


How UpdateEXPERT Checks For HotFixes

Some anonymous hacker trying to make a name for himself alleged that UpdateEXPERT (UE in short) is insecure. This is false data. Here is a short explanation that shows UE is a perfectly valid tool. Thanks to Marc Maifret from eEye for his comments from which I have freely borrowed. While UpdateExpert could be using improved methods of checking for installed patches, it is a perfectly secure product.

There is a problem that all too often people release "advisories" which mislead people into thinking that software is "insecure" when the reality of the situation is that the software is actually perfectly fine. It is hard sometimes to weigh what makes software secure or not. You must assess the possible threat level, if there even is one.

The "problem" pointed out in UpdateExpert is that UE uses the windows registry hotfix keys as a method to see if patches are installed or not. It is possible that an attacker who has already penetrated the system, changes the registry keys and makes it appear as if a system is fully patched, or unpatched.

So the "vulnerability" would be that if you compromise a system, and then gain Administrator or SYSTEM level access (which is required to alter the registry keys) that you could then fool UE into thinking a patch is installed when it is not, therefore leaving the system "vulnerable".

Once again, you have to have administrator/SYSTEM level access to be able to perform this attack. It should be a rule of thumb that any "advisory" that claims "you need admin/sys access to perform the attack", be regarded as a bogus advisory done by amateurs looking to get their name out.

Is this a vulnerability in UE? Nope! If UE had implemented their patch checking in a different manner would it have made it harder for people to trick UE? No. You see, no matter what method of patch checking you choose to implement, if you compromise a system then it is TRIVIAL to make it seem as if a patch is installed or not.

It does not matter if you are checking the registry, file versions, crypto hashes, etc. All of that can be manipulated or altered once you are the administrator or SYSTEM. The fact of the matter is that this is a not a security vulnerability in UE. If anything it is a usability/reliability feature enhancement request for a future version of the product.

The only reason it should even be thought of as a feature enhancement is in the rare case that an administrator deliberately goes and modifies the registry keys and therefore that breaks the results of an UE audit of your system for patches. This will be a rarity though.

Retina, the network security scanner, in fact does use the registry method (same as UpdateExpert, same as Windows Update) to be able to see if a patch is installed or not. However, it also tests for the vulnerability as existing or not, from a network layer perspective. That way the system is a bit more foolproof.

So in the case of an IIS unicode attack, Retina checks the registry for the patch, and it checks the web server itself via network queries, to see if it is vulnerable. This extra step that Retina does in truly auditing for the vulnerability, as a hacker would, is something that just makes Retina a bit more thorough. Retina as a product has a different goal than UE which is why they work differently in checking for system vulnerabilities and the two complement each other nicely.

Retina can find holes a bit better but doesn't do anything with patch management or deployment as UE does. Hence the bundle that you can find at:

Comprehensive Group Policy Management With Fazam 2000

When you implement AD, you need tools. FAZAM 2000 Version 3 provides complete Life-Cycle Management for Group Policy. It builds on Microsoft's native Windows 2000/XP infrastructure to provide the most comprehensive Group Policy management solution on the market.

Version 3 is due next month. Customers love FAZAM 2000 version 2 but because GPOs are so powerful, they asked FullArmor to come up with a way to manage the entire change process. The key feature is the 4 W's - who, what, when, why. Also it was a site that is managing over 90,000 AD users who asked for these features.

FAZAM 2000 Version 3.0 includes Change and Release Management, a critical best practice for implementing Group Policy. You can track the history of Group Policy Object (GPO) changes, including who made the changes, from initial design to retirement.

Version 3 New Benefits:

  • Tracks changes to GPOs
  • Provides version control for GPOs
  • Allows new or changed GPOs to move into production only after being tested and approved
  • Eliminates the risk of making changes in a live, production environment
  • Prevents two or more individuals making simultaneous changes to the same GPO
  • Reduces risk when delegating GPO administration.
Comprehensive GPO Management

Version 3 New Features:

  • 4 Ws of GPO Change: Version history for any GPO is available on-line and shows who made the change, what was changed, when the change was made, and why the change was made. For any entry in the GPO history, FAZAM 2000 can produce a full report of the GPO as it existed at that point in time. The report may be viewed online, printed, saved as an HTML file or saved as a MS Access file.
  • Check-out and Check-in: GPOs must be checked out before editing. Only the person that checked out the GPO may edit it
  • Difference and Comparison Reports: Determining differences and comparing GPOs is automated. A GPO can be compared to: another version of the same GPO; an archived version of a different GPO; or to the version of the GPO that is in live Active Directory. The reports can be viewed online, printed, saved as an HTML file or saved as a MS Access file.
  • Off-line Changes: GPO changes are made off-line and can be subject to normal testing, approval and release processes
  • Delegation: Authority to create and change GPOs can be limited preciously so that GPO administration can be delegated with reduced risk
  • Rollback: GPO rollback allows a prior version of a GPO can be put back in production
  • Documentation: GPO documentation may be viewed on-line, printed, saved as an HTML file or saved as a MS Access file
  • Replication and Synchronization: GPOs can be easily replicated and synchronized from domain to domain and across forests ? even disconnected forests ? for consistency
  • Baselining: GPOs can be baselined so that it is always possible to go back to the original settings
  • GPO Health: Administrators can determine the health of their Group Policy environment by running reports to discover GPO corruptions and replication problems.
  • GPO Reporting: Allows IT Administrators to view detailed reports on GPOs in Active Directory through the MMC console or Web Browser
  • Resultant Set of Policies: Provides Resultant Set of Policies (RSoP) or the set of effective policies that apply to a user when logging on to a machine. Also allows for enhanced ?What/if? Scenarios
  • Policy-centric view of AD: Provides a view of Active Directory with Group Policy links and filters
  • Backup/Restore: Allows administrators to backup and restore individual GPOs on a domain including filters and links
  • Troubleshooting and Diagnostics: Provides administrators with the ability to perform remote diagnostics from a central administrator console
  • Search: Provides searching for GPOs and settings within GPOs
  • Scripting: Provides scripting of the backup, import, and reporting of GPOs.
Get an eval copy of 2.0 now and get a taste of the power!

Opalis And Sonic Mobility Work Together

Opalis Software and Sonic Mobility today announced a partnership that will enable users to directly and securely execute jobs and gather the status of Windows Server processes via a wirelessly enabled PDA. The partnership involves OpalisRobot task automation for Windows environments and the SonicAdmin mobile system administration application for PDAs. The combined solution, which is planned for availability late spring, will enable SonicAdmin users to extend server automation and monitoring capabilities with OpalisRobot while managing jobs, servers and network infrastructure remotely using their RIM Blackberry or Pocket PC device. Network administrators will be able to view vital server information such as reports, logs, statistics and alerts. They will also be able to trigger OpalisRobot jobs with SonicAdmin.


This Week's Links We Like. Tips, Hints And Fun Stuff

  • A nice site that compiles all MS Security Bulletins.

  • http://www.w2knews.com/rd/rd.cfm?id=020422FA-Security_Bulletins
  • Small racecars with even smaller cameras mounted on them. Fun to watch!

  • http://www.w2knews.com/rd/rd.cfm?id=020422FA-CarCam
  • Getting into Wi-Fi wireless internet access, but cannot find the right antennas?

  • http://www.w2knews.com/rd/rd.cfm?id=020422FA-WiFi

    The Hacker Diaries

    No, I'm not making this up. The book exists and I have it on my desk here. The subtitle is even more interesting: "confessions of teenage hackers". Now normally I would not take this serious but the publisher Osborne is a respectable outfit. Then I started reading. This is about the 14-year old kid that brought down Yahoo, CNN, Datek and E-Trade! So, if you want to stop hackers, better start thinking like one. Entertaining reading and "job enhancement" in-one: