Sunbelt W2Knews Electronic Newsletter
The secret of those "who always seem to know" - Over 500,000 Readers!
Mon, Apr 22, 2002 (Vol. 7, #32 - Issue #363)
Sunbelt Unscheduled Downtime: Postmortem
This issue of W2Knews contains:
- EDITORS CORNER
- Sunbelt Unscheduled Downtime: Postmortem
- TECH BRIEFING
- "OK, This Is What Happened". By The Man Who Did It.
- NT/2000 RELATED NEWS
- What Is The Source Of 2005 W2K Support Stop?
- HotFix Hint #1
- HotFix Hint #2
- NT/2000 THIRD PARTY NEWS
- How UpdateEXPERT Checks For HotFixes
- Comprehensive Group Policy Management With Fazam 2000
- Opalis And Sonic Mobility Work Together
- W2Knews 'FAVE' LINKS
- This Week's Links We Like. Tips, Hints And Fun Stuff
- PRODUCT OF THE WEEK
SPONSOR: Wireless Microsoft Enterprise Manager
The #1 Wireless Systems Management Tool, Now Starting at $149.
Limited Time Offer for W2Knews Subscribers only! Save Time while keeping
your systems running at top performance by managing them anytime and
from anywhere, using just your web-enabled mobile phone, PDA (PalmOS/
PocketPC/Symbian) or RIM Wireless Handheld. With StarAdmin, you start
working on system problems immediately because the MMC and all other
things like processes, services, events logs, etc), User Management,
File System, SQL, Telnet, SSH, and much more are in your hand. Demo
StarAdmin on PocketPC, RIM, or WAP. Download a 15-day eval copy now:
Visit Wireless Microsoft Enterprise Manager for more information.
Sunbelt Unscheduled Downtime: Postmortem
We were off the air for just under half an hour last night. Despite
all the precautions we take. We lost some data as well. Ouch. The
Tech Briefing explains what happened and how YOU can prevent this
from happening. In the mean time, the links to W2Knews stopped
working for a while, but we reconstructed them. If you tried to
read your W2Knews but could not get anywhere and found yourself
in an old W2Knews issue instead of the link destination, our apologies.
Here are all the links that now work again. And learn from the Tech
Briefing so this does not happen to you!!
Microsoft Patch Management? It's A Mess
Gartner: Securing Windows Takes 15% Longer Than Unix
Already Own One Half Of The Bundle? The Other One Now Discounted!
NEW: ScriptLogic V4.0- No More Messing With Logon Scripts!
PentaSafe OnLine Security Library Opened
Throwing More Hardware At It: "Storage Insanity"
Self Service For User Passwords!
The "HammerOfGod" site has a few good utilities
A new Linux desktop that's a spitting image of WinXP
New threats force intrusion-detection vendors to rearm.
Cool & scary at the same time: Key Katcher- record all keystrokes and URLS
Here is something to gross out some of the girls in the office
And when you think that your day is bad, take a look at THIS man's Monday
morning. This is one of the best ads for furniture I have seen in years and
every time I see it I'm in stitches. Forward this one to your friends!
Book Of The Week: Hacker Diaries
UNDO Dept: Last issue I said that ScriptLogic generates KiXtart. Bzzzzz.
Wrong! It is not a "code generator" nor a "graphical wrapper" for KiXtart.
Rather, ScriptLogic is a turnkey product that utilizes the KiXtart language.
Since the ScriptLogic engine is compiled and the ScriptLogic Manager does
not generate raw code, it cannot accidentally generate syntax errors: the
type of problem that could easily shut down your enterprise one morning.
(email me with feedback: [email protected])
Sunbelt Remote Admin Now Comes With "HelpDesk License"
You might want to give Remote Admin a try too. It's as if you are there,
with all the tools to control multiple remote systems. This puppy was
"made-by-and-for" system administrators: super fast and works great for
telecommuters. Pricing is very low: Just 35 bucks for Client/Server! (and
dirt cheap site-, helpdesk and corporate licenses). Available with instant
online delivery and mainframe quality Sunbelt Tech Support. Get your (eval)
Visit RAdmin for more information.
"OK, This Is What Happened". By The Man Who Did It.
Last night at about 6:15pm we started working on the SQL Server that is
the backend of our website. The purpose of the maintenance was to add 2
additional drives and reconfigure an existing set of drives that had been
put in temporarily for storage purposes. The drive configuration to begin
with was as follows:
Drive 0/1/2 = Array 0 = RAID 5 array with the OS and SQL installed
Drive 4/5 = Array 1 = RAID 0 stripe for temp storage of database backups
The addition of the two new drives, drives 6/7 as Array 2 in a RAID 1
configuration, went without any problems. After they had been setup
and installed the members database was copied from Array 0 and attached.
Then came the procedure of recreating Array 1 as a RAID 1 array. After about
90 minutes of time spent working within the raid controller software,
working in the raid controller's BIOS, and reading the colorful (but
useless) documentation for the raid controller software things went bad,
Luckily we had replicated all the website SQL data to our co-location
servers in Texas, so we told IIS to redirect everyone over there, and
we got to work fixing things.
In the BIOS of the RAID controller I had selected Array 1 and hit
initialize, nothing else had worked up to this point. After being
prompted if I wanted to initialize and saying yes a progress screen
popped up showing Array 0 and 2 being initialized, not Array 1. The
initialization took about 1 second to complete and that was the end of
any data that had existed on Array 0 and 2. As an FYI for those that
care, the RAID controller in question is a Dell PERC2 with an AMI
chipset. Up to this point I had been less than thrilled with the AMI
based controller, the Dell PERC2 Adaptec controller had always seemed
to be a superiorly designed controller.
After the RAID controller finished its business a reboot confirmed that
it had in fact completely wiped both of the crucial arrays, time to call
Dell. Call their support line, punch in the system service tag, follow
the prompts for a PowerEdge 6400 server support. After holding for a
few minutes shy of a full hour a tech finally came on the line. The
conversation in a nutshell went along the lines of:
#1. If the drives initialized there is nothing you can do, grab a backup.
If your backup isn't enough find a place that does data recovery.
#2. You must have selected an option to initialize all the drives.
#3. In order to remove an array on that type of controller you have to
clear the configuration and then recreate the containers.
Needless to say this was a completely useless call and we started to
rebuild the SQL server. Since I had opportunity to recreate the arrays
I once again went into the controller BIOS. Now that I could run from
a clean slate I setup the following configuration:
Drive 0/1 = Array 0 = RAID 1 mirror for OS, SQL binaries,
Drive 4/5 = Array 1 = RAID 1 mirror for most of the other databases
Drive 6/7 = Array 2 = RAID 1 mirror for members database
While setting up these arrays I did confirm for my own piece of mind
that there is no way that I could have selected all the drives for
initialization by accident, the controller had in fact just freaked out.
Then we reinstalled the OS, installed SQL, and ran through all the
patches and updates. Then we went to restore the database backups
from tape. Now for the second surprise, the most current database
backup was not data from Thursday morning. The data was from Wednesday
morning. Some quick research explained what happened. On Monday the
nightly integrity checks and backups had been changed from 11pm to
This was because the performance of the site would become extremely
poor during this time. However the backups still ran at 1 am and
completed backing up the SQL server at 2:25 am, missing the SQL backup
files by just over an hour.
As of about 1 am this morning everything crucial was restored but any
changes since Wednesday at 3:30am were lost. Augh! This included the
last newsletter, database processing done in the mean time, subscribe
and unsubscribe requests.
Today's project is going to come up with policies and procedures to
keep this sort of incident from occurring again. Last time we got burned
by a RAID failure was with the firewall and that was resolved by getting
2 PIX firewalls that have been > 99.999% reliable.
NT/2000 RELATED NEWS
What Is The Source Of 2005 W2K Support Stop?
A bunch of people asked me that, because it cannot be found on any
of the know MS webpages for that. Well, this is kinda new news and
was disclosed by Gartner vice president and research director Tom
Bittman at their "Nothing but .Net" Conference. The people from
SearchWin2000 were there and reported on it. Here is the source:
HotFix Hint #1
When I run into a hotfix that doesn't support the -z switch, I just
wait until the hotfix finishes installing. Then, when the dialog box
pops up telling you to reboot, go into task manager and kill the
hotfix. e x e process. You can then continue installing other hotfixes
with the same method and then run Qchain and reboot.
HotFix Hint #2
"The Patch Crashed My Server"... You sometimes run into this apparent
situation. Apply patch, reboot, problems! Would be easy to jump
conclusions, but hold your horses for a moment.
Russ Cooper pointed this out the other day, and MS has told me as well,
when a patch appears to cause a problem with your server it's more
likely that it wasn't the patch that caused a problem, it was something
else prior to the patch that caused the problem. So, the trick here
would be to reboot that server once before you apply the patch(es).
This would show you if any problems were there but yet undetected.
But, if you really run into troubles with a patch, call PSS, Microsoft
Product Support Services. They will help you out.
THIRD PARTY NEWS
How UpdateEXPERT Checks For HotFixes
Some anonymous hacker trying to make a name for himself alleged that
UpdateEXPERT (UE in short) is insecure. This is false data. Here is
a short explanation that shows UE is a perfectly valid tool. Thanks
to Marc Maifret from eEye for his comments from which I have freely
borrowed. While UpdateExpert could be using improved methods of checking
for installed patches, it is a perfectly secure product.
There is a problem that all too often people release "advisories"
which mislead people into thinking that software is "insecure" when
the reality of the situation is that the software is actually perfectly
fine. It is hard sometimes to weigh what makes software secure or not.
You must assess the possible threat level, if there even is one.
The "problem" pointed out in UpdateExpert is that UE uses the windows
registry hotfix keys as a method to see if patches are installed or
not. It is possible that an attacker who has already penetrated the
system, changes the registry keys and makes it appear as if a system
is fully patched, or unpatched.
So the "vulnerability" would be that if you compromise a system, and
then gain Administrator or SYSTEM level access (which is required
to alter the registry keys) that you could then fool UE into thinking
a patch is installed when it is not, therefore leaving the system
Once again, you have to have administrator/SYSTEM level access to
be able to perform this attack. It should be a rule of thumb that
any "advisory" that claims "you need admin/sys access to perform the
attack", be regarded as a bogus advisory done by amateurs looking
to get their name out.
Is this a vulnerability in UE? Nope! If UE had implemented their patch
checking in a different manner would it have made it harder for people
to trick UE? No. You see, no matter what method of patch checking you
choose to implement, if you compromise a system then it is TRIVIAL to
make it seem as if a patch is installed or not.
It does not matter if you are checking the registry, file versions,
crypto hashes, etc. All of that can be manipulated or altered once
you are the administrator or SYSTEM. The fact of the matter is that
this is a not a security vulnerability in UE. If anything it is a
usability/reliability feature enhancement request for a future version
of the product.
The only reason it should even be thought of as a feature enhancement
is in the rare case that an administrator deliberately goes and modifies
the registry keys and therefore that breaks the results of an UE audit
of your system for patches. This will be a rarity though.
Retina, the network security scanner, in fact does use the registry
method (same as UpdateExpert, same as Windows Update) to be able to
see if a patch is installed or not. However, it also tests for the
vulnerability as existing or not, from a network layer perspective.
That way the system is a bit more foolproof.
So in the case of an IIS unicode attack, Retina checks the registry for
the patch, and it checks the web server itself via network queries, to
see if it is vulnerable. This extra step that Retina does in truly
auditing for the vulnerability, as a hacker would, is something that
just makes Retina a bit more thorough. Retina as a product has a
different goal than UE which is why they work differently in checking
for system vulnerabilities and the two complement each other nicely.
Retina can find holes a bit better but doesn't do anything with patch
management or deployment as UE does. Hence the bundle that you can find at:
Comprehensive Group Policy Management With Fazam 2000
When you implement AD, you need tools. FAZAM 2000 Version 3 provides
complete Life-Cycle Management for Group Policy. It builds on Microsoft's
native Windows 2000/XP infrastructure to provide the most comprehensive
Group Policy management solution on the market.
Version 3 is due next month. Customers love FAZAM 2000 version 2 but
because GPOs are so powerful, they asked FullArmor to come up with a
way to manage the entire change process. The key feature is the 4
W's - who, what, when, why. Also it was a site that is managing over 90,000
AD users who asked for these features.
FAZAM 2000 Version 3.0 includes Change and Release Management, a critical
best practice for implementing Group Policy. You can track the history of
Group Policy Object (GPO) changes, including who made the changes, from
initial design to retirement.
Version 3 New Benefits:
Comprehensive GPO Management
- Tracks changes to GPOs
- Provides version control for GPOs
- Allows new or changed GPOs to move into production only after being tested and approved
- Eliminates the risk of making changes in a live, production environment
- Prevents two or more individuals making simultaneous changes to the same GPO
- Reduces risk when delegating GPO administration.
Version 3 New Features:
Get an eval copy of 2.0 now and get a taste of the power!
- 4 Ws of GPO Change: Version history for any GPO is available on-line
and shows who made the change, what was changed, when the change was made,
and why the change was made. For any entry in the GPO history, FAZAM 2000
can produce a full report of the GPO as it existed at that point in time.
The report may be viewed online, printed, saved as an HTML file or saved
as a MS Access file.
- Check-out and Check-in: GPOs must be checked out before editing. Only
the person that checked out the GPO may edit it
- Difference and Comparison Reports: Determining differences and comparing
GPOs is automated. A GPO can be compared to: another version of the same
GPO; an archived version of a different GPO; or to the version of the GPO
that is in live Active Directory. The reports can be viewed online, printed,
saved as an HTML file or saved as a MS Access file.
- Off-line Changes: GPO changes are made off-line and can be subject to
normal testing, approval and release processes
- Delegation: Authority to create and change GPOs can be limited preciously
so that GPO administration can be delegated with reduced risk
- Rollback: GPO rollback allows a prior version of a GPO can be put back
- Documentation: GPO documentation may be viewed on-line, printed, saved
as an HTML file or saved as a MS Access file
- Replication and Synchronization: GPOs can be easily replicated and
synchronized from domain to domain and across forests ? even disconnected
forests ? for consistency
- Baselining: GPOs can be baselined so that it is always possible to go
back to the original settings
- GPO Health: Administrators can determine the health of their Group
Policy environment by running reports to discover GPO corruptions and
- GPO Reporting: Allows IT Administrators to view detailed reports on
GPOs in Active Directory through the MMC console or Web Browser
- Resultant Set of Policies: Provides Resultant Set of Policies (RSoP)
or the set of effective policies that apply to a user when logging on to
a machine. Also allows for enhanced ?What/if? Scenarios
- Policy-centric view of AD: Provides a view of Active Directory with
Group Policy links and filters
- Backup/Restore: Allows administrators to backup and restore individual
GPOs on a domain including filters and links
- Troubleshooting and Diagnostics: Provides administrators with the
ability to perform remote diagnostics from a central administrator console
- Search: Provides searching for GPOs and settings within GPOs
- Scripting: Provides scripting of the backup, import, and reporting of GPOs.
Opalis And Sonic Mobility Work Together
Opalis Software and Sonic Mobility today announced a partnership that
will enable users to directly and securely execute jobs and gather the
status of Windows Server processes via a wirelessly enabled PDA.
The partnership involves OpalisRobot task automation for Windows
environments and the SonicAdmin mobile system administration application
for PDAs. The combined solution, which is planned for availability late
spring, will enable SonicAdmin users to extend server automation and
monitoring capabilities with OpalisRobot while managing jobs, servers
and network infrastructure remotely using their RIM Blackberry or Pocket
PC device. Network administrators will be able to view vital server
information such as reports, logs, statistics and alerts. They will
also be able to trigger OpalisRobot jobs with SonicAdmin.
This Week's Links We Like. Tips, Hints And Fun Stuff
A nice site that compiles all MS Security Bulletins.
Small racecars with even smaller cameras mounted on them. Fun to watch!
Getting into Wi-Fi wireless internet access, but cannot find the right antennas?
PRODUCT OF THE WEEK
The Hacker Diaries
No, I'm not making this up. The book exists and I have it on my desk
here. The subtitle is even more interesting: "confessions of teenage
hackers". Now normally I would not take this serious but the publisher
Osborne is a respectable outfit. Then I started reading. This is about
the 14-year old kid that brought down Yahoo, CNN, Datek and E-Trade! So,
if you want to stop hackers, better start thinking like one. Entertaining
reading and "job enhancement" in-one: