Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

McAfee Retracts Lowball Bug Damage Estimate

kdawson posted more than 4 years ago | from the had-our-fingers-crossed dept.

Security 233

bennyboy64 writes "McAfee has changed its official response [warning: interstitial] on how many enterprise customers were affected by a bug that caused havoc on computers globally. It originally stated the bug affected 'less than half of 1 per cent' of enterprise customers. Now McAfee's blog states it was a 'small percentage' of enterprise customers. ZDNet is running a poll and opinion piece on whether McAfee should compensate customers. ZDNet notes a supermarket giant in Australia that had to close down its stores as they were affected by the bug, causing a loss of thousands of dollars."

cancel ×

233 comments

Sorry! There are no comments related to the filter you selected.

XP SP3 (3, Insightful)

Enderandrew (866215) | more than 4 years ago | (#31955022)

I thought this affected anyone running XP SP3, which I expect would be a majority of enterprise desktops, not less than half of one percent.

Re:XP SP3 (4, Insightful)

SharpFang (651121) | more than 4 years ago | (#31955060)

I guess less than half of 1% of all corporate customers are customers of McAffee.
The right wording is everything.

Re:XP SP3 (1)

poetmatt (793785) | more than 4 years ago | (#31955390)

yeah, the media spin is strong with mcafee.

Reality? It affected everyone who has automatic updates on mcafee for enterprise, which roughly translates to a large majority of enterprise customers. Usually from a security perspective it's seen as bad form to not have updates available as soon as possible.

It also shows that mcafee's quality control is nothing short of crap. It's known that viruses do rename as svchost sometimes, but clearly they didn't test the heuristics here.

Re:XP SP3 (0)

Anonymous Coward | more than 4 years ago | (#31955718)

It only affected you if you were set to autoupdate right when the new DAT was released. No major threat announcements = set autoupdate a few hours behind release from McAfee; risk/benefit and others beta test the new DAT...

Re:XP SP3 (1)

EvilBudMan (588716) | more than 4 years ago | (#31955472)

I would guess there are more than that because of previous licensing. Luckily their licensing ran out on us and we switched to Norton since McAfee hasn't really done much since 2003. There enterprise stuff has really sucked for a while now but we had to wait to get out of the deal with them because of "you know" the economy.

Re:XP SP3 (1)

coniferous (1058330) | more than 4 years ago | (#31955742)

I really wouldn't trust Norton any more then McAfee.

Honestly - I don't know what the right answer for a corporate entity is... There is just something really scummy about both companies that I don't like.

Re:XP SP3 (2, Informative)

Enderandrew (866215) | more than 4 years ago | (#31955814)

Microsoft Forefront is what I'd suggest.

Re:XP SP3 (2)

coniferous (1058330) | more than 3 years ago | (#31955886)

I really like MS Security essentials... I hate to say it.. but I actully do trust Microsoft much more then McAfee and Symantec. I would try this out in a heartbeat.

Re:XP SP3 (1)

berwiki (989827) | more than 4 years ago | (#31955852)

Trend Micro has performed well for us.
Kaspersky is *supposed* to be the up-and-comer, but we've had our share of issues with it.

But none of them are immune from a rushed update.

Re:XP SP3 (1)

ircmaxell (1117387) | more than 4 years ago | (#31955066)

Well, it depends. How many have their computers set to pull updates hourly? If you pulled the updates daily, and it was released an hour after you checked, you were fine (considering they pulled it the same day). So the only computers affected were those that polled in the several hour window that the update was available (Something like 8 hours IIRC). And that's not to mention those configurations that are set to pull updates weekly or more.

Re:XP SP3 (1)

kiehlster (844523) | more than 4 years ago | (#31955150)

You should also add to this the statistic of how many corporations use their own distribution server (middleman). Even if clients poll daily, the corporation as a whole may only deliver updates weekly or may stagger updates to ensure they are tested in the wild before pushing them out to corporate clients.

Re:XP SP3 (2, Informative)

Jazz-Masta (240659) | more than 4 years ago | (#31955274)

You should also add to this the statistic of how many corporations use their own distribution server (middleman). Even if clients poll daily, the corporation as a whole may only deliver updates weekly or may stagger updates to ensure they are tested in the wild before pushing them out to corporate clients.

Not only this, but many Administrators manually review virus' before they are cleaned. I have caught a few false positives by doing manual checks.

Only under certain conditions. (1)

khasim (1285) | more than 4 years ago | (#31955126)

Well, one condition - that the v8.7 McAfee app scanned the svchost.exe file of a WinXPsp3 machine.

Which could happen under three situations:

1. You manually launched a scan.
or
2. A scheduled scan launched.
or
3. A setting in your policy said "scan processes on enable".

Re:Only under certain conditions. (1)

Enderandrew (866215) | more than 4 years ago | (#31955170)

In most enterprise environments McAfee is going to have real time protection against running processes. Can you point me to an enterprise environment where this wouldn't be the case?

Yep. (1)

khasim (1285) | more than 4 years ago | (#31955282)

In most enterprise environments McAfee is going to have real time protection against running processes.

It is "real time protection" even if that setting is set to "off".

McAfee's documentation specifically mentions turning it off because there is a high processor utilization bug still in it. Although you'd need to read the "read me" file that came with the patches.

Other than that, unless you choose the highest security setting, it is off by default in a BRAND NEW VANILLA install. But not if you had upgraded from a previous version where it was set to "on" by default.

This is 100% McAfee's fault on so many levels.

Re:Only under certain conditions. (-1, Offtopic)

moogied (1175879) | more than 3 years ago | (#31956172)

Unix.
BAMN MOTHER FRAKKER. Super obvious answer to your god awfully ignorant statement.

Re:XP SP3 (2, Interesting)

GIL_Dude (850471) | more than 4 years ago | (#31955226)

It really depends on the intersection of folks running McAfee along with SP 3 in the enterprise. My company is just finishing a migration to Vista, but we still do have about 15,000 Windows XP SP3 desktops (not done deploying yet). However, late last year, I was at a MS Global Accounts meeting (35 very large companies) and NONE of the rest of them had deployed SP 3 for their XP machines. They were all on SP 2 and were harping on Microsoft about the end of support for SP 2 that was fast approaching. None of them wanted to deploy SP 3. It was flabbergasting to me, but they just didn't want to do it. So none of those companies were impacted - even if they ran McAffee.

Re:XP SP3 (1)

swb (14022) | more than 4 years ago | (#31955532)

None of them wanted to deploy SP 3. It was flabbergasting to me, but they just didn't want to do it.

Some fucktard in a suit gets told that they don't care about problems caused by not running SP3, running SP3 requires a bunch of money to get spent and if he spends it he doesn't get a new BMW 7 series this year.

Really, so many of these decisions have nothing to do with rationality. At some high level it comes down to some guy in a suit angling for a new car, a new house or some other luxury/status symbol.

Re:XP SP3 (1)

The MAZZTer (911996) | more than 4 years ago | (#31955286)

At my work we run XPSP3 and McAfee, had no problems here.

@WithinRafael on Twitter (from www.withinwindows.com) was trying to reproduce it and had problems, I think he recently succeeded but hasn't provided details yet.

Re:XP SP3 (1)

coniferous (1058330) | more than 4 years ago | (#31955808)

He tried to reproduce it and had problems? The summary of the problem made it seem like all svchost.exe's would get deleted no matter what.

I wonder what sort of specific conditions had to be met? Not that I like coming to the defense of McAfee... But has this been overblown?

Re:XP SP3 (0)

Anonymous Coward | more than 4 years ago | (#31955636)

It knocked out roughly 3500 of Suncor's North American Desktop computers. They have 2000 fixed as of this morning and there are still enough computers knocked out to impact daily work.

Re:XP SP3 (1)

proxima (165692) | more than 4 years ago | (#31955686)

I thought this affected anyone running XP SP3, which I expect would be a majority of enterprise desktops, not less than half of one percent.

You had to be running versions 8.7 or 8.9 it seems to be affected. 8.0 or 8.5 did not exhibit this problem, even if the virus definitions were updated to 5958.

It wouldn't surprise me if the enterprise rollouts of McAfee often used 8.5 (released in Nov 2006) rather than 8.7 (released in Sep 2008) or newer.

Re:XP SP3 (1)

Col. Klink (retired) (11632) | more than 3 years ago | (#31955924)

Presumably at least a few enterprise customers have enough brains to internally test updates before rolling them out. I expect McAfee doesn't consider those customers "affected".

Re:XP SP3 (1)

Piranhaa (672441) | more than 3 years ago | (#31955984)

Everyone that received the patch running XP SP3, yes. However, where I work, they download the patches in the morning and deploy them later on in the evening. So yes, there is a window of attack there, but it saved us from having to go through every SP3 machine and copying the deleted OS file. Basically, everyone else that gets the patches instantly are 'our' guinea pigs.

Really? (2)

ircmaxell (1117387) | more than 4 years ago | (#31955024)

ZDNet notes a supermarket giant in Australia that had to close down its stores as they were affected by the bug, causing a loss of thousands of dollars.

A chain of supermarkets close down, and they only lose thousands

of dollars? Really? I would expect that figure to be a lot higher than that for a single store... Think about all the fresh produce that'll go bad (that have daily deliveries). Think of the power usage (lights, refrigerators). And that's assuming that they aren't paying any of their employees while the store is closed. I'd imagine the loss would be on the order of tens of thousands of dollars per store. Not thousands of dollars across all of the stores...

Re:Really? (0)

Anonymous Coward | more than 4 years ago | (#31955244)

Tens of thousands is still thousands... just more of 'em.

Re:Really? (5, Funny)

pinkj (521155) | more than 4 years ago | (#31955296)

Maybe Australia only has one big grocery store somewhere in the Outback. Kinda of like what we have in Canada except it's a giant igloo in northern Toronto.

Re:Really? (-1, Redundant)

Anonymous Coward | more than 3 years ago | (#31955912)

Mod parent up! Funny:5

Re:Really? (1)

kiehlster (844523) | more than 4 years ago | (#31955354)

I would think the same, but it could be a discount supermarket with really low profit margins on dirt-cheap products from second-rate suppliers. We have a chain like that in our area where they leave out the produce until it gets moldy and then offer a replacement guarantee. So if you're 5-day old fruit turns moldy on you, you can return it, but they don't have to toss out as much because people tend to use the fruit within a day or two of purchase. If this was a reputable supermarket, I could see shorter shelf-lives affecting them more, but for discount supermarkets with cheap computer systems they don't care and just leave the food out for an extra day.

Re:Really? (2, Interesting)

Cimexus (1355033) | more than 4 years ago | (#31955498)

Nah - this is Coles. That'd be one of the "big two" Australian grocery retailers, with thousands of stores nationwide. I expect that 'loss of thousands of dollars' was many, many thousands (either that or it only affected a very small number of stores for a very small time before getting fixed).

Actually I used to work at Coles (it was my first job!). Our store was the smallest one in the state but still had revenue of ~$300,000 a day...

Re:Really? (1)

eggoeater (704775) | more than 4 years ago | (#31955494)

Agreed. And that's just the immediate cost. When things like this happen, stores/businesses lose loyal customers to competitors and it takes months to recover.
And what about the IT costs? I guarantee you, there is now an effort underway in all major businesses to (1) test new anti-virus patches before rolling them out, (2) re-review all anti-virus software being used, (3) developing and testing mitigation plans for another failure. All of this is VERY expensive.
Here's another example: Airlines shut down because of a volcano. You think when the volcano stops that their business is going to go back to the previous levels? Nope. Even for something like airlines where people often don't have a choice, it will take quite some time to recover. 9/11 is another example of this; it took years for airlines to get back to pre-9/11 levels, although there were other economic factors that led to the decline in '01.

I'm still wondering ... (4, Insightful)

khasim (1285) | more than 4 years ago | (#31955050)

... why they didn't test the new dat file against Windows system files.

Seriously, we pay them a LOT of money for their product licenses and they cannot even test against known system files?

Re:I'm still wondering ... (1, Funny)

Anonymous Coward | more than 4 years ago | (#31955146)

Maybe it was a "reminder" so we don't get complacent about license renewal?

"Gee, that's a nice operating system ya got there..."

Re:I'm still wondering ... (1)

bstreiff (457409) | more than 4 years ago | (#31955378)

Not every XP SP3 machine was bitten. There were some XP SP3 machines here that were affected, but just as many that weren't.

It's possible that they did test against XP SP3, and just got 'lucky'.

No, not possible. (1)

khasim (1285) | more than 3 years ago | (#31956100)

Not every XP SP3 machine was bitten. There were some XP SP3 machines here that were affected, but just as many that weren't.

There's no magic here. They have a signature that matches a specific version of svchost.exe.

They did not test the scan engine with that dat against that version of the file.

That's all it is.

I wonder (2, Interesting)

mr_da3m0n (887821) | more than 4 years ago | (#31955090)

...If McAfee has a clause in their EULA somewhere that limits their responsibility, and should that be the case, if it is legally enforcable.

Maybe someone with access to said EULA could look it up?

Microsoft once pushed their accountability as a selling point for the Windows Server platform against Linux, if I recall well -- however their maximum responsibility was something like 50$. I wonder what is McAfee's stance in this regard.

Re:I wonder (2, Informative)

ProdigyPuNk (614140) | more than 4 years ago | (#31955520)

Here's an online version of their EULA: http://home.mcafee.com/Root/AboutUs.aspx?id=eula [mcafee.com] Of course there's a limited liability clause: Limitation of Liability UNDER NO CIRCUMSTANCES AND UNDER NO LEGAL THEORY, WHETHER IN TORT, CONTRACT, OR OTHERWISE, SHALL MCAFEE, OR ITS AUTHORIZED PARTNERS OR SUPPLIERS BE LIABLE TO YOU OR TO ANY OTHER PERSON FOR LOSS OF PROFITS, LOSS OF GOODWILL, OR ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, OR DAMAGES FOR NEGLIGENCE OF ANY CHARACTER INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF GOODWILL, WORK STOPPAGE, LOSS OF DATA, COMPUTER FAILURE OR MALFUNCTION, OR FOR ANY OTHER DAMAGE OR LOSS. IN NO EVENT SHALL MCAFEE, OR ITS AUTHORIZED PARTNERS OR SUPPLIERS BE LIABLE FOR ANY DAMAGES IN EXCESS OF THE PRICE PAID FOR THE SOFTWARE, IF ANY, EVEN IF MCAFEE, OR ITS AUTHORIZED PARTNERS OR SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. This limitation shall not apply to liability for death or personal injury to the extent that applicable law prohibits such limitation. Furthermore, some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this limitation and exclusion may not apply to you. Nothing contained in this Agreement limits McAfees liability to you for McAfees gross negligence or for the tort of fraud. McAfee is acting on behalf of its suppliers and Authorized Partners for the purpose of disclaiming, excluding and/or limiting obligations, warranties and liability as provided in this Agreement, but in no other respects and for no other purpose. The foregoing provisions shall be enforceable to the maximum extent permitted by applicable law. And under warranties: Warranty Disclaimer. Except for the limited warranty set forth herein, THE SOFTWARE IS PROVIDED "AS IS" AND McAfee MAKES NO WARRANTY AS TO ITS USE OR PERFORMANCE. EXCEPT FOR ANY WARRANTY, CONDITION, REPRESENTATION OR TERM THE EXTENT TO WHICH CANNOT BE EXCLUDED OR LIMITED BY APPLICABLE LAW, (The warranty they give is basically just for defective install media). It's rather telling if you look at the selling points on their website, and then look at the EULA.... I understand that most places have EULA's like this now, but they aren't standing behind their product when it comes down to it one bit.

Re:I wonder (1)

The Wooden Badger (540258) | more than 3 years ago | (#31955988)

I think they would still have a case even with that EULA. A victory for McAfee in the courts would set an incredibly bad precedent.

McAfee (1)

rocket97 (565016) | more than 4 years ago | (#31955094)

I don't know which one anymore I dislike more, McAfee or Symantec. I stopped using both several years ago, I not run Avast Home on my gaming system at home.

Re:McAfee (5, Funny)

Anonymous Coward | more than 4 years ago | (#31955292)

I, too, not run Avast Home. Me switch to MS Security Essentials.

Re:McAfee (1)

rocket97 (565016) | more than 4 years ago | (#31955328)

I not sleep last night. Tarzan sleepy.

Re:McAfee (0)

Anonymous Coward | more than 4 years ago | (#31955326)

Why not just use Microsoft Security Essentials? It's free for personal use, isn't nagware like AVG (dunno about Avast), and is probably as lightweight as you can get for a real-time scanner. When you're the company writing the low-level system APIs, you probably know how to use them in the most efficient way.

Anything remotely fishy gets run in a disposable VM anyway.

Chile affected (0)

Anonymous Coward | more than 4 years ago | (#31955098)

In Chile, this bug affected operations of the judiciary systems. They had to suspend hearings and other proceedings for the day.

Delayed update (1)

z4ns4stu (1607909) | more than 4 years ago | (#31955124)

I would bet that the reason the affected numbers are so low is because a large number of corporations know to delay the application of patches for at least a day. This isn't the first time McAfee has done this, and it definitely won't be the last. It's the same concept with Microsoft/Apple/other OS patches. Every organization needs a patch strategy and the good ones include some kind of lab environment to make sure stuff isn't going to break before it's rolled out.

My employer dodged a bullet on this one. (1)

Nadaka (224565) | more than 4 years ago | (#31955134)

Everything here is windows xp sp3 with McAfee installed.

Fortunately for us, all software updates are filtered through and managed by an internal server due to security restrictions on some of the work we do for the government.

Re:My employer dodged a bullet on this one. (1)

Mister Whirly (964219) | more than 4 years ago | (#31955666)

all software updates are filtered through and managed by an internal server due to security restrictions on some of the work we do for the government.

And this is a perfect example of why an internal server to distribute updates is a Good Thing(TM). Hey, the government got something right!

Re:My employer dodged a bullet on this one. (1)

chill (34294) | more than 3 years ago | (#31956242)

Hey, the government got something right!

Whoa there, pardner! Before jumping to any wild conclusions, re-read what he said.

...on some of the work we do for the government.

That most likely means contractor, not actual government employee.

The gov't didn't do something right. The world is not going to end. Moped Jesus was not spotted on I-55 heading west.

Necessary Evil (2, Interesting)

RayRuest (1417225) | more than 4 years ago | (#31955168)

It could only effect that few if the policies were set up update infrequently (ever few days or so). My policies are set to check for updates and push them frequently, so I got bitten. I have less than 100 desktops but am a 1 person shop. 4 hours of sneaker net repairs and corporate downtime. Thanks McAfee. There was at least 1 hospital in the area that had to resort to turning non-critical patients away. Don't these things get testing before release? These products are a necessary evil... they don't need to be more evil than the purpose they are attempting to provide.

Re:Necessary Evil - bull (1, Interesting)

Anonymous Coward | more than 3 years ago | (#31956214)

Don't businesses run their own update server and categorize, verify, and deploy those updates based on what software THEY have running?

If you're telling me that a hospital IT system is setup to take any and all updates directly from vendors( McAfee, Microsoft, etc ) all I can say is they get what they deserve for doing that and it's nobodies fault but their own. Let me guess, this is how most Windows shops are run these days and that is why Windows admins cost much less than *nix admins. IMO

so 4 hours of corporate downtime for this one issue. And why do you not have a few machines configured to represent your standard corporate computers and run the updates on them before expecting some other company to have tested their update with _your_ software configuration? Does Microsoft Windows not give you the power to push out updates locally? The very first time I setup a classroom configuration using Linux it dawned on me that I did not want every computer doing auto updates so I mirrored the Ubuntu repo, setup a cron to keep that updated, and configured all the lab computers to pull from a secondary local mirror where I'd move updates over as they got tested. dah.

LoB

In the many millions of dollars... (1)

mother_reincarnated (1099781) | more than 4 years ago | (#31955192)

Heck I was at a small IT security trade event yesterday and like a quarter of the attendees had to cancel because they were dealing with the aftermath...

McAfee had almost a 50% corporate AV market share, and nearly all of those companies still run many XP SP3 boxes. If 10% pulled the DAT before it was yanked, that's a metric buttload of machines...

My estimate (1)

Monkeedude1212 (1560403) | more than 4 years ago | (#31955212)

Is that it would only take 1 oil and gas company who usually handles Million Dollar deals. Lets see.
International Corporation... Lets say 3000+ Employees... lets say just half the company goes down. Rule of thumb is 1 IT guy for every 100 computers (but we all know thats in a perfect world).
So, the simplest way to get out of downtime is to go into safe mode and disable the Antivirus, right? Lets say it takes on average 5 minutes to walk to each machine and preform the steps. 500 minutes, or 8.3repeating hours.

MCaffee basically put you out of business for the day.

You morons are still using Windows? (-1, Troll)

Anonymous Coward | more than 4 years ago | (#31955216)

Then you deserve crap like this.

Re:You morons are still using Windows? (1)

X0563511 (793323) | more than 4 years ago | (#31955436)

This was hardly the fault of Windows.

100% third-party problem, here... troll.

Not Windows' fault, but still its problem... (3, Informative)

Animaether (411575) | more than 3 years ago | (#31956016)

( Title after the VirtualDUB developer's excellent post entitled "Just because it is not your fault does not mean it is not your problem"; http://www.virtualdub.org/blog/pivot/entry.php?id=245 [virtualdub.org] )

Here's the thing.. it's not Windows' fault that some random program deletes svchost.exe , just as it isn't Windows' fault that any app or user can delete ntldr (e.g. a badly designed uninstaller).

But it -is- a Windows problem because without those, it won't start up. So why is Windows even allowing these files to be deleted?
I can't delete by hiberfil.sys even though all it is, is pre-allocated space for the hibernation functionality. If I deleted it, nothing would be lost, and upon hibernation it could re-allocate the required space or tell the user the drive is too full and they're SOL. But no - I simply can't delete it. But I -can- delete vital system files.

So, no.. it's not Windows' fault that McAfee's virus scanner deleted the file. It -is- Windows' problem that they -can- in the first place.

I realize that sometimes there may be a need for a 3rd party application to modify a system file - however rare - but then provide this through a proper mechanism that backs up the original and deletes/replaces on reboot only, with the option to deny the change on boot-up. ( System Restore points only go so far as you'll need the Windows CD/DVD in order to get to the restore utility if you can't boot into Windows anymore. It's also an overly complex solution to the simple problem of renaming files on bootup. )

WHAT???? (1)

FearKratos (1794192) | more than 4 years ago | (#31955222)

People still use McAfee for support, that's laughable.

Re:WHAT???? (2, Funny)

FearKratos (1794192) | more than 4 years ago | (#31955248)

Symantec is so much better.

Re:WHAT???? (0)

Anonymous Coward | more than 4 years ago | (#31955298)

Sure, if you don't love your RAM or processor. And have nothing against an AV scanner that's helpless against viruses, and whose tech-support team uses ComboFix instead of their own software.

Impact Probably Much Higher (1)

jonnyboy3us (1176709) | more than 4 years ago | (#31955246)

I imagine this impact was much higher than they stated. One of the small operations I support on the side called a couple of days ago about this issue when it cropped up. The Windows XP computer would not even allow him to do a system restore let aloneuse his computer. Luckily, we found out about the fix yesterday or it could have cost them a couple hundred dollars to fix. Along with the lost productivity time, this isn't a good thing for McAfee. While we use other solutions for our systems, this highlights how much testing needs to take place before a patch is deployed. It's amazing these types of 'issues' occur in today's world. Time for McAfee to step up QA.

Testing before deploying? (1)

adosch (1397357) | more than 4 years ago | (#31955264)

I've read a few interviewed accounts where the story was much like this:

We applied the updates, and rebooted, then I went on to kick off the others. When I went back to the first couple of servers, I noticed they had rebooted again... then I knew something was wrong.

I know things can't be 100% perfect in an IT world, and yes, virus definitions can be touchy when sometimes zero-day shit can really cause havoc, but I, myself, have of test boxen on my network that I test all patches/updates/virus definitions on for *NIX and Windows boxen. It's not perfect, because to test and interrogate everything is impossible, but I don't apply things blindly. And yes, I've had a few fallout where the package/patch/update applied fine, but there was a bug in it that affected something. But at least you had some comforting notion that you prepared as best as you could. It just is mind numbing that 1) things still get deployed blindly at the enterprise level and 2) for the amount we all in an IT organization fork out for trust and support from these companies for services and big fallouts are happening.

Re:Testing before deploying? (3, Insightful)

X0563511 (793323) | more than 4 years ago | (#31955468)

I know assumptions are bad, but is it really that big a stretch to assume the vendor tests their updates on their supported platforms?

It's not like these were weird corner-cases.

Re:Testing before deploying? (1)

alen (225700) | more than 4 years ago | (#31955554)

i've been using Winders since the mid 1990's along with AV software. I have never seen an issue where a definition update has caused something like this. i've seen plenty of times where you can't run an old version on a new OS or issues with games or some software. but letting something out like this into the wild just shows that there was no testing done just to make sure it's OK

AV on POS computer?? (4, Insightful)

wvmarle (1070040) | more than 4 years ago | (#31955288)

I feel sorry for that super market chain but: wtf is AV doing on a POS computer?

POS should be a dedicated computer, running one and only one application (the POS software), on a thoroughly shielded LAN, talking to only a centralised server (or small network of servers if one is not enough) that collects the sales data and distributes prices etc. That server should itself be connected only to the POS network and a corporate LAN. In other words: no direct access out of the Internet, no web browsing, no local storage of any data files, no downloading, nothing that could have the most remote risk of a virus.

Or am I missing something here?

Re:AV on POS computer?? (0)

Anonymous Coward | more than 4 years ago | (#31955522)

... how naive....

Many of these POS (piece of ...) devices are either 3rd party managed, or are connected via slow (dial-up) links to the central office.

Many of them run Windows XP or before (some still running '98 "until it breaks"), so AV is a good idea on many of these machines...

Re:AV on POS computer?? (3, Funny)

ifrag (984323) | more than 4 years ago | (#31955566)

Or am I missing something here?

That it was in Australia?

Re:AV on POS computer?? (1)

SmallMonkeyPirate (932116) | more than 4 years ago | (#31955568)

The Co-Op a huge grocery retail chain in the UK use XP based tills. I only noticed them because the customer facing part of them often displays Win errors :)

http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=4000001958 [microsoft.com]

Also I know of a large Radiography company whose X-Ray machines all had Dell workstations running XP inside..now that's scary.

Re:AV on POS computer?? (1)

JaCKeL 1.0 (670980) | more than 4 years ago | (#31955578)

This comic explain AV on POS : http://xkcd.com/463/ [xkcd.com]

Re:AV on POS computer?? (0)

Anonymous Coward | more than 4 years ago | (#31955588)

It's called a dream world, good luck achieving it. I would consider any windows based system not running Antivirus to be considered a red light district or petri dish just waiting to be infected.

Re:AV on POS computer?? (4, Funny)

Anonymous Coward | more than 4 years ago | (#31955612)

wtf is AV doing on a POS computer?

This setup also seems somewhat redundant, since McAfee's AV itself is a POS.

Exactly what I was thinking (2, Informative)

Freaky Spook (811861) | more than 4 years ago | (#31955674)

McAfee must have had a really good sales guy to convince a Project manager that the POS machines needed AV, either that or who ever developed the POS machines didn't decide to secure them with Enhanced Write Filter, SteadyState, DeepFreeze or some other disk write protection so every time the machine is rebooted it loses all its write cache.

Even though it is Windows, there is absolutely no need for AV when the application is so limited.

Re:AV on POS computer?? (2, Insightful)

EMG at MU (1194965) | more than 4 years ago | (#31955850)

I agree.
However, when you have 200,000+ POS machines, management wants an AV.
I hate McAfee, I hate using a AV instead of isolating a machine from removable media and the Internet. I hate spending money on AV when we could use it on something else. But when a franchise manager on the other side of the world lets one of his employees use the wifi or a printer or something, I'm glad there's an AV to protect my ass. Even though there shouldn't be a way the POS machines get a virus, the AV is kind of like car insurance: It protects you from accidents, costs too much money per year, someone else forced you to get it, and in the end when something shitty happens it kind of saves your ass.

Re:AV on POS computer?? (1)

Dragee (881700) | more than 3 years ago | (#31956060)

You're confusing "should be" vs. "real-world." Even if the actual POS machines are dumb terminals, they'd connect back to the server, which is probably Windows. Marketing Department and/or Customer demand says that more POS systems will be sold if Corporate can push out pricing updates to the servers and check inventory levels across the Internet, instead of having leased lines all over the place. And since there's Internet connectivity, you need Antivirus. The fact that POS machines aren't quarantined the way they should be isn't the worst of it. I walked up to my bank branch once, and the ATM was showing a WinXP BSOD.

Re:AV on POS computer?? (1)

Artifakt (700173) | more than 3 years ago | (#31956174)

Most small businesses that are service related have at least one Point Of Sale machine up front at their physical store, but the person operating it is also the person who makes appointments, so they just about have to be able to bring up a scheduler and appointment manager. A separate terminal for appointments is a serious cost, as would be keeping separate people to operate it, or training across skill sets (your cosmetologist or hair stylist or auto mechanic now needs to be trained to schedule appointments and take payments - and to log out or whatever every time they go back to what they think of as their main job - that's already likely to create an environment with much more serious security problems than having POS run other programs).
      When your auto mechanics make 25 an hour, the solution is to hire a receptionist at half that, but splitting that job to a clerk for payments and a 'customer service associate' for scheduling appointments just wiped out much of the savings you would normally get by keeping the more skilled workers on their primary jobs. Meanwhile customers are constantly going to the 'wrong' desk to try to do the 'wrong' task there (Yes, even if there's a big sign saying "Payments" hanging over one desk - many customers go to whichever one's closest, and they can't all be closest).
      I work for a large company, a fortune 500 - the 'skilled professionals' usually float between multiple offices to handle appointments (one of my areas of specialization is taxation and planning on author's often very irregular royalties, particularly as those of the deceased authors are being passed through LLCs set up for their heirs - you can imagine how that increases my normal operations range, but several of my co-workers have some area that they support, but for which the market is simply too spread out to draw all the customers into one location. I put up with the driving by scheduling several clients in the same area back to back, and because I often get free autographed copies as tips). Even for us, a secretary handling both appointments and payments on the same machine is common, especially where the physical layout of an office doesn't allow a separate entire room for a separate client manager to handle the appointment process. Plus, we have pros both constructing their billable hours statements and scheduling for themselves, to further link the processes.
      Since some of our pros are self scheduling at distant locations, our appointment manager actually runs as an Internet app! Of course, we have VPN, damned good encryption, and UNIX based systems between anything Windows and the net, but still, I can't see most small companies doing all those things, yet they may face the same sort of problems, and the others I have outlined above, and probably many more. Saying POS should always run on a dedicated machine by itself is saying that there has to be some other good solution to all these sorts of issues, and if you really have that, there are people who would gladly pay you at least 500 K a year to implement it.

Re:AV on POS computer?? (2, Interesting)

Scyth3 (988321) | more than 3 years ago | (#31956192)

Typically the POS desktops are talking directly to a server in the backroom. The server in the backroom is typically where a manager will check their emails (via Outlook), take training via a web site, etc. and it's also where the database for the POS client desktops is stored. Every night that small store server submits the data to a main server at the "home base". So, if the virus scan is on the server (typically is), and the machine goes down, then the business is effectively closed. It's not that the POS machines had a virus scanner on them, it's that the server does since it's used as a work machine for the manager as well. That's how one of the biggest auto part chains in the US operates. It wouldn't surprise me to see this elsewhere.

Re:AV on POS computer?? (1)

c (8461) | more than 3 years ago | (#31956256)

> Or am I missing something here?

Slavish adherence to corporate IT policies which require AV software on any system which can run it?

c.

Re:AV on POS computer?? (1)

Trailer Trash (60756) | more than 3 years ago | (#31956276)

You're missing nothing except one minor point: no POS system - or anything else in the chain - should be running Windows. This should be a non-issue. My advice to the Australian grocery chain is to fire whomever in the IT department thought this was a reasonable idea.

Which is more harmful? (4, Funny)

goffster (1104287) | more than 4 years ago | (#31955308)

McAfee or being part of a botnet?

Re:Which is more harmful? (0)

Anonymous Coward | more than 4 years ago | (#31955644)

What is more likely in most instances? If you are running a fairly well managed Firewall and IPS, then I would say that McAfee is more likely...
So it becomes a cost-benefit.
Delaying the AV signatures by ~10 hours would be a good balance, but if everyone did that, it would be useless...

I'm not sure what the right answer is... but I would factor likelyhood into the equation..

Re:Which is more harmful? (1)

JaCKeL 1.0 (670980) | more than 4 years ago | (#31955732)

I think by definition, Mcafee and other AV are not very far from a Botnet. Hackers could find a way to exploit their update systems and put down MILLIONS of machines.

Re:Which is more harmful? (1)

TheVelvetFlamebait (986083) | more than 3 years ago | (#31955940)

What, no Cowboy Neal option?

Getting real about things here (4, Interesting)

onyxruby (118189) | more than 4 years ago | (#31955380)

First, McAfee blew this big time, that such a bug made it to production shows a complete breakdown in their internal processes. XP with SP3 is the number one OS combination in enterprise environments, and should have been the first thing that they tested on. Without doubt McAfee has liability on this and needs to get aggressive about damage control with clients.

That being said, every one of these clients that was hit by this is just as guilty as McAfee is! They are in no better shape and those responsible need to be going management review for their failure. Enterprise Management 101 - nothing goes into production that has not been tested in a lab for pre-pilot and a small group of production computers for pilot! This is as basic as enterprise management gets. Every single environment that was taken down by this shows professional incompetence by their requisite IT departments.

The only question is if it is the fault of management for failing to allow the budget and support needed for a lab for testing or if it is the fault of the IT staffer who never tested things as they should. This is without doubt one of the most public examples of IT incompetence to make the news in years. This is a case of sheer and utter incompetence by every affected party and no pity should be given. If pity were to be given, give it to the poor desktop techs that have to go around making apologies and manual fixes for everything.

Re:Getting real about things here (0)

Anonymous Coward | more than 4 years ago | (#31955594)

I would kill to mod you up.

how somebody deploys something into a mass production environment without even testing it on a few virtual machines, is beyond me!

I'd love to claim that "I had deployed the update in a controlled environment and watched it kill things, thus deciding to not deploy"

but I've never even met an admin that seriously uses McAfee in a production environment. that's like deploying a d-link 8 port switch to "add ports" to the 3550.

Re:Getting real about things here (1, Informative)

Anonymous Coward | more than 4 years ago | (#31955784)

How is this also the IT departments fault? This bug was in a virus definition file (DAT file) not a application update. Do you expect offline lab testing of every singe virus definition file that is released? Do you realize that there is a new definition file released at least once a day and sometimes up to 3 per day? If you have the time to test each one in a lab great. But who's fault is it when while you are "testing" in the lab a new worm spreads through your corporate network?

We use McAfee in our environment (6000 PC's) and were not affected due to running version 8.5 of the software, apparently only 8.7 clients had the issue. Just to recap the bad DAT file was released 4/21 at 6 AM PST, in our environment we look for and pickup DAT files every hour and update the clients automatically on a staggered schedule. By the time we were made aware of the issue via a email from our McAfee rep. (4/21 9AM) 2500 of our PC's already had the bad dat file, if we would have been impacted by the bug we would have been screwed.

I do agree that McAfee has quite a bit of explaining to do and also will nee provide some type of compensation for companies that were impacted by their screw-up.

Re:Getting real about things here (4, Informative)

onyxruby (118189) | more than 3 years ago | (#31955936)

As a matter of fact I do expect that. I have designed and set up processes for patch management, software distribution and similar testing for large enterprise environments for years. I have done so everywhere from very large financial institutions to health-care and government. The fact that you need to test daily does not change any principal of what I have said. For any enterprise not to have a dedicated lab to do exactly this kind of testing, or ever worse, not to to use it is sheer and utter incompetence.

In no case should an automated update for an environment ever be released into production without testing. Even Microsoft gets this point and allows you to disable automatic patching to ensure that proper testing can be conducted. I'm not trying to sound harsh, but in all seriousness if you can't learn why testing /every/ production change is necessary from this debacle, than you do not belong in enterprise management. It really is that simple.

Re:Getting real about things here (0)

Anonymous Coward | more than 3 years ago | (#31956024)

I agree. While the anti-virus software should have been tested better before it was pushed to customers, the IT departments should have tested the deployment on a test machine before rolling out the update. Granted, that doesn't help really small shops or home users, but medium and large IT groups should have seen this coming.

Re:Getting real about things here (1)

idjohnston (1350181) | more than 3 years ago | (#31956058)

Up until this week, I was actually wondering why in the hell we hadn't upgraded to SP3, now I'm extremely glad we're not. All of our McAfee just went to 8.7i a week ago, this would have been an insane hit on us. Especially since the one member of our IT Security team is on vacation! I wouldn't have been surprised if he had turned on auto-updates before he left since he wouldn't be around to lab test them.

Re:Getting real about things here (1)

Kimen (1594743) | more than 3 years ago | (#31956098)

You would only be correct for the clients where they are a large enough organization to justify their own testing and deployment infrastructure. However, I suspect a very large number of customers are small enough that they just let the software update itself on a routine schedule and do not have the resources to build a complete IT test and deploy infrastructure. In that regard, the smaller McAfee clients had no responsibility for the failure at all.

Made quite a mess of some college networks, too. (5, Interesting)

ProdigyPuNk (614140) | more than 4 years ago | (#31955386)

A buddy of mine is in IT at a college in the area. This affected almost all of their computers. Although it's harder to put a dollar figure on, the students and professors were NOT happy when all of the computer labs on campus went down, along with a "server" or two. Ever seen professors gets mad ? Now imagine your an IT guy and the professors can't access their online grade books that you pushed them into using. I really think McAfee is going to have a big problem on it's hands come contract renewal time. Pissed off IT people have long memories!

I am sure they "forgot" to count third party AV. (2, Interesting)

JaCKeL 1.0 (670980) | more than 4 years ago | (#31955428)

We use Sonicwall's security services, their anti-virus is a crippled version of Mcafee business. And we've been hit hard: Machine where going down but WITHOUT any explanation or any warning messages (this version is silent to the user) and since svchost was killed, no chance of getting in the event monitor or using any tools, it took me couple of hour to figure it was the AV. I am sure they "forgot" to add all those third party security solution who rebrand Mcafee solutions. What is making me mad is the way they try to play with "numbers" (a small percentage, half of a percent...) and the way they hide everything and to act like it didn't happen(go navigate on their website and try to find any information about this bug, they even closed their support form in the peak of the crisis). C'mon if you screwed up, at least PLAY FAIR and be sorry, we might forgive you.Pplaying the ostrich game will make us angrier.

Oblig. xkcd (4, Insightful)

wvmarle (1070040) | more than 4 years ago | (#31955438)

Quite apt, even though not POS: http://xkcd.com/463/ [xkcd.com] .

Huge impact where I work (1)

Jon_Hanson (779123) | more than 4 years ago | (#31955442)

At a certain large semi-conductor manufacturer this false positive wreaked havoc. Most of our IT-supported laptops are running XP. Fortunately I figured out what was going on pretty quickly and knew how to fix it. Other people here weren't so lucky and it took the IT department at least half the day to figure out a solution. Most people were down the entire day.

Re:Huge impact where I work (0)

Anonymous Coward | more than 4 years ago | (#31955734)

Huge impact here, some locations had as many as 80% to 95% of the machines affected by this update.

My location had the least problems (10-20%), and our department (development) even less, because we noticed what was going on very fast and acted on it.

this exposed internal Q&A flaws too (0)

Anonymous Coward | more than 4 years ago | (#31955676)

i work at a Fortune 25 company that was CLOBBERED by the antivirus virus. because we span all timezones, the impact was greatest on the east coast while the west coast was minimal [due to halt in DAT push]--except for those early-risers who connected to the network before business hours.

yes, mcafee really dropped the ball. but it's equally careless not to have it internally tested before allowing ANY updates; moreover, because our corporate image is XP-SP2, our Q&A team could've easily--but didn't/dont--test the DAT on SP2 & SP3.

Damage Limitation (2, Informative)

MrNemesis (587188) | more than 4 years ago | (#31955726)

"McAfee Interwebs Secrutiny has detected that your outgoing mail to customerservices@mcafee.com, subject "You f**king idiotic t**tballs of a son of a ****** in the ******** with a hatstand!!!!" has been detected as Offensive Spam and will be deleted. Thank you for Trusting in McAfee! [TM]"

On a more serious note, I ran into a few small shops that were badly hit, but most of the people I know who work in the enterprise have a time delay before the updates hit the machines, which is usually a hangover from the last time $av_vendor bollocksed up an update.

Personally, I'm still a believer in most AV's being worse that the viruses themselves, and don't run any on my windows boxes - I don't think I've used a single one that hasn't fucked up at some point. Most of my colleagues feel the same way (and, IMHO, by the time it's hit your filesystem and you have that 20% chance of the AV detecting it, it's already too late anyway) and the only reason we run it at work is because of compliance issues... that and the majority of machines being a poorly patched IE6. Yay!

screw corporatespeak (0)

Anonymous Coward | more than 4 years ago | (#31955750)

screw corporatespeak

Compensate customers? (2, Funny)

northernfrights (1653323) | more than 4 years ago | (#31955800)

"ZDNet is running a poll and opinion piece on whether McAfee should compensate customers."

Poll? Opinion piece??? This is fucking America. Spare me the nonsense, show me the lawyers.

what it did to my 11'000 computers (3, Informative)

Atreide (16473) | more than 4 years ago | (#31955818)

we have 11K computers

only XP SP3 computers were impacted
whether running Virus Scan 8.7 or 8.5

but in fact less than 100 computers were impacted,
1% compared to our total

one thing that helped
was employees had started to leave after work when update propagated
and they shutdown computer when they leave

it could have been a nightmare
we were very lucky

Isn't this a problem with the IT departments? (0)

Anonymous Coward | more than 4 years ago | (#31955834)

From my perspective the IT departments that had issues should be to blame. The patch or dat file for an anti-virus program should be treated like any software update. Update one system only, test that nothing serious goes wrong, then deploy the patch to production machinces. Do these guys just allow the anti-virus application to update itself? That seems seriously wrong, and I only blame the IT group for that.

I lost... (2)

Schnoogs (1087081) | more than 3 years ago | (#31955952)

about a day and a half of productivity time at work. Granted some of that was because of how slowly information was passed out. It wasn't til the next day that I found the solution on my own using my own personal notebook and internet connection.

Regardless it was a massive disruption and when you work for a company that has 50,000 customers world wide the task of fixing the problem is massive and the effects of downtime can be disastrous as it spans entire divisions, etc.

On a correctly designed OS... (1, Insightful)

Anonymous Coward | more than 3 years ago | (#31955960)

On a correctly designed OS:

a) there's no need to run an anti-virus

b) a third-party party software does NOT need to know the admin/root password to do its job

c) a software running without admin/root priviledges CANNOT break havoc in anything but the user account

Tech-savvy companies who switched tens of thousands of XP machines to Linux and were
criticized for doing so by MS fanbois/astroturfers (don't forget to add *that* to your CTO reports
if they were running Mc Afee) are now laughing all the way to the bank.

But, I know dear MS fanbois/astroturfers: nothing to see here, move along, Windows has
nothing to do with this issue right!? Because the Windows family are the most well-designed
OSes on earth right!? It's of course the fault of McAfee (nonetheless on *my* OS there's
no third-party software that can render my system unusable)... And all the paid "reporters"
that make a living by ever only talking about the Microsoft ecosystem would be silly to
cut the grass under their feet by pointing out the *real* guilty one here.

But, no, dear paid MS astroturfer/fanboi, I won't find your answer compelling.

Read the EULA (0)

Anonymous Coward | more than 3 years ago | (#31956078)

Read the EULA people, your software, written by "Software Engineers" comes with:

NO WARRANTY
NO FITNESS FOR A PARTICULAR PURPOSE
CONTAINS KNOWN DEFECTS

You paid your money, now you take your chances.

Unlike real engineers, you can't sue a software engineer, report them to some sort of professional body, or seek any type of remedy, besides a possible refund of the money paid for the software.

Aren't you glad you paid the full retail price of windows, the most secure OS ever?

Enjoy the FREE*:

-Digital-Restrictions-Managements
-Viruses, Trojans, Etc
-Internet Explorer
-Shiny New Icons(TM)

*: some restrictions apply, co

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>