OK – yesterday was one of those days that I’d like to do over again. It started with a very late night before when I went to the airport to pickup my in-laws and their flight was delayed a few hours – ultimately I got home in the early hours of the morning and therefore had little sleep. My first appointment was an 8am visit with one of my long term clients and went well – planning for a 170 location VPN solution, VLAN’s in the head office and cool stuff like that. I got into the office around 10am and was almost immediately hit by my phone support guy Daniel with a “we’ve got a virus outbreak and what should we do“. Now Dan has only been with us a few months, he’s a great guy and what he lacks in knowledge he makes up for with enthusiasm and desire to learn. But we had two sites call in within a few minutes of each other telling us that they had massive outbreaks of a virus on many workstations and their servers. This had also affected their terminal servers and users were kicked off and when they tried to reconnect, they timed out. Now I am (and still remain) a very big advocate of Trends Antivirus products – for me they have saved the day every time, and the only infection that I’ve had has been a single PC that was hit by a virus in the first few hours of it’s life, before the AV vendors became aware of it. So I have a great faith in Trend and it’s abilities to keep me clean and secure. So when you get something like this, you start to worry. We could not TS into the clients site (which were a side effect of the problem) so we got the client to tell use what the name of the virus was.. We got the name HKTL_LSASSSBA.A from the client and did a lookup on Trends site. The infected file was always the same on my sites – c:\winnt\system32\netapi32.dll . At that point in time Trends site reported this as a low risk but that they had only discovered it less than 15 hours beforehand. Given what appeared to me to be a wide spread infection combined with the fact that I’d not heard about this virus in the community earlier in the day, I was concerned that we were at “ground zero“. I used IM to ping about 18 other SBS MVPs around the globe to see if they had seen it either on their customers sites or in the community – the response was No. After a few minutes, one of the guys found that one of his sites had just started to report it too.
So we hit the panic button – started to warn the community and our customers to be on the lookout for this as we felt that we would not be alone. Given that the apparent description was the it installed a hackers toolkit, we didn’t know what it would then do. I advised my infected clients to shutdown their internet connection as first step and then to shutdown all workstations – we left the servers running so we could gather information about it. I spoke to Trend locally and they didn’t know anything other than what was in the on line information, but proceeded to try to assist. The problem was that we didn’t know how this virus got into our systems in the first place – was it email borne, web browser based or who knows what.
I ultimately believed the best way to get a handle on this was to go to site myself, so I jumped into the car and lucky for me, the nearest client was 15 minutes away. Got there and was able to run the Trend Console and this showed that the server and a few workstations were affected by the virus – the customers terminal server (sitting next to the SBS server) was apparently clean. We went over the logs, and found that the virus pattern had updated to 2.333.00 at around 2:30am that morning – and the first reported infection of the netapi32.dll file was at 9:17am – other PCs were shown as infected after that time – therefore we believed that the server was the “zero point“ in this system for the infection. We still didn’t know how it got infected though which was a big worry. I spoke with a Melbourne reseller – Daryl Maunder who also had two sites affected by this – what made it more interesting was that one of his sites was a Lotus Notes site and therefore it was not likely that it was email bourne. His other infected site was actually a single PC that sat in the corner and did not have email or web browsing done on it at all – it was used just for running a scheduled FTP download from an external site to the internal system. NOW WE WERE REALLY FREAKED OUT.
So, given I had already spoken with first level support at Trend and didn’t seem to get too far, I got on the phone with my mate Andy Huntrods at Trend (Andy is the Aussie Channel Manager) and said “we’ve got a problem and I need this kicked up to the top ASAP please“ – Andy proceeded to walk around the office and passed me on to the Aussie Technical Manager – Anthony. He was not aware of anything big “out there“ right now which made me both happy and worried at the same time. But he took it on board and went digging into his information. After a few phone calls back and forth, he came back to me with a pattern file to apply to our infected system. This ultimately resolved the problem for us as it was a “false positive“. We had NO infection, but Trend thought we did and did it’s job of protecting us. So what happened???
Well first you need to understand something here and that is that Trend alerts show an infection by spyware in the exact same way as an infection by a virus. When Trend detects a virus, it attempts to block access to the infected file (which it should do of course). The netapi32.dll is a critical system file and as such access to it was being blocked even from the system itself. This we suspect caused the terminal services components to cease functioning and was an unfortunate side effect of the bigger issue. So shortly at 9:11am, the clients sites started to download an update for the spyware signatures (v0.195.00) from Trend (these are different files from the normal virus signatures) and this had incorrectly identified the netapi32.dll file as being the HKTL_LSASSSBA.A “virus“. Trend did it’s job and then distributed this new pattern file to all the workstations which was what we saw as the “spread of the infection“. Now I’ve learned that HKTL is actually Trends short version of “HacKer TooL“ which means it’s not actually a virus, but potentially spyware. The updated pattern file that Anthony provided was a “Bandage“ solution – one that they use in times of high risk virus outbreaks to quickly get a customers site under control – it’s version was 0.196.00. We applied that to the server, pushed it out the the client PC’s and then scanned the network to be sure – all was good and systems returned to normal. Later in the day (actually about 1hr after I got this sorted), Trend pushed 0.197.00 out to the world via their normal channels. So the problem was fixed – all that remained for me was to go to the other infected site and manually push the 196 file out to the server and workstations.
So what did I learn from this? What feedback can I provide to Trend?
- In our standard configuration we elect to turn on the Spyware scanning for workstations AND servers (which is by default turned off in CSM Suite for SMB) – I plan to modify this slightly and only have it enabled for workstations.
- For Trend – Give us a different type of warning for spyware and virus infections – don’t make us and the customers decipher HKTL – it should tell us in big clear letters “Virus infection“ or “Spyware Infection“.
- Also for Trend – Give me a console that I can use (as a VAR/Reseller) to manage ALL my clients sites from my office – I had to initiate a person to login to each and every site in the early stages to check each clients CSM installation to see what signature versions they had etc. I want a console that I can use to control my sites installations of Trend products – you have this for the enterprise already, you just need to make it available to us for this end of the market as a reseller!
- You can never have too many friends in too many places – thanks for the SBS MVPs worldwide who got onto this and helped investigate what may have been a potential outbreak.
So the last question is – do I still trust Trend? Will I still be as enthusiastic about it?
YES – for sure – although it caused me some pain this time around, it is the first time it’s done so and I would rather be having it detect things than not. The people and relationships I have with Trend are one of the main reasons I love working with the product. They help make supporting it even easier.