Server slowness

[jimnyc]

A short while ago we had severe slowness. Figured out that we had about 4,500 users online, and of course about 4,400 of them being "bot spam". There were many IP's hitting us, and searching the forum for indexing or whatever.

Nighttrain was kind enough to get on the phone with me. Took forever due to the slowness, but one by one we started blocking IP subnets. Eventually it started picking up speed. We still have 3,000+ online, but they will disappear soon as they have been blocked. I will block more if I find them.

Most of them were foreign, and one massive USA company that apparently we were getting nailed by. Hopefully we'll be back to normal soon.

[Drummond]

A short while ago we had severe slowness. Figured out that we had about 4,500 users online, and of course about 4,400 of them being "bot spam". There were many IP's hitting us, and searching the forum for indexing or whatever.

Nighttrain was kind enough to get on the phone with me. Took forever due to the slowness, but one by one we started blocking IP subnets. Eventually it started picking up speed. We still have 3,000+ online, but they will disappear soon as they have been blocked. I will block more if I find them.

Most of them were foreign, and one massive USA company that apparently we were getting nailed by. Hopefully we'll be back to normal soon.

I suspected something of the kind (.. and posted on it ..). Yes, this site is now loading at something approaching normal speed. Glad to see that it's being sorted out. Thanks to all concerned ... I'd hate to see this site rendered inoperable !

[NightTrain]

That was fun.. I've never participated in defending a site against a botnet before.

So, I'm just curious as to who the butthurt party was. Was it the hippie from CA? A random liberal that doesn't have the balls to debate?

[jimnyc]

Rick sat on the telephone with me, he was reading off multiple IP's to me to look up and helping me identify and block them. We would still likely be down/slow right now if not for his help!!!

I don't know if it was an "attack", or if it was runaway bots. Usually it would be just one bot scouring the entire site, where in this case it was a bunch. For example, there were many in the USA, and also a few from Romania!!

[Drummond]

Rick sat on the telephone with me, he was reading off multiple IP's to me to look up and helping me identify and block them. We would still likely be down/slow right now if not for his help!!!

I don't know if it was an "attack", or if it was runaway bots. Usually it would be just one bot scouring the entire site, where in this case it was a bunch. For example, there were many in the USA, and also a few from Romania!!

'A bunch' suggests targeting to me. If it was a wholesale attack against many such sites in the US, surely there would be news reports saying that such a thing had happened. It'd surely also make the news, even in the UK ? But I've come across no such report. So your site was chosen ?

Jim, if so ... should you be flattered ?? Seems that someone out there is really bothered about your site's successful operation.

Stands to reason that a targeted attack comes from a source that finds outright attack preferable to attempts at debate ? To testing out standpoints publicly, instead, to test their worth ?

... so, from a Leftie source, then .. ? My thinking may be speculative. Seems to fit, though ...

[Max R.]

That was fun.. I've never participated in defending a site against a botnet before.

So, I'm just curious as to who the butthurt party was. Was it the hippie from CA? A random liberal that doesn't have the balls to debate?

More likely someone seeking to cash in on email addresses or other information. With bots, as with spammers, it's a net as big as the Internet.

[NightTrain]

More likely someone seeking to cash in on email addresses or other information. With bots, as with spammers, it's a net as big as the Internet.

Well... maybe, but I really don't think so.

There were over 4,000 bots all on this site, performing searches and trying to PM users. At one point, the site didn't respond to me for over 22 seconds, and a few times it actually timed out. We identified the primary offending server (most likely being used as a proxy and it has a bad reputation for allowing attacks / spam traffic) and it began to recover after Jim began blocking swaths of addresses from that server. But it wasn't just the one server, there were others from overseas doing the same thing at the same time. It seemed to me like it was coordinated botnet assault - really, you're not going to gain any information by attempting to PM members and a lot were doing that over and over.

I can see where it could have been a mistake and there were runaway spiders because someone screwed up and added a zero somewhere, but then all those users would be from the same origin and there were quite a few different origins all doing the same pattern of crippling activity here on the board.

Legitimate spiders are up front with who they are - they'll say something like rate-limited-Googlebot or Yahoo or MSN... none of these identified themselves as such, and many were duplicated IPs doing intensive queries to the site all at once.

I think it was an attack, albeit a clumsy one that was called off when countermeasures were taken... but I'm not an IT guy and Jim knows a lot more about it than I do.

[jimnyc]

Just to assure some, all data is fine, including PM's and email addresses. What these folks want to do at a time like that is to make tons and tons and tons of requests to the site, no matter the request, with the intent of making so many, that the site will be dead/unusable/slow.

But those hitting the site, whether live people or bots, have the same rights to the board as any other guest. They will see an error message if attempting to send a PM or even view something they don't have rights to. When viewing "Who's Online", it shows us the IP address of each visitor, and what they are doing. All of these visitors also had a stop sign next to their names, or an error sign, meaning they were having an error message displayed on their side. Unfortunately, although they get nothing from us, it still uses resources to go through the request. And as you can imagine, even just a few thousand over a short time can kind of cripple a smaller site like this. Imagine some of the larger sites with redundancy that get hit with millions of requests in a very large botnet attack!

All is safe and all is well, and all of your data is fine - and no one's email addresses have been compromised.

[aboutime]

Jim. Romania? Really? Ring any bells....maybe???

[jimnyc]

'A bunch' suggests targeting to me. If it was a wholesale attack against many such sites in the US, surely there would be news reports saying that such a thing had happened. It'd surely also make the news, even in the UK ? But I've come across no such report. So your site was chosen ?

Jim, if so ... should you be flattered ?? Seems that someone out there is really bothered about your site's successful operation.

Stands to reason that a targeted attack comes from a source that finds outright attack preferable to attempts at debate ? To testing out standpoints publicly, instead, to test their worth ?

... so, from a Leftie source, then .. ? My thinking may be speculative. Seems to fit, though ...

Jim. Romania? Really? Ring any bells....maybe???

Guys, I'm no network expert like this. I'm good on certain things, but identifying and finding attacks are difficult. It wasn't even until I saw the users online that I realized something was up. We set a record today! LOL

I would have thought if it were someone commandeering, via a botnet from malware/virus, we would have seen IP requests from thousands of machines. We had thousands of requests during that time, but it was from about 12 IP addresses in total.

I've seen Google bots come here and have about 150 of their IP's here at once, and the same for MSN. I've seen it from foreign search engines as well, but I block most of them.

Could have been a few search engines gone awry, but I don't think so. I believe some weird behavior going on for sure, but it doesn't always mean it's directed. Could have been random too. Who knows.

Suffice to say, I'll be keeping a close eye on things. I have the webhost looking into a few things for me as well.

[Gunny]

A short while ago we had severe slowness. Figured out that we had about 4,500 users online, and of course about 4,400 of them being "bot spam". There were many IP's hitting us, and searching the forum for indexing or whatever.

Nighttrain was kind enough to get on the phone with me. Took forever due to the slowness, but one by one we started blocking IP subnets. Eventually it started picking up speed. We still have 3,000+ online, but they will disappear soon as they have been blocked. I will block more if I find them.

Most of them were foreign, and one massive USA company that apparently we were getting nailed by. Hopefully we'll be back to normal soon.

So this was YOUR fault? I was dismantling my freakin' computer and checking connections. You owe me a steak or something.

[jimnyc]

So this was YOUR fault? I was dismantling my freakin' computer and checking connections. You owe me a steak or something.

Nevermind the geeky guy in the closet playing with the wires!!

[Gunny]

Nevermind the geeky guy in the closet playing with the wires!!

I think you did it on purpose because Jeff and I were talking about you.

[jimnyc]

I think you did it on purpose because Jeff and I were talking about you.

Hmmmmm.... Ever hear of the old "tachy goes to coventry" in the older vbulletin days? I think you were able to set a member as such, and they can login, post and do everything, only the rest of the board wouldn't even see any of it. Kinda like putting that person on everyone's ignore list! LOL

[Gunny]

Hmmmmm.... Ever hear of the old "tachy goes to coventry" in the older vbulletin days? I think you were able to set a member as such, and they can login, post and do everything, only the rest of the board wouldn't even see any of it. Kinda like putting that person on everyone's ignore list! LOL

I only did it to ONE person.