51 replies to this topic

[patpikunas]

Honestly, the only time I have had any issues with the MB was on the morning that Reid got fired.

True, I am not on here nearly as much as some others, but I have no complaints.

[VaBeach_Eagle]

I had some problems early (very early) this morning, they only seemed tolast about 5 or 10 minutes though.

[MIKESCR]

So if I go to the full list of people on the forum and most are listed as Guest Viewing Community Index thats what you are talking about?

[Rome]

Eliminating the bots from the equation would make things much less congested here.  That's up to Dave to pull the trigger on.  I do find Invision's responses to be, well, interesting, however.

MIKESCR, on 26 January 2013 - 09:14 AM, said:

So if I go to the full list of people on the forum and most are listed as Guest Viewing Community Index thats what you are talking about?

http://en.wikipedia....iki/Web_crawler

Si, Senor.

[dave moss]

patpikunas, on 25 January 2013 - 03:44 PM, said:

Honestly, the only time I have had any issues with the MB was on the morning that Reid got fired.

True, I am not on here nearly as much as some others, but I have no complaints.

i had problems getting on when it was announced that they would hire chip kelly.  not suprising.

[VaBeach_Eagle]

MIKESCR, on 26 January 2013 - 09:14 AM, said:

So if I go to the full list of people on the forum and most are listed as Guest Viewing Community Index thats what you are talking about?

Not all of those would likely be spiders, but I'd bet that at least half, if not more are.

Dave would be able to tell, if interested, by getting the IP addresses of various guests and checking them or he could go to the ACP (Admin Control Panel) and go to the Spider Logs to see when the spiders are on and how much they are indexing.

I'd be very interested in seeing how many hits the Google spider has. Like I said earlier, a very small message board of fewer than 4,000 members with about 800,000 to 1,000,000 posts has been indexed almost 1.3 million times in the past 12 or so months by Google. So the EMB must be 10 to 50 times that number.

[VaBeach_Eagle]

Here's a good, visual example of a spiders work: http://web.archive.o...lphiaeagles.com

That's Archive.org, their spider indexes as much of a webpage as it can so it can archive as much of the internet as possible.

On that list, click on any date up until March 21, 2011. All you get is the Login Page. That's because that's all the spider had access to. It couldn't get into the board and use any of the EMB's bandwidth to archive it.

Next click on April 27, 2011. All of the sudden, the spiders have access to the content of the board and can now archive actual pages other than the login page. Click through the links on any given day in their archive, see how many pages open and show you the posts. Then take note of the web address in the address bar. It's not on the EMB, it's on Archive.org.

What does that mean? It means that archive.org is downloading everything you see and saving it on their servers. What difference does that make? The Archive.org spider is using EMB bandwidth to download everything. What difference does that make? Add to that all of the other spiders that are doing the same thing, along with normal member activity and at worst you get "Over Capacity" errors or a very slow board for human members.

[LacesOut]

Va,

So, Invision needs to put a stop to the spiders/bots/EMB Invaders?

[VaBeach_Eagle]

LacesOut, on 26 January 2013 - 08:31 PM, said:

Va,

So, Invision needs to put a stop to the spiders/bots/EMB Invaders?

It's not that Invision needs to put a stop to them. They are useful, they are how you can get good hits when you go to a search engine. If you decide to open your own web page for whatever you want to do, you want spiders to index your site, often.

In the case of the EMB though, there is no logical (that I can see) reason for spiders to be constantly indexing the EMB. The EMB isn't a commercial site, nothing is being sold here and no advertising is here. The spiders should, and do, index PE.COM. That's logical. It's a commercial site that has advertising all over it. You want search engine hits all over PE.COM... or at least the Eagles should want it anyway.

Here though, with so much new content being added every minute of the day, it's silly to have the spiders constantly indexing and re-indexing and re-re-re-indexing.

Also, it's something that Dave or any other admin can do in the Admin Control Panel, it doesn't have to be Invision that does it. There's a setting in there that allows any Admin with access to the ACP to close the board to guests and only registered members can log in. As seen below:

Spoiler

A spider would come to the board as a "guest" and the only thing the spider would see would be the "Log in or Register" screen. That's all it would be able to do, leaving the forum to real people.

Dave, someone ought to look into this, how much bandwidth is being used by Spiders? How much is Invision charging you for the boards bandwidth? If you're paying more for extra bandwidth, cutting the spiders out of the equation may (should) reduce the required bandwidth and thus cut the amount you have to pay to Invision for extra bandwidth.

If you can save $300 to $500 on bandwidth by doing this, you can just send me an authentic #17, autographed Harold Carmichael throwback...

lol

[evilwaldo]

Random Spider Facts:  Spiders can sustain hundreds of atmospheres of pressure, and can work flawlessly in a vacuum. Scientists have discovered spiders working at both the bottom of the sea and in the ultra-thin atmosphere thirty miles above the Earth.

I would create a clone but it would take too long.

[VaBeach_Eagle]

It's been running fairly smoothly for the last couple of days for me, but at this time of the year I'd expect that regardless.

It does hang every morning at 3 am though.

[NorCalEagle]

VaBeach_Eagle, on 29 January 2013 - 02:58 AM, said:

It's been running fairly smoothly for the last couple of days for me, but at this time of the year I'd expect that regardless.

It does hang every morning at 3 am though.

could be backups are running at that time...

that said...i just tried to get into atnfl...& got a service unavailable message...

[VaBeach_Eagle]

NorCalEagle, on 29 January 2013 - 05:58 PM, said:

could be backups are running at that time...

that said...i just tried to get into atnfl...& got a service unavailable message...

A powerful enough server should be able to do a backup without interrupting service though.

[VaBeach_Eagle]

Dave, have you looked at the possibility of restricting the board to only registered members yet? Or is there a reason to keep it open to guests and spiders?

[DaveSpadaro]

I have not, VA Beach. We have a meeting on it this week.

[VaBeach_Eagle]

Well, I don't get a vote but my vote is to eliminate guests and spiders. Guests can register if they would like to enter and read TATE wisdom. Spiders, for the EMB are useless.

Ask the question, either to yourself or to whoever is in the meeting, of why the EMB needs spiders indexing the board. Ask someone to look into the boards stats in the ACP and see how much bandwidth the spiders are using, how much bandwidth are "guests" using? Not all guests are people and not all spiders will show up as spiders.

If spiders are using a lot of bandwidth and if you have to pay extra for extra bandwidth, then it would seem logical that spiders (and guests) should be eliminated.

[Runtherock]

Probably has nothing to do with anything but I'll throw it out there. This site functions much better for me with Google Chrome.

[VaBeach_Eagle]

Runtherock, on 02 February 2013 - 08:16 PM, said:

Probably has nothing to do with anything but I'll throw it out there. This site functions much better for me with Google Chrome.

I've tried it with that, Internet Explorer and Firefox and never saw any difference.

[E v 2.0]

I could be wrong but wouldn't locking out unregistered guests just send them to a different screen yet still tax the same server?

[VaBeach_Eagle]

E v 2.0, on 02 February 2013 - 10:24 PM, said:

I could be wrong but wouldn't locking out unregistered guests just send them to a different screen yet still tax the same server?

No, the amount of content being delivered to them on only the login screen is minimal and the hits on the database would also be minimal because very little data needs to be delivered.

On the other hand, if there's 500 guests and/or spiders accessing 500 different pages (or even 100 different pages) then all of that data on those pages need to be delivered to those 500 guests which will hit the database and slow the server and cost bandwidth.

[EagleJoe8]

Runtherock, on 02 February 2013 - 08:16 PM, said:

Probably has nothing to do with anything but I'll throw it out there. This site functions much better for me with Google Chrome.

Same with me, but then, everything has worked much better since I switched to GC. NFL.com used to take forever to load on IE, and I've had no issues with GC. But, even with GC, I have still gotten the occasional Over Capacity errors.

[VaBeach_Eagle]

A potential "Crash the EMB" day could be if Vick gets cut...