Improving mySQL performance on IPB board

After 8 days of waiting for the server stress to slow down and decrease, it seems that it is even getting higher along with more demanding requests.

I really thought we were having a huge ammount of new visitors, but this is no longer a friendly wave of visitors and resembles a <a href="http://en.wikipedia.org/wiki/Denial-of-service_attack">DDoS</a> attack targeted to hog and overload the server.

Checking back the logs from the hosting company I've noticed that SQL usage has multiplied 5 times the regular usage since the 21st of June.

This excess of usage makes this site consume an exorbitant amount of bandwidth - luckily enough that we can endure this proportion of data transfer (and even bigger if necessary).

Unfortunately there isn't much to be done with the mySQL database which is taking most of the stress and was often giving the "error 500" messages when acessing the forums. I was hoping to place our database on other external servers where these requests could be balanced, but this is a bit dificult because I have no experience with this sort of handling - and it would be better not to risk the safety of our posts.

At the moment, I've removed all the extra "fluff" from the IPB forum software and disabled statistics displayed on the main page, search function, members viewing a forum, members who viewed a topic, similar topics, aniversary box were also disabled.

Reduced the counting of number of visitors to the last 5 minutes, also decreased the number of displayed topics on each category, removed gzip encoding while exchanging pages (more bandwith, but also less CPU overhead) and ensured that pages are cached on user's disk instead of always retrieving them from the server on multiple requests.

On the portal software, I've disabled the display of the last 1000 posts (will reduce google rank) and the left bar which was displayed while browsing the forum.

Some other internal improvements were also added to ensure the minimum SQL usage.

This removes much of the user friendly functions that keep everyone in touch with what is being discussed during the day but I hope to bring these addons back when we step over these server overload issues.

At least in the meanwhile we still have a working site instead of semi-offline one.

Peter has started a good post <a href="http://www.boot-land.net/forums/Error-500-t2482.html">here</a> to further comment this matter where everyone is welcome to write their comments or ideas.

Have a good weekend!

-- Nuno Brito

New sub-forum for wb on 911cd.net

A new sub forum was created on 911cd.net to discuss wb projects, more specifically intended to the customization of bartPE based projects but also intended to be used as an alternative place whenever our servers get overloaded with too many people has we have been experiencing during this week. This way we always ensure a contact point if boot-land.net gets temporarily offline.

To improve server performance a new static html page to directly download WinBuilder was also created and can be found <a href="http://boot-land.net/winbuilder">here</a>

This new page contains very few images and is very fast to load. I hope this way we can free the CPU and RAM resources of our server since it won't even use any PHP scripts or mySQL queries. Once people download Winbuilder.exe, the remaining projects and script can also be retrieved using direct download links from the internal Download Center which is a very efficient and well supported method for massive downloads.

Our current download system will continue to be available in the future, the important thing at this moment is to be able to move this massive wave of visitors to a place where they can get the desired projects and that we can continue to talk with better conditions.

Once again, sorry for these issues, let's hope to get them all solved by the end of this week.

:)

-- Nuno Brito

Sony Reader Update

After having the <a href="http://www.engadget.com/2006/09/26/sony-reader-prs-500-hands-on-connect-reader-screenshots/" title="PRS-500">PRS-500</a> for awhile I've realized it's best for plain text files. PDF files expect to be displayed on a bigger screen. RTF files don't always display properly or simply don't open. These problems are not necessarily the fault of the unit. I'm quite satisfied to read plain old text files. Long live <a href="http://en.wikipedia.org/wiki/Edit_(MS-DOS)" title="DOS EDIT">DOS EDIT</a>!

I had a little scare this morning when the unit rebooted and scrambled its display. It subsequently failed to power on or allow a reset. I tried to recharge it via USB with no avail. Turns out the battery was drained and USB charging only works if the Sony Connect software is installed.

I've gotten used to cellphones and iPods that charge without drivers installed. PDAs usually give more proactive warnings when the battery is low. The eReader is supposed to blink a pinhole sized indicator light on its side when low (which I obviously missed.) The moral of this story is: if you don't read the manual expect surprises.

- moongoon

Error 500??

Sorry for this incovenience but it seems that simply too many people are recently visiting the forums at the same time and this causes the server to refuse more connections when it reaches the limit. This happened after the official support from bootdisk.com and removal of user limitations on the new download system

Our hosting conditions are quite good in general terms since we enjoy ilimited bandwith provided on a decent speed, but I wasn't predicting that so many people would be connecting to the boot land forums so soon and this obviously pushes the CPU to a frenzy state - more people are visiting our community and this should be expected to grow more and more over the next months.

One option would be shifting to a dedicated server instead of shared hosting with enough conditions to survive this demand but at this moment this upgrade is simply too expensive for monthly support and survive on it's own over the years. Sad enough that our datacenter in Lisbon - Portugal has excellent hardware conditions to host a small server since they have a dedicated 5Tb optic fiber internet uplink connection - on which I'm sure that nobody would mind if I joined a small machine there to host our community, but unfortunately I'm just a bit too far away from Lisbon until the next year and would only trust myself to set up the machine and keep an eye on it to fix whenever needed.

Or maybe I can contact my hometown university in Coimbra where I have some trusted childhood friends on the IT department to see if they have good hosting conditions to support this demand.

Another option would be the help from an external company with enough resources to freely give dedicated hosting, but I don't also have many contacts on this area and I think that most companies would only help in exchange for placing their advertisements. A logo on the bottom of the page saying something like "Hosted by ..." would be reasonable to be accepted, but annoying ads like the ones provided by google are not. <em>(have nothing against google ads but I like to keep a good distance from any sort of ads)</em>

If anyone knows a trustable company willing to help us on this dedicated server, please write a topic on the forums where we can further discuss it.

I will do my best to find solutions that can solve this sort of annoyance within a short period of time. First step will be moving WinBuilder.exe to a static page or external server since our only issue is when people get on the forums to download this binary and start running php scripts and mySql queries which overload the CPU / Memory when entering here.

Having a direct link to winbuilder should allow to later download all needed scripts and projects from the download servers which is very well supported and is very far from reaching limits. We spend around 1Tb of monthly bandwith and can still use well more than 5Tb and this value can be adjusted as needed. This would still allow people to later visit our forums for support or join the daily talks if they wish.

I will start working on this right away, it surely isn't anywhere pleasant having to press reload several times until a connection is accepted.

Was expecting around a rate of 100~200 visitors but &gt;800??

I'm quite surprised how the server still (barely) supports this workout.

Is MS evil?

Tough question raised by Jaclaz <a href="http://www.boot-land.net/forums/index.php?s=&amp;showtopic=2195&amp;view=findpost&amp;p=16493">here</a> about the resistance to use IE as default method to download files on wb, but what real alternatives exist out there?

Sad enough that I probably know exactly what he means and have to agree with his perspective that using a IE function has it's serious downsides when we consider their code quality when compared to other available browsers.

Unfortunately I think that we still need a solution for this case and all the other methods have proven to be not compatible with most machines available. Some would fail to start because were they were not prepared for x64 windows OS versions. Others would complain about another door open up on their firewall security which could be pointed like a suspicious behavior and last but not least was the proxy headache.

Proxies are a lovely piece of design - when they work as we expect. Otherwise it's a very lengthy task to figure which type of proxy is being used and then try to correctly add up the needed settings to make the proxy access authorized.

This seems wrong but unfortunately most home users / companies / schools / universities use IE and so we need to use IE as well to use their settings and connection functions.

<strong>Jaclaz</strong>, sorry because I've really tried to use other methods as much as possible, but these are things that were clearly monopolized to a single "compatible" solution. Would really like to fight against this system, but if we don't use this tool then it's our own loss and I really doubt that MS or the other browser alternatives would ever care.. :(

-- Nuno Brito

New download system

There is a new download system available on boot land, you can access the older system visiting <a href="http://www.boot-land.net/downloads.html">here</a>.

A lot of files had been posted recently over the old system, so I'm sorry for the inconvenience of having to repost these scripts again. The main advantage of the new system is a much more customized way of doing things. Developers can post their scripts and a new topic is automatically generated from where everyone can read back the feedback or discuss things that need to be improved.

So, everyone is welcome to post their files on this new download location. Hope you like it. :)

Memory leaks and speed improvement

It's been a while since the last blog post so there's a lot to catch up. At moment wb is being worked to solve memory leaks and start handling properly big sized files. This wasn't a priority before, but since more people are starting to use scripts sized over 10Mb we are also noticing that performance gets sluggy. Happily enough that this is somewhat easy to solve, but I'm also taking the time to do a complete recoding over the most important areas to see that they are also improved.

One other important step is complete the download center to make it more compatible with machines were network connectivity is heavily filtered and restricted. So far wb can also completely disable the builtin network functions, but ideally it would be good to add more support for all sort of web connections. So far it should work just fine for most home users, but it is still hard to connect on local intranets that use proxies to access the web.

Another positive step is the development of a common script standard that introduces a general purpose script that can be used across different projects without adding any changes on the script file itself. This is very important to start creating an easier way to code and also allow to make all written code reusable within a few years as projects evolve and change themselves. Meaning that I can write a script and be assured that it will still work after 10 years. Is it possible? I'm not sure, but would be good to think this way.

-- Nuno Brito

Sony Reader

Got a <a href="http://en.wikipedia.org/wiki/Sony_Reader" title="Sony Reader">Sony Reader</a> last week. Its main purpose is to read text files and pdfs. Because the display uses <a href="http://en.wikipedia.org/wiki/Electronic_paper" title="e-ink or e-paper">eink</a> some claim that you get 7500 page turns before the battery runs out. The display uses no power after a page is "printed". Unlike LCD displays you can view from any angle and even in direct sunlight. Also, unlike other Sony gadgets, this one has a SD card slot.

So far the only downsides are ghosting (residue from previous pages) and its small size. Some PDF files have text too fine to read.

- MoonGoon