You should assume your site is going to be hacked, and design with that in mind.

Blaming things on "poor coding" is a get out. There will always be bugs. You must be sure your procedures, your protocols, are faultless despite these bugs.

The majority of bitcoins should be in offline storage, and your hot wallet should be on its own dedicated server, with access through some kind of very limited API that does sanity checks and rate limiting on everything. (Access could be through a custom and very carefully written daemon i.e. not just some Apache HTTP API with associated vulnerabilities!)

Ideally, the user owns an encryption key that is used to somehow client side sign transactions (or any other action they take) which are applied to their account or the hot wallet.

Even then, a hacker could presumably hijack the serving of your pages, modify the crypto etc to steal their key. But at least if the user's key was only generated once and is only used by them occasionally, the chance for massive theft is reduced. Two factor auth may help too.

You could have daemons on other hidden servers that regularly check your site and its resources for tampering, suspending everything if necessary. Perhaps release a browser add-on which verifies your site. Your entire site could be signed, and the add-on could request a new signature to verify any changes. (And you perform this signing offline). As long as the user's browser or the add-on publishing system doesn't get compromised, you'll be alright ;)

To summarise all that:

• client side encryption/authorisation of any action a user can take
• this simply tunnels through the terribly insecure WWW/apache etc layer to a very limited access highly secure "hot wallet" server
• some way of verifying that client side encryption is not compromised, e.g. watchdogs running on IPs unknown to attackers
• (and lots of other watchdogs all over the place which halt service at the slightest cause for concern)

You'll do all this and then your hosting provider will be compromised by social engineering.

At the end of the day, all said and done, your most effective protection will be the offline cold wallet with manual (and slow and annoying) transfers of customer's funds, and regular audits to check all funds are actually there (mtgox lol).

btw, this is written by somebody with no background in security, imagine what somebody who did have such a background might write ;)

Final point: If you are thinking of running a wallet service, don't. Just don't. Give up now. It's going to end in tears, and death threats.

There are two things a hacker may try to get hold on your coins first is vulnerability in your code and second is to bruteforce your server to get access. In most of the cases PHP vulnerabilities are found when they make an effect on the website because every programmer thinks that his code is secure enough to be hacked. Among both of these PHP is more easy to get compromised(I am not saying that PHP is not secure). I suggest you to ask people(if you may know any security expert or hacker) to penetrate the code and try to find a vulnerability or you may start a bounty for that.

When designing a security system first you must decide on the level of security needed. What level of security are you looking for? How many $/coin will this be protecting? What are the attacks you are protecting against?

There are high end devices that self-destruct on detection of tampering or attack used by the banking industry and other places keys need to be secured, if you are creating a large-scale site definitely look into that option. As an example Apple uses HSMs (Hardware Security Modules)) to secure their message service user keys.

If you are concerned about PHP or scripting languages in general use a compiled language for the secure portion of the site. Separate the secure portion from the user facing portion of the site. By segregating the security portion into a self contained section the actual security portion will be smaller and thus easier to vet. Also reduce the interface to the secure portion and validate all inputs. You do not want the secure code intermingled with the entire site.

When I designed mass market security products I paid a top level security expert to vet my design and implementation. His first questions were along the line of what I asked above and more. Cost was about $2,000/day.

Pay attention to the answer by @user18443.