I know several people have chimed in and said that it is the block size that is limiting the transactions per second, but that doesn't make sense to me, I don't see, nor understand, what block size has to do with transactions per second...

A block can hold no more than 1 MB of transactions. This sets the network's transaction speed limit.

A block is generated once every 10 minutes on average. We can calculate the block chain's write speed with:

w = s * g

where w is write speed MB/hour, s is block size limit in megabytes (1 MB/block), and g is the rate of block generation per hour (6 blocks/hour).

Plugging in the value yields:

w = 1 MB/block * 6 blocks/hour = 6 MB/hour

Converting to bytes/second, that's:

6 MB/hour * (1 hour/60 minutes) * (1 minute/60 seconds) * (1,000,000 bytes/1 MB) = 1,700 bytes/second

In other words, think of the blockchain as a hard drive. We can write to this hard drive at a rate of 1,700 bytes/second.

Even if we develop faster computers, we can't write data faster than this because the network rejects any block larger than 1 MB. We need to make some allowances for SegWit, but for now, let's keep things simple.

A one-input, one-output Alice-Pays-Bob transaction requires ~ 200 bytes. This is the absolute best case. We can, therefore, obtain an upper bound on the transaction rate using:

v = w / z

where v is the maximum transaction volume, w is the write speed calculated above (1,700 bytes/second), and z is the average size of a transaction (200 bytes/transaction).

Plugging in the numbers:

v = 1,700 bytes/second * (1 transaction / 200 bytes) = 8 transactions/second

The answer to my question should be something like verification of sufficient funds per transaction input address takes 0.1 seconds, or as a whole verification of each transaction takes 0.3 seconds.

The network might appear to have a limit based on hardware or validation, but it does not. At least not directly.

As to why we should limit the block chain's write speed in the first place, the answer has to do with network security. Increasing write speed means:

1. data flows through the network at a higher rate;
2. more data must be stored by every node;
3. more transactions must be validated by every node.

Every node sends and receives every block, and every block must be stored. Increasing the blockchain write speed means that every node pays the price. Some won't be able to keep up, regardless of write speed.

Transaction validation is a resource hog due to signature validation and UTXO set manipulation. Storage space and bandwidth are also considerations, but less resource-intensive than validation.

Now, if you're asking why we have a 1 MB limit given XXX computer has such-and-such technical capabilities, that's the realm of the Never Ending Debate. Calculations have been done and simulations have been reported. They are not hard to find. They are, however, hard to believe given their wildly-differing claims and underlying assumptions.

The best thing I can recommend is to download a copy of Bitcoin Core 0.16.0 and try syncing a node from scratch. Report back when you're done.

The write speed limit we see is a compromise between inclusivity (more people using bitcoin) and security (difficulty of double-spending attacks). The limit may not have been set correctly, but figuring out where (or even how) to set it is exceedingly difficult. Not because of lack of ideas, but because of too many and lack of widely-accepted selection criteria.

After reading the existing answers, and your comments on this, I believe what you're looking for is this.

Your question is based on a misunderstanding. The figure "Bitcoin can process 7 transactions per second" is not a burst rate. It is talking about the long-term network-wide average, and results from the fact that (a) blocks are limited in size and (b) the block production rate is limited.

Individual Bitcoin node implementations can process far more transactions per second than that, but they necessarily need to be many times faster than the rate at which the whole network operates.

When a new block is created, nodes in the network need to be able to process it quickly, in order to not delay how long it takes before other miners can start building on top of that block. Due to the inherent variance in block production, the next block has a small but significant probability of being found just seconds later. If network nodes can't verify the previous block within that time, they are disadvantaged. However, the miner who created the block himself has no such penalty (he already knows his own block is valid). Generally these delays in processing tend to benefit the larger and well-connected miners proportionally more than smaller ones - an effect that is very detrimental to decentralization of the network, so to be avoided at all cost. For that reason, the time between blocks must be many times larger than the time it takes to verify the transactions in it.

The bottleneck is the number of transactions that can fit in a block, and the frequency of blocks.

You can create an infinite number of transactions per second and broadcast them to the network, but that is irrelevant because those transactions are not considered final. They are considered reversible and insecure.

For a transaction to be considered final and secure, it must be included in a block. So the transaction rate that is often cited as being 3 to 7 transactions per second is limited by the number of transactions that can fit in a block and the frequency of blocks as that metric only considers transactions that are final and secure.

The actual number of transactions per second that bitcoind can verify is somewhere in the thousands of transactions per second. Transactions can be verified very quickly and the limiting factor becomes hardware. Furthermore, it depends on your hardware; a slow CPU could be the bottleneck for some devices, while a slow hard drive may be the bottleneck for others.

However the number of transactions that bitcoind can verify is irrelevant as those transactions are not confirmed and thus not final, which is what really matters. The rate at which transactions become confirmed and final is limited by the block size as explained earlier.

I am curious, what is the bottle neck for the transaction rate?

If I broadcast several thousand valid transactions per second onto the Bitcoin network;

1. Those transactions will be validated when received by each node and enter each node's individual mempool until the individual mempool resources are exhausted, in which case each node has its own rules about how to handle additional new transactions. Some nodes with slower hardware may not keep up but that is irrelevant as the mempool is not ever in consensus.

2. A miner draws a number of transactions from their own mempool using any selection criteria that they wish (transactions in mempool have already been validated), not more than the maximum block size in block weight per consensus rules, constructs a proposed block and attempts to make it valid by incrementing nonce (simplified). This process of solving a block takes on average ten minutes and is regulated to maintain that average block time per consensus rules. Ten minutes is currently considered the optimum for security and reliability.

3. Since there is a fixed maximum block size in block weight AND a regulated average interval, this is the limiting factor for transaction bandwidth rate since a non-infinite number of transactions may be compiled into a valid block and will take an average of ten minutes to solve. This can be shown as: (average number of transactions that will fit in a block in block weight)/average block time in seconds = approximately 7 transactions per second maximum, however, this ideal figure would be seldom possible to maintain using real data.

Note that I have marked valid at the top of these statements in bold.

My question is, what process is hogging the resources limiting the amount of transactions per second...

It is not a process, it is the application of consensus rules.