The address prefix determines how a wallet will interperet the address. Let's look at an example.

Generate a P2PKH address

1. Take a public key, e.g. 023cba1f4d12d1ce0bced725373769b2262c6daa97be6a0588cfec8ce1a5f0bd09

2. Apply HASH160: 3a38d44d6a0c8d0bb84e0232cc632b7e48c72e0e

3. Prepend the version prefix for P2PKH (0x00): 003a38d44d6a0c8d0bb84e0232cc632b7e48c72e0e

4. Base58Check encode it: 16JrGhLx5bcBSA34kew9V6Mufa4aXhFe9X

See How to Generate a Bitcoin Address — Step by Step for command line steps.

Prefixes

Bitcoin has 2 main address prefixes (mainnet):

P2PKH (Pay to Public Key Hash): 1
e.g.: 17VZNX1SN5NtKa8UQFxwQbFeFc3iqRYhem

P2SH (Pay to Script Hash): 3
e.g.: 3EktnHQD7RiAE6uzMj2ZifT9YgRrkSgzQX

Addresses are just representations of the information necessary (scriptPubKey) to create a transaction output, which is part of the transaction and which is validated and stored on the network. It is only used by wallet application software, addresses are not stored on the Blockchain in this format.

The wallet will determine, based on the address prefix, when it is decoded, how to construct this scriptPubkey. For P2PKH this would be in this form:

OP_DUP OP_HASH160 <pubKeyHash> OP_EQUALVERIFY OP_CHECKSIG

Where you will notice, the only info it needs is the pubKeyHash which is contained in the decoded address (step 2 above). However, the format for a P2SH scriptPubkey is OP_HASH160 <scriptHash> OP_EQUAL. If you used the wrong prefix, the wallet would create the wrong script and a tx will most likely burn those coins because no one will be able to guess keys/scripts that will successfully satisfy the scriptPubkey.