Mempools between nodes will be different. There is no "supposed to match after a while" as the mempool does not have anything that makes all nodes' mempools converge to the same thing. There is no "the mempool", there is only "a node's mempool" and these will differ based on the settings for accepting transactions.
The settings on your node's mempool are likely much stricter than the settings for blockchain.info's mempool since blockchain.info shows data for a variety of transactions that most nodes will reject (e.g. transaction fees less than 1000 sat/kB, creating dust outputs, unconfirmed transaction chains longer than 20 txs, etc.).