When creating a commitment transaction, you consume the multi-sig output of the funding transaction as the input. CSV is a relative timelock opcode which means the time specified in the CSV is a function of when the funding transaction was mined. For example, if you create a 100 block CSV output, that output is only consumable after 100 blocks have passed since the funding transaction was created.
When sending a HTLC, the first node it passes through will be the one you have direct channel with. Since you and your channel counterparty are on the same page related to the funding transaction you can freely use CSV opcode when timelocking the to_self_delay for the local node output. However, from there on it will be sent to nodes you do not have direct open channels. These nodes have channel open with other nodes whose funding transaction time will be different from the one you had opened with your channel counterparty. Thus specifying CSV in a HTLC output, would create ambiguous timeouts from the point of view of different channel parties. To ensure consistency, you need to use absolute locktime, which is why CLTV is chosen.