Give this a read
https://yegorshytikov.medium.com/magento-2-msi-multi-source-inventory-performance-issue-fix-3380509df073
Also, make sure have it in production mode, aren't loading jquery/ui and use PHP 7.4 or 8.1. 8.0 in all my testing is slower than 7.4 by double
Forget VARNISH, go for Redis...