By using first principles cluster calculations, we identified that Ta or W substitution for V is useful for decreasing the lattice thermal conductivity of the Fe2VAl Heusler alloy without greatly affecting the electron transport properties. It was clearly confirmed that the Fe2(V1−xTax)Al0.95Si0.05 (x=0, 0.025, 0.05), Fe2(V0.9−xTaxTi0.1)Al (x=0, 0.10, 0.20), and Fe2(V0.9−2xWxTi0.1+x)Al (x=0, 0.05, 0.10) alloys indeed possessed large Seebeck coefficient regardless of the amounts of substituted elements, while their lattice thermal conductivity was effectively reduced. As a result of partial substitution of Ta for V, we succeeded in increasing the magnitude of the dimensionless figure of merit of the Heusler phase up to 0.18, which is five times as large as the Ta-free compound.