(fwMath): use glm vector when computing barycenters and fix mistake

Description

• Replace all fwVec3 structure by ::glm::dvec3, along with glm maths functions.
• Fix mistake in dot product when computing barycentric coordinates.
• Update unit-test

Closes #510 (closed)

How to test it?

1. Compile fwMath
2. Run fwMathTest