Let the forces be F1 and F2.
probably you mean |F1|+|F2| = 16 ---------------- (1) , and
|F|² = 64 = |F1|²+|F2|² +2*|F1|*|F2|*cos theta ----------------------- (2)
Squaring (1) we get
|F1|²+|F2|² +2*|F1|*|F2| = 256 ---------------------- (3)
(3) - (2) gives
2*|F1|*|F2|*[1 - cos theta] = 192 ------------------- (4)
2*|F1|*|F2| - (2*F1.F2) = 192 or
|F1|*|F2| - (F1.F2) = 96 ----------------------------------(4a)

Let |F1| < |F2|
We have F1.(F1+F2) = |F1|² + F1.F2 = 0 or F1.F2 = -|F1|² -------------- (5)
Substituting (5) in (2), we get
|F1|² - |F2|² = 256 ------------ (6)
(|F1| +|F2|)*(|F1| -|F2|) = 256 ------------------- (6a)
Substituting (5) in (4a), we get
|F1|*|F2| -|F1|² = 96 or
|F1|*[|F2| - |F1|] = 96 ------------------- (7)
Dividing (6a) by (7), we get
(|F1| +|F2|)/|F1| = 8/3 or
3|F1| + 3|F2| = 8|F1| or
5|F1| - 3|F2| = 0 ---------------------- (8), multiplying (1) by 3 we get
3|F1| + 3|f2| = 48 --------------------- (9), adding (8) and (9) we get
8|F1| = 48 or |F1| = 6 N and |F2| = 10 N