Let the ball A have mass m and initial velocity u. 

Then E_i = 1 x m x u²/2 = E

And momentum = p mu, its a constant. 

The potential energy becomes maximum when the kinetic energy is minimum. So this can occur when the bodies moves with the same velocities. 

Minimum kinetic energy =E_f = p²/2(m + nm) = m²u²/2m(1 + n)

Thus, maximum potential energy = E_{f  ‾}  E_i = [1 x m x u²/2] – m²u²/2m(1 + n)

So, E [1- (1/{1+n})] 

That is, nE/(n + 1).

Thus the maximum potential energy will be nE/(n + 1).