Improves rgb -> gray16 conversion

Fixes Ticket3422

The pam and png output files look visually similar, in both cases the
dynamics increase to 0x0 -> 0xfffb.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 37c07d45)