Would the opponent know that I've pressed the Down button?

Nope! Your previous matches will never be able to know who pressed the down button.
So if you ever meet someone who behaved inappropriately, please be sure to tap the down button.
Your contribution makes Vividi more safe and enjoyable! 🙂