It stores an obfuscated version of the password in the given file (default $HOME/.vnc/passwd). The vncserver program runs vncpasswd if necessary the first time you start a VNC desktop, and invokes Xvnc with the appropriate -PasswordFile option. vncviewer can also be given a password file to use via the -passwd option.
The password must be at least six characters long. When allowing legacy viewers to connect (SecurityType includes VncAuth) then only the first eight characters are significant. All characters of the password are significant when using RA2 or RA2ne as the SecurityType.
Note that the stored password is not encrypted securely - anyone who has access to this file can trivially find out the plaintext password, so vncpasswd always sets appropriate permissions (read and write only by the owner). However, when accessing a VNC desktop a challenge-response mechanism or other encryption is used over the wire making it hard for anyone to crack the password simply by snooping on the network.