I'm just worried.

Downloading it always from DaFont... not from other crappy sites...

not from ♦ other ♦ crappy sites... which implies ...

To be active, a virus need to be executed, from an exacutable file, a file that include program or macro or the boot of a file system.

Font don't include this possiblities.
To my knowledge, font can't be infected with a virus (like all files) but can't transmit a virus because this virus can't be activate.

If you're that worried, have an antivirus software check your files.