I wanted to know, I make album covers and lyric videos using some free fonts from this website, is it legal to use the fonts without paying if I'm not making any money from them?

Commercial use includes any redistribution of images outside of your PERSONAL use. So, no, if you're using them and making album covers for other people, you're doing a commercial use, even if you're not charging for them. (IE, it's one thing to give one to a friend, it's a whole 'nother thing to give them to thousands of people over the internet.)

If it's realy free font, no matter if you make money with your product.
For the other ones, contact the author, explain your project and ask him if you can use his creation.

I think that a lot of the confusion about personal use is caused by the use of the word personal which imho it far too vague and leaves a bundle of opportunities for those who wish to interpret 'personal' as 'I am only using it myself' - for whatever I want to.
It might be a solution changing 'personal' to something that has as much more strict meaning. One might think about 'private' or maybe even 'strictly private', which leaves little room for free interpretation and by definition excludes any other use.