If you want to browse your music by album or artist on the iPod Nano (4th generation), you can. The option is kind of hidden in a way though.

From the Now Playing screen, press and hold the Center button to browse by the album first.

1. Select Browse Album or Browse Artist.