I’m quoting an answer I found online, which quotes from Rachel Brown’s The Weaving, Spinning, and Dyeing Book, ed. 1987. I probably have other books that explain this as well, but I’m feeling lazy today.

“Cotton, linen, woolen, and worsted yarns can all be described by count. #1 describes the size of the yarn that will be produced when spinning a certain base yardage from 1 pound of fiber…. For cotton, No. 1 is the size thread that results from spinning 840 yards from 1 pound of cotton fiber. #2 would be the size thread if twice this yardage (1,680 yards) was spun from 1 pound of cotton… So the higher the number, the finer the thread.”

Different fibers might use different numbers for their base yardage, but for all fibers, the higher the first number, the finer the thread.