They're used to exercise the hands, during meditation, or to relax the hands and dissipate nervous tension. In the book, a psychiatrist says that Queeg's hands tremble, so he rolls the balls as a way to steady them. Queeg was actually using steel ball bearings, much easier to obtain aboard a ship. Edit