First, it is important to note that the following information is not exhaustive. There will always be exceptions. Through many years and after having sold many thousands of rare and valuable books, we tend to see recurring themes. Usually (again, not always), when a 19th century book is valuable, it will tend to fit within one of the following categories:
1: Important and recognizable fiction such as first editions by Mark Twain, Edgar Poe, Charles Dickens, Thomas Hardy, Flaubert, and other household names. You will probably recognize the titles of these types of books: The Adventures of Huckleberry Finn, Great Expectations, A Christmas Carol, Uncle Tom's Cabin, etc. Remember that just because an author and title are recognizable is not enough. It must be a first edition. If you see a copy of Shakespeare's Romeo and Juliet printed in 1850, then you know it cannot be a first edition because Shakespeare died about 200 years earlier. Key point here: During the 1800s (particularly the mid to late 1800s), reprints of classic works are extremely common. You will see plenty of Shakespeare, Milton, and similar books reprinted. In fact, you will also see many reprints of works by 19th c. authors such as the ones listed above. The edition is all important. Complicating matters, it was common in the late 1800s for a reprint publisher not to date the book but only provide the copyright date. Always look for a date at the bottom of the title-page.
2: Books dealing with exploration. Some of the more valuable books printed during this period include those dealing with exploration. For example, works dealing with the exploration of the American West, exploration of Africa, exploration of the seas/islands, and etc. Often, these books tend to contain maps or illustrations which are important. A common value-killing issue tends to be books which are lacking maps or illustrations, so completeness is important. Having a first edition is important, as usual. Many of these types of books are important because they contain new discoveries.
3: Illustrated books, Atlases with hand-colored maps and other works with hand-colored "plates" (engraved illustrations.) Always pay attention to any 19th century book that contains color illustrations. Often, the coloring was done by hand. It takes a bit of a trained eye, but if you look very closely you can often see coloring slightly outside of the image lines. But other books from this period also contained chromolithographs which essentially was a way of printing illustrations in color. These can be of value as well. So, to keep it simple, if you see colored illustrations, then pay close attention. Black and white illustrations are rather common; however, if you see a slight impression around the border of the image, then you know the image is a plate. A plate is an image that is carved or engraved, inked and then pressed onto the paper (leaving an impression around the border.) These are less common than other b/w illustrations, so those are worth checking as well.
4: Civil War era books printed in the Southern U.S. These are called Confederate Imprints. Essentially, these are books that were printed in the Confederate States of America during the Civil War. Due to the war, fewer books were printed in the South; also, after the war, many of those books were destroyed. So, if you have book that was printed in the Confederate states during the Civil War, it is typically quite rare. The collecting of Confederate Imprints is an entire field of its own. However, most books about the Civil War itself, mostly printed in the North during and after the Civil War, are quite common. For Civil War books, you want to look for Confederate imprints, military manuals such as field artillery manuals, soldier's pocket-Bibles, first edition accounts of specific military engagements, companies or regiments. Stay away from general histories of the War, or the general "War of the Rebellion," the "Life of Gen. Grant", and other such general biographies and accounts; such books are common.
5: Books printed by fine presses such as the Kelmscott press. Typically, these books are limited editions (stated as such at the front of back of the book.) These books are finely printed and when you open them typically you recognize this when you open the book and see the paper and printing quality.
The above is not comprehensive, but most valuable books from the 1800 to 1900 period tend to fall into those categories in some way. There are always exceptions.