The United States government normally gives what's known as a Certificate of Naturalization to new U.S. citizens, either on Form N-550 or N-570. The certificate serves as proof that the person whose name and photo the form bears has obtained U.S. citizenship via the process known as naturalization.
Naturalization is the process by which U.S. lawful permanent residents (green card holders) may, after a certain number of years, apply to become U.S. citizens. The process involves filling out and submitting U.S. Citizenship and Immigration Services (USCIS) Form N-400, attending an interview, demonstrating the ability to speak, read, and write English, and passing a test on U.S. history and government. Once approved and sworn in at an oath ceremony, they are officially U.S. citizens.
No, people who have successfully applied to become naturalized citizens will automatically be given a certificate of naturalization at their swearing-in ceremony. You will likely be scheduled for a swearing-in ceremony after the USCIS interview at which you are approved for U.S. citizenship.
You will separately need to apply for a U.S. passport. This will be an important document to have if you travel outside the United States. It is also easier to carry around with you than the naturalization certificate (especially if you decide to frame the certificate).
You should be given an opportunity to fill out an application for a U.S. passport at your swearing-in ceremony. If you miss that opportunity, passport applications are handled by the U.S. State Department.
By the way, if you decide to frame your certificate of naturalization, you might want to make a photocopy first, particularly if you there's a possibility you will want to file petitions (Form I-130) in order to help any of your family members immigrate. (Yes, copying the certificate is illegal for most purposes, but it's okay if you're lawfully trying to petition for a family member.) See The I-130 Petition: Information for Family Sponsors.
It's possible to apply for a replacement certificate, using USCIS Form N-565. You'll need to submit a copy of the lost or destroyed certificate, if you can, along with two passport-style photos of yourself and a processing fee. If someone stole or destroyed your certificate, you'll also want to contact the local police, then obtain a police report to send to USCIS.
See the further instructions on the USCIS website page concerning this form.
No, a certificate of U.S. citizenship is the document given to people who became citizens automatically, through their parents or grandparents, via either the legal process of acquisition or derivation of citizenship. If you are a naturalized citizen, you do not need to worry about this type of certificate.