You can create lots of different types of content on your site.
Posts can be anything, it's just information that is grouped into different categories. They can be galleries, essays, blog posts... you just need to click the correct category to tell it where to exist on the site.
Posts are useful when there is many of them. Say a sports portfolio could be full of gallery posts called football, hockey, rugby, swimming, track and field etc. This way you can send a client a link to a page where all those can be chosen from and allow you to seperate out fashion work or general portraiture from this section.
Suited to content that rarely changes such as bio information, contact etc. Your homepage is also a page, so try not to delete this unless you really know what you're doing.
Categories are a way to group posts. You can have groups of blog posts called a blog or maybe diary. You can have groups of gallery posts called portfolios, or maybe sports with posts called things like football, hockey, etc. Imagine categories as a draw and your content is all the things inside it. Labels help people find things quicker.
If you have this enabled on your site, it's just means you create your gallery posts via a different tab on the menu, but the rest is the same.
is normally created as a 'Page'; The content of these pages rarely change and tend to exist as a stand alone piece of content such as a contact page or as a child of a standalone piece of content, say how a page listing exhibitons could be a sub page of a bio.
is usually created as a 'Post'; Posts can be grouped into categories and can have many categories associated with it. Posts are suited to blog posts, news, projects and portfolios. A couple of examples of this are how a portfolio could be in a group of featured portfolios, commissions and portraits at the same time.