GeoServer supports a range of custom parameters for WMS GetMap requests:

FILTER: filter rendered content beyond SLD through OGC Filter predicate language
CQL_FILTER: filter rendered content beyond SLD through the OGC Common Query Language predicate language
FEATUREIDS: filter rendered content beyond SLD through a list of Feature identifiers
BUFFER: get seamless tiles by indicating the server to use a virtual image a bit larger so it picks border features
PALETTE: force the use of a specific 8-bit color pallete
TILED: WMS Tiling client extension
TILESORIGIN: Tiles grid origin, as per the WMS Tiling client extension
STARTINDEX: Start rendering a vector layer at a specified Feature index
MAXFEATURES: Limit the amount of features to render for vector layers

Filter extensions

WMS base spec does not allow for filtering data beyond the BBOX parameter, and provides some user driven filtering in the WMS-SLD extensions, but requires the user to provide a full SLD document (that is, not only filtering, but also styling).
Geoserver allows for the same filtering capabilities WFS provides:

• FILTER: the filter parameter is a list of OGC filters enclosed in () parenthesis. And empty parenthesis couple means no filtering. When the FILTER parameter is used, either a single filter is provided (that will be applied to all layers) or a list or filters, with the same number of filters as the layers in the WMS request. For example (request has been split over multiple lines for better readability, the filter has been URL-encoded as required by HTTP standard):

  http://localhost:8080/geoserver/wms?
  request=GetMap
  &layers=tiger:poly_landmarks,tiger:tiger_roads
  &styles=,
  &bbox=-73.97,40.74,-73.93,40.78
  &Format=image/png
  &width=344
  &height=500
  &srs=EPSG:4326
  &filter=(%3CFilter%3E%3CPropertyIsEqualTo%3E%3CPropertyName%3ELANAME%3C/PropertyName%3E%3CLiteral%3EEast%20River%3C/Literal%3E%3C/PropertyIsEqualTo%3E%3C/Filter%3E)
  (%3CFilter%3E%3CPropertyIsEqualTo%3E%3CPropertyName%3ENAME%3C/PropertyName%3E%3CLiteral%3EWest%20Rd%3C/Literal%3E%3C/PropertyIsEqualTo%3E%3C/Filter%3E)

• CQL_FILTER: a filter list encoded in CQL. Same filter as before, but encoded in CQL. Beware, at the time of writing CQL does not allow for all kinds of filter OGC filter supports, in particular, it's not possible specify FID filters.

  http://localhost:8080/geoserver/wms?
  request=GetMap&
  layers=tiger:poly_landmarks,tiger:tiger_roads
  &styles=,
  &bbox=-73.97,40.74,-73.93,40.78
  &Format=image/png
  &width=344
  &height=500
  &srs=EPSG:4326
  &cql_Filter=LANAME%20=%20%22East%20River%22;NAME%20=%20%22West%20Rd%22

  If you do not want to filter one of the layers, you can use the INCLUDE filter. For example,

  &cql_Filter=LANAME%20=%20%22East%20River%22;INCLUDE

• FEATUREIDS: a list of feature ids separated by comma

Both FILTER and CQL_FILTER can be specified in two ways (same as WFS filters):

• single filter: in this case, the filter will be applied to all layers
• filter list: one filter per layer, each filter will be applied to the corresponding layer

Rendering extensions

• BUFFER: the number of extra pixels that should be taken into account when rendering a map for catching features that are outside the current bounding box, but whose styling is thick enough to make them visible inside the painted area. Think, for example, points depicted as big red circles. This is especially useful when working with tiled clients, such as OpenLayers. Geoserver will try to compute the buffer automatically by parsing the SLD specification, but that will work only if line widths and point symbolizers sizes are literal values. When the size is linked to attributes you'll have to specify a buffer in the WMS request.
• PALETTE: forces the generation of a 256 colors (with eventual transparency) image using the specified palette. This allows for fast and small image generation, but forces antialising off and adapts colors to whatever is in the palette. The palette can be "SAFE", to use the internet safe palette, or else, create a "/palettes" directory in the GeoServer data directory, and put there .gif or .png files with a 256 colors palette and refer to the file using just the name (without the extension. For example, if you provide a "/palettes/mypalette.gif" then you can refer it using "...&palette=mypalette".
  Using a 256 colors palette is especially intersting for simple vector layers that do use few colors and have simple geometries. When dealing with a map with imagery, we suggest, if the client allows it, to serve the imagery as a separate layer with full color and jpeg format, and the vector layers as png/gif layers with 256 colors palette.
  See also the paletted images research page for some samples and measures.

On the fly meta tiler

When using a tiled clied such as OpenLayers there are issues with labels being heavily duplicated. To cope with this, on the fly meta tiling has been created, that is, server side tiles are rendered in a bigger 3x3 meta tile, and the split into smaller tiles that are cached for a certain (usually short) amount of time in memory.
In order for meta-tiling to figure out that the request is tiled, tile size must be 256x256 request needs to include two extra vendor options:

• TILED=yes (conforming to the OSGEO WMS tiling client recommendation specification).
• TILESORIGIN=x,y, where x and y are the coordinates of the lower left corner or the layer in OpenLayers, that is, the origin or the tile grid.

Google Earth KML/KMZ extensions

• KMATTR Determines whether the KML returned by GeoServer should include clickable attributes.
• KMSCORE Whether GeoServer should render data as vector or raster. See: KML Output page for more information

Paging

Starting at version 1.6.5, the official GeoServer WMS output formats supports the concept of maxFeatures and startIndex. Both can be used together to provide "paging" support, or in isolation.

Though paging for map production may seem strange at first, there are a number of use cases where it can be of great use, mostly because we don't support only raster output formats, but some vector based too, like KML and SVG.
One use of paging is KML crawling, where for a large number of features it is desirable to be able of getting the whole map in bunches.

Note, however, that not every layer will support paging. So you better get sure your target layer does or WMS Exception report will be generated instead of your map.

• STARTINDEX A positive integer value that specifies at which index in an ordered list of Features to start rendering. The concept is the same than the SQL OFFSET clause. For a layer to be queried this way, the underlying feature source shall support paging. At the time of writing the only data source known to do that is the PostGIS back end.

• MAXFEATURES Integer valyue that sets a limit on the amount of features rendered. If used in conjunction with STARTINDEX, the Features rendered will be the ones starting at STARTINDEX. The concept is the same than the SQL LIMIT clause.