FGLM is a Groebner basis conversion algorithm. This means it takes a Groebner basis of an ideal with respect to one monomial order and changes it into a Groebner basis of the same ideal over a different monomial order. Conversion algorithms can be useful since sometimes when a Groebner basis over a difficult monomial order (such as lexicographic or an elimination order) is desired, it can be faster to compute a Groebner basis directly over an easier order (such as graded reverse lexicographic) and then convert rather than computing directly in the original order. Other examples of conversion algorithms include the Groebner walk and Hilbertdriven Buchberger.
FGLM performs conversion by doing linear algebra in the quotient ring R/I, where I is the ideal generated by the original Groebner basis in the polynomial ring R. This requires that I is zerodimensional.
In Macaulay2, monomial orders must be given as options to rings. For example, the following ideal has monomial order given by graded reverse lexicographic (which is also the default order in Macaulay2).


If we want a Groebner basis of I1 with respect to lexicographic order we could substitute the ideal into a new ring with that order and compute directly,



but it may be faster to compute directly in the first order and then use FGLM.


Further background and details can be found in the following resources:
The C++ implementation of the algorithms in Version 1.2.0 was sponsored by an IMA Coding Sprint at the Cornell University.
The ideal generated by the Groebner basis must be zerodimensional.
This documentation describes version 1.1.0 of FGLM.
The source code from which this documentation is derived is in the file FGLM.m2.