The dynamics of soft mechanical metamaterials provides opportunities for many exciting engineering applications. Previous works have shown tremendous success in describing the unique nonlinear dynamics of certain types of soft mechanical metamaterials. However, capturing the nonlinear dynamic response of these materials especially those with complex geometries, can be a challenge due to the strong nonlinearity and large computational cost. An efficient and reliable framework to predict the overall response of the metamaterials based on the geometry of their building blocks is not only key to understanding the unique behavior of metamaterials, but also vital to the rational design of such materials. In this work, we propose metamaterial graph network (MGN), a machine learning approach to address this challenge. MGN is based on a graph that represents the lattice-like metamaterial structure. The trained MGN is capable of simulating the dynamics of a metamaterial structure with over 200 by 200 unit cells, a task that is practically impossible for traditional direct numerical simulation using the finite element method. We also verify the accuracy of the proposed MGN against several representative numerical examples. In the first two examples, we show that MGN successfully captures the well-known pattern transformation behavior of porous metamaterials. In the later examples, we consider wave propagation in a dynamic setting and show that MGN produces quantitatively accurate results compared with direct numerical simulation. An additional feature of MGN is that defects/inhomogeneities can be easily incorporated into the metamaterial structure. We expect our method to open a new avenue for the study and modeling of mechanical metamaterials.