The dynamics of large networks is an important and fascinating problem. Key examples are the Internet, social networks, and the human brain. In this paper we consider a model introduced by DeVille and Peskin  for a stochastic pulse-coupled neural network. The key feature and novelty in their approach is that they describe the interactions of a neuronal system as a discrete-state stochastic dynamical network. This idealization has two benefits: it captures essential features of neuronal behavior, and it allows the study of spontaneous synchronization, an important phenomenon in neuronal networks that is well-studied but unfortunately far from being well-understood. In synchronous behavior the firing of one neuron leads to the firing of other neurons, which in turn may set off a chain reaction that often involves a substantial proportion of the neurons. In this paper we rigorously analyze their model. In particular, by applying methods and tools that are frequently used in theoretical computer science, we provide a very precise picture of the dynamics and the evolution of the given system. In particular, we obtain insights into the coexistence of synchronous and asynchronous behavior and the conditions that trigger a "spontaneous" transition from one state to another.