Building of atlases plays a crucial role in the analysis of brain images. In scenarios where early growth, aging or disease trajectories are of key importance, longitudinal atlases become necessary as references, most often created from cross-sectional data. New opportunities will be offered by creating longitudinal brain atlases from longitudinal subject-specific image data, where explicit modeling of subject’s variability in slope and intercept leads to a more robust estimation of average trajectories but also to estimates of confidence bounds. This work focuses on a framework to build a continuous 4D atlas from longitudinal high angular resolution diffusion images (HARDI) where, unlike atlases of derived scalar diffusion indices such as FA, statistics on dODFs is preserved. Multi-scalar images obtained from DW images are used for geometric alignment, and linear mixed-effects modeling from longitudinal diffusion orientation distribution functions (dODF) leads to estimation of continuous dODF changes. The proposed method is applied to a longitudinal dataset of HARDI images from healthy developing infants in the age range of 3 to 36 months. Verification of mixed-effects modeling is obtained by voxel-wise goodness of fit calculations. To demonstrate the potential of our method, we display changes of longitudinal atlas using dODF and derived generalized fractional anisotropy (GFA) of dODF. We also investigate white matter maturation patterns in genu, body, and splenium of the corpus callosum. The framework can be used to build an average dODF atlas from HARDI data and to derive subject-specific and population-based longitudinal change trajectories.