Understanding of early brain changes has the potential to investigate imaging biomarkers for pre-symptomatic diagnosis and thus opportunity for optimal therapeutic intervention, for example in early diagnosis of infants at risk to autism or altered development of infants to drug exposure. In this paper, we propose a framework to analyze longitudinal changes of structural connectivity in the early developing infant brain by exploring underlying network components of brain structural connectivity and its changes with age. Structural connectivity is a non-negative sparse network. Projective non-negative matrix factorization (PNMF) offers benefits in sparsity and learning fewer parameters for non-negative sparse data. The number of matrix subcomponents was estimated by automatic relevance determination PNMF (ARDPNMF) for brain connectivity networks for the given data. We apply linear mixed effect modeling on the resulting loadings from ARDPNMF to model longitudinal network component changes over time. The proposed framework was validated on a synthetic example generated by known linear mixed effects on loadings of the known number of bases with different levels of additive noises. Feasibility of the framework on real data has been demonstrated by analysis of structural connectivity networks of high angular resonance diffusion imaging (HARDI) data from an ongoing neuroimaging study of autism. A total of 139 image data sets from high-risk and low-risk subjects acquired at multiple time points have been processed. Results demonstrate the feasibility of the framework to analyze connectivity network properties as a function of age and the potential to eventually explore differences associated with risk status.