A search is performed for resonant and non-resonant Higgs boson pair production in the γγbb¯ final state. The data set used corresponds to an integrated luminosity of 36.1 fb−1 of proton-proton collisions at a centre-of-mass energy of 13 TeV recorded by the ATLAS detector at the CERN Large Hadron Collider. No significant excess relative to the Standard Model expectation is observed. The observed limit on the non-resonant Higgs boson pair cross-section is 0.73 pb at 95% confidence level. This observed limit is equivalent to 22 times the predicted Standard Model cross-section. The Higgs boson self-coupling (κλ=λHHH/λSMHHH) is constrained at 95% confidence level to −8.2<κλ<13.2. For resonant Higgs boson pair production through X → HH → γγbb¯, the limit is presented, using the narrow-width approximation, as a function of mX in the range 260 GeV <mX< 1000 GeV. The observed limits range from 1.1 pb to 0.12 pb over this mass range.