Abstract
Searches are performed for nonresonant and resonant di-Higgs boson production in the b b ¯ γγ final state. The dataset used corresponds to an integrated luminosity of 139 fb-1 of proton–proton collisions at a center-of-mass energy of 13 TeV recorded by the ATLAS detector at the CERN Large Hadron Collider. No excess above the expected background is found and upper limits on the di-Higgs boson production cross sections are set. A 95% confidence-level upper limit of 4.2 times the cross section predicted by the Standard Model is set on pp → HH nonresonant production, where the expected limit is 5.7 times the Standard Model predicted value. The expected constraints are obtained for a background hypothesis excluding pp → HH production. The observed (expected) constraints on the Higgs boson trilinear coupling modifier κλ are determined to be [ -1.5,6.7 ] ( [ -2.4,7.7 ] ) at 95% confidence level, where the expected constraints on κλ are obtained excluding pp → HH production from the background hypothesis. For resonant production of a new hypothetical scalar particle X ( X → HH → b b ¯ γγ ), limits on the cross section for pp → X → HH are presented in the narrow-width approximation as a function of mX in the range 251 GeV≤mX≤1000 GeV . The observed (expected) limits on the cross section for pp → X → HH range from 640 fb to 44 fb (391 fb to 46 fb) over the considered mass range.