Abstract
A search for the nonresonant production of Higgs boson pairs in the HH -> b (b) over bar tau(+)tau(-) channel is performed using 140 fb(-1) of proton-proton collisions at a center-of-mass energy of 13 TeV recorded by the ATLAS detector at the CERN Large Hadron Collider. The analysis strategy is optimized to probe anomalous values of the Higgs boson self-coupling modifier kappa(lambda) and of the quartic HHVV (V = W, Z) coupling modifier kappa(2V). No significant excess above the expected background from Standard Model processes is observed. An observed (expected) upper limit mu(HH) < 5.9(3.3) is set at 95% confidence-level on the Higgs boson pair production cross section normalized to its Standard Model prediction. The coupling modifiers are constrained to an observed (expected) 95% confidence interval of -3.1 < kappa(lambda) < 9.0 (-2.5 < kappa(lambda) < 9.3) and -0.5 < kappa(2V) < 2.7 (-0.2 < kappa(2V) < 2.4), assuming all other Higgs boson couplings are fixed to the Standard Model prediction. The results are also interpreted in the context of effective field theories via constraints on anomalous Higgs boson couplings and Higgs boson pair production cross sections assuming different kinematic benchmark scenarios.