Fractional‐order operators on nonsmooth domains

The fractional Laplacian(−Δ)𝑎,𝑎∈(0,1),anditsgen-eralizations to variable-coefficient2𝑎-order pseudodif-ferential operators𝑃, are studied in𝐿𝑞-Sobolev spacesof Bessel-potential type𝐻𝑠𝑞. For a bounded open setΩ⊂ℝ𝑛, consider the homogeneous Dirichlet problem:𝑃𝑢 = 𝑓inΩ,𝑢=0inℝ𝑛⧵Ω. We find the regularityof solutions and determine the exact Dirichlet domain𝐷𝑎,𝑠,𝑞(the space of solutions𝑢with𝑓∈𝐻𝑠𝑞(Ω))incaseswhereΩhas limited smoothness𝐶1+𝜏,for2𝑎 < 𝜏 <∞,0⩽𝑠<𝜏−2𝑎. Earlier, the regularity and Dirichletdomains were determined for smoothΩby the sec-ond author, and the regularity was found in low-orderHölder spaces for𝜏=1by Ros-Oton and Serra. The𝐻𝑠𝑞-results obtained now when𝜏<∞arenew,evenfor(−Δ)𝑎. In detail, the spaces𝐷𝑎,𝑠,𝑞are identified as𝑎-transmission spaces𝐻𝑎(𝑠+2𝑎)𝑞(Ω), exhibiting estimates interms ofdist(𝑥, 𝜕Ω)𝑎near the boundary.The result has required a new development of methodsto handle nonsmooth coordinate changes for pseudod-ifferential operators, which have not been availablebefore; this constitutes another main contribution ofthe paper