RAPID COMMUNICATIONS PHYSICAL REVIEW A 72 051604 R 2005 Engineering superfluidity in Bose Fermi mixtures of ultracold atoms 1 D W Wang 1 M D Lukin 2 and E Demler2 Department of Physics National Tsing Hua University Hsinchu Taiwan 300 Republic of China 2 Physics Department Harvard University Cambridge Massachusetts 02138 USA Received 19 October 2004 published 17 November 2005 We investigate many body phase diagrams of atomic boson fermion mixtures loaded in the two dimensional optical lattice Bosons mediate an attractive finite range interaction between fermions leading to fermion pairing phases of different orbital symmetries Specifically we show that by properly tuning atomic and lattice parameters it is possible to create superfluids with s p and d wave pairing symmetry as well as spin and charge density wave phases These phases and their stability are analyzed within the mean field approximation for systems of 40K 87Rb and 40K 23Na mixtures For the experimentally accessible regime of parameters superfluids with unconventional fermion pairing have transition temperature around a percent of the Fermi energy DOI 10 1103 PhysRevA 72 051604 PACS number s 03 75 Mn 03 75 Hh 74 20 Fg 74 20 Rp Mixtures of quantum degenerate atoms recently became a subject of intense studies Examples include recent experimental observations of instabilities in Bose Fermi mixtures 1 superfluidity of fermion pairing 2 and condensation of molecules in fermionic mixtures 3 Many other intriguing many body effects have been proposed theoretically They include formation of composite particles 4 appearance of charge density wave order 5 phonon induced fermion pairing 6 7 and polaronic effects 8 In this paper we study quantum phases of boson fermion mixtures BFM in twodimensional 2D optical lattices The fermionic atoms are prepared as a mixture of two hyperfine spin states which interact via short range repulsive interaction Density fluctuations in a condensate of bosonic atoms induce an attractive interaction between fermions which is of finite range Competition between these two types of interactions results in several many body phases of fermions by appropriately choosing atomic and lattice parameters These include charge and spin density wave phases CDW SDW as well as superfluid states with unconventional pairing of fermions We discuss how these phases can be observed in realistic parameter regime of interest 9 Experimental realization of such systems should provide critical insights into understanding several important strongly correlated electron systems including quasi 2D unconventional superconductors such as high Tc cuprates 10 and organic conductors 11 displaying d wave superconductivity as well as ruthenates 12 and Bechgaard salts 11 displaying p wave superconductivity We first describe the microscopic theory for the BFM systems in 2D optical lattice When the lattice potential is strong enough the BFM system can be described by the single band Hubbard type Hamiltonian 8 13 f f f k f k k f k f k H kb b k b k k k 1 Ubb b b f f f Ubf kb k U f f k k k 2 k k 1 where b k and f k s are the annihilation operators for bosonic b p and fermionic atoms with momentum k kb pb p k b b is the boson density operators and k k b where 1050 2947 2005 72 5 051604 4 23 00 kb tb k is the single particle energy with tb being the tunneling amplitude of bosons between neighboring sites and k 2 cos kx cos ky lattice constant is set to be unit b is the boson chemical potential Similar notations also apply to f f k in Eq 1 fermions with superscript f and kf k Ubb Ubf and U f f are respectively boson boson bosonfermion and fermion fermion onsite interaction energy which can be calculated from the s wave scattering length and the lattice potential 13 14 is the system volume For simplicity we neglect the global trapping potential and consider systems with uniform densities We are interested in the low temperature regime where the bosonic atoms are condensed b b0 Using Bogoliubov approximation one can obtain an effective fermionphonon coupling Hamiltonian 6 15 The phonon field can be integrated out 6 to provide an effective attractive interaction between fermion atoms and hence cause the fermion pairing If the phonon velocity c is much larger than the Fermi velocity f i e in the fast phonon limit the resulting interaction between fermions is instantaneous and given by Vind k V 1 2 4 k where V U2bf Ub is the strength of the phonon induced attractive interaction and tb 2nbUbb is the boson correlation healing length Such antiadiabatic limit may not be easily achieved in typical 40 K 87Rb system c f 1 because the boson atom mass is larger than the fermion atoms In this paper therefore we also consider a BFM system composed by 40K and 23Na atoms where the phone velocity can be several times larger than the Fermi velocity c f 5 We note that including the retardation effects just changes the prefactor in the BCS expression for Tc from the Fermi energy E f to some characteristic bosonic frequency 16 and therefore it should provide similar quantum phases as obtained within the fast phonon limit see also Refs 6 7 As a result we may still apply the fast phonon approximation and obtain the following effective fermion Hamiltonian f k s Heff kf f k s k s 1 Vs s k s k s 2 k s s eff 2 s s k U f f s s Vind k where Veff 051604 1 2005 The American Physical Society RAPID COMMUNICATIONS PHYSICAL REVIEW A 72 051604 R 2005 WANG LUKIN AND DEMLER Following the early work of Micnas et al 17 we apply the mean field approximation to calculate the Tc of fermion pairing phases and that of the competing SDW CDW phases For the superfluid states the single particle excitation energy Ek has a gap at Fermi surface Ek kf k 2 ks s 2 s s k p f p s f p s where the gap function ks s 1 pVeff is determined by the gap equation ks s ps s 1 Ep s s V k p tanh 2 p eff Ep 2T 3 s s k p f p s f p s is fermion exchange Here k 1 pVeff self energy within Hartree Fock HF approximation Finally the fermion chemical potential corrected by the Hartree energy is fixed by the known total density of fermions nf pf 1 1 Ep f p s f p s 1 tanh E p s p p 2T 4 To analyze the results of different gap symmetries we consider the following ansatz for the gap function 17 ks s s s s0 s1 k d k s s p sin kx where k 2 cos kx cos ky Here s0 and s1 are for the onsite and extended s wave pairing phase while p d is for the p d wave pairing phase The transition temperature Tc is s s 0 in Eqs 3 then numerically