The term Deep space is generally used when referring to large distances from Earth. Deep space missions provides a great challenge for the communication with the spacecraft. Due to huge distances and limitations of radio frequency(RF) communications. Optical communication or LASER communication can be considered for such missions in the future.
RF is most widely used communication medium for space applications. The frequency range used for communication is within the microwave spectrum ranges from 1GHz- 300 GHz. This range is also broken down into number of bands. The bands which used for deep space communications are L-band(1.67-1.71GHz), S-band(2.025-2.3GHz), X-band(8-9GHz) and Ka-band(20-30GHz). Higher frequencies offer higher data rates. But higher frequencies are more affected by atmospheric interferences, especially frequencies above 30GHz. Also Radio wave's power or intensity reduces with distance. The loss in intensity is roughly inversely proportional to the square of the distance. A single steady wave can not be used to transmit data as there is no way to separate signal into zeros and ones required to achieve digital communication.
Optical communication uses lasers instead of radio waves. Optical systems have the advantage of extremely broad and unregulated bandwidth as compared to RF systems, which allows optical systems carry more information. For modulation of optical signals can be use simple OOK(On-Off keying) technique. Optical systems typically have smaller receive aperture and lower power efficient transmitter and receivers. As well as optical systems suffer less from atmospheric interference. Optical system mass requirement is less than RF system mass requirements. As distance increases the difference between RF mass and optical mass requirement increases at all data rates. Only fraction of the total spacecraft power will be used for optical communication.
NASA demonstrated Lunar LASER communication by the Lunar atmosphere and Dust Environment Explorer mission. And as a result much higher data rates were received. In the figure the NASA's plan for the optical links for this demonstrative experiment is shown.
There has been several challenges to be faced while establishing the optical communication links. Damage of optical equipment due to direct solar illumination, for deep space missions; increasing the optical transmitting aperture diameter for achieving larger EIRP(effective isotropic radiated power) implies narrower beam divergence making downlink beam pointing requirements tighter, deep space range rates results in larger Doppler shifts, these are some of the challenges for optical communication. The RF communication is still competitive till mars at 100Mbps data rate and beyond mars it is competitive at 10mbps data rate. Optical communication to be used for communication with the spacecraft in deep space mission needs to demonstrate yet.

Very interesting topic to read.
Lot of can be done with Optical Communication challenge.