Confronting Galactic and Extragalactic $\gamma$-ray observed by Fermi-LAT with Annihilating Dark Matter in Inert Higgs Doublet Model