Primary adrenal lymphoma (PAL) is extremely rare although involvement of malignant lymphoma into adrenals is common. We report a case of a 58-year-old man with bilateral PAL who demonstrated adrenal insufficiency. Primary large B-cell lymphoma was proven by a computed tomography-guided needle biopsy of the adrenal tumor. Although a complete remission was once achieved by combination chemotherapy plus rituximab, a recurrence occurred with brain metastasis leading to his death. We concluded that PAL should be considered as a possible cause of bilateral adrenal incidentalomas with progressive adrenal insufficiency.