Although deep learning has achieved a milestone in forecasting the El Nino-Southern Oscillation (ENSO), the current models are insufficient to simulate diverse characteristics of the ENSO, which depends on the calendar season. Consequently, a model was generated for specific seasons which indicates these models did not consider physical constraints between different target seasons and forecast lead times, thereby leading to arbitrary fluctuations in the predicted time series. To overcome this problem and account for ENSO seasonality, we developed an all-season convolutional neural network (A_CNN) model. The correlation skill of the ENSO index was particularly improved for forecasts of the boreal spring, which is the most challenging season to predict. Moreover, activation map values indicated a clear time evolution with increasing forecast lead time. The study findings reveal the comprehensive role of various climate precursors of ENSO events that act differently over time, thus indicating the potential of the A_CNN model as a diagnostic tool.